Re: [Lightning] Memory Allocation on Mac OS X

[Paulo César Pereira de Andrade]

Em 27 de setembro de 2010 06:32, Paolo Bonzini <address@hidden> escreveu:
> On 09/27/2010 01:29 AM, Paulo César Pereira de Andrade wrote:
>>
>> 2010/9/26 Noah Lavine<address@hidden>:
>>>
>>> Hello Lightning developers,
>>>
>>> The attached patch fixes an issue with memory allocation on Mac OS X.
>>> The trouble was that the test programs allocate memory for their
>>> functions as static variables, but in Mac OS X statically allocated
>>> memory is not marked as executable. This patch uses posix_memalign and
>>> mprotect instead, which brings the number of failed tests on my
>>> machine from 21 to 2.
>>
>>   I think most tests are somewhat bogus, and the way jit_flush_code()
>> works, and calls mprotect() is wrong, as it needs to adjust it to
>> unaligned
>> memory, what means marking random, but on the same process, memory
>> as executable.
>
> As you suggest using mmap to allocate memory will do.  Alternatively you can
> use posix_memalign with the page-size as the alignment.

  What you think about adding not only an interface to allocate memory for
a code buffer, but also adding a linked list with information about:
1. Start of code buffer up to first jit_prolog() call; it is valid to
have code there,
  for example, in my assembler I used it usually just to jump to a "main"
  label, and define functions before it, but later I added the undocumented
  jit_patch_calli() call, so it is not much important now that it can call
  forward jit functions, and could previously do an indirect call.
  Anyway, this area could be used for other things, like a jump to a
exit routine,
  like is done in clobber.tst, where it prints an error messages and call exit.
2. Knowing the range from two jit_prolog() calls, it can figure the range of a
  function. Could even know about dead code, but probably better to not try
  to be too much smart about jumps and jump targets, just slightly smart :-)
3. Most important, this linked list should have information about non relative
  labels, so that it could even reallocate the buffer and adjust
offsets. This is
  tricky, because to have it to work transparently, it would need to modify the
  api, that is, instead of having a label as a "jit_insn *", it would
need something
  more like:
    jit_movi_p(&my_label, r0, pointer);
    jit_bner_i(&my_other_label, r0, 0);
  otherwise, if it resizes the buffer, with either allocate a new one and copy
  contents, or try to extend with realloc or mremap, it may need to adjust
  "pending" labels.

  There is also the issue of if it can or not call malloc to create such linked
list, that would usually be released automatically in jit_flush_code(). May need
to add some interface for alloc/free function pointers, in case lightning is
being used with some language that does not allow malloc/free, or wraps it
and could gc the linked list.

  Just some ideas... This would also make it easier/safer to use the change
I did to adjust the stack alignment in i386/x86_64 by patching the immediate
to adjust %esp in the jit_prolog call, as it could figure out what
routine it did
jump to, but forward jumps to another function would still be tricky;
the problem
is that if it jumps to another function, and call another from there,
if it patches
the stack adjustment due to building a large frame, it will patch the adjustment
of the wrong function, same if it calls jit_allocai to have some %ebp relative
buffer, like is done for x87/sse conversion (but this one is mostly for testing,
maybe should always allocate the first 8 stack bytes for it, or just push/pop
always).

> Paolo

Paulo