Re: [Tinycc-devel] tcc_relocate() and tcc_relocate_ex()

[Jared Maddox]

> Date: Tue, 28 Aug 2012 14:04:10 -0400
> From: Sean Conner <address@hidden>
> To: address@hidden
> Subject: Re: [Tinycc-devel] tcc_relocate() and tcc_relocate_ex()
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=us-ascii
>
> It was thus said that the Great grischka once stated:
>> Sean Conner wrote:
>> >  Can you at least commit the change that makes tcc_relocate_ex()
>> > global?
>>
>> Yes, perhaps at least.  You wrote:
>> >... but fixing my code to match the current definition
>> >of tcc_relocate() was out of the question (the surrounding code manages
>> > the
>> >lifetime of both the TCCState and the resulting executable code compiled
>> >into memory).
>>
>> I'd actually like to know some details why you can't just use
>> tcc_relocate()?
>> Admittedly we know little about how people actually use the libtcc API.
>
>   I have three use cases right now in using libtcc.
>
> Case 1:  Synthesis OS (http://valerieaurora.org/synthesis/SynthesisOS/)
>
>   Synthesis OS was an experiment operating system from 1992 written for a
> Ph. D. dissertation.  Dispite it being written in assembly language
> (MC68030) it basically would "recompile" system calls on the fly (the whole
> thing is mind blowing and its worth reading the dissertation)---think of a
> "just-in-time" compiler but one that recompiles the kernel as needed.  This
> is a technique I'm interested in (on my own time).
>
>   I have a "wow---it can work!" type of prototype using libtcc (nothing
> even
> close to a kernel, just some simple code).  I have a custom "open()"
> function that will compile a custom "read()" routine based upon the
> paramteres given it---if I open a file read-only for character-by-character
> access, and if the file can be mmap()'ed into memory, then you get a
> specialized version of read() that returns successive characters from an
> mmap()'ed file; if the file can't be mmap()'ed, you get a more traditional
> read() function.  This re-compilation happens on a file-by-file basis (each
> file returned from my custom open() will have a custom read() function
> compiled that deals only with the given file).
>
>   The issue here with tcc_relocate() is that once the compilation phase is
> done, I don't need (or want) the symbol table, section definitions,
> buffers,
> or anything else of the TCCState *except for the code just compiled*, and
> the memory for that (the compiled code) is reclaimed on the call to a
> custom
> close() function.
>
>   Yes, I could skip the compilation step and load pre-written functions (as
> a shared object) but the ability to compile a custom version, quickly (one
> of the strengths of TCC) in memory and use it immediately, is not something
> I want to give up here.
>

Quajects are indeed very neat. I've played around with them myself in
C & writing some VMs, though I was focusing on the mutability and
never have compiled any of them.


> Case 3: Closures in C (or, a mixture of case 1 and case 2)
>
>   Why this exists doesn't matter, but I have written Lua bindings to Xlib
> (the lowest level of X programming without talking the protocol directly).
> Xlib, however, reports some errors via a callback function you register
> using XSetErrorHandler():
>
>       int (*XSetErrorHandler(int (*handler)(Display *, XErrorEvent *)))();
>
>   I would like to push the error into Lua, but the issue I have is---I need
> a global variable pointing to the current Lua state, which is something I
> don't particularly want (the global variable), because the following:
>
>       int foo(lua_State *L)
>       {
>         XSetErrorHandler(int (Display *display,XEerrorEvent *event)
>               {
>                 lua_pushlightuserdata(L,display);
>                 lua_gettable(L,LUA_REGISTRYINDEX);
>                 lua_getfield(L,-1,"error_handler");
>                 mylua_pushevent(L,event);
>                 lua_call(L,1,0);
>                 return 0;
>               }
>         )
>         return 0;
>       }
>
>   is not legal C.  However, using libtcc, I can fake a closure with C:
>
>       #include <stdlib.h>
>       #include <X11/Xlib.h>
>       #include <syslog.h>
>       #include <lua.h>
>       #include <lualib.h>
>       #include <lauxlib.h>
>
>       lua_State *L;
>
>       int Xcallback_function(Display *display,XErrorEvent *event)
>       {
>         lua_pushlightuserdata(...);
>         /* ... */
>       }
>
> and compile it with (excluding error checking for clarity):
>
>       lua_State  *L; /* assume it's set properly */
>       TCCState   *s;
>       int         size;
>       void       *mem;
>       lua_State **pL;
>         int       (*cb)(Display *,XErrorEvent *);
>
>       s = tcc_new();
>       tcc_set_output_type(s,TCC_OUT_MEMORY);
>       tcc_add_file(s,"xcallback.c");
>       size = tcc_relocate_ex(s,NULL);
>       mem = lua_newuserdata(L,size);
>       tcc_relocate_ex(s,mem);
>       pL = tcc_get_symbol(s,"L");
>       cb = tcc_get_symbol(s,"Xcallback_function");
>       *pL = L;
>       XSetErrorHandler(cb);
>
>       /*----------------------------------------
>       ; I don't need the compiler info any more
>       ;----------------------------------------*/
>
>       tcc_delete(s);
>
>   Yes, it's a bit of a ways to go to get closures under C, but it works.
>

It's better than when I worked out Continuations for C (stack copying
AND assembly code!), so I wouldn't say it's all that bad.


> Date: Wed, 29 Aug 2012 16:04:55 +0200
> From: grischka <address@hidden>
> To: address@hidden
> Subject: Re: [Tinycc-devel] tcc_relocate() and tcc_relocate_ex()
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>

> Anyway, we're not in a hurry, are we?
>
> What about this:  Keep the 0.9.25 declaration of tcc_relocate, which was:
>
>     /* copy code into memory passed in by the caller and do all relocations
>        (needed before using tcc_get_symbol()).
>        returns -1 on error and required size if ptr is NULL */
>      LIBTCCAPI int tcc_relocate(TCCState *s1, void *ptr);
>
> but define a special value for 'ptr' to get behavior as of 0.9.24, as in:
>
>     /* Do all relocations. Needed before using tcc_get_symbol().
>        Possible values for ptr:
>          - TCC_RELOCATE_AUTO : Allocate memory internally
>          - NULL              : return required memory size for the step
> below
>          - memory address    : copy code into memory passed by the caller
>        returns -1 on error. */
>
>      LIBTCCAPI int tcc_relocate(TCCState *s1, void *ptr);
>      #define TCC_RELOCATE_AUTO (void*)-1
>
> Is this ugly?  Confusing?  0.9.26 isn't out yet so we can still decide.
>
> --- grischka
>

If you're going to use that method, then I'd suggest this definition
of TCC_RELOCATE_AUTO:
#define TCC_RELOCATE_AUTO (void*)&tcc_relocate

If I recall the C standard correctly, that will result on a value
that's unique from all other valid addresses even on Harvard
architectures (though porting TCC to one of those would be
"difficult"), thus ensuring that the new behavior can only be invoked
by accident if the calling code has a bug.

It also makes it easy to not run into a conflict with anything else,
which could be relevant, since I half recall some architectures having
valid negative addresses for some reason..