RE: [Chicken-users] RE: Win32/cygwin dynamic loading

[Jonah Beckford]

Changed web site to a permanent location ... is now:

http://beckford.netfirms.com/hobbies/chicken/

(and SWIG CHICKEN is at http://beckford.netfirms.com/hobbies/swig/)

MinGW has also been tested with the dynamic loading (use the
all-mingw.sh script).

Jonah

-----Original Message-----
From: Jonah Beckford [mailto:address@hidden 
Sent: February 21, 2003 1:59 AM
To: 'felix'
Cc: 'Felix Winkelmann'; 'address@hidden'
Subject: RE: [Chicken-users] RE: Win32/cygwin dynamic loading


I have the modifications to Chicken that allow dynamic loading of
modules in a portable fashion.  Right now only cygwin has been tested
(mingw had also been, but not lately).  If you have cygwin, then run the
all-cygwin.sh script first because the bootstrapping process is not easy
to do by hand.

http://beckford.homeip.net/chicken/

The web site is up most but not all of the time.

Major changes:
- (declare (uses xx yy)) is basically translated into (load "xx-unit")
(load "yy-unit") conditionally at the C source code level.
- while not enforced, the idea is that 1 unit == 1 dynamic module.  No
bundling of many units into one shared library (although you still may
if you don't care about portability).
- (load "xx-unit") will look for xx-unit, xx-unit.dll, cygxx-unit.dll
and libxx-unit.dll on cygwin.  It does .so and .sl and its variants as
well, just hasn't been tested on those platforms yet.  If can't find a
module, then will proceed with the original Chicken rules (like looking
for a xx-unit.scm).
- on "primitive shared" platforms, all function calls and variable
accesses that are defined in chicken.h are done through a data structure
(of function pointers and variables).  This is done transparently
through macros.  So a function call C_string(x,y,z) is macro expanded to
(*(C_shlib->priv_C_string))(x,y,z) and a variable access to
C_collectibles is macro expanded to (*(C_shlib->priv_C_collectibles)).
When a unit is loaded through (load "xx-unit"), its own C_shlib is
initialized with the proper values.  Basically, we simulate the
indirection method used by ordinary DLLs, but we define it in a way
(through C_shlib) such that it works on most platforms.
- there is a script gen_shlib.sh that writes all the macros and all the
code for you ... just add a new variable or function to chicken.h, run
gen_shlib.sh and nothing else to do.
- MinGW is done through the ./configure script, not makefile.mingw.

Benefits:
- Should work on all shared library platforms because each xx-unit is
completely self-contained.  In fact, the only requirement is that the
platform have some object file format that supports relocatable code
(ie. gcc -PIC) and a predefined entry point (either a symbol or an
absolute address).
- Can have circular dependencies between xx-unit and yy-unit because the
units are dynamically loaded.
- Quite unexpectedly, the technique can be used to make Chicken thread
safe as well (a thread can get its own C_shlib).  The remaining static
variables in runtime.c would have to be moved to chicken.h (trivial;
I've already done half), the static C_[xxx_]toplevel_initialized in
Chicken generated code would have to be removed (trivial) and each
C_shlib access would have to go into thread local storage (not trivial).

Drawbacks:
- Is slower because of the pointer dereference of C_shlib.  This becomes
an issue inside loops (good example: looping over C_collectibles).  The
C code developer has to be careful and introduce temporary variables
above large loops.  With the temporaries (which is nothing more than a
hint to the C compiler to not dereference the pointer every step in a
loop), the C_shlib method would be the exact same speed as access
__declspec(dllimport/export) variables.
- I'm sure there are more ...

Jonah

-----Original Message-----
From: felix [mailto:address@hidden 
Sent: February 18, 2003 4:02 PM
To: Jonah Beckford
Cc: 'Felix Winkelmann'; address@hidden
Subject: Re: [Chicken-users] RE: Win32/cygwin dynamic loading


Jonah Beckford wrote:
> I took your comment about solution #2 into mind ... I don't need to
> pass in the toplevel pointer.  I did need to split up libchicken into 
> two libraries though (a static library that has defns that must be 
> statically linked [CHICKEN_initialize/run/invoke, main/WinMain/DLLMain

> and global data], and a runtime library for everything else).
> 

Sounds reasonable. But we still have the option of generating `main'
entry-points into compiled code. Yet, the method youu describe seems
better.

> Also, my 'better' solution to #3 is basically to resolve the 
> C_<unit>_toplevel at runtime instead of link time.  Circular 
> dependencies should be fine using the new method; no need to touch the

> srfi-37 unit :)

Too late. I removed the serialization stuff from the lolevel unit and
srfi-37 does the option-parsing by hand (which is more efficient
enyway).

I'm checking in the changes just at this moment...

> 
> Also, I was using a Win32 dload_2 stub that called 
> LoadLibrary/GetProcAddress.  This works on all Win32 platforms ... 
> Cygwin does have a dlopen/dlsym, but it is just a tiny wrapper around 
> LoadLibrary.

Yup.

> 
> It's mostly finished ... I will tar.gz it up and make it available for

> testing in a day or two.

Great!


cheers,
felix