Re: advice needed for multi-threading patch

[Ken Raeburn]

On Aug 26, 2009, at 12:08, Tom Tromey wrote:
> Ken> I hope you and Daniel and I aren't doing redundant work.
>
> Unless your changes apply to Emacs I don't think they are really
> redundant.  I think the key thing is agree how it ought to work, so we
> end up with the same model, but I don't anticipate any problems there.

Well, in their current form, my Emacs changes are kind of ugly, so  
they won't be applied in their current form.  Some of them are hacks  
to get things working, so we can (eventually) evaluate the feasibility  
of merging in Guile for real.  It's not that I don't want to fix  
things properly; but fixing some of them properly requires either  
large-scale divergence from the Emacs repository -- in small but  
pervasive and less well automated ways -- or changes to how things are  
done in the Emacs repository that can't be enforced very cleanly, and  
which my only argument for is, "well, my Guile work that may or may  
not ever get folded in is going to need it."

The big issue -- the *huge* one -- is that all-bits-zero, the default  
C initializer for a static Lisp_Object and the result of bzero, is an  
integer-zero in Emacs currently and an invalid object representation  
in Guile.  There are lots of places where such values get used, or  
Lisp_Object variables or fields are made visible to the GC system  
before being properly "initialized" with valid representations.

I could go change all of the places like that to first initialize a  
location with a valid object before using it, but I don't see how to  
tweak the non-Guile version to help me catch such cases without, say,  
changing Lisp_Int to use a nonzero tag, which would affect performance  
(how much?), and would stop XINT/XUINT from being a shortcut for  
"gimme all the bits" in the hash code and perhaps a few other places I  
haven't tracked down yet.

Or, I could change things like staticpro to first store a valid object  
value, which then requires that the assignment of the desired value in  
the various init_foo functions *must* happen after the staticpro call,  
whereas now the ordering is inconsistent.  That might be more  
enforceable, because I can make staticpro and the like complain if  
they see anything other than all-bits-zero, which implies that the  
code tried to store the desired value (other than integer-zero)  
first.  I'd still have to patch up anything using bzero to follow it  
up with explicit initialization of some fields.

It's a bit annoying that the unexec setup basically prohibits the use  
of static initializers for Lisp_Object variables because they might  
become read-only; otherwise static initialization of V* and Q*  
variables with a macro expanding to a valid object representation  
would be an option.

I was figuring on pushing this work a bit further along, and having  
something I could show off better, to make a stronger case for  
integrating the guile-emacs work, before trying to argue for such  
changes in the main Emacs code base.  (Even if the decision eventually  
went against integrating Guile specifically, I might be able to make a  
case for going through a little pain to better isolate the Lisp engine  
so it could perhaps be replaced someday by something else.)  And in  
the meantime, I'd prefer to keep it easier to resync with upstream  
changes, by not rewriting all that code, just as long as I know  
roughly what needs to be fixed later.

But, I digress. :-)

My own changes are, in part, to insulate the C Lisp implementation in  
Emacs as much as possible from the rest of Emacs, and in part, to  
replace the most fundamental layer -- representation, allocation and  
GC -- with Guile.  Whether this project eventually leads to replacing  
the Lisp evaluator itself, I don't know; some people seem to think it  
would be a great idea, and some seem to think it would be a bad idea.   
I'm on the fence until I can see some performance numbers and serious  
testing, and that requires a more or less complete implementation;  
there's been some encouraging work happening on the Guile side, but  
it's not done yet.  But it's appealing to think we might someday be  
able to have elisp available as an extension language for other  
programs, via Guile; and if that happens, do we really need two  
implementations?  (And if we go with only one for both Emacs and  
Guile, would it be the current Guile one, or taken from Emacs and  
dropped into Guile?  Dunno.)

I'm not worrying about things like multithreading, but my changes will  
need to ensure that symbol value lookups and updates are sufficiently  
isolated through macros or functions that it should be easy to slip in  
thread-specific processing.

Daniel Kraft and others are working on the Guile elisp implementation,  
which in part requires learning the specs of how elisp functions in  
various corner cases as well as normal usage.  There has been some  
discussion of multithreading, but as far as I know there's currently  
no implementation of buffer-local variables.

Daniel's work is using Guile's string and symbol representations.  I  
started with the "smob" application-defined-object handles so I could  
store pointers to Lisp_String and Lisp_Symbol and so forth, and not  
have to rewrite all that code right away; that's coming up.  So  
there's a bit of work needed before Daniel's work and mine can  
possibly work together.

> Ken> I've written similar elisp code for doing transformations of  
> patterns
> Ken> within the Emacs code -- mostly more localized things like "XCONS
> Ken> (expr)->car" to "XCAR (expr)", but with support for somewhat  
> complex
> Ken> values of "expr" that have to have balanced parens, etc.
>
> If you still have these, I would appreciate a copy, so that I don't  
> have
> to reinvent them.  Are they in your git repository?

No, but I've got them floating around somewhere, and I'll send you a  
copy.
They're not pretty, but they get the job done.

> I will have to think about this.  This would imply removing all those
> slots from struct buffer; I suppose my concern would be that the  
> result
> would be too inefficient.

Not necessarily.  With Lisp_Misc_Buffer_Objfwd, one of the existing  
fields is the offset into the buffer structure.  Maybe that could be  
where the thread-local object handle is stored.

> I've also been considering a similar thing for the Vfoo globals:  
> remove
> them and have the C code always refer to them via SYMBOL_VALUE.

I think that might also simplify some things for my work.

Ken