Re: Very interesting analysis of "the state of Emacs"

[Stefan Monnier]

>> The issue is when we start putting global variables into the mix:
> ...
>> Anyway, hopefully someone has some ideas on what to do here.  I admit I
>> haven't looked at how sxemacs handles this yet. Maybe we can just deal
>> with locks?  At least in that case my IMAP mail could download while I
>> am typing in another buffer :)

> If the multi-threading were cooperative (as rms suggested), then such
> problems would obviously be a bit easier to manage -- you can basically
> just say "no context switches except at well defined points", and define
> these "points" to be (1) user interaction/recursive edits [where the
> user can do something to "screw up the state" even today], or (2)
> explicit calls to yield.

W.r.t to concurrency, I think we need to think about how to introduce it
into Emacs, indeed.  The reason why I think so is because in order to
evolve Emacs needs to eat up more CPU.  You may think Emacs is not CPU
bound, but it's only the case because it only uses what is available.
Try to run Emacs-22 on a 486 to get an idea of what I mean.
Single-thread CPU power is very unlikely to increase significantly in
the future, so if we want to get access to the additional CPU power
we'll need concurrency.

Now we already have concurrency, in the form of start-process.
`flyparse' and `flymake' do just that.  And maybe that's good enough.
But I expect that closer integration of additional threads will be
needed/useful at some point.

Other than separate processes, we can add threads without affecting
Elisp, by using separate threads for the GUI and for the redisplay.
Even the redisplay could be parallel by using separate threads per
frame/window.

As for adding concurrency to Elisp, the core difficulty will be the heap
of code that assumes concurrency doesn't exist, as well as the
interaction between buffer-local and let-bound variables.

But indeed, I think a good first step is to add coroutines.  This should
not be terribly difficult to do (especially since it's OK for it to
break code as long as it's only broken when you use `yield').
Patches welcome.

> However I think there are potentially additional problems with dynamic
> scope:  remember, elisp uses shallow scoping, where binding a variable
> is basically "save old value, and set global".  For normal variables,
> this could be replaced by deep-binding, which is more multi-threading
> friendly (my "lexbind" branch already uses deep-binding in the
> interpreter), but afaik, the use of shallow-binding in elisp is kind of
> intertwined with the implementation of buffer-local variables and the
> like, and I'm not so sure how easy it would be to handle such things
> with a new deep-binding implementation.

The way I see it, multithreading within a buffer is a problem we can
ignore for the next 20 years.  So let-binding a buffer-local var can
be handled via shallow-binding (whereas let-binding of global variables
needs to use deep binding).  So the only difficulty is to deal with
variables which are not make-variable-buffer-local but only
make-local-variable where we need to make sure we can always
unequivocally know whether to use deep or shallow binding.


        Stefan