Re: Concurrency via isolated process/thread

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: yantar92@posteo.net,  emacs-devel@gnu.org
> Date: Sun, 09 Jul 2023 08:37:47 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > IOW, I think you have a very narrow idea of "number and text crunching
> > tasks" that could benefit from threads.  For example, one of the
> > frequent requests is to run the part of Gnus that fetches email and
> > articles in a separate thread -- if this is okay for "number and text
> > crunching tasks", then it is likely to prompt users.
> 
> Prompting for options should take place before the thread is started, or
> after the data is retrieved and about to be displayed.

That is of course not always possible, or even desirable.  For
example, some prompts are caused by specific aspects of the
processing, which may or may not happen, depending on the data.
Prompting for options unrelated to the actual processing would mean
annoying users unnecessarily.  Etc. etc.

> > We are mis-communicating.  My point is that it is almost impossible to
> > take an existing non-trivial Lisp program and let it run from a
> > non-main thread without bumping into this issue.  Your responses
> > indicate that you agree with me: such Lisp programs need to be written
> > from scratch under the assumption that they will run from a non-main
> > thread.
> 
> I agree with this completely.  From my POV, such requirements are
> reasonable and not very different from the requirements for doing so in
> other GUI toolkits and programs.

So basically, what you have in mind is a way of accruing a body of
special-purpose Lisp programs written specifically for running from
non-main threads.  Which means reusing existing code or packages not
written to these specifications will be impossible, and we will in
effect have two completely separate flavors of Emacs Lisp programs.
It would mean, in particular, that many functions in simple.el,
subr.el, and other similar infrastructure packages will need to have
specialized variants suitable for running in non-main threads.

And all this will be possible if -- and it's a large "if" -- the
necessary support on the C level will be written and prove reliable.

If this is the plan, it might be possible, at least in principle, but
is it really what is on people's mind when they dream about "more
concurrent Emacs"?  I doubt that.

> > How is this compatible with the goal of having threads in Emacs, which
> > are to allow running Lisp code with less hair than the existing timers
> > or emacs-async?
> 
> I thought the concern was one of efficiency, and not ease of use:
> process IO is slow compared to sharing the same VM space, and
> subprocesses also utilize more memory.

Memory is cheap these days, and "slow IO" is still a gain when it
allows us to use more than a single CPU execution unit at the same
time.  So yes, efficiency is desirable, but ease of use is also
important.  What's more, I don't think anyone really wants to have to
write Lisp programs in a completely different way when we want them to
run from threads.

> >> > Fixed how?
> >> 
> >> By replacing `sit-for' with `sleep-for' (and in general avoiding
> >> functions that call redisplay or GUI functions.)
> >
> > So programs running in non-main threads will be unable to do stuff
> > like show progress etc.?  That's not very encouraging, to say the
> > least.
> 
> They should be able to run code from the main thread's command loop, via
> mechanisms such as `unread-command-events'.

Those mechanisms only work when Emacs is idle, which is bad for
features like progress reporting.  Doing this right requires to
redesign how redisplay kicks in, and probably have several different
kinds of redisplay, not one.

> > This basically means rewriting most of Emacs.  Because most APIs and
> > subroutines we use in every Lisp program were not "specifically
> > written for running outside the main thread".  So we'll need special
> > variants of all of those to do those simple jobs.
> 
> I wasn't thinking about Lisp functions used for text editing tasks, but
> rather raw data crunching functions.  Most of these are already
> side-effect free, and some are even truly reentrant.

I think none of them are side-effect free, if you look closely.  They
access buffer text, they move point, they temporarily change the
selected window and the current buffer, the access and many times
modify the obarray, etc. etc.