Re: Concurrency via isolated process/thread

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

> It's not completely clear (to me) what you mean by "locking".  But I 
> assume you mean two operations lock/unlock with which a thread can request 
> and waive exclusive access to an object, and sleeps until that exclusive 
> access is granted.  IOW, mutexes.

Yes. Although mutexes are not the only way to achieve this.

> If my guess is correct, that is not 
> possible.  You cannot use mutexes in a program whose data structures have 
> not been organized in a way that makes the use of such synchronization 
> primitives possible, without having deadlocks.  Given that in Emacs 
> objects are part of an enormous unstructured graph, with pointers leading 
> from anywhere to anywhere, that's clearly not the case, and all you can 
> use is a single global lock.

I do not think so.
There is usually no need to ensure that a given Elisp object is locked
recursively, with all the linked objects (like all elements in a list).
We just need to ensure that we protect individual Lisp_Object's from
data races.

>> But the very need to access global Elisp variables/objects from async 
>> threads should not be considered a good practice (except near start/end 
>> of thread execution). Most of processing should be done using threads' 
>> local lexical scope.
>
> ... if what you have in mind are async threads that should in fact not 
> access objects of the main thread, why is it necessary to lock
> objects?
> You can prepare the arguments to the async thread in the main thread, 
> start the async thread, and when its execution completes process the 
> return values of the async thread in the main thread, which is what 
> emacs-async already does.

> May I ask you if you have a concrete example of a task that you would like 
> to perform with such threads, and that cannot already be done with 
> emacs-async?  In other words, what are the limitations of emacs-async you 
> try to overcome?

You cannot pass non-printable objects like (like markers of buffer). You
cannot pass Emacs local state, like user customization, except a small
subset. You have to go through print/read loop to pass the objects
around. You have to deal with startup overheads and large memory
overheads associated with creating a new Emacs process.

Examples:

1. Background parsing of large buffers, where you cannot afford to pass
   the whole buffer and parser state back-and-forth every time the user
   makes changes.
   
2. Searching across multiple buffers, while keeping all the buffer-local
   customizations (including runtime, this-session-only, customizations)
   honored.

3. Stealth fontification of buffers. Not just on idle, but
   asynchronously.

4. Doing something as basic as loop parallelism on multiple CPUs for
   CPU-heavy processing. (passing the data between Emacs process would
   defer the whole purpose here - that's inter-process communication
   overheads + sexp parsing !!!)

5. The very async process communication where process sentinels are not
   trivial and have to utilize significant CPU processing.

6. Basically, anything more complex than what can be done using a bunch
   of foo& + wait() in bash.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>