[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Async IO and queing process sentinels (was: Concurrency via isolated pro
From: |
Ihor Radchenko |
Subject: |
Async IO and queing process sentinels (was: Concurrency via isolated process/thread) |
Date: |
Tue, 18 Jul 2023 12:39:08 +0000 |
Hugo Thunnissen <devel@hugot.nl> writes:
>>> I'm not familiar with the C core at all, do you think it might be more
>>> realistic to add a little more facilities for asynchronous data streams
>>> than to rework the execution model of Emacs to add multi threading? I'm
>>> mainly thinking of something like "channels" or "queues" for lisp
>>> objects, with an easy to use scheduling mechanism that makes it
>>> straightforward for people to not stall the main thread for too long.
>> AFAIK, sentinels do this already. Of course, they are quite basic.
>> As for queues, see https://emacsconf.org/2022/talks/async/
>
> Sentinels are for external processes. I was thinking more in the vain of
> a queue that is owned by two lisp threads, where one thread can write
> and prepare input for the other, and the other thread yields as long as
> there is no new input to process. When a thread receives an input
> message from the other, it will come alive at the next idling moment and
> process the queue while yielding in between messages. This would make it
> easier to process data in chunks without halting the main thread long
> enough to bother the user, as long as the main thread is prioritized
> over others.
What you describe sounds very similar to `condition-wait'/`condition-notify'.
That said, I have seen scenarios when I have several dozens of network
connections open, receive data from server, and fire all their sentinels
near the same time. This effectively blocks Emacs because all the
sentinels are suddenly scheduled to run together.
It would be nice to have some better balancing in such case.
> I still suspect that async filesystem IO could make a considerable
> difference in some scenarios. My package is for PHP projects and it is
> not unheard of for PHP files to be edited over network mounts where IO
> could have a lot more latency. For example, some people have asked me
> whether my package works over tramp. I haven't profiled this scenario
> yet as I have to make some tweaks first to make it work at all, but I
> fear that blocking on `insert-file-contents' may not be ideal in that
> scenario.
AFAIR, Michael Albinus (the maintainer of TRAMP) is working on better
support for async processes in TRAMP. And he had troubles with managing
async process queues as well.
CCing him, as this discussion might be relevant.
> And then there also is `directory-files-recursively':
>
> 457 77% - phpinspect-index-current-project
> 120 20% -directory-files-recursively
> 79 13% -directory-files-recursively
> 21 3% -directory-files-recursively
> 11 1% directory-files-recursively
Note how functions involving actual IO do not show up in the backtrace:
`file-name-all-completions' and `file-symlink-p' are not there.
Most of the 120ms spend inside `directory-files-recursively' is Elisp
recursive calls.
I also did testing locally, on my $HOME dir, and got similar results
with recursive calls taking most of the CPU time:
463 29% - directory-files-recursively
434 27% - directory-files-recursively
305 19% - directory-files-recursively
111 7% - directory-files-recursively
85 5% - directory-files-recursively
62 3% - directory-files-recursively
24 1% - directory-files-recursively
10 0% - directory-files-recursively
10 0% - directory-files-recursively
7 0% - directory-files-recursively
7 0% directory-files-recursively
3 0% sort
3 0% sort
3 0% sort
If you want to improve performance here, you likely need to rewrite
`directory-files-recursively' without recursion. Doable, and nothing to
do with IO slowness.
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
- Re: Concurrency via isolated process/thread, (continued)
- Re: Concurrency via isolated process/thread, Po Lu, 2023/07/04
- Re: Concurrency via isolated process/thread, Eli Zaretskii, 2023/07/04
- Re: Concurrency via isolated process/thread, Hugo Thunnissen, 2023/07/17
- Re: Concurrency via isolated process/thread, tomas, 2023/07/18
- Re: Concurrency via isolated process/thread, Ihor Radchenko, 2023/07/18
- Re: Concurrency via isolated process/thread, Po Lu, 2023/07/18
- Re: Concurrency via isolated process/thread, Ihor Radchenko, 2023/07/18
- Re: Concurrency via isolated process/thread, Hugo Thunnissen, 2023/07/18
- Async IO and queing process sentinels (was: Concurrency via isolated process/thread),
Ihor Radchenko <=
- Re: Async IO and queing process sentinels (was: Concurrency via isolated process/thread), Ihor Radchenko, 2023/07/18
- Re: Async IO and queing process sentinels, Michael Albinus, 2023/07/18