Re: call-process should not block process filters from running

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: sbaugh@catern.com
>> Date: Tue, 04 Jul 2023 08:20:39 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> What is complex about this?
>> >
>> > Whatever caused you to be surprised that just calling
>> > wait_reading_process_output doesn't process selection requests.  How
>> > many Lisp programmers understand what is going on in Emacs when such a
>> > request is processed, and what that means for their Lisp programs?
>> >
>> >> What problems can it cause that aren't already covered by "arbitrary
>> >> Lisp will run during call-process"?
>> >
>> > Processing selection requests does not really qualify as "running
>> > arbitrary Lisp".  It is much more subtle, and I'm quite sure the
>> > mental model of that which most Emacs users have is very simplistic.
>> 
>> I take your point, but can you be a little more concrete?  I genuinely
>> don't know what potential issues you're referring to that can be caused
>> by processing selection requests in a new place.  I'm asking just to
>> improve my knowledge of Emacs.
>
> I never closely analyzed that, but just follow the control flow of
> process_special_events and the functions it calls, and you will see
> quite a few non-trivial things.
>
> My point is that most people aren't aware of how this works, so they
> will be unable to decide whether their program can or cannot handle
> this kind of processing while call-process is in progress.
>
>> > Still not important enough, IMO, for us to have to solve this right
>> > away, since you will have the same issue when Emacs is busy with
>> > something else, like redrawing the display or even executing some
>> > long-running Lisp program.
>> 
>> Redrawing the display is much quicker
>
> Except when it isn't.  E.g., with long lines, redisplay could take
> several hundreds of milliseconds, and sometimes longer.
>
>> and I don't use any long-running Lisp programs.
>
> Maybe you don't, but others might.  "Long" here doesn't have to be
> minutes or hours, btw, it could be seconds.  For example, sorting
> incoming email could take that long if you've got a lot of it.
>
>> In fact, I move long-running operations out into
>> asynchronous subprocesses so they can run concurrently with Emacs.  It's
>> only call-process usage that makes Emacs noticeably busy for me.
>
> We are talking about a general-purpose API used a lot in Emacs.  We
> are not talking about a feature only for your personal usage -- for
> that you can do whatever you want locally.

I mean as a general principle when writing Emacs Lisp.  "Do expensive
operations in separate processes" is advice I've heard and given many
times, and certainly Emacs follows it in many places.

>> OK.  So what's the next step in this step-by-step from your perspective?
>> Should we install the option on trunk, let people use it, and gradually
>> make what it does broader and more fine-grained over time?
>
> I'd prefer that you run with this locally for some time, and report
> back any complications or issues.  You have posted the patch, so if
> someone else is interested, they can apply that locally and use it.

OK, I'll install the patch I've posted at my site.  It will include the
process_special_events call, since that fixes the part that I've gotten
user complaints about.  I'll report back about how it goes in a couple
weeks.