bug#33018: 26.1.50; thread starvation with async processes and accept-process-output

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 17 Oct 2018 21:20:22 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: michael.albinus@gmx.de, 33018@debbugs.gnu.org
>> 
>> > From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> > Cc: <michael.albinus@gmx.de>,  <33018@debbugs.gnu.org>
>> > Date: Wed, 17 Oct 2018 19:05:59 +0100
>> > 
>> > > Looks like the buffer of the pipe through which Emacs reads the stuff
>> > > is full, and wget waits for some space there?
>> > 
>> > Would that imply that different threads/processes are (re)using the same
>> > buffer/pipe?
>> 
>> Could be, but it's more likely that Emacs simply doesn't read the
>> output from wget.
>
> I think the relevant code should be instrumented to show which thread
> waits for what process(es).

Each thread launches a single wget process which it then waits for
before dying, and current-thread is always eq to that thread around
calls to accept-process-output.  Or are you talking about some other
type of thread?

Either way, I'll report back when I've had a deeper look into what Emacs
is doing, unless someone beats me to it.

> Btw, are you sure this is not a bug in your program?

No, but the fact that it reliably works when Emacs is run with -batch,
and reliably hangs when run with -Q is at least somewhat intriguing.

> Michael caused your program to work twice by simple changes, AFAIU,
> no?

Michael avoided the hang by rewriting the create-join-...-create-join
sequence as create-create-...-join-join.  But if the latter is done
twice from within the same master thread, the hang still occurs.  As I
said:

> There's something about going through a complete create-join cycle more
> than once within a non-main-thread which is triggering this behaviour.

(Actually, I haven't checked whether the hang occurs when two
 create-join cycles are completed within main-thread; I was just
 specifically describing my sample program.)

In his second rewrite, Michael replaced wget with echo, which does not
suffer from any hangs.  As I said:

> Indeed, there are many subprocess-within-a-thread examples which don't
> suffer from a hang, e.g. by using a different URL.  I would like to get
> to the bottom of why network programs like wget/curl in particular
> eventually hang, though.

In other words, I don't (yet) see why my recipe shouldn't work, and I'm
curious to eventually get to the bottom of this.

I'm sorry I've been talking more than doing, but university and house
hunting will dominate my free time for the next few weeks.

Thanks,

-- 
Basil