[Pan-users] Re: Pan 0.120 only queuing tasks

[Duncan]

SciFi <address@hidden> posted
address@hidden, excerpted below, on  Mon, 30 Apr 2007
23:24:38 +0000:

> I also have some highly tweaked sysctl settings that help keep the
> network pipe being filled while Pan2 or Unison takes its time finishing
> the single-threaded task, then I see our network monitor sends a huge
> wad of ACKs when Pan2/Unison gets that task freed up.  ;)

Wait wait wa' wa' wait!

That's putting up all /sorts/ of red flags here.  Something's SERIOUSLY 
wrong with that picture, but if it indicates what I think it /might/ 
indicate, it goes quite some way to explaining some things.  (OTOH, maybe 
I'm wrong, and all this can be written off as a misunderstanding of a mix 
of the implications of TCP dynamics and certain microkernel 
implementations, with priority inversion thrown into the mix as well.)

/Whatever/ pan might be doing with the downloaded packets once it gets 
them, and no matter /what/ sort of CPU time it's /wanting/ to take to 
sort stuff out or whatever, this should NOT interfere with the return of 
TCP ACKs by the system networking stack, which should be returning them 
independently (on Linux, in kernel context and at a rather high priority, 
the implications of user mode threads doing this on a a BSD based 
microkernel being suspect here, but something I don't know enough about 
to go any further than that and it's entirely possible this is just the 
bashings of ignorance), as packets are sucked out of the receive buffer.

Specifically, there should be no burst of ACKs AFTER pan does its thing, 
as in ordered to do its thing on the packets, it will have to have been 
pulling them out of the receive buffer first, in turn freeing room in 
that buffer for additional packets, so a properly functioning (and not 
blocked by priority inversion) network stack will have been returning 
ACKs as the room was freed in the buffer, thus signaling the other end 
that said packets had been received and processed at least to the extent 
that there is now room in the buffer for additional packets, which should 
now be sent if there are additional packets to send.

Thus, a huge burst of ACKs NOT sent until pan finishes doing its thing 
indicates something SERIOUSLY wrong, not really with pan, but with the 
implementation of networking stack of the OS pan is running on, and/or 
the priority handling thereof, if it's the case that pan's priority is 
high enough to cause it to schedule ahead of the networking stack ACK 
return mechanism.  (The latter of course assumes the user hasn't screwed 
with the system's default priority settings, thus causing a priority 
inversion the system wouldn't have had on its own.)

OTOH, depending on the priority of the mechanism used to capture and 
report on those ACKs, it's ALSO possible that the system was sending ACKs 
as it should, and simply that they weren't getting reported when they 
happened as the monitoring mechanism was operating at a priority low 
enough it wasn't able to log the reports in a timely manner, and didn't 
do so until after pan got thru doing what it was doing, at which point 
there was the backlog of events to process, thus resulting in a heap of 
ACKs /reported/ at the same time, even tho they actually occurred rather 
more steadily, over a rather longer period.  If this is the case, 
/moderately/ boosting the priority of the logging app (or reducing the 
priority of pan), just enough so it gets scheduled ahead of pan, such 
that it can log the acks as they occur, should cure what is in that case 
just a reporting issue.

Finally, it's entirely possible that the ACK logging itself is 
sufficiently resource intensive that it's disrupting the system's 
otherwise normal ACK processing.  Again, priority handling could be 
critical here, if an otherwise high priority networking stack thread is 
being forced to wait on the ordinary user priority network logging 
utility, which is in turn getting scheduled behind pan's particularly 
resource intensive activities.

Regardless of which of the above is actually happening, unless my 
understanding is /totally/ screwed up, pan should NOT be causing a 
significant delay in ACK return.  If it in fact is, and it's not just an 
artifact of the ACK event logging itself, there's something SERIOUSLY 
wrong with the system.  If that's the case, it's /possible/ pan can be 
engineered to work around the issue, voluntarily yielding CPU back to the 
system so the system can take care of the business it needs to do, but 
make no mistake, it's still the system's problem, not pan's, as pan is 
simply using the resources the system is allowing it to use.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman