Re: [Pan-users] Groups w/huge header count won't load?

[David Kelly]

On Jul 26, 2004, at 5:30 PM, Keith MacDonald wrote:

> I've tried leaving the machine alone to process the headers for,
> like, 8 hours at a time, and Pan never manages to parse them
> properly.  If I close Pan and restart it, I'm prompted to resume
> the last batch, at which point Pan proceeds to download all the
> headers from scratch.

8 hours may not be long enough. For fun look at the OS processes with 
something like top and notice how much swapping is going on, and how 
much CPU time pan is really getting. Sometimes as little as 2% during 
swapfests on my 800 MHz Athlon running FreeBSD.

Many years ago I gave up on Linux *because* it behaved so badly under 
heavy loads. Seemed to be heavily tuned for low latency under light 
single user loads. Then a filesystem bug crept in and trashed my drives 
3 times in one week. So I tried FreeBSD and haven't looked back. I 
*have* laughed at the plight of Linux users dealing with files over 2G 
and many other little things which a good OS should not have trouble 
with.

> So, can anyone help out here?  Is this a known issue?  My problem
> is probably related to this guy's :
> http://lists.gnu.org/archive/html/pan-users/2004-05/msg00030.html
> , however, the details of that thread don't exactly match-up with
> my situation.

That's me. With 768M of RAM core and swap spread across three drives 
I've learned that 1.5M messages in a single group is about my limit 
before exponential unreasonable slowdowns occur due to swapping.

Have seen in the To-do or wishlist for pan the use of a database to 
maintain headers. Apparently its maintained in core now. Storing this 
data in a file-based database would be faster for really large groups 
but slower for small. No doubt it will make things more complicated.

The auto-load of groups preference thing I haven't figured out. Pan was 
unusable without it enabled because it would never list the group. If I 
haven't visited a large group in about 500k or million messages I've 
found the thing to do is once its already trying to download a million 
or so to select Download Options and start another restricted to 500k 
then go to the Task Manager and stop the 2nd from the top task which 
was the unlimited new header download. Gotta do this before swapping 
gets out of hand.

On that topic my ISP limits 2 simultaneous sessions. If I "stop" a 
running task in pan, usually the connection is not closed cleanly and 
continues to consume one of my connection quota counts. News server 
bitches about "too many connections" while pan thinks there is only 1. 
Eventually pan decides to carry only 1 connection. Each connection has 
an independent bandwidth throttle, as a result pan operates at half 
bandwidth until exit and restart.

The work-around for the above is to push to the bottom of the list any 
pan task I wish to stop. It completes the current chunk it has bitten 
off of the binary attachment. Then re-examines the queue for its next 
chunk and cleanly stops that task. Don't think this works for header 
download.

I believe many of my error messages from pan are/were due to memory 
problems. Its impossible to say as no matter what it passes all memory 
tests I can throw at it but for pan. With only Crucial 512MB and 256MB 
PC133 modules installed it seems to work best. Have ordered a new 
machine with ECC memory to put an end to this problem once and for all.

--
David Kelly N4HHE, address@hidden
========================================================================
Whom computers would destroy, they must first drive mad.