[Pan-users] Re: 0.129 - a few minor issues

[Duncan]

"Jim Henderson" <address@hidden> posted
address@hidden, excerpted
below, on  Thu, 10 May 2007 17:23:18 -0600:

> Seems to be related to the number of headers in the group - I had
> forgotten to set the purge for the server, so I had messages going back
> several years.
> 
> Unlike your issue, I'm not seeing the processor utilization spike - it
> could just be the raw number of headers having to be sorted each time
> that was throwing it off.
> 
> But I'll poke through the open bugs and see if any of the issues I've
> run into are in there - I'd forgotten that Charles had set this up to
> track bugs.
> 
> Jim
> 
> On 5/10/07, SciFi <address@hidden> wrote:
>> On Thu, 10 May 2007 19:05:38 +0000, Jim Henderson wrote:
>> >
>> > One other thing that I had forgotten to mention - the download
>> > performance is AWESOME (as I knew it was in these newer builds), but
>> > I've had points where Pan totally seizes (it eventually does come
>> > back to life, but while it's stuck it's seemingly hung) when
>> > downloading groups or switching into a group that has many tens of
>> > thousands of headers in it.
>>
>> I opened this bug:
>> <http://bugzilla.gnome.org/show_bug.cgi?id=430628> There are a few
>> others depending on exact scenario. Please check bugzilla and chime-in
>> there on which ones may be affecting you.  It's a sure way to "vote"
>> for which bugs are, ummm, bugging most ppl.  ;)
>>
>> > [...]

Please don't top post.  It messes up quote context.  There's a reason pan 
warns about it.  Please respect that on the pan list, even if you top 
post elsewhere.

Old-pan was fully multithreaded.  This increased performance some and 
responsiveness dramatically, but made coding and debugging timing issues 
between the threads incredibly difficult.  In the rewrite, Charles 
decided that the added complexity wasn't worth it in the main, so while 
pan now spins off worker threads in some cases (when first negotiating 
connections, consolidating back to a single thread after they are 
established, and either just added or actively being tested, for 
decoding), the main loop, group updating, and the GUI are all single 
threaded.

What's happening is that pan is only multi-threaded now where a single 
thread creates a noticeable bottleneck.  The first place that was noticed 
was at initial connection, where it would sometimes take 10-20 minutes to 
establish all the available connections and get up to full speed.  If one 
was only downloading a moderate amount, and at full speed could have 
accomplished the entire download in 10-20 minutes, this was obviously 
very frustrating as it could increase the time to completion by 50-100%.  
I was one of the ones that complained about that.  So Charles made 
connection establishment multithreaded, and that bottleneck went away.

Then someone that was saving attachments directly noticed that pan would 
idle the connections sometimes while it decoded and saved big attachments 
(consisting of 50 yencoded parts or so).  Since it's generally the 
download that takes the time, seeing it sit there idle, increasing the 
total time to completion just to do the relatively fast local decode and 
save was very frustrating.  (I didn't see this one, as I tend to work in 
multiple steps, downloading to cache while I do other things, then coming 
back when everything's in local cache to actually sort thru things and 
decide what I want to keep and where, and what I want to kill.  Thus the 
decode and save wasn't occurring at the same time as the actual 
downloading, and couldn't slow the downloading down.) 

As for this new issue, bottlenecking on loading a group...

* If you have download new headers when entering group set, you'll see 
more of a slowdown.  Try turning that off and triggering your header 
downloads only manually.  That should help.

* pan now message-threads dynamically as it gets headers, saving the 
threaded list to disk.  This takes FAR less memory as various tricks are 
used to combine similar strings and on multisegment posts, display only 
one header for however many individual segments make up the part.  This 
is more work (and accomplished in that single thread) when downloading 
headers, but where old-pan would do it every time it started (thus the 
minutes long load time if you had many headers to thread), new-pan does 
it once.  Of course, loading the pre-threaded headers still takes some 
CPU and perhaps more importantly disk i/o time.  Since pan now doesn't 
keep everything in memory at once as it used to do, only a single group, 
it reloads this pre-threaded list every time you enter the group.

* The follow-on from the above is that at least if you have enough memory 
to give the system cache to work with in addition to the loaded group, 
you can decrease the stalls further by loading each group before you 
start your download, then starting your download in one before reloading 
the other group.  pan will still load the pre-threaded message list when 
you reload the group, but it'll already be in cache, so you won't have 
the disk activity to worry about.

* As mentioned above, I'm not been following the decode-and-save multi-
threading very closely, as I don't have the issue the way I do things, 
but I'm not sure if it has been merged into trunk or is still being 
tested outside of trunk.  If it hasn't been merged to trunk yet, you'll 
see slowdowns during decode and save, and thus be able to avoid them by 
using my technique of downloading to cache first (while working on 
something else, or sleeping/working/shopping/whatever), then coming back 
when everything's in local cache to do your actual sorting and saving of 
attachments to disk.  You'll need to increase your cache size, but I 
cover that in other posts.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman