[Pan-users] Re: Re: A couple of .95 problems.

[Duncan]

Charles Kerr posted
<address@hidden>, excerpted
below,  on Fri, 05 May 2006 02:48:14 -0500:

>> Duncan wrote:
>>> Talking about which, how does the caching work in 0.9x (second time
>>> I've asked)? I don't see a size preference.
>>>
>> Here's how I changed the cache size.
>> gui/pan.cc line 186
>>
>> ArticleCache cache (cache_path,100); // replace 100 with the max size in
>> MB.  the default is 10.
>>
>> Now I just have to remember to do this for each new tarball.
> 
> I can put in a preferences line to let you change this, but let me ask a
> naive question -- why do you want to?  Since Pan won't expire any article
> that a task's got a reference to, the queue can grow far larger than the
> preferred maximum so long as the task exists. You could, for example,
> download a dvd iso without having to increase the cache size past 10M.

I believe the issue is one of folks (me) using PAN in ways the author
didn't envision.  <g>  As is common in such instances, one must explain
the use in ordered to get the point across, since the author obviously
hadn't considered that particular use.  <g>

There are two issues, the one I originally ran into back when old-PAN had
a 1 gig limit, and especially now, a newer/different one.

The issue back then was a basic one of how one approaches downloading and
sorting binaries.  Your example, downloading a single ISO complete, saving
it, and moving on, is one way, and may work.  Even still, I'm not sure
how well it will work if one is grabbing from multiple groups at a time. 
Will the second ISO start overwriting the cached parts of the first while
they are both downloading, the first overwriting the second, so neither
one ends up with more than about 5MB saved after downloading 10 /gigabytes/!!?

That's essentially what used to happen to me, but for a different reason. 
Rather than saving everything directly to binary and moving on, then
sorting and deleting or final-saving the binaries later, working directly
with the filesystem, not the newsclient, I normally prefer to use a
newsclient in a dramatically different way.  After deleting obvious spam
and reposts I already have, I'll select everything remaining and set PAN
downloading it all.  Then I do the second step within PAN, viewing
downloaded messages within PAN (if it's a simple jpeg or whatever, that
PAN will display), or saving them out individually and opening them,
deciding whether I want to keep them or not, and where I want to save them
if so, before moving on.  This way, I have not just the filename as
attached, but all the extra data in the posts that went with it, the
author, the date posted, the title of the post, etc, that I can use to
help sort and categorize the resulting binaries.  Saving the binaries out
directly, rather than caching the posts and processing them within PAN,
I'd lose the benefit of all that extra metadata!

Naturally, this sort of processing requires a much larger cache, typically
a gig at minimum, for an active still-image binary downloader, several gig
for an active MP3 downloader, tens of gigs if one is downloading full DVD
ISOs.  Without that, I was **VERY** dismayed to find after downloading
several gigs of content, over a session lasting several hours, that most
of it was already deleted before I'd even looked at it, because PAN was
practically operating with no cache (well, a few megs, but that's
practically no cache, when one's dealing with content in the hundreds of
megabytes or tens of gigabytes range!).  

(Not exactly no topic, but I've found it helpful, and this seems a good
place to mention it...  For many years, from back in my MSWormOS days, in
fact, I've actually run a dedicated news partition.  Formerly, it was my
news cache, and I setup an 8 gig partition for that purpose when I
switched to Linux.  After I found PAN couldn't use it, I switched to using
that for my intermediate stage of binary processing -- after I'd decided I
wanted to save the stuff and had performed the initial sort into subdir
categories, but before I saved it to its semi-permanent location on my
media partition, for eventual burn-off to ISO or deletion if I decided it
wasn't worth keeping around after all, a few months later.  I like that
dedicated partition management, and as a result thereof, would actually
find an "unlimited" cache option quite useful as well, with PAN not
managing cache until it ran into filesystem full errors.)


Naturally, I soon discovered the cache setting in the old (old) PAN and
set it to maximum, which helped, but back then, it was only a gigabyte,
which still wasn't enough.  That's why I requested that the max be raised
to at least 4 gigs and was very happy to see you raise it to 20 gig, must
have been about 0.13 or so I'd guess.  Of course, with broadband speeds
increasing and full DVD ISOs now commonly traded, that 20 gig is only
about 4 DVDs worth of content, so even that's not likely to be enough for
those serious DVD downloaders.  Personally, I'd consider making it a
terabyte or similar, which should stave off requests for upping it, for
awhile anyway.  <g>

I should also mention that I almost dumped PAN due to this "bug", before I
figured out what was happening.  Setting up your newsclient to download
several gigs of content before you go to work, returning after work
expecting to find it all ready and waiting, only to find less than 1%
actually still there, is a STRONG signal the newsclient you are testing is
NOT going to workout!  =8^(  While complaints haven't been common on the
list, I'm not the only one who has come across this, as I recall
explaining the situation to a couple of others over the years who've
bothered to complain about PAN's not retaining stuff they downloaded, on
the user list.  I'd guess that's the tip of the iceberg in terms of the
number that have actually tried PAN and simply moved on to something else,
when it appeared not to be able to even reliably keep track of what it was
downloading.

That's the first problem.  The second problem is eventually going to hit
people even if they save binaries directly, rather than downloading to
cache as I prefer and processing later.  With PAN headed for better
multi-server support, it's going to be used for binaries more and more
often, and we're just heading for a collision with this one if something
isn't changed to fix it.

Ask yourself what happens to the two months of saved messages configured
for a text group, when someone downloads that ISO you mentioned.  BAM!!! 
ALL GONE!!!  How many text group users, certain ones of which I know
actually keep a permanent archive of messages going back /years/, are
going to be happy with a "feature" that autodeletes all those messages --
again, going back /years/ in some cases -- when they download just a
/single/ largish (10MB) MP3?

I think it should be obvious at this point where that impending crash I
mentioned is coming from.  =8^(  Hopefully, I've explained it well enough
to get at least an edit-the-config-file setting, if not a spot in
preferences. 

Maybe, considering the text-message deletion scenario I mentioned, thought
should actually be given to something seriously high (in PAN terms) as the
default.  A gig default would allow folks to downoload a full CD ISO with
a reasonable safety margin for saved text posts.  A 6 gig default would
allow folks to download a 4.7 gig DVD with a reasonable safety margin,
even if they never touched their cache settings.

As I mentioned above, an "unlimited" option (say, a -1 in the config file)
would also be of some use, where PAN wouldn't manage cache until it
started running into filesystem error messages.  That's perfect for those
of us that like dedicated news-cache partitions, among others, tho I'd not
advocate making it the default.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html