[Pan-users] Re: Forcing an expire?

[Duncan]

Ron Johnson <address@hidden> posted
address@hidden, excerpted below, on  Thu, 09 Jul 2009 08:24:36
-0500:

>> It seems to me that's more likely to cause huge issues than the
>> comparatively small 1/3 gig tasks.nzb file that you've been complaining
>> about, particularly since the kernel should be caching the file and it
>> will normally be updating fast enough that the writes won't all get to
>> disk.
> 
> ????
> 
> I see tasks.nzb getting rewritten almost on a continuous basis.  The
> issue is much more noticeable now that I'm on giganews and my nntp
> bandwidth has sextupled.

Yes, but what I'm saying is that if your write-caching is set correctly 
(you may have to tweak the parameters a bit as they're normally set to ~5 
seconds and 5 & 10% of RAM background and foreground flush, vm.dirty* in 
sysctl.conf, at least here, /proc/sys/vm/dirty* to access the file 
interface directly), with decent speed downloading, the rewrites should 
be triggered fast enough that the file never expires out of write cache 
and actually gets written to disk, because before it does, it's updated 
again.  Thus, that should be mostly memory-only writes, to the write-
cache, with another write before it actually gets updated on disk, and 
the actual disk file should only be updated when there's enough of a lull 
in activity to allow the write-cache to expire without getting updated 
and resetting the clock before it's written.

At least, that's the way I understand that it's supposed to work.

>> But a 3.6 gig single file, ESPECIALLY on 32-bit, is going to cause
>> **HUGE** issues if pan's trying to work with the whole thing in memory
>> at once, as I suspect it is.  If you're 64-bit and are working with a
>> decent filesystem, the issues won't be as bad, but it's still a huge
>> amount of data to be trivially shifting around!

This one's going to be updated similarly frequently when processing a 
header/overview download, but should be change-free during actual message 
download.  However, it's simply huge.  You say 8 gig RAM which will 
definitely help, but handling that much data even if it's just in memory 
is an issue.

> Which was the genesis of my "Better processing..." thread back on
> July-02.
> 
> Since then, I've discovered the 5-Minute Rule: anything you'll need
> within the next 5 minutes should be kept in RAM.  Everything else should
> be on disk.
> 
> http://en.wikipedia.org/wiki/Five-minute_rule

Did you happen to discover it reading the same /. article featuring the 
latest update to it that I did? =:^)

But meanwhile, that's pretty much what I'm saying about tasks.nzb.  While 
you probably don't want a 5 minute write-cache expiry (WAY too much data 
can be lost in 5 minutes), that default 5-second expiry between flushes 
is probably too small for your usage.  Something like say, 15-20 seconds 
is probably more realistic, and would likely keep it from actually 
flushing due to constant updates before write-cache expiry, much of the 
time.

Let me know if you need a better explanation of what those parameters do, 
before you attempt tweaking them.  I know I'd want more than I mentioned 
here, if I was going to be tweaking them and didn't already know a decent 
amount about what I was doing.  But no sense explaining it needlessly if 
you're already aware of how it works.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman