[Pan-users] Re: Pan Memory usage options?

[Duncan]

Kelly, Mark posted
<address@hidden>,
excerpted below,  on Thu, 08 Apr 2004 09:22:42 -0400:

> Pan is by far my favorite newsreader but the memory usage is driving me
> crazy.
> 
> I was playing around trying to find groups with 1 million plus articles to
> measure some memory usage.  Well I was exploring one with 1.4 million and
> here's a dump from TOP sorted by memory usage.  This is right after I had
> hit "get new articles" when I already had 1.1 million existing articles.
> 
[]
> 
> I got tired of waiting on pan to come back to life because of the massive
> swap file usage going on, so I killed the process shortly after.  Just glad
> I upgraded this box to 1 Gig of RAM, but it looks like even that isn't near
> enough.
> 
> How many other see this type of usage with huge article counts?

PAN has known memory scaling and performance issues with overview counts
approaching one million. It used to happen at about 200K overviews, but
with optimization, Charles and Chris managed to raise it to ~1M overviews.

Unfortunately, that isn't likely to change near-term, due to PAN's
dependencies.  The problem is the scaling issues on the GTK widget PAN
uses to list the overviews.  The one that would NORMALLY be used for the
job (treeview or some such, not a developer here, but followed the
discussion) doesn't scale well AT ALL in GTK2, at least not into the 100k
entry range.  They had to use a different widget to get ANY decent
scaling, and that came with its own issues and STILL doesn't scale so well
at several hundred k entries.

Mid-term, and there was recently some activity on the pan-devel list on
this so it's possible this is finally going to happen.. PAN will switch to
a database library back-end, probably SQLite.  While GTK widgets are
designed primarily for GUI applications and just happen to have some
convenient data management hooks built-in, thus, aren't really optimized
for data processing and likely never will be, that's the PURPOSE of
databases, and in theory, at least, the switch to SQLite could/should
bring dramatic improvements in this area, as the same display widget would
be used but the data handling would be offloaded to a library designed and
optimized for it.  There are a number of other advantages too, and the
switch will open up the door to a whole host of fancy features (like
integrated virtual server handling, similar to BNR2 but hopefully more
stable, and of course open source) now on hold due to implementation
issues, so the development has been one regulars such as myself have been
anticipating for some time.  However, one of the bigger and more immediate
benefits, since it won't require additional coding beyond the core
implementation, should be that PAN should scale dramatically better,
raising the bar to 5 million overviews or so, at minimum, hopefully.

Of course, that's the theory.  Actual implementation may take awhile to
match the ideal, even if it IS possible to reach it in practice, which it
might not be for some reason or other.  However, I'd guess a factor of two
scaling should be pretty reasonably achievable, bringing the now ~1M
overview ceiling b4 memory thrashing occurs to something over 2M overviews.

If the discussions on the devel list, with someone with some real db
experience volunteering to help with the migration and Charles saying now
might be a good time to do the migration because PAN's current code base
has reached a fairly stable plateau, continued in as promising a vein once
they were taken to private mail, it's possible the back-end switch might
be as early as version 0.15.0,  (betas 0.14.9x), and released later this
year.  If not, it's anyone's guess, as Charles seems to be have had his
life full with meat-space complexities lately and hasn't had as much time
to work on PAN, meaning it hasn't been progressing as fast lately as it
did for awhile.  (Here's hoping his life simplifies.. only in a good way,
of course.. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"They that can give up essential liberty to obtain a little
temporary safety, deserve neither liberty nor safety." --
Benjamin Franklin