Re: [Pan-users] Feature request: compressed header lists

[Duncan]

Heinrich Mueller posted on Wed, 08 Jun 2011 09:35:37 +0200 as excerpted:

> On 06/07/11 21:39, Duncan wrote:
>> Arnd posted on Tue, 07 Jun 2011 14:22:27 +0000 as excerpted:
>>
>>> Compressing the files in $HOME/.pan2/groups/ would save a lot of space
>>> for big groups, and therefore make it easier to take .pan2/ with you
>>> onto space limited devices like netbooks, and or syncing it between
>>> computers.
>>>
>>> Ideally this would be a per group setting, with a choosable compress
>>> format (gzip, bzip2, etc.).
>>>
>>> Or is there a trick to do this right now?
>> In theory it could be done now, if you're using a filesystem like
>> reiser4 or btrfs, both of which are supposed to have builtin/plugin
>> on-the-fly compression available.
>>
>> A rather more practical alternative would be using a pan wrapper script
>> that among other things, decompresses these files before pan starts,
>> then waits around until it closes and recompresses them.  I use a
>> wrapper script here for other reasons, but not for file compression.
>>
>> FWIW, ordinary messages, both text and non-yenc binaries, should
>> compress extremely well too, since text and both UUE and base64 encoded
>> binaries are not particularly efficient, particularly UUE/base64
>> encoded binaries.
>> However, pan's cache is only 10 MB normally anyway, reasonable as a
>> text cache, but it'll blow away REAL fast as a binary cache, so unless
>> you have manually increased the cache to multiple gigs as I do...
>>
> Are we speaking of decompression into ram or to disk?
> The former could be achieved with some fiddling in the encode-cache
> constructor...

I was speaking of decompressing to disk, altho if you have enough RAM, you 
could always use tmpfs or the like, which would be in ram but look like 
disk.

The reason being that for many people, particularly on legacy 32-bit 
hardware or OSs, pan already has memory issues, especially in the large 
binary groups with many millions of headers.  Ram-based header 
decompression is thus not particularly practical for 32-bit pan (even 32-
bit in PAE mode, which can in theory handle up to 64-gig, has problems, 
because I believe the largest possible userspace available to a single 32-
bit app is 4 gig, even if the system as a whole has 64-gig), tho for 64-
bit machines with > 4 gig of physical RAM , it may still be reasonable.  

But there's still enough 32-bit out there that I don't believe it likely 
that pan would do memory-based decompression.

Unless someone comes up with a clean patch and submits it for inclusion, 
of course.  There's a lot that can be done if someone with the skills (or 
money to pay someone with the skills) cares enough about an issue to 
create a patch, even if most devs wouldn't consider it a good use of their 
time, given the other stuff they could be doing instead.

And it'd be an even BIGGER bonus if someone came up with a patch that 
allowed pan to work with primarily compressed headers and only decompress, 
say, the page it's displaying and the page before and after that, thus 
killing two birds with one stone, if pan then used LESS memory than 
before, as WELL as allowed on-disk compression (presumably controlled with 
an option, altho it might be a file-only option without a UI switch, a 
number of pan features work that way in ordered to keep the complexity of 
the options dialogs down).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman