Re: [Pan-users] 64 bits fails as solution to large binary groups

[Duncan]

Ron Johnson posted on Sun, 09 Oct 2011 22:15:24 -0500 as excerpted:

> On 10/09/2011 09:58 PM, Lacrocivious Acrophosist wrote:
>> Ron Johnson<address@hidden>  writes:
>>
>>> [64-bit pan]
>>>
>>> One of the first things that I did was try out Pan on a binary group.
>>>
>>> Many hours later, it had fetched 6 weeks of headers and consumed 6.8GB
>>> of RAM.  The 2+ years of data in Giganews would require 123GB of RAM.
>>>
>>> :(
>>>
>> [I]s this 64-bit performance different from 32-bit performance
> 
> It's a fact that 32-bit Pan runs out of *process* address space at
> around 2GB.  64-bit Pan doesn't technically have that problem, but
> effectively it does, although it does for all practical intents.

Well, the 32-bit part isn't quite accurate, or at least it's accurate for 
only a subset of 32-bit.

The 32-bit memory limit has to do with the amount of memory addressable 
by a 32-bit number.  2^32 = 4 Gi, so if the addressable unit is single
8-bit bytes as is the case on x86 (both 32-bit and 64-bit), that's a 
directly addressable memory area of 4 GiB.  However, from the viewpoint 
of an application, traditionally that address space is divided in half 
between user-space and kernel-space addresses, 2 GiB for each.  That's 
where the above 2 GiB limit comes from.  (Do note, however, that this 
viewpoint is per-app -- each app gets its own 2-gig userspace address-
space view, it's not shared.)

But, at least on Linux, as that 4 GiB total space began to look smaller 
and smaller, a number of alternatives were developed.  Generally, these 
options are applied to the kernel configuration pre-build, so it depends 
on what kernel you choose to run.  As is often the case, that means the 
choices available to those who configure and compile their own sources 
have far more choices exposed to them than those who choose to run pre-
built binaries.[1]

In addition to the 2:2 split, there's also 3:1 and 1:3 split choices 
available, so it should be possible to run a 3-gig pan on a 32-bit kernel 
set for 3-gig of userspace and 1-gig of kernel-space.

Additionally, there's a 4:4 mode option, where userspace and kernelspace 
each have their own separately addressed 4-gig space.  This allows apps 
to utilize a full 4 GiB userspace, but at a measurable performance cost, 
since the hardware must now switch between address modes and fully flush 
the TLB (address translation lookaside buffer) each time it switches 
between usermode and kernelmode, even if it's the same app.  A full TLB 
flush costs a significant amount of CPU cycles, so this means a 
significant drop in efficiency for apps that make a lot of calls to 
kernel services.  As such, it's not recommended for desktop use, but it 
*IS* available, for those who need to use the full 4 GiB address space in 
either usermode or kernelmode, for a particular workload.

So if one chooses the right kernel, it should be possible to run even 32-
bit pan up to close to 4 GiB userspace memory.  However, given the kernel 
calls pan makes for access to both the network and block-storage devices, 
this could be somewhat slow.

Meanwhile, it should be noted that back in the 16-bit era, a segmented 
memory space addressing scheme was generally used, as opposed to the 
"flat address space" scheme generally used four 32-bit and 64-bit 
processing.  Were it to be necessary, the same segmented memory 
addressing scheme could certainly be used again.  However, a flat address 
scheme is *VASTLY* simpler, and as it happens, 64-bit machines that 
eliminated the 4 GiB addressing barrier were introduced at just the right 
time to avoid forcing people back into segmented address schemes, as 
would have certainly become the case had 64-bit consumer-level systems 
not been introduced.

That brings us to 64-bit.  As mentioned, the additional address space 
available in 64-bit eliminates the problem for the foreseeable future, 
allowing flat addressing to be used up into the PiB, IIRC.  As it 
happens, the additional number-space addresses a number of other problems 
as well, including the fact that it's quickly becoming feasible to brute-
force a 32-bit random number space in many cases, so there's significant 
security advantages to switching to 64-bit as well.

All that said, the base fact remains true; per-app usable memory on a 32-
bit system is QUITE restricted, for usage measured in the GiB.  64-bit 
removes that barrier.

But, while 8-16 GiB physical memory machines aren't entirely unusual in 
the consumer space any more, much beyond that certainly remains so.  
Thus, a 128 GiB magnitude memory requirement remains almost as much out 
of practical range as it always has.  It at least should be possible to 
buy systems that have that sort of memory slot capacity, these days, but 
it'll be a few years yet before consumer level machines will be shipping 
with that sort of populated memory, for sure.

But hey, at least it's possible to reasonably project the sort of memory 
it'd require, on a reasonably consumer level system, now.  That's 
definitely progress from a few years ago, both because pan uses far less 
memory than it did before the rewrite (back in the C pan days, version 
0.14 era, pan would choke on ~2 million headers, regardless of the memory 
one had in the machine; it simply didn't scale at all), and because of 64-
bit and 4-8 GiB memory being reasonably common, these days.

>> As for the multi-bazillion-header binary groups... is there *any* 'old
>> style'
>> newsreader capable of downloading all their headers? By 'old style' I
>> mean newsreaders intended to include conversation. Giganews, for one,
>> would seem to me to make this nearly impossible due to their vast
>> retention span.
>>
>>
> Any "straight to file" news reader could do it, given the time to d/l
> all the headers.
> 
> Pan's fatal binary group flaw is that it stores all the headers in
> memory before writing them out to disk.

Not only that, but it reads them all back into memory every time it 
starts!  That's why pan takes so long to load from cold-disk-cache if you 
up the cache size and set no-expiration, even on text groups, as I do 
here.

The biggest problem is pan's assumption that it has all the information 
necessary to maintain its threading structure in memory at all times.  In 
ordered to really allow pan to become a disk-based client, to store most 
of that info on disk and only read in a rather smaller limited working 
set at once, pan really needs some sort of header indexing or at least 
hashing system devised, such that it can figure out what info it needs to 
read in from disk, from a vastly larger on-disk store, in ordered to work 
with and properly thread, at a minimum, the currently displayed article 
headers, likely plus some pages from the article list before and after 
the currently displayed set.

But curing that "it's all in memory and immediately accessible for 
threading purposes" logic, is likely to be quite a big job, 
unfortunately.  I wonder if it's that challenge that had Charles finally 
giving up, after leading pan's development all those years, thru at least 
one rewrite?

---
[1] This is of course one of the big advantages Gentoo touts in general, 
that for the most part, it deliberately exposes these choices to the end 
user, yet still within a structured and highly automated framework, so 
while the builds do take time, for the most part they can be done in the 
background and the time spent actually administering the system, at least 
once the basic choices are made, can compare reasonably favorably to that 
for other distros.  However, Linux is modular enough and the kernel 
separate enough from the rest of the system, that it's not /that/ 
difficult to learn how to configure and build just the kernel on a 
regular distro, if one so desires, as one might if they aren't happy with 
the available pre-compiled kernels on their distro of choice.  In fact, I 
was configuring and building my own kernel, on the then Mandrake (now 
Mandriva/Mageia) binary distro I was running as a Linux newbie, even 
before I had finished my switch to Linux and before I had chosen my 
desktop environment and my defaults for apps such as the browser, mail 
and news clients.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman