Re: libdiskfs or not mapping the entire store

[Marcus Brinkmann]

On Sun, Nov 11, 2001 at 01:04:35PM +0100, Neal H Walfield wrote:
> I am been thinking about what it would take to reimplement libdiskfs
> to not map the entire underlying store into memory.  I have a few
> ideas, however, I am sure that you also have some brilliant ones.  In
> fact, I remember tossing this idea around with Roland in an elevator
> in the AI building at MIT two years ago.

First, for any approach you must create memory objects for all 4GB chunks of
the partition.  Any memory object can only cover 4GB, so that's a hard
limit.  Any approach also requires to make sure everywhere the locations on
the partition are referred to by block, and not byte offsets.

Then, you have basically two options:

* The T-approch.  Keep a hash of mappings.  Whenever you access a part of
the partition, allocate a suitable mapping (or set of mappings), and use it.
After that, release it.  Mappings are cached, and freed if not enough
virtual memory is available.  It is not determined yet what is a suitable
slicing for the individual memory mappings.  Remember that metadata is
somewhat dense (inode blocks) and somewhat sparse (indirect blocks).
The disk pager will still cover the whole disk.

* The R-approach.  Reorganize the metadata to fill a 1 GB virtual range of
addresses.  Roland did an off-the-cuff calculation that 1 GB of metadata
would be enough to handle terabytes of disk space for ext2fs.
This would require to reorganize the metadata, which is simple to do for
metadat that covers a fixed linear range, but more difficult for indirect
blocks (which are allocated like file blocks).  Or, you can use several
pagers for different kind of metadata (which requires hierarchical locking).
This would make it easier to access the metadata, as the logic to find the
metadata is in the pager.

Thanks,
Marcus


-- 
`Rhubarb is no Egyptian god.' Debian http://www.debian.org address@hidden
Marcus Brinkmann              GNU    http://www.gnu.org    address@hidden
address@hidden
http://www.marcus-brinkmann.de