bug#10877: Wimpy external files.

[Rogier Wolff]

On Sat, Feb 25, 2012 at 10:41:05AM -0800, Paul Eggert wrote:
> A problem with that idea is, suppose we have many independent 'sort'
> invocations running at at the same time, as part of a shell pipeline
> say?  If they each grab 1/8 of physical RAM, merely because they
> want to sort piped data of a few bytes, they may exhaust swap space.

Two things. 

First: Many modern operating systems do "lazy" allocation. With 8G RAM
+ 8G SWAP you can have hundreds of apps each allocating a gigabyte, as
long as they don't all use that gigabyte. 

Second: If you're afraid for a lone operating system that DOES show
this problem, then a slight change in the codebase might be in order
for "unknown sort size". Then the "1/8th of memory" becomes an upper
bound and the buffer should be dynamically expanded. If you do not
have to "move" the data from the old buffer to a new buffer you can
just allocate a 2x larger buffer each time you reach the limit (but
are below the upper bound). If you DO have the expense of having to 
move all the data up to that point, I would suggest allocating a
4x larger buffer up to the upper bound. 

        Roger. 

-- 
** address@hidden ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
The plan was simple, like my brother-in-law Phil. But unlike
Phil, this plan just might work.