Re: see file size limits

[Jeremy Kepner]

Hi Bob,
  I believe the issue is that my text file had no newline characters
and so appeared as a line > 2^31 characters.  It appears that this is fixed
in 4.2.2, which I hope to test soon.

Regards.  -Jeremy

On Tue, Feb 05, 2013 at 01:37:33PM -0500, Bob Proulx wrote:
> Kepner, Jeremy - 0553 - MITLL wrote:
> > When I try and run sed on files larger than 2GB the resulting file is empty.
> > -u flag doesn't help.
> > 
> > System is 32 core, 96 GB, Linux 2.6.32
> > sed version is 4.2.1
> 
> I cannot recreate your result.  Your system resembles my 64-bit amd64
> Debian Squeeze 6.0 system where I performed this quick and dirty test
> to verify the operation.
> 
>   echo now is the time $(seq 1 300) > testdata1
>   : > testdata2
>   while [ $(stat --format "%s" testdata1) -lt $((2*1024*1024*1024)) ]; do
>     cat testdata1 >> testdata2
>     cat testdata2 >> testdata1
>   done
>   sed 's/^now/Now/' testdata1 > testdata2
> 
>   ls -log
>   -rw-rw-r-- 1 3905232424 Feb  5 11:26 testdata1
>   -rw-rw-r-- 1 3905232424 Feb  5 11:29 testdata2
> 
> The first part was just the first way I thought of to create a large
> file.  It creates a text file just under 4G in size called testdata1.
> I am sure there are more clever ways to do this.
> 
> The second part makes a simple sed operation on that file and writes
> testdata2.  Both files are large, nonzero, and appear to be correct.
> Therefore I believe your problem to be elsewhere.
> 
> What filesystem are you trying to use for this operation?
> 
> Are there file size quotas limiting total space in operation?
> 
> Bob