Re: see file size limits

[Bob Proulx]

Kepner, Jeremy - 0553 - MITLL wrote:
> When I try and run sed on files larger than 2GB the resulting file is empty.
> -u flag doesn't help.
> 
> System is 32 core, 96 GB, Linux 2.6.32
> sed version is 4.2.1

I cannot recreate your result.  Your system resembles my 64-bit amd64
Debian Squeeze 6.0 system where I performed this quick and dirty test
to verify the operation.

  echo now is the time $(seq 1 300) > testdata1
  : > testdata2
  while [ $(stat --format "%s" testdata1) -lt $((2*1024*1024*1024)) ]; do
    cat testdata1 >> testdata2
    cat testdata2 >> testdata1
  done
  sed 's/^now/Now/' testdata1 > testdata2

  ls -log
  -rw-rw-r-- 1 3905232424 Feb  5 11:26 testdata1
  -rw-rw-r-- 1 3905232424 Feb  5 11:29 testdata2

The first part was just the first way I thought of to create a large
file.  It creates a text file just under 4G in size called testdata1.
I am sure there are more clever ways to do this.

The second part makes a simple sed operation on that file and writes
testdata2.  Both files are large, nonzero, and appear to be correct.
Therefore I believe your problem to be elsewhere.

What filesystem are you trying to use for this operation?

Are there file size quotas limiting total space in operation?

Bob