Re: diff: memory exhausted

[Bill Rugolsky Jr.]

On Fri, Dec 12, 2003 at 03:27:50AM -0500, Michael Elizabeth Chastain wrote:
> This looks reasonable.  Your system has 112 megabytes of free memory
> and 80 megabytes of swap, and you want to diff 2 * 200 megabytes of
> data.
> 
> I know three things you can try:
> 
>   diff -H
>     heuristic to find small changes in large files
> 
>   split(1) on each input file and then diff the splits:
>     mkdir z1 z2
>     cd z1 ; split ../file1 ; cd ..
>     cd z2 ; split ../file2 ; cd ..
>     diff -r z1 z2
> 
>   add more swap space:
>     dd if=/dev/zero of=/some.file.system/foo.swap bs=1048576 count=512
>     mkswap /some.file.system/foo.swap
>     swapon /some.file.system/foo.swap

A fourth:

Calculate a per-line hash for each file, then diff that.
If you can, increase the number of columns output by xxd, to increase
the compression ratio.

Post-process the diff to substitute the lines back (using perl or
whatever), and check for spurious collisions.

Regards,

        Bill Rugolsky