Re: Memory leak when catting(/sedding/...) large binary files with backticks

[Chet Ramey]

benj@pml.ac.uk wrote:
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc
> Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
> -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' 
> -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
> -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib  -D_GNU_SOURCE  -O2 -g -pipe 
> -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector 
> --param=ssp-buffer-size=4 -m64 -mtune=generic
> uname output: Linux pmpc983.npm.ac.uk 2.6.24.4-64.fc8 #1 SMP Sat Mar 29 
> 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> Machine Type: x86_64-redhat-linux-gnu
>
> Bash Version: 3.2
> Patch Level: 33
> Release Status: release
>
> Description:
>         Using echo `cat ...` on a large binary file causes lots of memory to be 
> used (fine), but if you ctrl-c while it's running
>         it doesn't die properly and doesn't return used memory when finished. 
> Originally found by screwing up a sed command (can
>         also reproduce bug using sed rather than cat) while trying to rename a 
> group of files.
>
> Repeat-By:
>         Every time
>         1. Find large binary data file for test (mine is ~3.2GB)
> 	2. echo `cat filename` 
> 	3. Ctrl-C previous command while running (doesn't terminate)
>         4. When step 2 eventually returns it does not release memory

I'm not sure what you mean by `doesn't return used memory', but if you mean
a process's size as reported by ps or similar, that does not indicate a
memory leak.  A memory leak is memory that has been allocated by a program
to which it retains no handles.

malloc acts as a cache between an application and the kernel.  Memory
obtained from the kernel using malloc may, under some circumstances, be
returned to the kernel upon free, but this may not always be possible.
Memory that is not returned to the kernel by freeing pages or using sbrk
with a negative argument is retained and used to satisfy future requests.

I ran your test using valgrind to check for memory leaks (but with only
a 330 MB file), and it reported no leaks after ^C.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                       Live Strong.  No day but today.
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/