[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Memory leak when catting(/sedding/...) large binary files with backt

From: Chet Ramey
Subject: Re: Memory leak when catting(/sedding/...) large binary files with backticks
Date: Sat, 10 May 2008 22:01:55 -0400
User-agent: Thunderbird (Macintosh/20080421)

benj@pml.ac.uk wrote:
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' 
-DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
-DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib  -D_GNU_SOURCE  -O2 -g -pipe 
-Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector 
--param=ssp-buffer-size=4 -m64 -mtune=generic
uname output: Linux pmpc983.npm.ac.uk #1 SMP Sat Mar 29 
09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu

Bash Version: 3.2
Patch Level: 33
Release Status: release

        Using echo `cat ...` on a large binary file causes lots of memory to be 
used (fine), but if you ctrl-c while it's running
        it doesn't die properly and doesn't return used memory when finished. 
Originally found by screwing up a sed command (can
        also reproduce bug using sed rather than cat) while trying to rename a 
group of files.

        Every time
        1. Find large binary data file for test (mine is ~3.2GB)
2. echo `cat filename` 3. Ctrl-C previous command while running (doesn't terminate)
        4. When step 2 eventually returns it does not release memory

I'm not sure what you mean by `doesn't return used memory', but if you mean
a process's size as reported by ps or similar, that does not indicate a
memory leak.  A memory leak is memory that has been allocated by a program
to which it retains no handles.

malloc acts as a cache between an application and the kernel.  Memory
obtained from the kernel using malloc may, under some circumstances, be
returned to the kernel upon free, but this may not always be possible.
Memory that is not returned to the kernel by freeing pages or using sbrk
with a negative argument is retained and used to satisfy future requests.

I ran your test using valgrind to check for memory leaks (but with only
a 330 MB file), and it reported no leaks after ^C.


``The lyf so short, the craft so long to lerne.'' - Chaucer
                       Live Strong.  No day but today.
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/

reply via email to

[Prev in Thread] Current Thread [Next in Thread]