bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: String replacements leak small amounts of memory each time


From: Matthew Woehlke
Subject: Re: String replacements leak small amounts of memory each time
Date: Tue, 22 Jun 2010 17:57:31 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.23) Gecko/20090825 Fedora/2.0.0.23-1.fc10 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0

address@hidden wrote:
Description:
When used in a script that iterates over several thousand lines of
logs or similar data, the bash string replacement functions seem to
leak memory. The Repeat-By list uses "ls -lR" to generate input, but
any data will do (try your system logs)

Repeat-By:
        Start a shell, and start "top" or some other resource monitoring tool

        Try one of the following:
        while read line; do test=${line%%/*}; done<  <(ls -lR)
        while read line; do test=${line//a/b}; done<  <(ls -lR)
        while read line; do test=${line#\ }; done<  <(ls -lR)

No question something bad is going on here.

Using bash 'bash-4.0.35-3.fc12.x86_64' (from Fedora 12); I know it's not latest but it's what I have convenient, and anyway the OP is reporting the issue against 4.1.5, so it probably hasn't been fixed.

I started two shells (each has exactly processed my startup scripts and nothing else); both start out at 109m VIRT.

I then ran:
shell1$ while read line; do temp=$line; done < <(seq 1 5000000)
shell2$ while read line; do temp=${line//1/2}; done < <(seq 1 5000000)

(This was so one could be sure the problem is the replacement and not something related to e.g. buffering the command output.)

After both completed, here's what top says:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30042 mwoehlke  20   0  109m 2040 1536 S  0.0  0.1   1:44.21 bash
30097 mwoehlke  20   0  264m 157m 1536 S  0.0  4.0   2:42.62 bash

Notice that not only did shell2 balloon significantly in VIRT, but it ate up 157m RES as well, while shell1's RES is practically non-existent. (I am confident the RES would drop significantly in the face of memory pressure, but it seems to indicate solidly that shell2 touched a much larger chunk of memory than shell1.)

Also, when run under valgrind, I see a number of leaks with block count exactly equal to the number of lines processed.

The line numbers here are probably off, but FWIW:

==30566== 16 bytes in 1 blocks are definitely lost in loss record 110 of 245
==30566==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==30566==    by 0x463F1A: xmalloc (xmalloc.c:86)
==30566==    by 0x428B39: alloc_word_desc (make_cmd.c:92)
==30566==    by 0x428C25: make_bare_word (make_cmd.c:104)
==30566==    by 0x43752B: copy_word (copy_cmd.c:61)
==30566==    by 0x4376F0: copy_word_list (copy_cmd.c:75)
==30566==    by 0x44867F: expand_word_list_internal (subst.c:8736)
==30566==    by 0x460177: redirection_expand (redir.c:253)
==30566==    by 0x460B5F: do_redirection_internal (redir.c:751)
==30566==    by 0x4615C9: do_redirections (redir.c:200)
==30566==    by 0x42EA06: execute_command_internal (execute_cmd.c:678)
==30566==    by 0x468D89: parse_and_execute (evalstring.c:315)
==30566==
==30566== 16 bytes in 1 blocks are definitely lost in loss record 111 of 245
==30566==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==30566==    by 0x463F1A: xmalloc (xmalloc.c:86)
==30566==    by 0x428B39: alloc_word_desc (make_cmd.c:92)
==30566==    by 0x428C25: make_bare_word (make_cmd.c:104)
==30566==    by 0x447A96: expand_word_internal (subst.c:8018)
==30566==    by 0x4487CE: expand_word_list_internal (subst.c:8664)
==30566==    by 0x460177: redirection_expand (redir.c:253)
==30566==    by 0x460B5F: do_redirection_internal (redir.c:751)
==30566==    by 0x4615C9: do_redirections (redir.c:200)
==30566==    by 0x42EA06: execute_command_internal (execute_cmd.c:678)
==30566==    by 0x468D89: parse_and_execute (evalstring.c:315)
==30566==    by 0x41B6D9: run_one_command (shell.c:1312)
==30566==
==30566== 79,968 bytes in 4,998 blocks are definitely lost in loss record 245 of 245
==30566==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==30566==    by 0x463F1A: xmalloc (xmalloc.c:86)
==30566==    by 0x428B39: alloc_word_desc (make_cmd.c:92)
==30566==    by 0x428C25: make_bare_word (make_cmd.c:104)
==30566==    by 0x447A96: expand_word_internal (subst.c:8018)
==30566==    by 0x4487CE: expand_word_list_internal (subst.c:8664)
==30566==    by 0x42DB8E: execute_simple_command (execute_cmd.c:3577)
==30566==    by 0x42EF56: execute_command_internal (execute_cmd.c:738)
==30566==    by 0x42F99D: execute_command (execute_cmd.c:373)
==30566==    by 0x4305DC: execute_while_or_until (execute_cmd.c:3016)
==30566==    by 0x42F0B3: execute_command_internal (execute_cmd.c:2984)
==30566==    by 0x468D89: parse_and_execute (evalstring.c:315)

--
Matthew
Please do not quote my e-mail address unobfuscated in message bodies.
--
What, no punchline?




reply via email to

[Prev in Thread] Current Thread [Next in Thread]