bug#20220: severe memory leak on emacs 24.4.1

[Mario Valencia]

Then the shell execute function is worthless. I had used it for opening the browser and also for opening files with an external program, or to open them in the explorer, i guess i will have to remove all use of that function from my scripts.

2015-03-29 9:35 GMT-06:00 Eli Zaretskii <eliz@gnu.org>:

> Date: Sat, 28 Mar 2015 17:39:38 -0600
> From: Mario Valencia <mariovalspi@gmail.com>
>
> to reproduce it, i create a small emtpy html document. then i evaluate the
> following _expression_:
>
> (dotimes (i 100) (browse-url-of-file))
>
> This causes emacs to start opening the file using google chrome. In the task
> manager, i can see emacs' memory usage go up by about 5 megabytes each time a
> tab is opened. When it opens about 30 tabs, emacs is using 138 megabytes of
> memory, and it gives the error below.
> the translation is something like this: "ShellExecute failed: Storage space
> insufficient to process this command"
> My harddrive has enough storage space btw.

(The error is not about disk storage, it's about reserving the address
space in virtual memory.)

FWIW, I don't see such a large memory increase when I reproduce this,
I see something around 1MB, sometimes 1.5MB.  But I didn't try on
Windows 8.

Anyway, Emacs doesn't allocate any memory when it calls the
ShellExecute function, so I see no way we could leak something here.

My guess would be that invoking ShellExecute causes Windows to start a
thread in the context of the Emacs process, and reserve 8MB of stack
space for that thread.  (On one of 3 systems I tried your recipe, I
actually saw a thread per invocation, each thread was running some
function inside shlwapi.dll, the shared library which implements the
ShellExecute API.)  The memory actually used by that thread for its
stack is much smaller than 8MB, of course, but Windows attempts to
reserve 8MB of address space for its stack.

The 8MB figure comes from the way we link Emacs: we need such a large
stack due to regular expressions, stack-based Lisp objects, and GC
which is deeply recursive.  By default, each thread reserves the same
stack space as the program to which it belongs.  For threads we launch
in Emacs, we override the default 8MB stack space by a much smaller
value, but we have no such control on threads that Windows itself
starts on behalf of the Emacs process.

The error message and the failure to launch too many browser tabs are
caused by the fact that Emacs itself reserves about 1.7GB of the
address space for its memory management, leaving only a small portion
of the 2GB address space that 32-bit programs can use on Windows.
Start enough threads with 8MB stack reservation, and you will run out
of address space.

Emacs 25 uses a different memory allocation scheme for buffers and
strings, which allows to avoid the large memory reservation, so at
least the out-of-memory error should happen there only after many more
ShellExecute invocations (because threads started by Windows on behalf
of ShellExecute will still reserve 8MB of memory for their stack).

If someone knows how to force threads started by Windows to reserve
less memory, without also lowering the stack space available to Emacs
itself (i.e. its main thread), please tell.  Otherwise, I guess we
will have to live with this limitation on Windows.