Re: [libmicrohttpd] Memory leak and global variables

[Nicolas Mora]

Hello again,

Le 2014-08-28 15:51, Kenneth Mastro a écrit :
> Glad my comments helped.
>
> One quick thing/clarification - that bug in the latest 'released'
> version of MHD was related to cleaning up the list of internal
> connection/request structures.  (This occurred during the 'stop daemon'
> call - manifesting itself with a very long shutdown time.)  To get the
> fix for this, you'd have to download the latest source - the released
> version still has the bug.
I will try it, but since my program is a standalone server, it doesn't 
stop the daemon unless the server is stopped.

>  From what I can tell, MHD re-uses the same internal structure/thread if
> it's not servicing multiple requests.  I.e., if you do one request at a
> time, the same one gets used over and over.  If you do 3 at once, new
> ones get created so 3 get used.  Something related to this could explain
> why you don't see the memory leak when you do the requests one at a
> time.  Maybe there's a leak in MHD for each new structure/thread that
> gets created and/or each one that gets created beyond the first?
> Alternatively, maybe your code just isn't thread-safe?
My code is designed to be thread-safe but maybe it isn't because I 
forgot something.

> Does the memory leak continue to occur/grow over time, or is it just
> 20kB per connection/request the first time?  In other words - if you hit
> it with 6 simultaneous requests and get 120kB of memory leak, wait 10
> seconds, then hit it again with another 6 simultaneous requests .. do
> you get ANOTHER 120kB of memory leak, or does it stop getting worse?
> Knowing that may provide you with some clues.  (E.g., if it doesn't go
> up any more, it seems more likely to be an MHD thing in its internal
> structures.  If it continues to go up, it could be either/or.)
I'm talking about memory used getting higher, every time I call the 6 
requests simultaneously.

The command I use is 'ps u p `pgrep angharad`', and I monitor the RSS value.
For example, when the program starts, the result is the following:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
pi        6076  2.0  0.4  20384  2276 ?        Sl   23:30   0:00 
/usr/local/bin/angharad /etc/angharad.conf

then I make the first 6 calls, I have the new values:
pi        6076  2.3  0.4  20384  2308 ?        Sl   23:30   0:00 
/usr/local/bin/angharad /etc/angharad.conf

So after the first 6 calls, the memory used increases from 2276 to 
2308Kb. Then the next calls raise the memory used to 2340, 2356 and 
2456Kb, and so on. That's where my suspicion of memory leaks come from.

> Have you tried Valgrind or some printfs to make sure that free is called
> for every malloc you're doing?  Even a simple counter would help.  I'm
> guessing you really need to make absolutely sure the free is getting
> called for every one - especially with lots of early-returns.
I traced my program with printf every time I malloc and free variables.
I have made a few tests, and as far I can tell, all mallocs variables 
are freed every time.

Unfortunately, Raspbian uses a hack that won't be supported by valgrind, 
so valgrind detects an early error. Here is valgrind output on the rpi 
(my program is compiled with the debug options).

In the next days, I will monitor the program on another platform with 
valgrind. By taking a closer look, I found one leak, but it's not over yet.

> Another thought - maybe one of your (or MHD's, I guess) global variables
> is getting overwritten with each request.  Doing one request at a time
> gives the code a chance to de-allocate the memory before the pointer
> gets overwritten.  That seems more likely to me than some cleanup code
> never being called.  What I'm getting at - I'm assuming your code is
> getting called by MHD's multiple threads simultaneously rather than
> serially.  Any globals/static storage would get overwritten - you can't
> just assume the function can clean up before the next gets called
> without using mutexes or semaphores or something.  Are any of your
> globals keeping track of memory?  What about static variables?
I don't use any more global variables (thanks to you also), and no 
static as well

> Oh - and how are you determining you have a memory leak, anyway?  (I'm
> not doubting you have one, just curious as to how you know.)
My suspicion for the memory leaks is the amount of memory used by the 
program with the command 'ps u p `pgrep angharad`' that increases every 
time I make multiple access at the same time­. But I may be wrong indeed.

I will use more carefully valgrind in a test environment to make sure of 
what I assume.

Also, I will monitor the program in normal use for several days, and see 
how the memory is going then.

Thanks again for your help, much appreciated

/Nicolas