[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [libmicrohttpd] Memory leak and global variables

From: Nicolas Mora
Subject: Re: [libmicrohttpd] Memory leak and global variables
Date: Fri, 29 Aug 2014 14:59:25 -0400
User-agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.0

Hello again,

Le 2014-08-28 15:51, Kenneth Mastro a écrit :
Glad my comments helped.

One quick thing/clarification - that bug in the latest 'released'
version of MHD was related to cleaning up the list of internal
connection/request structures.  (This occurred during the 'stop daemon'
call - manifesting itself with a very long shutdown time.)  To get the
fix for this, you'd have to download the latest source - the released
version still has the bug.

I will try it, but since my program is a standalone server, it doesn't stop the daemon unless the server is stopped.

 From what I can tell, MHD re-uses the same internal structure/thread if
it's not servicing multiple requests.  I.e., if you do one request at a
time, the same one gets used over and over.  If you do 3 at once, new
ones get created so 3 get used.  Something related to this could explain
why you don't see the memory leak when you do the requests one at a
time.  Maybe there's a leak in MHD for each new structure/thread that
gets created and/or each one that gets created beyond the first?
Alternatively, maybe your code just isn't thread-safe?

My code is designed to be thread-safe but maybe it isn't because I forgot something.

Does the memory leak continue to occur/grow over time, or is it just
20kB per connection/request the first time?  In other words - if you hit
it with 6 simultaneous requests and get 120kB of memory leak, wait 10
seconds, then hit it again with another 6 simultaneous requests .. do
you get ANOTHER 120kB of memory leak, or does it stop getting worse?
Knowing that may provide you with some clues.  (E.g., if it doesn't go
up any more, it seems more likely to be an MHD thing in its internal
structures.  If it continues to go up, it could be either/or.)

I'm talking about memory used getting higher, every time I call the 6 requests simultaneously.

The command I use is 'ps u p `pgrep angharad`', and I monitor the RSS value.
For example, when the program starts, the result is the following:
pi 6076 2.0 0.4 20384 2276 ? Sl 23:30 0:00 /usr/local/bin/angharad /etc/angharad.conf

then I make the first 6 calls, I have the new values:
pi 6076 2.3 0.4 20384 2308 ? Sl 23:30 0:00 /usr/local/bin/angharad /etc/angharad.conf

So after the first 6 calls, the memory used increases from 2276 to 2308Kb. Then the next calls raise the memory used to 2340, 2356 and 2456Kb, and so on. That's where my suspicion of memory leaks come from.

Have you tried Valgrind or some printfs to make sure that free is called
for every malloc you're doing?  Even a simple counter would help.  I'm
guessing you really need to make absolutely sure the free is getting
called for every one - especially with lots of early-returns.

I traced my program with printf every time I malloc and free variables.
I have made a few tests, and as far I can tell, all mallocs variables are freed every time.

Unfortunately, Raspbian uses a hack that won't be supported by valgrind, so valgrind detects an early error. Here is valgrind output on the rpi (my program is compiled with the debug options).

In the next days, I will monitor the program on another platform with valgrind. By taking a closer look, I found one leak, but it's not over yet.

Another thought - maybe one of your (or MHD's, I guess) global variables
is getting overwritten with each request.  Doing one request at a time
gives the code a chance to de-allocate the memory before the pointer
gets overwritten.  That seems more likely to me than some cleanup code
never being called.  What I'm getting at - I'm assuming your code is
getting called by MHD's multiple threads simultaneously rather than
serially.  Any globals/static storage would get overwritten - you can't
just assume the function can clean up before the next gets called
without using mutexes or semaphores or something.  Are any of your
globals keeping track of memory?  What about static variables?

I don't use any more global variables (thanks to you also), and no static as well

Oh - and how are you determining you have a memory leak, anyway?  (I'm
not doubting you have one, just curious as to how you know.)

My suspicion for the memory leaks is the amount of memory used by the program with the command 'ps u p `pgrep angharad`' that increases every time I make multiple access at the same time­. But I may be wrong indeed.

I will use more carefully valgrind in a test environment to make sure of what I assume.

Also, I will monitor the program in normal use for several days, and see how the memory is going then.

Thanks again for your help, much appreciated


reply via email to

[Prev in Thread] Current Thread [Next in Thread]