Re: [libmicrohttpd] Memory leak and global variables

[Kenneth Mastro]

Glad my comments helped.

One quick thing/clarification - that bug in the latest 'released' version of MHD was related to cleaning up the list of internal connection/request structures.  (This occurred during the 'stop daemon' call - manifesting itself with a very long shutdown time.)  To get the fix for this, you'd have to download the latest source - the released version still has the bug.

From what I can tell, MHD re-uses the same internal structure/thread if it's not servicing multiple requests.  I.e., if you do one request at a time, the same one gets used over and over.  If you do 3 at once, new ones get created so 3 get used.  Something related to this could explain why you don't see the memory leak when you do the requests one at a time.  Maybe there's a leak in MHD for each new structure/thread that gets created and/or each one that gets created beyond the first?  Alternatively, maybe your code just isn't thread-safe?

Does the memory leak continue to occur/grow over time, or is it just 20kB per connection/request the first time?  In other words - if you hit it with 6 simultaneous requests and get 120kB of memory leak, wait 10 seconds, then hit it again with another 6 simultaneous requests .. do you get ANOTHER 120kB of memory leak, or does it stop getting worse?  Knowing that may provide you with some clues.  (E.g., if it doesn't go up any more, it seems more likely to be an MHD thing in its internal structures.  If it continues to go up, it could be either/or.)

Have you tried Valgrind or some printfs to make sure that free is called for every malloc you're doing?  Even a simple counter would help.  I'm guessing you really need to make absolutely sure the free is getting called for every one - especially with lots of early-returns.

Another thought - maybe one of your (or MHD's, I guess) global variables is getting overwritten with each request.  Doing one request at a time gives the code a chance to de-allocate the memory before the pointer gets overwritten.  That seems more likely to me than some cleanup code never being called.  What I'm getting at - I'm assuming your code is getting called by MHD's multiple threads simultaneously rather than serially.  Any globals/static storage would get overwritten - you can't just assume the function can clean up before the next gets called without using mutexes or semaphores or something.  Are any of your globals keeping track of memory?  What about static variables?

Oh - and how are you determining you have a memory leak, anyway?  (I'm not doubting you have one, just curious as to how you know.)

Ken

On Thu, Aug 28, 2014 at 2:54 PM, Nicolas Mora <address@hidden> wrote:

Hi Kenneth, thanks a lot for your answer, you helped me moving on with my problems.

I fixed a lot of small problems thanks to your answers, but the memory leaks are still here with the same conditions: about 20kb is lost each time I call the 6 main webservices simultaneously with jquery, but no leak at all when I call each webservice on by one.

I'm sure I'll find the solution, I'm so close.

Here are my feedbacks.

* I'm far from an expert, but are you sure you can use
thread-per-connection and use-select-internally together?  (You very
well may be able to, I just thought they weren't normally used
together.  Could be completely wrong about that, though.)

Not really, maybe it's not recommended, but the program never yelled at me for that. I skipped select-internally flag, to keep just thread-per-connection, but no obvious change since then.

* There's a minor bug in the last release (0.9.37 and earlier) of MHD
where it doesn't properly cycle through the existing connections when in
thread-per-connection mode.  The fix was simple (essentially deleting
one character), but if you're detecting leaks by looking at the result
just before your program terminates, there's a remote chance that has
something to do with it.

Actually, I'm using raspian repository, which provides a 0.9.20-1 release. I tried yesterday using the last release by downloading, compiling and installing it, but not change then too.

* I haven't noticed any leaks in MHD.  I'm not saying there aren't any,
but I haven't seen anything like what you're describing.  (Granted - I'm
not hammering it like you would be in a web-service-based solution.)

I'm pretty sure the leaks come from my code somewhere, but most likely because of my misuse of MHD, that's why I'm asking the questions here.

* You have a lot going on in that code.  If it were me, I'd strip it way
down to a simple test case to try and isolate the leak(s).  You're doing
a lot of mallocs in general - any chance you could (at least
temporarily) remove them and reply with fake data?  The point is to try
and see if the problem is memory that MHD is allocating or if it's
something you're doing.

You're right about the useless mallocs. I changed all the fixed size mallocs to local variables, there is one (the page variable) that can't be change to a local variable since its size may vary.
I'll try with fake data as you suggest it.

* You're doing a few mallocs at the start of the request - are you sure
that's necessary (e.g., for the 'page' and 'urlcopy')?  Also, are you
sure they're all getting cleaned up?  For testing, maybe you could just
create a stack-based character string (i.e., no malloc) and tell MHD to
copy the response string instead of de-allocating the memory later.  I'm
not sure that would be any less efficient, anyway - malloc is kind of
slow, and your strings are really short.  (I.e., instead of 'MUST_FREE'
use the one that tells MHD to copy the buffer).  I mean - you're
returning in the middle of the code all the time (MHD_YES, MHD_NO), but
your cleanup code executes at the end of the function.  That seems very
very suspicious to me.  In fact - I bet that's what your problem is (or
at least one of them).  That connection request function gets called
multiple times - but you'll return early some times and the cleanup code
at the end never gets executed.  That leaves your 'urlcopy' (and
possibly other things) not freed, right?

You were totally right, I forgot about those return statements, and the 'free' calls that were forgotten there.

* I see you have a 'request completed' callback that frees the memory
you allocated for/in your connection info.  I assume you've verified
that it's getting called?

Yes, the request_completed function is called at the end of each angharad_rest_webservice function.

* As for the globals - I didn't analyze your code in any kind of depth,
but you probably already know that MHD can take a pointer to any object
you want and keep giving you the pointer through the connection cycle
(typically a structure).  At a glance, it looks like you're taking
advantage of that.  For things that you need across connections.. I'm
not sure what the suggested usage pattern is.  I'm using C++, so I just
use class data members.

I finally found it, indeed the parameter dh_cls in MHD_start_daemon is designed for that ! :)

That's all I've got.  Hope it helps.

That helped, thanks

/Nicolas