Re: [libmicrohttpd] Odd Stop Behavior

[Christian Grothoff]

Dear Kenneth,

The first thing you should provide is the name of the operating system
you are on (if non-Linux, that might by itself be a big hint).

Regardless, given that MHD does terminate after 115s suggests that some
timeout is involved.  Did you, by chance, set the per-connection timeout
to like 115-120 seconds?  Or is that your kernel's TCP timeout?  Again,
that's just for diagnostics -- there clearly still is an issue as that
timeout should not matter for shutdown.


When MHD is stopped, it should terminate existing ongoing requests
immediately (client gets a closed TCP socket); this is usually done
by calling 'shutdown()' on the TCP socket if HAVE_LISTEN_SHUTDOWN
(which should unblock the pthread), or by writing to a pipe.  Again,
what happens exactly depends a lot on your thread mode, MHD options
and kernel.

However, shutdown issues like this are usually easy to debug by
attaching a debugger and seeing where that thread hangs around,
so if you could provide a testcase I'm sure we can find a solution
rather quickly.


Happy hacking!

Christian

On 06/06/2014 08:50 PM, Kenneth Mastro wrote:
> I'm seeing an odd behavior when I shut down the daemon (MHD_stop_daemon).
> The shutdown hangs for about 115 seconds, then it finally terminates
> cleanly.
> 
> I'm using the 'thread per connection' mode, which may be relevant.  After
> digging into microhttpd's code, I found that it was hanging on a thread
> join during 'close_all_connections'.
> 
> I began doing some heavy debugging, and determined that I don't have any
> lingering connections that I never finished with.  I even went so far as to
> print out the thread PIDs in both microhttpd and my code to make sure I saw
> the requests that were being handled by the hanging thread.  Nothing fancy
> - just some css and/or image files being served up (about 25 requests or
> so).  Through this debugging, I'm nearly certain that I'm ending the
> connection/request callback every time (I also did a global
> increment/decrement of requests, and it gets back to 0 every time - Ergo,
> I'm not holding onto the thread.).
> 
> Some other useful info:
> * When I set the maximum number of connections/threads to 1 (instead of 20)
> when I start the daemon, it does not hang when I stop the daemon - it just
> ends immediately, as expected.
> 
> * After I start shutting down, if I make ANOTHER request from the browser
> (e.g.., refresh the page), the daemon stops immediately and cleanly.
> 
> * After I load the page the first time (and it makes a bunch of requests to
> the server, causing 1 or more threads to be used), if I wait a while (e.g.,
> a minute) and then stop the server, it shortens the time it takes the
> thread to terminate.  I.e., if I wait 65 seconds, it only takes 70 seconds
> for the server to stop instead of 115.  If I wait longer than 115 seconds,
> it will end immediately, as expected.
> 
> * If I make NO requests (i.e., I just start, then stop a few seconds
> later), it terminates cleanly.  The 115 second delay only occurs after I've
> made a request to the server from the browser and then try to stop it.
> 
> * I'm using a fairly recent version: 0.9.35
> 
> 
> I came across some vaguely related issues regarding stopping the daemon:
> -----------------
> I saw some old archived messages about a problem that existed in
> microhttpd, but that was several years ago, but it sounds like it was a
> full-on deadlock:
> https://lists.gnu.org/archive/html/libmicrohttpd/2009-10/msg00013.html
> 
> Also saw this, which may be similar to what I'm dealing with (only it
> sounds like this was solaris specific, and I'm on Linux):
> http://lists.gnu.org/archive/html/libmicrohttpd/2011-09/msg00024.html
> 
> Also, this issue feels similar as well:
> https://gnunet.org/bugs/view.php?id=1901
> ------------------
> 
> It seems like there's been some trouble in this area in the past, so I
> figured it was worth asking about.
> 
> So - any idea if I'm doing something wrong?  I'm at a bit of a loss since
> my code doesn't have any control over the internal threads.  I'm happy to
> try anything in my application and/or modify microhttpd to test, but I'm
> not sure where to focus my efforts.
> 
> 
> Thanks much,
> Ken
>