[libmicrohttpd] Problems with latency and getting "stuck"

[maurice barnum]

Hi.

I'm working on a project where I want to use libmicrohttpd to handle
incoming HTTP connections that are then forwarded to a ZeroMQ server.
Three problems I'm encountering:
 * latency is bad due to a significant delay between when I wake up
a connection with MHD_resume_connection and when MHD_run calls the
corresponding handler callback

 * each run of MHD_run will accept several incoming connetions but
only "retire" one of the connections I've resumed.

 * eventually, everything stops:  I call zmq_poll with a timeout of -1
(MHD_get_timeout() returned 0), but the epoll fd never signals read-in
even when new connections come in.

My event loop listens on a ZeroMQ socket and the epoll fd returned from
MHD.  The loop looks basically like (pseudo-code):

   while true:
     MHD_run(md)
     timeout = MHD_get_timeout() / 1000
     if not timeout:
       timeout = -1
     zmq_poll(items, len(items), timeout)
     if items[0].revents:
        # this is the zeromq socket
        if items[0].revents and ZMQ_POLLIN:
          # loop elided
          m = zmq_read()
          context = find_context(m.identity)
          context.response = m.data
          MHD_resume_connection(context.mc)

On the callback side, I send a request over the ZeroMQ socket and
then suspend the connection until my event loop picks up a response.

Curiously, I found that if I didn't call MHD_run() each time through
the loop, my server would get wedged immediately, which is why I don't
call it iff zmq_poll signals that the epoll fd is readable.

This is how I init the daemon:

    const unsigned mhdflags =
        MHD_USE_DEBUG |
        MHD_USE_SUSPEND_RESUME |
        MHD_USE_EPOLL_LINUX_ONLY;

    DaemonPtr result(MHD_start_daemon(
                mhdflags,
                httpPort,
                NULL, NULL, // accept callback
                handlerCallback, self,
                MHD_OPTION_NOTIFY_COMPLETED, connectionComplete, self,
                MHD_OPTION_CONNECTION_TIMEOUT, 5,
                MHD_OPTION_CONNECTION_LIMIT, 25,
                MHD_OPTION_END),
            MHD_stop_daemon);

Any ideas on where to look/debug is appreciated.  I've attached a 
digested trace of my debug output that shows the behavior.

Thanks.

timeline.txt.xz
Description: Binary data