[libmicrohttpd] strange latencies using libmicrohttpd w/sendfile

[Eivind Sarto]

I ran into some very long HTTP latencies using libmicrohttpd and the linux 
sendfile syscall.
It turned out that the problem had nothing to do with the library, but with the 
linux kernel itself (RedHat Enterprise 5.5).
The workaround for the problem was solved by a change to the library, so I 
figured I might as well share the results
with others who may also run into the same problem.

I have an HTTP streaming server that uses libmicrohttpd.  For streaming most 
type of content, everything works
just great using sendfile (via  MHD_create_response_from_fd_at_offset).  But, 
when I upload the content from
a live encoder (using PUT+DELETE) I started seeing huge latencies when I 
attempted to stream the live content.

When I ran strace on the server, I noticed that the latencies came from either 
sendfile and close of the open file.
The latencies were coming from the kernel and not the library.  It appeared to 
be triggered by the encoder
writing the file (HTTP-PUT) and then the client streaming it a few seconds 
later.
I am guessing that there is some interaction between writing the data into the 
page cache and writing it to the socket with sendfile
(which does a zero-copy transfer from the page cache to the socket head).  And 
maybe the pdflush kernel daemon deciding
to flush dirty pages to disk.

libmicrohttpd does not open the connection socket in non-blocking mode. so I 
decided to add an fcntl(O_NONBLOCK)
to the connection socket when the connection is created.
And the delays disappeared.

When streaming about 5gbps of Apple HLS Live protocol data that is being PUT 
from the encoder, the max latencies went from more than
7 seconds to less than 5 microseconds (measuring the latencies between the HTTP 
request and the response from the
clients point of view).

I am pretty sure that there is something going on in the linux kernel that 
causes the latencies.
My question is:  Is there any reason why the connection socket should not be in 
O_NONBLOCK mode?
And if not, could this be added to the library?

I have a few patches I need to add to the library, so adding one more is not a 
big deal.
But, I would not object if you decided to make connection sockets O_NONBLOCK :-)

-eivind