[libmicrohttpd] unexpected "Server reached connection limit." failures

[Frank Ch. Eigler]

Hi -

I manage a libmicrohttpd based public service, and the software
that runs it (elfutils debuginfod & its fedora service).  Under
some usage patterns, it can get into a state where it starts
rejecting new incoming connections with $SUBJECT stderr message.
But it's a mystery why.

The server is throttled to a few dozen tasks via systemd limits.  It
uses MHD_USE_THREAD_PER_CONNECTION | MHD_USE_INTERNAL_POLLING_THREAD,
and received front-end reverse-proxy services with keepalive etc.
That part's fine most of the time.  So we're servicing some dozen
concurrent connections, no problem.

But somehow, libmicrohttpd (0.9.73, fedora34) gets into a state where
the daemon->connections number is weirdly bloated (1020), and so
internal_add_connection reports $SUBJECT.  But there are still only a
few dozen keepalive sockets alive (and their threads), and matching
number of entries on the daemon->connections list, and zero or one
entry on the daemon->cleanup list.

I wonder if the thread throttling (via linux cgroups) or other
transient problem can let that number bloat.  In
internal_add_connection(), ->connections++ is done, but in case of
later error, the cleanup: path does not propagate the halfbroken
connection object to MHD_connection_cleanup_() for a subsequent
->connections--.  So that ->connections number could grow and grow,
and eventually saturate to the ->global_connection_limit.

Do I read the code right?  Other than a fix to this part of
libmicrohttpd, do I have an option other than using
MHD_OPTION_THREAD_POOL_SIZE instead of MHD_USE_THREAD_PER_CONNECTION?


- FChE