[libmicrohttpd] Re : Re: External select mode limited to 1024 FDs

[louisbenoit]

Hi Christian,

I wanted to use external threads because my system has the following constraints:
1) One connection per thread: the response to the client is time critical and I cannot afford that a connection delays another one.
2) Many listening sockets are opened, hence many MHD daemon: some for HTTP and some for HTTPS.
3) Up to 2000 concurrent connections (shared between HTTP and HTTPS).

As I don't know how the traffic will be shared amongst the daemons, I would need to configured each MHD daemon with epoll and 2000 threads.
This solution does not scale. Just imagine the number of threads created at start up or debugging a single HTTP request with gdb.

The best solution would be to share the worker threads pool between the daemons but if I understand correctly the current implementation does not allow it.
This why I fall back on external thread: I create 2000 daemons for each listening socket and dispatch the HTTP/HTTPS request to a pool of 2000 threads, which reuse the daemons.

Is there a plan to implement worker threads sharing between daemons? If not, are there other MHD solutions that I miss?

Thanks,

Louis

Le 05/03/15, Christian Grothoff  <address@hidden> a écrit :

Hi!

I had no plans for an 'external poll' API, as we have epoll() which is
better for very large FDsets (as long as you're on Linux).  Also,
redefining FD_SETSIZE is also IMO a very good possibility.

Regardless, select/poll with > 1024 FDs and external mode is likely not
the greatest idea from a scalability/performance perspective: if you
have that many concurrent / open FDs, you really want epoll() and
possibly think about using threads / thread pool for processing.

Happy hacking!

Christian

On 03/04/2015 10:22 PM, address@hidden wrote:
> Hi,
>
> I would like to use the external select mode, as I need to manage the connection threads externally.
> Because there could be much more than 1024 FDs used at run time, I cannot use MHD_get_fdset() and select ().
> Apart from redefining FD_SETSIZE, are there other ways to use external threads when FD > 1024?
> If not, is there a plan to add an MHD_get_poll_event API, the poll version of MHD_get_fdset()?
>
> Thanks,
>
> Louis Benoit
>