Re: [libmicrohttpd] Hang with MHD_OPTION_THREAD_POOL_SIZE on Windows

[Jonathan McDougall]

Found it. The bug was introduced in 0.9.49 with a change[1] to how
sockets are created. On 0.9.48, create_socket() used socket() on all
platforms. On 0.9.49, Windows started using WSASocketW() instead, I'm
guessing it was to create it as non inheritable directly instead of
calling make_noninheritable() afterwards.

The problem is that the WSASocket() family doesn't create a socket with
the overlapped attribute by default, whereas socket() does[2]:

    [...] sockets created with the socket function will have
    the overlapped attribute. However, in order to be more
    compatible with the rest of the Windows API, sockets
    created with WSASocket will not, default, have the
    overlapped attribute. This attribute will only be applied
    if the WSA_FLAG_OVERLAPPED bit is set.

On Windows, a non-blocking socket without the overlapped I/O attribute
behaves all sorts of weird[3]. The support article links to another page
called "179942 INFO: WSA_FLAG_OVERLAPPED Is Needed For Non-Blocking
Sockets", but it's a 404. I found a cache of it[4], which says:

    A socket must be created with its overlapped I/O attribute
    set if overlapped I/O is to be performed using the socket.
    This can be done either by creating the socket with the
    WSASocket API with the WSA_FLAG_OVERLAPPED, or by creating
    the socket via the socket API. However, if the socket will
    be set to non-blocking mode (on Windows NT 4.0), the socket
    must also be created with its overlapped attribute set,
    even if no overlapped I/O will be performed with the socket

The documentation for WSASocket() also mentions this about the
WSA_FLAG_OVERLAPPED flag[5]:

    Create a socket that supports overlapped I/O operations.
    Most sockets should be created with this flag set.

Some of the things a non-overlapped, non-blocking socket seems to do is
randomly fail to wake up correctly in a select() call with multiple
sockets, or block in accept() when called from multiple threads on the
same socket.

I found a post on this mailing list from 2017[6] about this very same
bug, where it was suggested to reverse the order of sockets in the read
set that's given to MHD_SYS_select_(). Surprisingly, it actually fixed
my immediate problem, but it's really just a symptom of having
non-overlapped sockets.

Therefore, the fix is to simply add the WSA_FLAG_OVERLAPPED flag to the
WSASocketW() call in MHD_socket_create_listen_() in mhd_sockets.c, so it
looks like this instead:

    fd = WSASocketW (pf,
                     SOCK_STREAM,
                     0,
                     NULL,
                     0,
                     WSA_FLAG_OVERLAPPED | WSA_FLAG_NO_HANDLE_INHERIT);

On an unrelated note, w32/common/ seems to be missing some files in the
tarball, particularly largepost-files.vcxproj and
simplepost-files.vcxproj.


[1] 
https://gnunet.org/git/libmicrohttpd.git/commit/?id=3959a99922d5bf358ec96ba2efe5e13198918ac8
[2] 
https://docs.microsoft.com/en-us/windows/desktop/winsock/default-state-for-a-socket-s-overlapped-attribute-2
[3] 
https://support.microsoft.com/en-ca/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode
[4] http://www.voidcn.com/article/p-cabdrnfn-sp.html
[5] 
https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsasocketa
[6] http://lists.gnu.org/archive/html/libmicrohttpd/2017-07/msg00005.html