Re: [libmicrohttpd] shutdown of listen socket does not work on solaris 10

[Will Bryant]

Coincidentally I discovered the same problem on Mac OS X last night, while 
trying to debug issue #1760.  There we had noticed that the shutdown call was 
failing and I had also noticed that my app does not shut down ever since I 
upgraded past 0.9.7, and the library's tests were not running properly.

It turns out that libmicrohttpd has been failing tests (well, they hang rather 
than failing) on OS X ever since 0.9.8.  I did a bisect to find the offending 
commit and found it was r14584, "also eliminating use of pipe, thereby 
addressing #1662".

So basically because the original requestor on FreeRTOS couldn't use pipe(), a 
solution has been adopted that I believe doesn't work on many unixes other than 
Linux - it's definitely broken on OS X and Solaris, and my understanding from 
#1674 is that it was also broken on Cygwin - though the #1674 workaround which 
has been committed presumably works on cygwin but doesn't seem to work on OS X 
when I try it.

Can anyone confirm if shutdown still works on the BSDs?  I would have guessed 
not since OS X tends to be closer to the BSD behavior.

I agree with Christian that using close() on the socket is not an acceptable 
solution due to the race condition.

I personally think that the best thing to do would be to revert the r14584 
commit to start with, because the pipe solution was a good one for most 
supported platforms, and if the FreeRTOS support is still required, let the 
requestor work on another solution for platforms missing most of the posix spec 
(it could use a localhost socket pair, for example) or patch the library to use 
the shutdown technique on that platform only.

Adding complicated workarounds for platforms that don't have many basic OS 
features seems like an unfair burden on libmicrohttpd.


On 16/09/2011, at 11:53 PM, Frank Meier wrote:

> hi
> 
> I tried to use libmicrohttpd on solaris 10, and my simple programm only 
> starting and stopping the daemon (see below) did hang in.
> after some investigation (see debug code) I found, that the shutdown call in 
> daemon.c fails, since the daemon socket is not a connected fullduplex but a 
> listening socket. this means the daemon thread keeps hanging in select() and 
> main hangs in pthread_join() (see pstack output). problem is since the socket 
> doesn't get closed, select does not interrupt. After adding CLOSE (fd) after 
> the point of SHUTDOWN () the programm did work. It looks like solaris and 
> linux behave differently when shutdown is called on a listening socket. But 
> on the other hand I'm not sure if it is correct to call shutdown in the first 
> place, since the man page states "The  shutdown() call causes all or part of 
> a full-duplex connection on the socket associated with sockfd to be shut 
> down" and the listening socket does not have a connection.
> 
> I think the socket should be closed with CLOSE before calling pthread_join() 
> and use SHUTDOWN only on connection sockets.
> 
> regards, Frank
> 
> 
> /*test programm*/
> struct MHD_Daemon* mhdDaemon = 0;
> mhdDaemon = MHD_start_daemon (MHD_USE_THREAD_PER_CONNECTION, mhdPort, NULL, 
> NULL, &answer_to_connection, NULL, MHD_OPTION_END);
> if (mhdDaemon)
> {
>    MHD_stop_daemon (mhdDaemon);  // <-here it hangs
>    mhdDaemon = 0;
> }
> 
> /*debug code in daemon.c*/
> --- daemon/daemon.c     (revision 16873)
> +++ daemon/daemon.c     (working copy)
> @@ -2477,7 +2477,11 @@
>       daemon->worker_pool[i].shutdown = MHD_YES;
>       daemon->worker_pool[i].socket_fd = -1;
>     }
> -  SHUTDOWN (fd, SHUT_RDWR);
> +       int res = SHUTDOWN (fd, SHUT_RDWR);
> +       int tmpErrno = errno;
> +       printf("shutdown res:%d  %s\n", res, strerror(tmpErrno));
> +
> +       CLOSE (fd);
> 
> ->output with debug code:
> shutdown res:-1  Transport endpoint is not connected
> 
> /*pstack output -> damon thread is in select, main hangs in pthread join*/
> $ pstack 16802
> 16802:  ./test.bin
> -----------------  lwp# 1 / thread# 1  --------------------
> feee9365 lwp_wait (2, 80478d8)
> feee2c1f _thrp_join (2, 0, 8047918, 1) + 5a
> feee2d9e pthread_join (2, 8047918, feeec7e0, fef9665d) + 2b
> fef96777 MHD_stop_daemon (8060d40, 0, 0, fee63301) + 12b
> 0805080a main     (1, 8047988, 8047990) + 6f
> 080506ac _start   (1, 8047a88, 0, 8047a93, 8047abe, 8047b78) + 60
> -----------------  lwp# 2 / thread# 2  --------------------
> feee8da5 pollsys  (fed5ed50, 1, 0, 0)
> fee93afa pselect  (4, fed5ef30, fed5eeb0, fed5ee30, 0, 0) + 18e
> fee93df0 select   (4, fed5ef30, fed5eeb0, fed5ee30, 0, 0) + 82
> fef95ba4 MHD_select (fec50200, fec50200, fed5efec, feee59b9, 8060d40, 0) + 130
> fef96248 MHD_select_thread (8060d40) + 18
> feee59b9 _thr_setup (fec50200) + 4e
> feee5ca0 _lwp_start (fec50200, 0, 0, fed5eff8, feee5ca0, fec50200)