Re: [libmicrohttpd] Consider allowing SO_REUSEPORT when requested

[Milan Straka]

Hi Christian,

> -----Original message-----
> From: Christian Grothoff <address@hidden>
> Sent: 27 Oct 2014, 17:58
>
> Hi!
> 
> AFAIK the interpretations of what 'reuse port' means differ widely
> between operating systems (especially W32 vs. POSIX, but possibly also
> within POSIX).  So adding that option to our API creates the issue that
> we need to define a particular semantics and then apply it consistently
> among all supported platforms --- and decide on how we should fail if
> setting that option is not possible or fails (especially on exotic
> platforms).
> 
> So if you had a (1) well-defined (semantics, errors), (2) reasonablly
> portable (W32, iOS, BSD, Linux at least), and (3) well-documented,
> including error conditions and handling implementation of this, I'd
> strongly consider adding it.

I agree that the 'reuse port' semantics is quite wild.

I am not familiar with many systems, but here are my 2 cents:

- BSD: SO_REUSEADDR and SO_REUSEPORT originate from BSD implementation.
  The SO_REUSEADDR has weird semantics, allowing two requests to bound
  to overlapping address:port pairs, if they are not equal. That is,
  one can bind to 10.0.0.1:80 and 0.0.0.0:80, which would not be
  possible without SO_REUSEADDR. Nevertheless, each address:port is
  still captured by a single socket.

  SO_REUSEPORT means multiple applications can bind to same address:port
  pair (if whey both specify SO_REUSEPORT).

- FreeBSD / OpenBSD / ...: same semantics as BSD, SO_REUSEPORT is available

- Mac OS X, iOS: same semantics as BSD, SO_REUSEPORT is available

- Linux: same semantics, but SO_REUSEPORT available only with 3.9+
  kernel

- Windows: only SO_REUSEADDR with different semantics -- SO_REUSEADDR on
  Windows is very similar to SO_REUSEPORT on BSD, allowing for multiple
  applications to bind to the same address:port.

So I propose the following:

- add MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS with the following
  semantics: if set to true, the listening socket is opened in such
  a way that other sockets can also listen on the same address:port.
  If set to false, the listening socket is opened in such a way that
  other sockets cannot listen on the same address:port.

- implementation of the "true" case: on Windows, this is already true.
  On BSD/Mac/iOS, use setsockopt with SO_REUSEPORT. On Linux, either
  a) use macros to find out whether SO_REUSEPORT is defined and use it,
     or fail if it is not defined and the option is used,
  b) use compilation switch which either allows using SO_REUSEPORT or
     throws runtime
  c) use the value of SO_REUSEPORT (15), use it unconditionally and fail
     if unsupported

- implementation of the "false" case: nothing, only on Windows use
  SO_EXCLUSIVEADDRUSE instead of SO_REUSEADDR. (This is BTW recommended
  for all Windows server applications, see for example
  
http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx).


As for the error handling, the creation of the socket should fail if the
request cannot be honored. From this point of view, it would be best to
add compilation option "USE_SO_REUSEPORT", which would be enabled by
default on BSD/*BSD/OSX/iOS/Linux 3.9+. If the SO_REUSEPORT should be
used and this compilation option was off, the socket creation should
fail.

This has the nice property that it will compile on all platforms, and
for the unsupported platforms the socket creating will fail at runtime.
Then, we can investigate the platform and either enable the
USE_SO_REUSEPORT switch, or provide new platform-specific
implementation.

Cheers,
Milan Straka