Re: [libmicrohttpd] Consider allowing SO_REUSEPORT when requested

[Milan Straka]

Hi Christian,

> -----Original message-----
> From: Christian Grothoff <address@hidden>
> Sent: 27 Oct 2014, 21:28
>
> On 10/27/2014 06:52 PM, Milan Straka wrote:
> > Hi Christian,
> > 
> > ...
> >
> > I am not familiar with many systems, but here are my 2 cents:
> > 
> > - BSD: SO_REUSEADDR and SO_REUSEPORT originate from BSD implementation.
> >   The SO_REUSEADDR has weird semantics, allowing two requests to bound
> >   to overlapping address:port pairs, if they are not equal. That is,
> >   one can bind to 10.0.0.1:80 and 0.0.0.0:80, which would not be
> >   possible without SO_REUSEADDR. Nevertheless, each address:port is
> >   still captured by a single socket.
> > 
> >   SO_REUSEPORT means multiple applications can bind to same address:port
> >   pair (if whey both specify SO_REUSEPORT).
> > 
> > - FreeBSD / OpenBSD / ...: same semantics as BSD, SO_REUSEPORT is available
> > 
> > - Mac OS X, iOS: same semantics as BSD, SO_REUSEPORT is available
> > 
> > - Linux: same semantics, but SO_REUSEPORT available only with 3.9+
> >   kernel
> > 
> > - Windows: only SO_REUSEADDR with different semantics -- SO_REUSEADDR on
> >   Windows is very similar to SO_REUSEPORT on BSD, allowing for multiple
> >   applications to bind to the same address:port.
> > 
> > So I propose the following:
> > 
> > - add MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS with the following
> >   semantics: if set to true, the listening socket is opened in such
> >   a way that other sockets can also listen on the same address:port.
> >   If set to false, the listening socket is opened in such a way that
> >   other sockets cannot listen on the same address:port.
> > 
> > - implementation of the "true" case: on Windows, this is already true.
> >   On BSD/Mac/iOS, use setsockopt with SO_REUSEPORT. On Linux, either
> >   a) use macros to find out whether SO_REUSEPORT is defined and use it,
> >      or fail if it is not defined and the option is used,
> 
> Fail to compile? Not a great idea to require 3.9 plus matching headers
> and otherwise FTBFS.

I am again sorry that I cannot make myself clear.

I meant something like

MHD_start_daemon() {
  ...

  if (MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS and has true value) {
  #ifndef WINDOWS
  #ifdef SO_REUSEPORT
    // set SO_REUSEPORT on the socket
  #else
    // return NULL and report that SO_REUSEPORT is not available
  #endif
  #else
  ...
}

So no FTBFS, but only return NULL in MHD_start_daemon if support for
SO_REUSEPORT was not available.

Nevertheless, SO_REUSEPORT might not be a macro (although it is in
Linux).

> >   b) use compilation switch which either allows using SO_REUSEPORT or
> >      throws runtime
> 
> Not sure I understand this one. I understand we sometimes use
> *configure* switches to enable people to build a smaller MHD for
> embedded devices, but on most systems the API should be the API and
> never differ simply based on how configure went.  Also, given that you
> have a rather strict specification that says: definitively allow or
> definitively deny, that means that there cannot be something in the
> middle where it is "well, maybe allow, maybe deny, as we were configured
> without the option and the application didn't specify 'hard allow'".

What I mean is something in lines of enable_epoll option of configure.

Therefore, we would have HAVE_SO_REUSEPORT, which would be detected by
configure script. Then the code would look like in the previous case,
i.e., if SO_REUSEPORT was not available during compilation,
MHD_start_daemon with MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS set to
true would return NULL.

> >   c) use the value of SO_REUSEPORT (15), use it unconditionally and fail
> >      if unsupported
> 
> That we could do, use macros if available, fallback to hard-coded '15'
> and then runtime failure if REUSEPORT requested but not available.
> 
> > - implementation of the "false" case: nothing, only on Windows use
> >   SO_EXCLUSIVEADDRUSE instead of SO_REUSEADDR. (This is BTW recommended
> >   for all Windows server applications, see for example
> >   
> > http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx).
> 
> Technically that's (on W32) API-breakage (suddenly different behavior),
> but given the context I think that's legitimate.

No, it is not. There are three cases here:
- no MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS option given. Then the
  behaviour is the same as before. As existing programs cannot use the
  (currently unexistent) MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS
  option, their behaviour is unaffected.

- MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS option give with true
  (nonzero) value: add SO_REUSEPORT if available or return NULL.

- MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS option given with false
  (zero) value: use SO_EXCLUSIVEADDRUSE on Windows.

> > As for the error handling, the creation of the socket should fail if the
> > request cannot be honored. From this point of view, it would be best to
> > add compilation option "USE_SO_REUSEPORT", which would be enabled by
> > default on BSD/*BSD/OSX/iOS/Linux 3.9+. If the SO_REUSEPORT should be
> > used and this compilation option was off, the socket creation should
> > fail.
> 
> Ugh, please not a compilation option. I'd just make it a flag,
> "MHD_USE_SO_REUSEPORT = 32768" and then if the flag is set, we try
> SO_REUSEPORT (and if that fails, return NULL from MHD_start_daemon());
> if the flag is not set, we do the EXLUSIVEADDRUSE on W32 (and if that
> fails, we again return NULL from MHD_start_daemon()).  Hard-code to 15
> if macro not defined on Linux, never FTBFS.  Leaves open the question as
> to what to do on "other" OSes, but I'm OK to leave that up to porters
> (and porting guides if those semantics cannot be satisfied).
> 
> > This has the nice property that it will compile on all platforms, and
> > for the unsupported platforms the socket creating will fail at runtime.
> 
> I agree with that goal, but enabling the option must be runtime flag,
> not compiler flag.

Once again, I could not explain what I meant. I meant that we could add
compilation switch telling whether we have SO_REUSEPORT support compiled
in. Then we would have MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS which has
additional true/false value, which, if specified, uses SO_REUSEPORT when
needed (or returns NULL if not compiled in).

I currently think that this is the best way. We can either make the
enable_so_reuseport opt-in (being it disabled by default), or we can
detect in the configure script whether SO_REUSEPORT can be used and if
it does, to enable it automatically. Then any platform supporting
SO_REUSEPORT will have the SO_REUSEPORT support compiled in
automatically.

To summarize:
- configure scripts either
  - gets --enable-so_reuseport or --disable-so_reuseport
  - if it does not, it tries to compile a simple program with
    SO_REUSEPORT and sets --enable-so_reuseport if successful
- if --enable-so_reuseport, add HAVE_SO_REUSEPORT to MHD_config.h.
- then, add MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS, which takes
  an boolean parameter
- if in MHD_start_daemon, MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS is
  not present, do nothing (i.e., unchanged behaviour for existing
  programs.)
- if MHD_OPTION_ALLOW_REUSING_LISTENING_ADDRESS is specified, than
  - not Windows:
    - if it has true value, then
      - if HAVE_SO_REUSEPORT, use it
      - otherwise report error and return NULL
    - if it has false value, do nothing
  - Windows:
    - if it has true value, do nothing
    - if it has false value, use SO_EXCLUSIVEADDRUSE instead of
      SO_REUSEADDR.

Going to bed,
thanks and cheers,
Milan Straka