Savannah DDoS Attack

[Bob Proulx]

Savannah Community,

Savannah systems are getting hit by a DDoS attack.  A botnet is
browning out the web UIs on three of the systems.  This has been going
on all weekend.  The botnet is hitting the web interface randomly
selecting every possible URL.  If you can imagine every version of
every project file in every project you will know what is happening.

The attack started late Friday.  It is at least 10k IP addresses
strong and probably a lot bigger.  It's somewhat hard to tell the
exact size.  I know that vcs0 was hit by 45k addresses in 24 hours on
Saturday but I do not know how many of those were the botnet and how
many were just nice people like you and I clicking on the web browser.
But that seems a likely upper end.

Unfortunately we weren't previously collecting trend data on that
particular statistic for vcs0 and so I don't know what is a normal
daily rate.  Not that high by a lot however.  But at least for the
future moving forward we will have this data.  Things are running
about 30 requests per second on just vcs0 at this moment.  5/s on vcs1
and 10/s on frontend0.  And sometimes it spikes significantly higher.

We are working as best we can to try to block the attack and keep the
system limping along.  But you know how these DDoS attacks go.  If
someone wants you offline then there is really no way to stop them.

In the meantime I suggest using ssh:// protocol member access for all
of the version control backends.  Because that is not http/https it is
faring better.  Checkouts and commits should still be working.  It's
really just the web UI that is problematic.  The 502 Bad Gateway for
the interfaces that use it is somewhat transient in that if one
retries then it will eventually succeed through the botnet.

Bob