Re: [Savannah-hackers-public] 502 Bad Gateway for project called tops

[Bob Proulx]

Dale Williamson wrote:
> It is possible to log in and update sources at "cvs -z3
> -d:ext:address@hidden:/sources/tops" but from a web browser
> viewvc/tops has a bad gateway.

The problem is that savannah systems are getting hit by a botnet.  It
is browning out the web UIs on three of the systems.  This has been
going on all weekend.  The botnet is hitting the web interface
randomly selecting every possible URL.  If you can imagine every
version of every project file in every project you will know what is
happening.

The attack started late Friday.  It is at least 10k IP addresses
strong and probably a lot bigger.  It's somewhat hard to tell the
exact size.  I know that vcs0 was hit by 45k addresses in 24 hours on
Saturday but I do not know how many of those were the botnet and how
many were just nice people like you and I clicking on the web browser.
But that seems a likely upper end.

Unfortunately we weren't previously collecting trend data on that
particular statistic for vcs0 and so I don't know what is a normal
daily rate.  Not that high by a lot however.  But at least for the
future moving forward we will have this data.  Things are running
about 30 requests per second on just vcs0 at this moment.  5/s on vcs1
and 10/s on frontend0.  And sometimes it spikes significantly higher.

We are working as best we can to try to block the attack and keep the
system limping along.  But you know how these DDoS attacks go.  If
someone wants you offline then there is really no way to stop them.

In the meantime I suggest using ssh:// protocol member access for all
of the version control backends.  Because that is not http/https it is
faring better.  Checkouts and commits should still be working.  It's
really just the web UI that is problematic.

The 502 Bad Gateway is somewhat transient in that if one retries then
it will eventually succeed through the botnet.

> Screen shots in Chrome are shown below.

Thanks for the report but in the future please simply type it up.
Plain text reports are always better.  "502 Bad Gateway" is more than
sufficient.  Those two attachments were 485KB and 324KB in size for
the images making for an 810KB mail message!  That's really quite
large for a mail message.  But don't let this stop you from reporting
problems in the future.  Please we welcome problem reports!

Bob