Re: [Savannah-hackers-public] Anyone have any news on Savannah?

[Bob Proulx]

Paul Smith wrote:
> Unfortunately it seems to be down / under attack again this morning :(

I see that my attempts at mitigation of the current problem are
failing.  I am looking into things again now.  I'll type this in
stream of consciousness as I work the problem and then send it.

I may have fixed the insufficient mitigation for the apache web server
wedging up.  I had thought that "apachectl graceful" should be
sufficient to restart the apache server.  But no it was not.  The
apache logs were filled with attempts to restart using that command
and obviously it wasn't working.

It appears that what is happening is that external client agents are
connecting to the apache web server and hanging onto connections
forever.  This causes apache to spawn up to the MaxRequestWorkers
number and hold there.  And it doesn't matter the number.  Eventually
all server processes are consumed regardless of the number.  Processes
respond to other requests until the malicious agent eventually grabs
them and holds onto them.  That's when no one else can get response
from the site.

Meanwhile that number must be below the max memory available for use.
If it is larger than available then of course that's bad and we must
start swapping.  I reduced it from the large 255 value to 32 yesterday
on the previous pass at this tuning.

The graceful action politely waits for the client to finish
processing.  Which in this case never occurs because the abusive
agents are not polite and hold on forever.  Therefore using graceful
is insufficient.  That is why it requires the forced stop action to
have apache stop talking to the abusive agents forcibly and to drop
all connections.  And of course then a start works.

I changed that to be a full "apachectl stop" and then followed by a
"apachectl start" and that worked.  Rather than simply restarting I
changed the automatic mitigation test and let it work and so tested
that it would automatically detect and restart okay.

But I am going to configure the "restart" action.  It makes me nervous
that the automation triggers "stop;start" immediately with no delay.
I worry that it might take a moment for the stop to propagate and all
of the old apache server processes to actually exit.  And if they
don't exit then the restart action can't succeed.  It's a race
condition due to the asynchronous behavior of systemd.  Though the
automation would trigger again and that one might succeed if the
previous action failed.  Feels like the apachectl restart would handle
this action more reliably since that instructs the supervisor process
to perform the restart.  Will configure it while the current malicious
agents are active to help us test for it.

[[ In this investigation I see that on systemd systems there is an
"interesting" control loop.  apachectl calls systemd and systemd calls
apachectl.  This would be a loop except for a programmed in internal
variable that breaks the loop.  Isn't it a wonderful world that we now
live in?  Not! ]]

If the restart action proves insufficient (though I think it will
work) then I will script up a stronger more explicit restart action.
Let's hope that isn't needed.

Additionally I tuned the apache server to have a smaller number of
MaxRequestWorkers than the default 255 which is more than the memory
available.  I have it currently set to a more conservative 32.  Though
now that things are observable again it appears that each process uses
about 25MB of memory for about 800MB of memory consumed in this
configuration.  We could increase that number somewhat and use more
memory for server processes.  Increasing this would eat into available
file system buffer cache though.  Life is a tradeoff.  But this seems
pretty responsive right now.  I'll leave it this way for a while and
observe.

Additionally I configured MaxConnectionsPerChild to be 100 rather than
unlimited.  This will cause apache to restart server processes after
that number of client requests have been handled.  That's just a
useful "random bug" guard restarting processes ever so often rather
than never.

> Don't people have better things to do with their lives?  Sigh.

It's why we can't have nice things.  :-(

Bob