Re: authentication failures

[Bob Proulx]

Thien-Thi Nguyen wrote:
> I confirm that it is working on the client side now, as well.
> My guess is that i was too hasty in retrying, and didn't give
> savannah enough time to process the new key.

Hmm...  I hate intermittent failures! :-(

At this time there should be no delay between updating an ssh key in
the Savannah web UI and being able to use it across the vcs and
download servers with ssh.  The ssh key is stored in an SQL database
and upon storing it is immediately available with an SQL query which
is used for ssh in our configuration.

In the development of Savannah (and the SourceForge it is based upon)
a lot of actions were done through cron running every hour or every
half hour or whatever.  One thing happens one place and then after a
bit the cron job runs and a reaction happens elsewhere.  That paradigm
of action and cron task later reaction is so pervasive that it is easy
to think that everything just needs time to settle.  (I would like to
get rid of all of those cases...)  And ssh keys used to be that way
until about ten years ago as far as I can tell from the historical
record.  Take a look at the last paragraph at that SshAccess page.

    file:///home/bob/src/savannah-stuff/administration/sviki/SshAccess.html

    (Savannah admin info: specifically, in
    /etc/ssh/authorized_keys/USERNAME on the subhosts; a cron job
    sv_authorized_keys runs on each vm.)

Well...  That isn't true anymore.  But it says that it used to be
true.  I saw that while looking at it for your incident and have it
queued up to make a change to that doc to update that bit but haven't
had a chance to do it yet.

Some time ago not sure when or who but things were changed to use the
MySQL database directly to hold ssh keys.  That has many advantages.
Not the least of which being that the web UI can store the changes to
the database and then the new keys are immediately available to the
other systems that need them for ssh access.  Both ssh keys and also
libnss for user account data.  They are separate processes but both
using the SQL database over the network.

The libnss SQL module for user account data has worked pretty well.
It does have a dependency on the network between the front facing
system with ssh needing it and the SQL database system that is shared
among the collection.  Usually that is okay.  But if the network has a
problem then reports will be logged.

    libnss-mysql: mysql_query failed: Lost connection to MySQL server during 
query, trying again (2)

I'll say that I probably never saw that happen for several years and
then in the last few months have suddenly started to see this quite
frequently in the logs.  (I use logcheck to scan the logs continuously
and send alerts upon unusual events.)  Most often this happens in the
middle of the US nighttime (say 3am or so but really at any time) when
no one, admins or volunteers, are doing anything.  And therefore I
have this idea that there must be network maintenance happening in the
datacenter in those hours which causes transient network glitches.

I am sure this affects our overseas free software friends the worst
since it is happening during their daylight times.  I feel bad about
that.  But then whenever we look at the problem later no trouble is
ever found.  I expect this to be causing transient glitches in using
the vcs services.  Upon retrying I expect things to work fine.

It isn't a problem of something changing on the systems because as you
have heard our problem is that we need to upgrade them and are working
on doing that, on other systems, and so nothing has changed on the vcs
systems.  Also this happens to both vcs0 and vcs1 both and they each
have different OS versions.  Therefore I am pretty sure this
particular problem is due to external influences.

> In any case, many thanks again for your help resolving things.
> Happy hacking!

I am happy things have been resolved for you!

Happy Hacking!
Bob

signature.asc
Description: PGP signature