Re: [Savannah-users] CVS errors

[Sylvain Beucler]

On Wed, Oct 21, 2009 at 10:09:45AM +0200, Sylvain Beucler wrote:
> On Wed, Oct 21, 2009 at 09:25:41AM +0200, Sylvain Beucler wrote:
> > On Tue, Oct 20, 2009 at 07:48:52PM -0700, Kaz Kylheku wrote:
> > > It always takes longer than you expect, even if you take into account
> > > Hofstadter's law.
> > 
> > Well technically the migration was done in roughly 1/2h as expected ,
> > but _then_ there was an error and the filesystem was remounted
> > read-only, which I discover now :/
> > 
> > We're checking the filesystem right now, and will probably revert to
> > the previous hardware: we may lose a few recent commits, but to the
> > least we'll return to a trusted state.
> > 
> > In parallel we hope to contact the FSF system administrators soon
> > enough so they can check the hardware.
> 
> The service is back to its state of around ~12h ago.
> 
> We stop the migration plan until we have investigated what went wrong
> on the new hardware.

We traced the problem in the new backup script that I wrote, which
makes use of LVM snapshots, but sadly mounted the base volume twice
instead of making use of the snapshot, causing inconsistencies when
combined with Xen (where double-mount safety checks don't work).

Hardware is now pretty much out of cause, the backup script is fixed,
and the migration to the new hardware can resume.

There is no data corruption, we only lost at most one-hour's worth of
latest commits that happened before reverting the migration as
explained above.

Apologizes for the inconvenience.

-- 
Sylvain