Failover and grub.

[spoofy root]

I am trying to build a reliable linux applience which I can do remote
upgrades on and have it failover if something goes wrong with the
update. My idea is to setup a recovery partition , boot partition, and
system partition.  When I need to do an update... simply edit grub to
boot from the recovery partition and once there.. update the system
partition. Then have it boot the system partition. BUT .. the problem
is that if it fails.. for example if it has a bad kernel..once it is
rebooted it will boot back into the recovery partition and not the bad
partition.  I was thinking having the system do something like write a
file on a sucessful boot and if possible have grub check for it .. and
if it is there( or missing) then boot into the recovery partition. 
But I don't think grub can do something like this..

Anyway.. that is the problem..and I'm not sure of what solution there
is.. or how to solve the problem.. do you guys know of a way to do
something like this with or even without grub?  Or if not know where I
could get info on doing something like this...


thanks

Johnathan