[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Stable-8.0.1 44/59] migration: Handle block device inactivation failure
From: |
Michael Tokarev |
Subject: |
[Stable-8.0.1 44/59] migration: Handle block device inactivation failures better |
Date: |
Tue, 23 May 2023 13:15:04 +0300 |
From: Eric Blake <eblake@redhat.com>
Consider what happens when performing a migration between two host
machines connected to an NFS server serving multiple block devices to
the guest, when the NFS server becomes unavailable. The migration
attempts to inactivate all block devices on the source (a necessary
step before the destination can take over); but if the NFS server is
non-responsive, the attempt to inactivate can itself fail. When that
happens, the destination fails to get the migrated guest (good,
because the source wasn't able to flush everything properly):
(qemu) qemu-kvm: load of migration failed: Input/output error
at which point, our only hope for the guest is for the source to take
back control. With the current code base, the host outputs a message, but then
appears to resume:
(qemu) qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable:
bdrv_inactivate_all() failed (-1)
(src qemu)info status
VM status: running
but a second migration attempt now asserts:
(src qemu) qemu-kvm: ../block.c:6738: int
bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags &
BDRV_O_INACTIVE)' failed.
Whether the guest is recoverable on the source after the first failure
is debatable, but what we do not want is to have qemu itself fail due
to an assertion. It looks like the problem is as follows:
In migration.c:migration_completion(), the source sets 'inactivate' to
true (since COLO is not enabled), then tries
savevm.c:qemu_savevm_state_complete_precopy() with a request to
inactivate block devices. In turn, this calls
block.c:bdrv_inactivate_all(), which fails when flushing runs up
against the non-responsive NFS server. With savevm failing, we are
now left in a state where some, but not all, of the block devices have
been inactivated; but migration_completion() then jumps to 'fail'
rather than 'fail_invalidate' and skips an attempt to reclaim those
those disks by calling bdrv_activate_all(). Even if we do attempt to
reclaim disks, we aren't taking note of failure there, either.
Thus, we have reached a state where the migration engine has forgotten
all state about whether a block device is inactive, because we did not
set s->block_inactive in enough places; so migration allows the source
to reach vm_start() and resume execution, violating the block layer
invariant that the guest CPUs should not be restarted while a device
is inactive. Note that the code in migration.c:migrate_fd_cancel()
will also try to reactivate all block devices if s->block_inactive was
set, but because we failed to set that flag after the first failure,
the source assumes it has reclaimed all devices, even though it still
has remaining inactivated devices and does not try again. Normally,
qmp_cont() will also try to reactivate all disks (or correctly fail if
the disks are not reclaimable because NFS is not yet back up), but the
auto-resumption of the source after a migration failure does not go
through qmp_cont(). And because we have left the block layer in an
inconsistent state with devices still inactivated, the later migration
attempt is hitting the assertion failure.
Since it is important to not resume the source with inactive disks,
this patch marks s->block_inactive before attempting inactivation,
rather than after succeeding, in order to prevent any vm_start() until
it has successfully reactivated all devices.
See also https://bugzilla.redhat.com/show_bug.cgi?id=2058982
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Acked-by: Lukas Straub <lukasstraub2@web.de>
Tested-by: Lukas Straub <lukasstraub2@web.de>
Signed-off-by: Juan Quintela <quintela@redhat.com>
(cherry picked from commit 403d18ae384239876764bbfa111d6cc5dcb673d1)
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
diff --git a/migration/migration.c b/migration/migration.c
index bda4789193..cb0d42c061 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -3444,13 +3444,11 @@ static void migration_completion(MigrationState *s)
MIGRATION_STATUS_DEVICE);
}
if (ret >= 0) {
+ s->block_inactive = inactivate;
qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
ret = qemu_savevm_state_complete_precopy(s->to_dst_file, false,
inactivate);
}
- if (inactivate && ret >= 0) {
- s->block_inactive = true;
- }
}
qemu_mutex_unlock_iothread();
@@ -3522,6 +3520,7 @@ fail_invalidate:
bdrv_activate_all(&local_err);
if (local_err) {
error_report_err(local_err);
+ s->block_inactive = true;
} else {
s->block_inactive = false;
}
--
2.39.2
- [Stable-8.0.1 v2 00/59] Patch Round-up for stable 8.0.1, freeze on 2023-05-27, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 38/59] s390x/pv: Fix spurious warning with asynchronous teardown, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 35/59] s390x/tcg: Fix LDER instruction format, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 42/59] linux-user: Fix mips fp64 executables loading, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 39/59] util/async-teardown: wire up query-command-line-options, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 40/59] docs/about/emulation: fix typo, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 45/59] migration: Minor control flow simplification, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 41/59] tests/docker: bump the xtensa base to debian:11-slim, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 43/59] linux-user: fix getgroups/setgroups allocations, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 44/59] migration: Handle block device inactivation failures better,
Michael Tokarev <=
- [Stable-8.0.1 46/59] migration: Attempt disk reactivation in more failure scenarios, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 47/59] target/arm: Fix vd == vm overlap in sve_ldff1_z, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 48/59] scsi-generic: fix buffer overflow on block limits inquiry, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 50/59] target/i386: fix avx2 instructions vzeroall and vpermdq, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 51/59] vhost: fix possible wrap in SVQ descriptor ring, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 52/59] hw/pci: Disable PCI_ERR_UNCOR_MASK register for machine type < 8.0, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 49/59] target/i386: fix operand size for VCOMI/VUCOMI instructions, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 53/59] virtio-net: not enable vq reset feature unconditionally, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 54/59] virtio-crypto: fix NULL pointer dereference in virtio_crypto_free_request, Michael Tokarev, 2023/05/23
- [Stable-8.0.1 55/59] aio-posix: do not nest poll handlers, Michael Tokarev, 2023/05/23