[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH v1 2/2] virtio-balloon: disallow postcopy with VIRTIO_BALLOON

From: David Hildenbrand
Subject: Re: [PATCH v1 2/2] virtio-balloon: disallow postcopy with VIRTIO_BALLOON_F_FREE_PAGE_HINT
Date: Wed, 7 Jul 2021 21:14:00 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0

On 07.07.21 21:05, Michael S. Tsirkin wrote:
On Wed, Jul 07, 2021 at 04:06:55PM +0200, David Hildenbrand wrote:
Postcopy never worked properly with 'free-page-hint=on', as there are
at least two issues:

1) With postcopy, the guest will never receive a VIRTIO_BALLOON_CMD_ID_DONE
    and consequently won't release free pages back to the OS once
    migration finishes.

    The issue is that for postcopy, we won't do a final bitmap sync while
    the guest is stopped on the source and
    virtio_balloon_free_page_hint_notify() will only call
    virtio_balloon_free_page_done() on the source during
    PRECOPY_NOTIFY_CLEANUP, after the VM state was already migrated to
    the destination.

2) Once the VM touches a page on the destination that has been excluded
    from migration on the source via qemu_guest_free_page_hint() while
    postcopy is active, that thread will stall until postcopy finishes
    and all threads are woken up. (with older Linux kernels that won't
    retry faults when woken up via userfaultfd, we might actually get a

    The issue is that the source will refuse to migrate any pages that
    are not marked as dirty in the dirty bmap -- for example, because the
    page might just have been sent. Consequently, the faulting thread will
    stall, waiting for the page to be migrated -- which could take quite
    a while and result in guest OS issues.

OK so if source gets a request for a page which is not dirty
it does not respond immediately? Why not just teach it to
respond? It would seem that if destination wants a page we
should just give it to the destination ...

The source does not know if a page has already been sent (e.g., via the background migration thread that moves all data over) vs. the page has not been send because the page was hinted. This is the part where we'd need additional tracking on the source to actually know that.

We must not send a page twice, otherwise bad things can happen when placing pages that already have been migrated, because that scenario can easily happen with ordinary postcopy (page has already been sent and we're dealing with a stale request from the destination).

While we could fix 1), for example, by calling
virtio_balloon_free_page_done() via pre_save callbacks of the
vmstate, 2) is mostly impossible to fix without additional tracking,
such that we can actually identify these hinted pages and handle
them accordingly.
As it never worked properly, let's disable it via the postcopy notifier on
the destination. Trying to set "migrate_set_capability postcopy-ram on"
on the destination now results in "virtio-balloon: 'free-page-hint' does
not support postcopy Error: Postcopy is not supported".
Note 1: We could let qemu_guest_free_page_hint() mark postcopy
         as broken once actually clearing bits on the source. However, it's
         harder to realize as we can race with users starting postcopy
         and we cannot produce an expressive error message easily.

How about the reverse? Ignore qemu_guest_free_page_hint if postcopy
started?  Seems better than making it user/guest visible ..

Might be an option, but we let the user configure something that does not work in combination ... essentially ignoring one of both user settings. Also not perfect IMHO.

Note 2: virtio-mem has similar issues, however, access to "unplugged"
         memory by the guest is very rare and we would have to be very
         lucky for it to happen during migration. The spec states
         "The driver SHOULD NOT read from unplugged memory blocks ..."
         and "The driver MUST NOT write to unplugged memory blocks".
         virtio-mem will move away from virtio_balloon_free_page_done()
         soon and handle this case explicitly on the destination.

Fixes: c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT")

OK it's not too bad, but I wonder whether above aideas have been

TBH, it's been broken all along and I'd rather have a simple fix. If somebody ever cares about this, we could investigate making it work (or making postcopy overrule free page hinting). But I'm open for suggestions.


David / dhildenb

reply via email to

[Prev in Thread] Current Thread [Next in Thread]