Re: [PATCH v2 0/6] migration/ram: Optimize for virtio-mem via RamDiscardManager

[David Hildenbrand]

On 27.07.21 19:10, Peter Xu wrote:
> On Tue, Jul 27, 2021 at 11:25:09AM +0200, David Hildenbrand wrote:
> > For 2) I see 3 options:
> >
> > a) Sync everything, fixup the dirty bitmap, never clear the dirty log of
> > discarded parts. It's fairly simple and straight forward, as I can simply
> > reuse the existing helper. Something that's discarded will never be dirty,
> > not even if a misbehaving guest touches memory it shouldn't. [this patch]
> >
> > b) Sync only populated parts, no need to fixup the dirty bitmap, never clear
> > the dirty log of discarded parts. It's a bit more complicated but achieves
> > the same goal as a). [optimization I propose for the future]
> >
> > c) Sync everything, don't fixup the dirty bitmap, clear the dirty log of
> > discarded parts initially. There are ways we still might migrate discarded
> > ranges, for example, if a misbehaving guest touches memory it shouldn't.
> > [what you propose]
> >
> > Is my understanding correct? Any reasons why we should chose c) over b) long
> > term or c) over a) short term?
>
> My major concern is we could do something during sync() for not a very good
> reason by looping over virtio-mem bitmap for disgarded ranges - IIUC it should
> be destined to be merely no-op if the guest is well-behaved, am I right?

I think yes, as long as we have no initially-set bit stuck somewhere.

> Meanwhile, I still have no idea how much overhead the "loop" part could bring.
> For a large virtio-mem region with frequent plugged/unplugged mem interacted,
> it seems possible to take a while to me..  I have no solid idea yet.

Let's do some math. Assume the worst case on a 1TiB device with a 2MiB 
block size: We have 524288 blocks == bits. That's precisely a 64k bitmap 
in virtio-mem. In the worst case, every second bit would be clear 
("discarded"). For each clear bit ("discarded"), we would have to clear 
512 bits (64 bytes) in the dirty bitmap. That's storing 32 MiB.

So scanning 64 KiB, writing 32 MiB. Certainly not perfect, but I am not 
sure if it will really matter doing that once on every bitmap sync. I 
guess the bitmap syncing itself is much more expensive -- and not 
syncing the discarded ranges (b ) above) would make a bigger impact I guess.

> The thing is I still think this extra operation during sync() can be ignored by
> simply clear dirty log during bitmap init, then.. why not? :)

I guess clearing the dirty log (especially in KVM) might be more 
expensive. But, anyhow, we actually want b) long-term :)

> Clear dirty bitmap is as simple as "reprotect the pages" functional-wise - if
> they are unplugged memory ranges, and they shouldn't be written by the guest
> (we still allow reads even for virtio-mem compatibility), then I don't see it
> an issue to wr-protect it using clear dirty log when bitmap init.
>
> It still makes sense to me to keep the dirty/clear bitmap in-sync, at least
> before your plan b proposal; leaving the dirty bits set forever on unplugged
> memory is okay but just sounds a bit weird.
>
> Though my concern is only valid when virtio-mem is used, so I don't have a
> strong opinion on it as you maintains virtio-mem. I believe you will always
> have a better judgement than me on that. Especially, when/if Dave & Juan have
> no problem on that. :)

I'd certainly sleep better at night if I can be 100% sure that a page 
not to be migrated will not get migrated. :)

I'll play with initial clearing and see how much of a difference it 
makes code wise. Thanks a lot for your feedback!

--
Thanks,

David / dhildenb