Re: [Qemu-devel] [PATCH v7 0/5] virtio-balloon: free page hint reporting support

[Dr. David Alan Gilbert]

* Peter Xu (address@hidden) wrote:
> On Fri, Jun 01, 2018 at 03:21:54PM +0800, Wei Wang wrote:
> > On 06/01/2018 12:58 PM, Peter Xu wrote:
> > > On Tue, Apr 24, 2018 at 02:13:43PM +0800, Wei Wang wrote:
> > > > This is the deivce part implementation to add a new feature,
> > > > VIRTIO_BALLOON_F_FREE_PAGE_HINT to the virtio-balloon device. The device
> > > > receives the guest free page hints from the driver and clears the
> > > > corresponding bits in the dirty bitmap, so that those free pages are
> > > > not transferred by the migration thread to the destination.
> > > > 
> > > > - Test Environment
> > > >      Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> > > >      Guest: 8G RAM, 4 vCPU
> > > >      Migration setup: migrate_set_speed 100G, migrate_set_downtime 2 
> > > > second
> > > > 
> > > > - Test Results
> > > >      - Idle Guest Live Migration Time (results are averaged over 10 
> > > > runs):
> > > >          - Optimization v.s. Legacy = 271ms vs 1769ms --> ~86% reduction
> > > >      - Guest with Linux Compilation Workload (make bzImage -j4):
> > > >          - Live Migration Time (average)
> > > >            Optimization v.s. Legacy = 1265ms v.s. 2634ms --> ~51% 
> > > > reduction
> > > >          - Linux Compilation Time
> > > >            Optimization v.s. Legacy = 4min56s v.s. 5min3s
> > > >            --> no obvious difference
> > > > 
> > > > - Source Code
> > > >      - QEMU:  https://github.com/wei-w-wang/qemu-free-page-lm.git
> > > >      - Linux: https://github.com/wei-w-wang/linux-free-page-lm.git
> > > Hi, Wei,
> > > 
> > > I have a very high-level question to the series.
> > 
> > Hi Peter,
> > 
> > Thanks for joining the discussion :)
> 
> Thanks for letting me know this thread.  It's an interesting idea. :)
> 
> > 
> > > 
> > > IIUC the core idea for this series is that we can avoid sending some
> > > of the pages if we know that we don't need to send them.  I think this
> > > is based on the fact that on the destination side all the pages are by
> > > default zero after they are malloced.  While before this series, IIUC
> > > any migration will send every single page to destination, no matter
> > > whether it's zeroed or not.  So I'm uncertain about whether this will
> > > affect the received bitmap on the destination side.  Say, before this
> > > series, the received bitmap will directly cover the whole RAM bitmap
> > > after migration is finished, now it's won't.  Will there be any side
> > > effect?  I don't see obvious issue now, but just raise this question
> > > up.
> > 
> > This feature currently only supports pre-copy (I think the received bitmap
> > is something matters to post copy only).
> > That's why we have
> > rs->free_page_support = ..&& !migrate_postcopy();
> 
> Okay.
> 
> > 
> > > Meanwhile, this reminds me about a more funny idea: whether we can
> > > just avoid sending the zero pages directly from QEMU's perspective.
> > > In other words, can we just do nothing if save_zero_page() detected
> > > that the page is zero (I guess the is_zero_range() can be fast too,
> > > but I don't know exactly how fast it is)?  And how that would be
> > > differed from this page hinting way in either performance and other
> > > aspects.
> > 
> > I guess you referred to the zero page optimization. I think the major
> > overhead comes to the zero page checking - lots of memory accesses, which
> > also waste memory bandwidth. Please see the results attached in the cover
> > letter. The legacy case already includes the zero page optimization.
> 
> I replied in the other thread.  We can discuss there altogether.
> 
> Actually after a second thought I think maybe what I worried there is
> exactly the reason why we must send the zero page flag - otherwise
> there can be stale non-zero page on destination.  Here "zero page" and
> "freed page" is totally different idea since even if a page is zeroed
> it might still be in use (not freed)!  While instead for a "free page"
> even if it's non-zero we might be able to not send it at all, though I
> am not sure whether that mismatch of data might cause any side effect
> too. I think the corresponding question would be: if a page is freed
> in Linux kernel, would its data matter any more?

I think the answer is no - it doesn't matter; by telling the hypervisor
the page is 'free' the kernel gives freedom to the hypervisor to
discard the page contents.
Now, that is trusting the kernel to get it's 'free' flags right,
and we wouldn't want a malicious guest kernel to be able to read random
data, so we have to be a little careful that what actually lands
in there is something the guest has had at some point - or zero
which is a very nice empty value.

Dave

> Thanks,
> 
> -- 
> Peter Xu
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK