Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent

[Dr. David Alan Gilbert]

* Chunguang Li (address@hidden) wrote:
> 
> 
> 
> > -----原始邮件-----
> > 发件人: "Dr. David Alan Gilbert" <address@hidden>
> > 发送时间: 2016年9月26日 星期一
> > 收件人: "Chunguang Li" <address@hidden>
> > 抄送: address@hidden, address@hidden, address@hidden, address@hidden, 
> > address@hidden
> > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as 
> > dirty after they have been sent
> > 
> > * Chunguang Li (address@hidden) wrote:
> > > Hi all!
> > > I have some confusion about the dirty bitmap during migration. I have 
> > > digged into the code. I figure out that every now and then during 
> > > migration, the dirty bitmap will be grabbed from the kernel space through 
> > > ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. 
> > > However I think this mechanism leads to resendness of some NON-dirty 
> > > pages.
> > > 
> > > Take the first iteration of precopy for instance, during which all the 
> > > pages will be sent. Before that during the migration setup, the 
> > > ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce 
> > > the dirty bitmap from this moment. When the pages "that haven't been 
> > > sent" are written, the kernel space marks them as dirty. However I don't 
> > > think this is correct, because these pages will be sent during this and 
> > > the next iterations with the same content (if they are not written again 
> > > after they are sent). It only makes sense to mark the pages which have 
> > > already been sent during one iteration as dirty when they are written.
> > > 
> > > 
> > > Am I right about this consideration? If I am right, is there some advice 
> > > to improve this?
> > 
> > I think you're right that this can happen; to clarify I think the
> > case you're talking about is:
> > 
> >   Iteration 1
> >     sync bitmap
> >     start sending pages
> >     page 'n' is modified - but hasn't been sent yet
> >     page 'n' gets sent
> >   Iteration 2
> >     sync bitmap
> >        'page n is shown as modified'
> >     send page 'n' again
> >
> 
> Yes，this is right the case I am talking about.
>  
> > So you're right that is wasteful; I guess it's more wasteful
> > on big VMs with slow networks where the length of each iteration
> > is large.
> 
> I think this is "very" wasteful. Assume the workload writes the pages dirty 
> randomly within the guest address space, and the transfer speed is constant. 
> Intuitively, I think nearly half of the dirty pages produced in Iteration 1 
> is not really dirty. This means the time of Iteration 2 is double of that to 
> send only really dirty pages.

Yes, it's probably pretty bad; and we really need to do something like
split the sync into smaller chunks; there are other suggestions
for how to improve it (e.g. there's the page-modification-logging
changes).

However, I don't think you usually get really random writes, if you
do precopy rarely converges at all, because even without your
observation it changes lots and lots of pages.

Dave

> Thanks,
> 
> Chunguang
> 
> > 
> > Fixing it is not easy, because you have to be really careful
> > never to miss a page modification, even if the page is sent
> > about the same time it's dirtied.
> > 
> > One way would be to sync the dirty log from the kernel
> > in smaller chunks.
> > 
> > Dave
> > 
> > 
> > > 
> > > Thanks,
> > > Chunguang Li
> > --
> > Dr. David Alan Gilbert / address@hidden / Manchester, UK
> 
> 
> --
> Chunguang Li, Ph.D. Candidate
> Wuhan National Laboratory for Optoelectronics (WNLO)
> Huazhong University of Science & Technology (HUST)
> Wuhan, Hubei Prov., China
> 
> 
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK