Re: [Qemu-devel] is there a limit on the number of in-flight I/O operations?

[Chris Friesen]

On 07/18/2014 10:30 AM, Andrey Korolyov wrote:
> On Fri, Jul 18, 2014 at 8:26 PM, Chris Friesen
> <address@hidden> wrote:
> > On 07/18/2014 09:54 AM, Andrey Korolyov wrote:
> > > On Fri, Jul 18, 2014 at 6:58 PM, Chris Friesen
> > > <address@hidden> wrote:
> > > > Hi,
> > > >
> > > > I've recently run up against an interesting issue where I had a number of
> > > > guests running and when I started doing heavy disk I/O on a virtio disk
> > > > (backed via ceph rbd) the memory consumption spiked and triggered the
> > > > OOM-killer.
> > > >
> > > > I want to reserve some memory for I/O, but I don't know how much it can
> > > > use
> > > > in the worst-case.
> > > >
> > > > Is there a limit on the number of in-flight I/O operations?  (Preferably
> > > > as
> > > > a configurable option, but even hard-coded would be good to know as
> > > > well.)
> > > >
> > > > Thanks,
> > > > Chris
> > >
> > > Hi, are you using per-vm cgroups or it was happened on bare system?
> > > Ceph backend have writeback cache setting, may be you hitting it but
> > > it must be set enormously large then.
> >
> > This is without cgroups.  (I think we had tried cgroups and ran into some
> > issues.)  Would cgroups even help with iSCSI/rbd/etc?
> >
> > The "-drive" parameter in qemu was using "cache=none" for the VMs in
> > question.  But I'm assuming it keeps the buffer around until acked by the
> > far end in order to be able to handle retries.
> >
> > Chris
>
> This is probably a bug even if the legitimate mechanisms causing it -
> peak memory footprint for an emulator should be predictable. Never hit
> something like this on any kind of workload, will try to reproduce by
> myself.

The drive parameter would have looked something like this:

-drive 
file=rbd:volumes/volume-7c1427d4-0758-4384-9431-653aab24a690:auth_supported=none:mon_host=192.168.205.3\:6789\;192.168.205.4\:6789\;192.168.205.5\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=7c1427d4-0758-4384-9431-653aab24a690,cache=none 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1

When we started running dbench in the guest the qemu RSS jumped 
significantly.  Also, it stayed at the higher value even after the test 
was stopped--which is not ideal behaviour.

Chris