Re: [Qemu-devel] Large sized guest taking for ever to boot...

[Chegu Vinod]

On 6/8/2012 11:37 AM, Jan Kiszka wrote:
> On 2012-06-08 20:20, Chegu Vinod wrote:
> > On 6/8/2012 11:08 AM, Jan Kiszka wrote:
> > > [CC'ing qemu as this discusses its code base]
> > >
> > > On 2012-06-08 19:57, Chegu Vinod wrote:
> > > > On 6/8/2012 10:42 AM, Alex Williamson wrote:
> > > > > On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote:
> > > > > > On 6/8/2012 9:46 AM, Alex Williamson wrote:
> > > > > > > On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I picked up a recent version of the qemu (1.0.92 with some fixes)
> > > > > > > > and tried it
> > > > > > > > on x86_64 server (with host and the guest running 3.4.1 kernel).
> > > > > > BTW, I observe the same thing if i were to use 1.1.50 version of the
> > > > > > qemu... not sure if this is really
> > > > > > related to qemu...
> > > > > >
> > > > > > > > While trying to boot a large guest (80 vcpus + 512GB) I observed
> > > > > > > > that the guest
> > > > > > > > took for ever to boot up...  ~1 hr or even more. [This wasn't the
> > > > > > > > case when I
> > > > > > > > was using RHEL 6.x related bits]
> > > > > > > Was either case using device assignment?  Device assignment will map
> > > > > > > and
> > > > > > > pin each page of guest memory before startup, which can be a noticeable
> > > > > > > pause on smallish (<16GB) guests.  That should be linear scaling though
> > > > > > > and if you're using qemu and not qemu-kvm, not related.  Thanks,
> > > > > > I don't have any device assignment at this point . Yes I am using qemu
> > > > > > (not qemu-kvm)...
> > > > > Just to be safe, are you using --enable-kvm with qemu?
> > > > Yes...
> > > Unless you are using current qemu.git master (where it is enabled by
> > > default), --enable-kvm does not activate the in-kernel irqchip support
> > > for you. Not sure if that can make such a huge difference, but it is a
> > > variation from qemu-kvm. You can enable it in qemu-1.1 with -machine
> > > kernel_irqchip=on.
> > Thanks for pointing this out...i will add that.
> >
> > I was using qemu.git    not the master
> With "qemu.git master" I meant the latest version you can pull from the
> master branch or qemu.git. If you are using an older version, please
> specify the hash.
>
> BTW, you can check if irqchip is on by looking at the out of "info
> qtree" in the qemu monitor. One of the last devices listed must be
> called "kvm-apic".


Sorry for the confusion.  I was using the qemu.git  and I do see the 
"kvm-apic" stuff  (in the info qtree output) without specifying the 
-machine kernel_irqchip=on option..


> > > > -----
> > > >
> > > > /etc/qemu-ifup tap0
> > > >
> > > > /usr/local/bin/qemu-system-x86_64 -enable-kvm \
> > > >
> > > > -cpu
> > > > Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s
> > > >
> > > > s,+acpi,+ds,+vme \
> > > > -m 524288 -smp 80,sockets=80,cores=1,threads=1 \
> > > > -name vm1 \
> > > > -chardev
> > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait
> > > > \
> > > > -drive
> > > > file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> > > > \
> > > > -device
> > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> > > > \
> > > > -monitor stdio \
> > > > -net nic,macaddr=52:54:00:71:01:01 \
> > > > -net tap,ifname=tap0,script=no,downscript=no \
> > > > -vnc :4
> > > >
> > > > /etc/qemu-ifdown tap0
> > > >
> > > > ----
> > > >
> > > > > > The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the
> > > > > > host and the guest and the host and the guest seemed to boot fine..
> > > > > Note that RHEL is based on qemu-kvm.  Thanks,
> > > > Yep..knew that :)
> > > >
> > > > I was using upstream qemu-kvm and was encouraged to move away from
> > > > it...to qemu.
> > > And that is good. :)
> > >
> > > Is the problem present in current qemu-kvm.git? If yes, can you bisect
> > > when it was introduced?
> > Shall try out the qemu-kvm.git  (after finishing some experiments..).
> Yes, please.

Just did that and below are the results...

> > BTW,  another data point ...if I try to boot a the RHEL6.3 kernel in the
> > guest (with the latest qemu.git and the 3.4.1 on the host) it boots just
> > fine....
> >
> > So something to do with the 3.4.1 kernel in the guest and the existing
> > udev... Need to dig
> > deeper.
> Maybe. But lets focus on getting the problematic guest running fine in
> some configuration. If that turns out to be impossible, we may see a
> guest issue.
Host config. : 80 Cores + 1TB.    Guest config : 40VCPUs + 512GB.

I rebuilt the 3.4.1 kernel in the guest from scratch and retried my 
experiments and measured
the boot times...

a) Host :  RHEL6.3 RC1 + qemu-kvm (that came with it) &  Guest : RHEL6.3 
RC1:  ~1 min

b) Host :3.4.1 + qemu-kvm.git & Guest : RHEL6.3RC1      -   ~1 min

c) Host :3.4.1 + qemu-kvm.git & Guest : 3.4.1      -   ~10 mins

d) Host :3.4.1 + qemu.git & Guest : RHEL6.3 RC1  -    ~1 min

e) Host :3.4.1 + qemu.git & Guest : 3.4.1                   -    ~14 mins


In both the case (c) & (e) had quite a few warning/error messages from 
udevd.

FYI..
Vinod
PS:  Haven't had the time to do any further analysis...as the machine is 
being used for
other experiments...


> Jan