Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores

[Davide Guerri]

Of course I can do some test but a kernel upgrade is not an option here :( 

However I don't think this is only a guest problem since it has been triggered 
by something between KVM 1.0 and 1.2.



On 14/mar/2013, at 18:50, Michael S. Tsirkin <address@hidden> wrote:

> 
> Could be one of the bugs fixed in guest drivers since 2.6.32.
> For example, 39d321577405e8e269fd238b278aaf2425fa788a ?
> Does it help if you try a more recent guest?
> 
> On Thu, Mar 14, 2013 at 11:43:30AM +0100, Alexandre DERUMIER wrote:
>> I don't think it's fixed in 1.3 or 1.4, some proxmox users have reported 
>> again this bug with guest kernel 2.6.32. (proxmox host is rhel 6.3 kernel + 
>> qemu 1.4)
>> 
>> 
>> 
>> ----- Mail original -----
>> 
>> De: "Davide Guerri" <address@hidden>
>> À: "Alexandre DERUMIER" <address@hidden>
>> Cc: "Peter Lieven" <address@hidden>, "Michael S. Tsirkin" <address@hidden>, 
>> "Stefan Hajnoczi" <address@hidden>, address@hidden, "Jan Kiszka" 
>> <address@hidden>, "Peter Lieven" <address@hidden>, "Dietmar Maurer" 
>> <address@hidden>
>> Envoyé: Jeudi 14 Mars 2013 10:22:22
>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
>> 
>> I'd like to reopen this thread because this problem is still here and it's 
>> really annoying.
>> Another possible work-around is to pin the virtio nic irq to one virtual CPU 
>> (with /proc/smp_affinity) but (of course) this is still sub-optimal.
>> 
>> Here are some graphs showing the performance of a heavy loaded machine after 
>> and before a live migration from KVM-1.0 to KVM-1.2.
>> Host is a Ubuntu 12.10 Kernel 3.5, guest is a Ubuntu 10.04.4 LTS Kernel 
>> 2.6.32
>> 
>> History
>> -------
>> CPU time: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/CPU-10days_zps4a088ff0.png.html
>> CPU Load: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/Load-10days_zpsff27f212.png.html
>> NET pps: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/pps-10days_zps003dd039.png.html
>> NET Mbps: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-10days_zpsfc3cba8c.png.html
>> 
>> Current
>> -------
>> CPU time: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/CPU-2days_zpsd362cac6.png.html
>> CPU Load: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/load-2days_zpsd4b7b50d.png.html
>> NET pps: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/pps-2days_zps8f5458c9.png.html
>> NET Mbps: 
>> http://s1299.beta.photobucket.com/user/dguerri/media/Mbps-2days_zps299338b9.png.html
>> 
>> Red arrows indicate the kvm version change.
>> Black arrows indicate when I "pinned" the virtio NIC IRQ to only one CPU 
>> (that machine has 2 virtual cores).
>> As can be seen the performance penalty is rather high: that machine was 
>> almost unusable!
>> 
>> Does version 1.3 fixes this issue?
>> 
>> Could someone with the required knowledge look into this, please?
>> Please, this is a very nasty bug because I guess I'm not the only one who is 
>> unable to upgrade all the machines with a (not-so) old kernel... :)
>> 
>> Thanks!
>> 
>> Davide Guerri.
>> 
>> 
>> 
>> 
>> On 09/dic/2012, at 19:38, Alexandre DERUMIER <address@hidden> wrote:
>> 
>>>>> Have you had any further progress on this regression/problem?
>>> 
>>> Hi Peter,
>>> I didn't re-tested myself,
>>> but a proxmox user who's have the problem with qemu-kvm 1.2, with windows 
>>> guest and linux guest,
>>> don't have the problem anymore with qemu 1.3.
>>> 
>>> http://forum.proxmox.com/threads/12157-Win2003R2-in-KVM-VM-is-slow-in-PVE-2-2-when-multiply-CPU-cores-allowed
>>> 
>>> I'll try to redone test myself this week
>>> 
>>> Regards,
>>> 
>>> Alexandre
>>> 
>>> ----- Mail original -----
>>> 
>>> De: "Peter Lieven" <address@hidden>
>>> À: "Alexandre DERUMIER" <address@hidden>
>>> Cc: "Dietmar Maurer" <address@hidden>, "Stefan Hajnoczi" <address@hidden>, 
>>> "Jan Kiszka" <address@hidden>, address@hidden, "Michael S. Tsirkin" 
>>> <address@hidden>, "Peter Lieven" <address@hidden>
>>> Envoyé: Lundi 3 Décembre 2012 12:23:11
>>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple cores
>>> 
>>> 
>>> Am 16.11.2012 um 12:00 schrieb Alexandre DERUMIER <address@hidden>: 
>>> 
>>>>>> While trying to reproduce the bug, we just detected that it depends on 
>>>>>> the hardware (mainboard) you run on.
>>>>>> 
>>>>>> Sigh :-/
>>>> 
>>>> Hi,
>>>> 
>>>> I can reproduce the bug on all my dell servers,differents generation (R710 
>>>> (intel),R815 (amd), 2950 (intel).
>>>> 
>>>> They all use broadcom bnx2 network card (don't know if it can be related)
>>>> 
>>>> host kernel : rhel 63 with 2.6.32 kernel
>>>> 
>>>> guest kernel : 2.6.32 (debian squeeze, ubuntu).
>>>> 
>>>> No problem with guest kernel 3.2
>>> 
>>> Have you had any further progress on this regression/problem?
>>> 
>>> Thanks,
>>> Peter
>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ----- Mail original -----
>>>> 
>>>> De: "Dietmar Maurer" <address@hidden>
>>>> À: "Peter Lieven" <address@hidden>
>>>> Cc: "Stefan Hajnoczi" <address@hidden>, "Peter Lieven" <address@hidden>, 
>>>> "Jan Kiszka" <address@hidden>, address@hidden, "Michael S. Tsirkin" 
>>>> <address@hidden>
>>>> Envoyé: Vendredi 16 Novembre 2012 11:44:26
>>>> Objet: Re: [Qemu-devel] slow virtio network with vhost=on and multiple 
>>>> cores
>>>> 
>>>>>> I only tested with RHEL6.3 kernel on host.
>>>>> 
>>>>> can you check if there is a difference on interrupt delivery between those
>>>>> two?
>>>>> 
>>>>> cat /proc/interrupts should be sufficient after some traffic has flown.
>>>> 
>>>> While trying to reproduce the bug, we just detected that it depends on the 
>>>> hardware (mainboard) you run on.
>>>> 
>>>> Sigh :-/
>>> 
>>> 
>

smime.p7s
Description: S/MIME cryptographic signature