Re: [Qemu-devel] [Question] why need to start all queues in vhost_net_start

[Jason Wang]

On 2017年11月16日 16:11, Yan Vugenfirer wrote:
> Hi Jason,
>
> Windows driver will initialise only the amount of queue based on the amount of 
> available vCPUs. So if there will be more queues in the device than we have 
> vCPUs on the guest, the driver will not initialise “excessive” queues. This is 
> tied to the way RSS on Windows should be implemented.
> Exactly as in described scenario (7 queues, but only 4 vCPUs).

I see but is there any chance that driver can use the left 3 queues (e 
.g cpu hotplug)?

Thanks

> Best regards,
> Yan.
>
> > On 16 Nov 2017, at 07:53, Longpeng (Mike) <address@hidden> wrote:
> >
> > Hi Jason & Michael,
> >
> > Do you have any idea about this problem ?
> >
> > --
> > Regards,
> > Longpeng(Mike)
> >
> > On 2017/11/15 23:54, Longpeng(Mike) wrote:
> >
> > > 2017-11-15 23:05 GMT+08:00 Jason Wang <address@hidden>:
> > > > On 2017年11月15日 22:55, Longpeng(Mike) wrote:
> > > > > Hi guys,
> > > > >
> > > > > We got a BUG report from our testers yesterday, the testing scenario was
> > > > > migrating a VM (Windows guest, *4 vcpus*, 4GB, vhost-user net: *7
> > > > > queues*).
> > > > >
> > > > > We found the cause reason, and we'll report the BUG or send a fix patch
> > > > > to upstream if necessary( we haven't test the upstream yet, sorry... ).
> > > >
> > > > Could you explain this a little bit more?
> > > >
> > > > > We want to know why the vhost_net_start() must start *total queues* ( in
> > > > > our
> > > > > VM there're 7 queues ) but not *the queues that current used* ( in our VM,
> > > > > guest
> > > > > only uses the first 4 queues because it's limited by the number of vcpus)
> > > > > ?
> > > > >
> > > > > Looking forward to your help, thx :)
> > > >
> > > > Since the codes have been there for years and works well for kernel
> > > > datapath. You should really explain what's wrong.
> > > OK. :)
> > >
> > > In our scenario,  the Windows's virtio-net driver only use the first 4
> > > queues and it
> > > *only set desc/avail/used table for the first 4 queues*, so in QEMU
> > > the desc/avail/
> > > used of the last 3 queues are ZERO,  but unfortunately...
> > > '''
> > > vhost_net_start
> > >   for (i = 0; i < total_queues; i++)
> > >     vhost_net_start_one
> > >       vhost_dev_start
> > >         vhost_virtqueue_start
> > > '''
> > > In vhost_virtqueue_start(), it will calculate the HVA of
> > > desc/avail/used table, so for last
> > > 3 queues, it will use ZERO as the GPA to calculate the HVA, and then
> > > send the results
> > > to the user-mode backend ( we use *vhost-user* ) by vhost_virtqueue_set_addr().
> > >
> > > When the EVS get these address, it will update a *idx* which will be
> > > treated as  vq's
> > > last_avail_idx when virtio-net stop ( pls see vhost_virtqueue_stop() ).
> > >
> > > So we get the following result after virtio-net stop:
> > >   the desc/avail/used of the last 3 queues's vqs are all ZERO, but these vqs's
> > >   last_avail_idx is NOT ZERO.
> > >
> > > At last, virtio_load() reports an error:
> > > '''
> > >   if (!vdev->vq[i].vring.desc && vdev->vq[i].last_avail_idx) { // <--
> > > will be TRUE
> > >       error_report("VQ %d address 0x0 "
> > >                          "inconsistent with Host index 0x%x",
> > >                          i, vdev->vq[i].last_avail_idx);
> > >             return -1;
> > >    }
> > > '''
> > >
> > > BTW, the problem won't appear if use Linux guest, because the Linux virtio-net
> > > driver will set all 7 queues's desc/avail/used tables. And the problem
> > > won't appear
> > > if the VM use vhost-net, because vhost-net won't update *idx* in SET_ADDR ioctl.
> > >
> > > Sorry for my pool English, Maybe I could describe the problem in Chinese for you
> > > in private if necessary.
> > >
> > >
> > > > Thanks
> >
> > --
> > Regards,
> > Longpeng(Mike)