Re: [Qemu-devel] [RFC] create a single workqueue for each vm to update vm irq routing table

[Gleb Natapov]

On Thu, Nov 28, 2013 at 01:33:48PM +0200, Michael S. Tsirkin wrote:
> On Thu, Nov 28, 2013 at 01:22:45PM +0200, Gleb Natapov wrote:
> > On Thu, Nov 28, 2013 at 01:18:54PM +0200, Avi Kivity wrote:
> > > On 11/28/2013 01:02 PM, Gleb Natapov wrote:
> > > >On Thu, Nov 28, 2013 at 12:12:55PM +0200, Avi Kivity wrote:
> > > >>On 11/28/2013 12:11 PM, Gleb Natapov wrote:
> > > >>>On Thu, Nov 28, 2013 at 11:49:00AM +0200, Avi Kivity wrote:
> > > >>>>On 11/28/2013 11:19 AM, Gleb Natapov wrote:
> > > >>>>>On Thu, Nov 28, 2013 at 09:55:42AM +0100, Paolo Bonzini wrote:
> > > >>>>>>Il 28/11/2013 07:27, Zhanghaoyu (A) ha scritto:
> > > >>>>>>>>>Without synchronize_rcu you could have
> > > >>>>>>>>>
> > > >>>>>>>>>    VCPU writes to routing table
> > > >>>>>>>>>                                       e = entry from IRQ 
> > > >>>>>>>>> routing table
> > > >>>>>>>>>    kvm_irq_routing_update(kvm, new);
> > > >>>>>>>>>    VCPU resumes execution
> > > >>>>>>>>>                                       kvm_set_msi_irq(e, &irq);
> > > >>>>>>>>>                                       
> > > >>>>>>>>> kvm_irq_delivery_to_apic_fast();
> > > >>>>>>>>>
> > > >>>>>>>>>where the entry is stale but the VCPU has already resumed 
> > > >>>>>>>>>execution.
> > > >>>>>>>>>
> > > >>>>>>>If we use call_rcu()(Not consider the problem that Gleb pointed 
> > > >>>>>>>out temporarily) instead of synchronize_rcu(), should we still 
> > > >>>>>>>ensure this?
> > > >>>>>>The problem is that we should ensure this, so using call_rcu is not
> > > >>>>>>possible (even not considering the memory allocation problem).
> > > >>>>>>
> > > >>>>>Not changing current behaviour is certainly safer, but I am still 
> > > >>>>>not 100%
> > > >>>>>convinced we have to ensure this.
> > > >>>>>
> > > >>>>>Suppose guest does:
> > > >>>>>
> > > >>>>>1: change msi interrupt by writing to pci register
> > > >>>>>2: read the pci register to flush the write
> > > >>>>>3: zero idt
> > > >>>>>
> > > >>>>>I am pretty certain that this code can get interrupt after step 2 on 
> > > >>>>>real HW,
> > > >>>>>but I cannot tell if guest can rely on it to be delivered exactly 
> > > >>>>>after
> > > >>>>>read instruction or it can be delayed by couple of instructions. 
> > > >>>>>Seems to me
> > > >>>>>it would be fragile for an OS to depend on this behaviour. AFAIK 
> > > >>>>>Linux does not.
> > > >>>>>
> > > >>>>Linux is safe, it does interrupt migration from within the interrupt
> > > >>>>handler.  If you do that before the device-specific EOI, you won't
> > > >>>>get another interrupt until programming the MSI is complete.
> > > >>>>
> > > >>>>Is virtio safe? IIRC it can post multiple interrupts without guest 
> > > >>>>acks.
> > > >>>>
> > > >>>>Using call_rcu() is a better solution than srcu IMO.  Less code
> > > >>>>changes, consistently faster.
> > > >>>Why not fix userspace to use KVM_SIGNAL_MSI instead?
> > > >>>
> > > >>>
> > > >>Shouldn't it work with old userspace too? Maybe I misunderstood your 
> > > >>intent.
> > > >Zhanghaoyu said that the problem mostly hurts in real-time telecom
> > > >environment, so I propose how he can fix the problem in his specific
> > > >environment.  It will not fix older userspace obviously, but kernel
> > > >fix will also require kernel update and usually updating userspace
> > > >is easier.
> > > >
> > > >
> > > 
> > > Isn't the latency due to interrupt migration causing long
> > > synchronize_rcu()s?  How does KVM_SIGNAL_MSI help?
> > > 
> > If MSI is delivered using KVM_SIGNAL_MSI as opposite to via an entry in
> > irq routing table changing MSI configuration should not cause update to
> > irq routing table (not saying this is what happens with current QEMU, but
> > theoretically there is not reason to update routing table in this case).
> > 
> > --
> >                     Gleb.
> 
> Unfortunately all high performance users (vhost net,
> vhost scsi, virtio-blk data plane, vfio) switched to using
> eventfd.
> 
Right :(

> KVM_SIGNAL_MSI is used as a simple mechanism to avoid routing
> table hassles e.g. for hotplug MSIs.
> 

--
                        Gleb.