[Qemu-devel] Re: [RFC] Moving the kvm ioapic, pic, and pit back to userspace

[Anthony Liguori]

On 06/07/2010 01:42 PM, Avi Kivity wrote:
> On 06/07/2010 08:04 PM, Anthony Liguori wrote:
> > I think we could also move the local APIC.
>
> I'm not even sure we can safely move the ioapic/pic (mostly due to 
> churn).  But the local APIC is so heavily accessed by the guest that 
> it's impossible to move it.  Run an ftrace one day, especially on an 
> smp guest.  Every IPI requires several APIC accesses.  Before a halt a 
> tickless kernel sets the wakeup timer.  EOIs.
>
> > To optimize device models, we've tended to put the full device model 
> > in the kernel whereas the hardware vendors have tended to put only 
> > the fast paths of the devices models in hardware.
> >
> > For instance, we could introduce a userspace interface similar to 
> > vapic support whereas a shared page that mapped the APIC's layout was 
> > used with a mask to select which registers trapped on read/write.
>
> That leads to very problematic interfaces.  When you separate along a 
> device boundary, you have a spec that defines the software 
> interfaces.  When you separate along a boundary that you define, it's 
> up to you to get everything right.
>
> In fact with the ioapic/pic/lapic one of the problems is that the 
> interconnection between the devices that is not well defined, and 
> that's where we have bugs.
>
> > That said, I can understand an argument that the local APIC is part 
> > of the CPU state since it's a very special type of device.
> >
> > A better example would be a generic counter kernel mechanism.  I can 
> > envision such a device as doing nothing more than providing a 
> > read-only view of a counter with a userspace configurable divider and 
> > width.  Any write to the counter or read of any other byte outside 
> > the counter register would result in a trap to userspace.
>
> What about latches?  byte access to word registers?  There will be as 
> many special cases as there are timers.
>
> If the kernel supported a bytecode/jit facility I'd happily use that 
> to download portions of the device model into the kernel.
>
> > That should allow both the PIT and the HPET to be accelerated with 
> > minimal effort in the kernel.
>
> IMO it's probably more effort than porting HPET to the kernel.  Try 
> outlining an interface that supports PIT, HPET, RTC, and ACPI PMTIMER.

I was referring specifically to time sources, not time events.

An accelerated counter for HPET is pretty trivial.  It's a 32-bit 
register that's actually a nanosecond value in qemu.  We need to be able 
to set an offset from the host wall clock time, a means to stop it, and 
a means to start it.

The PIT is latched so the kernel needs to know enough about how to 
decode the PIT state to understand the latching.  There's very little 
state associated with latching though so I don't think this is a huge 
problem.  It's a fixed value write to a fixed register followed by a 
read to a fixed register.  The act of latching doesn't effect the state 
beyond the fact that you need to save the latched value in the event 
that you have a live migration before reading the latched value.

The PMTIMER is also pretty straight forward.  It's a variable port 
address (that's fixed during execution).

Even if we require three separate interfaces, the interfaces are so 
simply that it seems like an obvious win.

> > I'd be in favor of a straight port to userspace.  We already have the 
> > interfaces to communicate with an external device model for these 
> > devices so let's just take the kernel code and stick it into 
> > dedicated threads in userspace.
>
> Currently we support an all-or-nothing approach.  I don't think local 
> APIC in userspace is worthwhile.  Esp. as it will slow down vhost and 
> assigned devices significantly - interrupts will have to be mediated 
> by userspace.

Yeah, as I said, I can understand the arguments for keeping the lapic in 
the kernel.

> > I think it's easier to then work to merge the two bits of code in the 
> > same tree than it is to try and take out-of-tree code and merge it 
> > incrementally.
>
> Are you talking about qemu.git/qemu-kvm.git?  That's the least of my 
> concerns, I'm worried about kvm.git.

qemu.git.

> > > 5. Risk
> > >
> > > We may find out after all this is implemented that performance is 
> > > not acceptable and all the work will have to be dropped.
> >
> > That's another advantage to a straight port to userspace.  We can 
> > collect performance data with only a modest amount of engineering 
> > effort.
>
> Port what exactly?  We have a userspace irqchip implementation.  What 
> we don't have is just the ioapic/pic/pit in userspace, and the only 
> way to try it out is to implement the whole thing.

If you take the kernel code and do a pretty straight port: switching 
kernel functions to libc functions and maintaining all the existing 
locking via pthreads, you could then implement a very simple MMIO/PIO 
dispatch mechanism in the kvm code that shortcutted those devices before 
we ever hit the qemu_mutex and the traditional qemu code paths.  It 
should be a relatively easy conversion and it gives a proper vehicle for 
doing experimentations.

In fact, you could pretty quickly determine viability by porting the PIT 
to userspace and implementing a vpit interface in the kernel that 
allowed the channel 0 counters to be latched and read within lightweight 
exits.

Regards,

Anthony Liguori