Re: [Qemu-devel] Re: [RFT][PATCH 07/15] qemu_irq: Add IRQ handlers with delivery feedback

[Blue Swirl]

On Tue, May 25, 2010 at 8:16 PM, Anthony Liguori <address@hidden> wrote:
> On 05/25/2010 02:09 PM, Blue Swirl wrote:
>>
>> On Mon, May 24, 2010 at 8:13 PM, Jan Kiszka<address@hidden>  wrote:
>>
>>>
>>> From: Jan Kiszka<address@hidden>
>>>
>>> This allows to communicate potential IRQ coalescing during delivery from
>>> the sink back to the source. Targets that support IRQ coalescing
>>> workarounds need to register handlers that return the appropriate
>>> QEMU_IRQ_* code, and they have to propergate the code across all IRQ
>>> redirections. If the IRQ source receives a QEMU_IRQ_COALESCED, it can
>>> apply its workaround. If multiple sinks exist, the source may only
>>> consider an IRQ coalesced if all other sinks either report
>>> QEMU_IRQ_COALESCED as well or QEMU_IRQ_MASKED.
>>>
>>
>> No real devices are interested whether any of their output lines are
>> even connected. This would introduce a new signal type, bidirectional
>> multi-level, which is not correct.
>>
>
> I don't think it's really an issue of correct, but I wouldn't disagree to a
> suggestion that we ought to introduce a new signal type for this type of
> bidirectional feedback.  Maybe it's qemu_coalesced_irq and has a similar
> interface as qemu_irq.

I'd rather try harder to find a solution for the problem without this.

>> I think the real solution to coalescing is put the logic inside one
>> device, in this case APIC because it has the information about irq
>> delivery. APIC could monitor incoming RTC irqs for frequency
>> information and whether they get delivered or not. If not, an internal
>> timer is installed which injects the lost irqs.
>>
>> Of course, no real device could do such de-coalescing, but with this
>> approach, the voodoo is contained to insides of one device, APIC.
>>
>> We should also take a step back to think what was the cause of lost
>> irqs, IIRC uneven execution rate in QEMU.
>
> Not only that.  The pathological case is where a host is limited to a 1khz
> timer frequency and the guest requests a 1khz timer frequency.  Practically
> speaking, there is no way we'll ever be able to adjust timers to reinject
> lost interrupts because of the host timer limitation.

But APIC knows when the guest has acked a timer irq and so can
immediately issue another one to cover a lost irq. APIC could also
arrange a CPU exit. The irqs would not arrive at even rate, but in
this pathological case, the host can't ever schedule any delays less
than 1ms.

>>  Could this be fixed or taken
>> into account in timer handling? For example, CPU loop could analyze
>> the wall clock time between CPU exits and use that to offset the
>> timers. Thus the timer frequency (in wall clock time) could be made to
>> correspond a bit more to VCPU execution rate.
>>
>
> A lot of what motivates the timer reinjection work is very old linux kernels
> that had fixed userspace timer frequencies.  On newer host kernels, it's
> probably not nearly as important except when you get into pathological cases
> like exposing a high frequency HPET timer to the guest where you cannot keep
> up with the host.

There will always be some delays and overhead because of the
emulation. We can either hide this completely from the guest (by
stretching the guest time) or we can try to keep the guest time base
in synch with host but then the guest can't expect native class
performance to be available between timer irqs.

On a native hardware, you could have a 1kHz timer and a kernel task
always running for max. 999us with interrupts disabled, then for at
least 1us the interrupts are enabled and the timer interrupt handler
gets a chance to run. On QEMU, such a small time window may be
impossible to emulate. No coalescing would help in this case. With a
cycle counting emulation it's possible, but the speed would probably
be slower than native. With the interrupt coalescing you may be able
to cover some easy cases.

>
> Regards,
>
> Anthony Liguori
>
>>> Signed-off-by: Jan Kiszka<address@hidden>
>>> ---
>>>  hw/irq.c |   38 +++++++++++++++++++++++++++++---------
>>>  hw/irq.h |   22 +++++++++++++++-------
>>>  2 files changed, 44 insertions(+), 16 deletions(-)
>>>
>>> diff --git a/hw/irq.c b/hw/irq.c
>>> index 7703f62..db2cce6 100644
>>> --- a/hw/irq.c
>>> +++ b/hw/irq.c
>>> @@ -26,19 +26,27 @@
>>>
>>>  struct IRQState {
>>>     qemu_irq_handler handler;
>>> +    qemu_irq_fb_handler feedback_handler;
>>>     void *opaque;
>>>     int n;
>>>  };
>>>
>>> -void qemu_set_irq(qemu_irq irq, int level)
>>> +int qemu_set_irq(qemu_irq irq, int level)
>>>  {
>>> -    if (!irq)
>>> -        return;
>>> -
>>> -    irq->handler(irq->opaque, irq->n, level);
>>> +    if (!irq) {
>>> +        return 0;
>>> +    }
>>> +    if (irq->feedback_handler) {
>>> +        return irq->feedback_handler(irq->opaque, irq->n, level);
>>> +    } else {
>>> +        irq->handler(irq->opaque, irq->n, level);
>>> +        return QEMU_IRQ_DELIVERED;
>>> +    }
>>>  }
>>>
>>> -qemu_irq *qemu_allocate_irqs(qemu_irq_handler handler, void *opaque, int
>>> n)
>>> +static qemu_irq *allocate_irqs(qemu_irq_handler handler,
>>> +                               qemu_irq_fb_handler feedback_handler,
>>> +                               void *opaque, int n)
>>>  {
>>>     qemu_irq *s;
>>>     struct IRQState *p;
>>> @@ -48,6 +56,7 @@ qemu_irq *qemu_allocate_irqs(qemu_irq_handler handler,
>>> void *opaque, int n)
>>>     p = (struct IRQState *)qemu_mallocz(sizeof(struct IRQState) * n);
>>>     for (i = 0; i<  n; i++) {
>>>         p->handler = handler;
>>> +        p->feedback_handler = feedback_handler;
>>>         p->opaque = opaque;
>>>         p->n = i;
>>>         s[i] = p;
>>> @@ -56,22 +65,33 @@ qemu_irq *qemu_allocate_irqs(qemu_irq_handler
>>> handler, void *opaque, int n)
>>>     return s;
>>>  }
>>>
>>> +qemu_irq *qemu_allocate_irqs(qemu_irq_handler handler, void *opaque, int
>>> n)
>>> +{
>>> +    return allocate_irqs(handler, NULL, opaque, n);
>>> +}
>>> +
>>> +qemu_irq *qemu_allocate_feedback_irqs(qemu_irq_fb_handler handler,
>>> +                                      void *opaque, int n)
>>> +{
>>> +    return allocate_irqs(NULL, handler, opaque, n);
>>> +}
>>> +
>>>  void qemu_free_irqs(qemu_irq *s)
>>>  {
>>>     qemu_free(s[0]);
>>>     qemu_free(s);
>>>  }
>>>
>>> -static void qemu_notirq(void *opaque, int line, int level)
>>> +static int qemu_notirq(void *opaque, int line, int level)
>>>  {
>>>     struct IRQState *irq = opaque;
>>>
>>> -    irq->handler(irq->opaque, irq->n, !level);
>>> +    return qemu_set_irq(irq, !level);
>>>  }
>>>
>>>  qemu_irq qemu_irq_invert(qemu_irq irq)
>>>  {
>>>     /* The default state for IRQs is low, so raise the output now.  */
>>>     qemu_irq_raise(irq);
>>> -    return qemu_allocate_irqs(qemu_notirq, irq, 1)[0];
>>> +    return allocate_irqs(NULL, qemu_notirq, irq, 1)[0];
>>>  }
>>> diff --git a/hw/irq.h b/hw/irq.h
>>> index 5daae44..eee03e6 100644
>>> --- a/hw/irq.h
>>> +++ b/hw/irq.h
>>> @@ -3,15 +3,18 @@
>>>
>>>  /* Generic IRQ/GPIO pin infrastructure.  */
>>>
>>> -/* FIXME: Rmove one of these.  */
>>> +#define QEMU_IRQ_DELIVERED      0
>>> +#define QEMU_IRQ_COALESCED      (-1)
>>> +#define QEMU_IRQ_MASKED         (-2)
>>> +
>>>  typedef void (*qemu_irq_handler)(void *opaque, int n, int level);
>>> -typedef void SetIRQFunc(void *opaque, int irq_num, int level);
>>> +typedef int (*qemu_irq_fb_handler)(void *opaque, int n, int level);
>>>
>>> -void qemu_set_irq(qemu_irq irq, int level);
>>> +int qemu_set_irq(qemu_irq irq, int level);
>>>
>>> -static inline void qemu_irq_raise(qemu_irq irq)
>>> +static inline int qemu_irq_raise(qemu_irq irq)
>>>  {
>>> -    qemu_set_irq(irq, 1);
>>> +    return qemu_set_irq(irq, 1);
>>>  }
>>>
>>>  static inline void qemu_irq_lower(qemu_irq irq)
>>> @@ -19,14 +22,19 @@ static inline void qemu_irq_lower(qemu_irq irq)
>>>     qemu_set_irq(irq, 0);
>>>  }
>>>
>>> -static inline void qemu_irq_pulse(qemu_irq irq)
>>> +static inline int qemu_irq_pulse(qemu_irq irq)
>>>  {
>>> -    qemu_set_irq(irq, 1);
>>> +    int ret;
>>> +
>>> +    ret = qemu_set_irq(irq, 1);
>>>     qemu_set_irq(irq, 0);
>>> +    return ret;
>>>  }
>>>
>>>  /* Returns an array of N IRQs.  */
>>>  qemu_irq *qemu_allocate_irqs(qemu_irq_handler handler, void *opaque, int
>>> n);
>>> +qemu_irq *qemu_allocate_feedback_irqs(qemu_irq_fb_handler handler,
>>> +                                      void *opaque, int n);
>>>  void qemu_free_irqs(qemu_irq *s);
>>>
>>>  /* Returns a new IRQ with opposite polarity.  */
>>> --
>>> 1.6.0.2
>>>
>>>
>>>
>>
>>
>
>