Re: [Qemu-devel] [PATCH 09/16] Enable message delivery via IRQs

[Blue Swirl]

On Sat, Jun 12, 2010 at 8:32 PM, Blue Swirl <address@hidden> wrote:
> On Sat, Jun 12, 2010 at 7:33 PM, Blue Swirl <address@hidden> wrote:
>> On Sat, Jun 12, 2010 at 3:58 PM, Paul Brook <address@hidden> wrote:
>>>> I think message passing interrupts
>>>> are only used in bus based systems, like PCI or UPA/JBUS etc. I don't
>>>> know how LAPIC/IOAPIC bus works, it could be similar.
>>>
>>> PCI Message Signalled Interrupts use a regular data write, and we model it
>>> exactly that way. Under normal circumstances you program the device to write
>>> to memory mapped APIC control registers, but there's no real reason why you
>>> couldn't write to RAM.
>>>
>>> LAPIC/IOAPIC have their own bus. On some hardware this is multiplexed over 
>>> the
>>> main system bus, but logically it is a separate interconnect.
>>>
>>> The fact that these a bus based systems suggests against using qemu_irq, 
>>> which
>>> is an inherently point-point system.
>>>
>>>> > How is a
>>>> > receiver meant to know for format of the message being delivered, and who
>>>> > it's intended for?
>>>>
>>>> This depends on the architecture. On Sparc64 the message is specified
>>>> by the guest or OF.
>>>>...
>>>> In real HW (MSI or Sparc64 mondo interrupts) a bus delivers the
>>>> message. But our buses only handle address and data lines.
>>>
>>> IIUC you're trying to use qemu_irq as a generic interconnect between 
>>> devices.
>>> I think that's going to cause more problems than it solves.  If a bus has
>>> additional message passing capabilities then this can be part of the bus
>>> interface.
>>>
>>> If two devices need a private communication channel then we probably want 
>>> some
>>> implementation agnostic way of defining this link.  Connecting LAPIC to 
>>> IOAPIC
>>> is a potential example of this. However this needs to be done in a type-safe
>>> manner.  DMA channels may be another potential use of this linking. We'd 
>>> want
>>> to avoid connecting a DMA channel to an APIC.
>>>
>>> [*] A simple unidirectional dma request line is suitable for qmu_irq. A DMA
>>> system that transfers data outside of memory read/write transactions is not.
>>> e.g. ISA effectively defines a regular memory bus plus 8 bidirectional data
>>> streams (aka DMA channels). These are multiplexed over the same physical 
>>> pins,
>>> but logically distinct. The DMA channels could either be bundled up into the
>>> ISA bus interface, or modelled as links between devices and the DMA
>>> controller.
>>
>> Very very interesting. There's some out of band data related to DMA
>> (bus errors, IOMMU faults, ISA DREQ) which hasn't been covered in
>> previous Generic DMA proposals. Maybe all we need is a generic side
>> band link, which would be usable for point to point (qemu_irq,
>> coalescing, messages) and bus channels?
>
> I think we could solve all problems (well, maybe not world peace, yet)
> by switching to message based system for all of DMA and IRQs.
>
> Each device would have a message input port and way to output messages.
>
> Examples:
>
> Zero copy memory access from device D1 to D2 to host memory (D3) with
> access broken to page length units and errors occurring on the last
> byte:
> D1 send_msg(ID, MSG_MEM_WRITE, DMA address, length) -> D2
> D2 send_msg(ID, MSG_MEM_WRITE, DMA address2, length) -> D3
> D3 send_replymsg(ID, MSG_MEM_PTR, host address, 4096) -> D2
> D2 send_replymsg(ID, MSG_MEM_PTR, host address, 4096) -> D1
> D3 send_replymsg(ID, MSG_MEM_PTR, host address, length - 4096 - 1) -> D2
> D2 send_replymsg(ID, MSG_MEM_PTR, host address, length - 4096 - 1) -> D1
> D3 send_replymsg(ID, MSG_MEM_ERROR, DMA address2 + length - 1, 1, status) -> 
> D2
> D2 send_replymsg(ID, MSG_MEM_ERROR, DMA address + length - 1, 1, status) -> D1

This could be implemented with small modifications to existing
cpu_physical_memory_rw() interface. Improved devices would check if a
channel back has been registered. If not, current methods are used,
same with unimproved devices. If yes, translated pointers will flow
back for zero copy DMA.

> IRQ delivery chain D1->D2->D3 with coalescing, messages, delivery
> reporting and EOI:
> D1 send_msg(ID, MSG_IRQ_RAISE, payload) -> D2
> D2 send_msg(ID, MSG_IRQ_RAISE, payload) -> D3
> D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_COALESCED) -> D2
> D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_COALESCED) -> D1
> D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_DELIVERED) -> D2
> D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_DELIVERED) -> D1
> D3 send_replymsg(ID, MSG_IRQ_STATUS, MI_EOI) -> D2
> D2 send_replymsg(ID, MSG_IRQ_STATUS, MI_EOI) -> D1

Same here: send_msg could be in fact be qemu_irq_raise(). Without a
return chain, no replies will be sent. For messages, an augmented
forward chain must exist, otherwise the messages will be ignored.