Re: Questions about virtio DMA operations

[Peter Maydell]

On Sat, 28 Aug 2021 at 13:03, Arnabjyoti Kalita
<akalita@cs.stonybrook.edu> wrote:
> I am trying to understand how a virtio-block device updates "used" buffers in 
> the virtqueue. I intend to 'mimic' this process without the actual disk event 
> happening, by storing the used buffers and then re-using them later. I can 
> see that the function virtqueue_fill is primarily responsible for filling up 
> the virtqueue which eventually updates the guest memory. vring_used_write 
> eventually does a memcpy to write the used buffer contents to the memory.
>
> I do not understand what is the primary role of the virtqueue_unmap_sg 
> function. I have a few questions.
>
> - What does this function do?

This function calls dma_memory_unmap() to indicate that it has
finished working with the guest memory that it started working
with via a call to dma_memory_map(). (The map calls are in
either virtqueue_map_iovec() or virtqueue_map_desc(), I think.)

These functions boil down to calling address_space_map()
and address_space_unmap(). address_space_map() says "I have
this guest physical address, give me a host address I can
use to work with the guest memory". You have to say at map
time whether you are going to read, or to write (you can't
do both, ie no read-modify-write). When you're done, you must
call address_space_unmap().

In the common case, the guest physical address points at RAM.
In this case address_space_map() always succeeds, and returns
you a host address that points directly at the host memory
which contains that bit of guest RAM. It also takes a reference
to the MemoryRegion for the RAM; this prevents QEMU throwing
away that memory if somebody hot-unplugs it in the middle of
when you were working with it.

address_space_unmap() will release the reference map took.
It also marks the memory as 'dirty' if this was a write
operation. That is important for things like live migration.

If the guest physical address points at something that is
not RAM (ie it is a device), then address_space_map() will
use a bounce buffer: it will perform a series of accesses
to read from the device into a local memory buffer (if this
is a map-for-read), and hand you a pointer to that local buffer.
If it's a map-for-write, then unmap will perform the write
accesses to copy data from the bounce buffer to the device.
(There is only one bounce buffer in the whole system -- so
if multiple devices try to use it at once the second one
will get an error back from the map call.)

> - Does it in any way modify the guest's physical memory?

Only in the "this isn't backed by RAM" corner case above.

> - What happens if unmapping does not happen?

* You leak a reference count to the MemoryRegion, which will
  result in a memory leak if the RAM is hot-unplugged later.
* The memory is not marked as dirty, so if you were in the
  middle of a live migration then the changes to the memory
  might not be correctly copied across to the destination.
* If the access was to a device and was for writes, then if
  you don't call unmap then the writes never happen.

Put another way:
QEMU provides two different ways to do reads and writes
to guest physical addresses:
 (1) you can use functions like address_space_read()
     and address_space_write() that do accesses
     "one at a time", ie each access must be done via
     a function call that takes the guest physical address
 (2) you can use a sequence of:
      - address_space_map() to get a host pointer
      - perform direct operations on the memory pointed
        at by that host pointer
      - address_space_unmap()
The virtio code chooses approach 2.

(Throughout this explanation I have simplified by just talking
about 'guest physical addresses'. Guests can have multiple
address spaces, so these functions tend to take both an
AddressSpace and an address with in that AS.)

-- PMM