[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-block] [PATCH 36/40] aio: update locking documentation
[Qemu-block] [PATCH 36/40] aio: update locking documentation
Tue, 24 Nov 2015 19:01:27 +0100
Signed-off-by: Paolo Bonzini <address@hidden>
docs/multiple-iothreads.txt | 65 +++++++++++++++++++++++++++++++++++++++------
1 file changed, 57 insertions(+), 8 deletions(-)
diff --git a/docs/multiple-iothreads.txt b/docs/multiple-iothreads.txt
index 4197f62..729c64b 100644
@@ -47,8 +47,11 @@ How to program for IOThreads
The main difference between legacy code and new code that can run in an
IOThread is dealing explicitly with the event loop object, AioContext
(see include/block/aio.h). Code that only works in the main loop
-implicitly uses the main loop's AioContext. Code that supports running
-in IOThreads must be aware of its AioContext.
+implicitly uses the main loop's AioContext, and does not need to handle
+locking because it implicitly uses the QEMU global mutex.
+Code that supports running in IOThreads, instead, must be aware of its
+AioContext and of synchronization.
AioContext supports the following services:
* File descriptor monitoring (read/write/error on POSIX hosts)
@@ -79,6 +82,11 @@ iothread_get_aio_context() or for the main loop using
Code that takes an AioContext argument works both in IOThreads or the main
loop, depending on which AioContext instance the caller passes in.
+When running in an IOThread, handlers for bottom halves, file descriptors,
+and timers have to handle their own synchronization. Block layer AIO
+callbacks also have to handle synchronization. How to do this is the
+topic of the next sections.
How to synchronize with an IOThread
AioContext is not thread-safe so some rules must be followed when using file
@@ -87,13 +95,15 @@ descriptors, event notifiers, timers, or BHs across threads:
1. AioContext functions can always be called safely. They handle their
own locking internally.
-2. Other threads wishing to access the AioContext must use
-aio_context_acquire()/aio_context_release() for mutual exclusion. Once the
-context is acquired no other thread can access it or run event loop iterations
-in this AioContext.
+2. Other threads wishing to access block devices on an IOThread's AioContext
+must use a mutex; the block layer does not attempt to provide mutual
+exclusion of any kind. aio_context_acquire() and aio_context_release()
+are the typical choice; aio_context_acquire()/aio_context_release() calls
+may be nested.
-aio_context_acquire()/aio_context_release() calls may be nested. This
-means you can call them if you're not sure whether #1 applies.
+#2 typically includes bottom half handlers, file descriptor handlers,
+timer handlers and AIO callbacks. All of these typically acquire and
+release the AioContext.
There is currently no lock ordering rule if a thread needs to acquire multiple
AioContexts simultaneously. Therefore, it is only safe for code holding the
@@ -130,9 +140,48 @@ aio_disable_clients() before calling bdrv_drain(). You
can then reenable
guest requests with aio_enable_clients() before finally releasing the
AioContext and completing the monitor command.
+AioContext and coroutines
Long-running jobs (usually in the form of coroutines) are best scheduled in the
BlockDriverState's AioContext to avoid the need to acquire/release around each
bdrv_*() call. Be aware that there is currently no mechanism to get notified
when bdrv_set_aio_context() moves this BlockDriverState to a different
AioContext (see bdrv_detach_aio_context()/bdrv_attach_aio_context()), so you
may need to add this if you want to support long-running jobs.
+Usually, the coroutine is created and first entered in the main loop. After
+some time it will yield and, when entered again, it will run in the
+IOThread. In general, however, the coroutine never has to worry about
+releasing and acquiring the AioContext. The release will happen in the
+function that entered the coroutine; the acquire will happen (e.g. in
+a bottom half or AIO callback) before the coroutine is entered. For
+1) the coroutine yields because it waits on I/O (e.g. during a call to
+bdrv_co_readv) or sleeps for a determinate period of time (e.g. with
+co_aio_sleep_ns). In this case, functions such as bdrv_co_io_em_complete
+or co_sleep_cb take the AioContext lock before re-entering.
+2) the coroutine explicitly yields. In this case, the code implementing
+the long-running job will have a bottom half, AIO callback or similar that
+should acquire the AioContext before re-entering the coroutine.
+3) the coroutine yields through a CoQueue (directly or indirectly, e.g.
+through a CoMutex or CoRwLock). This also "just works", but the mechanism
+is more subtle. Suppose coroutine C1 has just released a mutex and C2
+was waiting on it. C2 is woken up at C1's next yield point, or just
+before C1 terminates; C2 then runs *before qemu_coroutine_enter(C1)
+returns to its caller*. Assuming that C1 was entered with a sequence like
+ qemu_coroutine_enter(co); // enters C1
+... then C2 will also be protected by the AioContext lock.
+Bullet 3 provides the following rule for using coroutine mutual exclusion
+in multiple threads:
+ *** If two coroutines can exclude each other through a CoMutex or
+ *** CoRwLock or CoQueue, they should all run under the same AioContext.
- [Qemu-block] [PATCH 27/40] aio: document locking, (continued)
- [Qemu-block] [PATCH 27/40] aio: document locking, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 25/40] aio-posix: remove walking_handlers, protecting AioHandler list with list_lock, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 29/40] quorum: use atomics for rewrite_count, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 30/40] quorum: split quorum_fifo_aio_cb from quorum_aio_cb, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 28/40] aio: push aio_context_acquire/release down to dispatching, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 31/40] qed: introduce qed_aio_start_io and qed_aio_next_io_cb, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 34/40] block: explicitly acquire aiocontext in timers that need it, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 33/40] block: explicitly acquire aiocontext in bottom halves that need it, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 32/40] block: explicitly acquire aiocontext in callbacks that need it, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 37/40] async: optimize aio_bh_poll, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 36/40] aio: update locking documentation,
Paolo Bonzini <=
- [Qemu-block] [PATCH 35/40] block: explicitly acquire aiocontext in aio callbacks that need it, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 38/40] aio-posix: partially inline aio_dispatch into aio_poll, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 40/40] dma-helpers: avoid lock inversion with AioContext, Paolo Bonzini, 2015/11/24
- [Qemu-block] [PATCH 39/40] async: remove unnecessary inc/dec pairs, Paolo Bonzini, 2015/11/24
- Re: [Qemu-block] [Qemu-devel] [RFC PATCH 00/40] Sneak peek of virtio and dataplane changes for 2.6, Christian Borntraeger, 2015/11/26