[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-block] Block layer complexity: what to do to keep it under con
From: |
Jeff Cody |
Subject: |
Re: [Qemu-block] Block layer complexity: what to do to keep it under control? |
Date: |
Wed, 29 Nov 2017 01:30:06 -0500 |
User-agent: |
Mutt/1.5.24 (2015-08-30) |
On Wed, Nov 29, 2017 at 11:55:02AM +0800, Fam Zheng wrote:
> Hi all,
>
> As we move forwards with new features in the block layer, the chances of
> tricky
> bugs happening have been increasing alongside - block jobs, coroutines,
> throttling, AioContext, op blockers and image locking combined together make a
> large and complex picture that is hard to fully understand and work with. Some
> bugs we've encountered are quite challenging already. Examples are:
>
> - segfault in parallel blockjobs (iotest 30)
> https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg01144.html
>
> - Intermittent hang of iotest 194 (bdrv_drain_all after non-shared storage
> migration)
> https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg01626.html
>
> - Drainage in bdrv_replace_child_noperm()
> https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg00868.html
>
> - Regression from 2.8: stuck in bdrv_drain()
> https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg02193.html
>
I agree, it seems the complexity is growing by quite a bit.
> So in principle, what should we do to make the block layer easy to understand,
> develop with and debug? I think we have opportunities in these aspects:
>
> - Documentation
>
> There is no central developer doc about block layer, especially how all
> pieces
> fit together. Having one will make it a lot easier for new contributors to
> understand better. Of course, we're facing the old problem: the code is
> moving, maintaining an updated document needs effort.
>
> Idea: add ./doc/deve/block.txt?
>
There are some bits of brilliance in what is already there; for instance,
devel/atomics.txt is very thorough. But I agree that a major piece missing
is an overall design document, that provides the "why" to the "what".
Even given the cost of maintaining a higher level design document, I
think your suggestion here is probably the one that can help mitigate the
complexity the most; the more we (developers) can keep a coherent design
model in mind, the better we are able to do your _other_ suggestions: create
effective tests, simplify code, and enhance debuggability.
> - Tests
>
> Writing tests is a great way not only to exercise code, verify new features
> work as expected and catch regression bugs, but also a way to show how the
> feature can be used. There is this trend that the QEMU user interface
> gradually moves from high level commands and args to small and flexible
> building blocks, therefore demostrating the usage in iotests is meaningful.
>
> Idea: Add tests to simulate how libvirt uses block layer, or how we expect
> it
> to. This would be a long term investment. We could reuse iotests, or create
> a
> new test framework specifically, if it's easier (for example, use docker/vm
> tests that just uses libvirt).
>
> Idea: Patchew already tests the quick group of iotests for a few
> formats/protocols, but we should really add it to "make check".
>
Perhaps higher level testing (like your example of how libvirt uses the
block layer) is a good candidate for avocado?
> - Simplified code, or more orthogonal/modularized architecture.
>
> Each aspect of block layer is complex enough so isolating them as much as
> possible is a reasonable approach to control the complexity. Block jobs and
> throttling becoming block filters is a good example, we should identify
> more.
>
> Idea: rethink event loops. Create coroutines ubiquitously (for example for
> each fd handler, BH and timer), so that many nested aio_poll() can be
> removed.
>
> Crazy idea: move the whole block layer to a vhost process, and implement
> existing features differently, especially in terms of multi-threading (hint:
> rust?).
>
> - Debuggability.
>
> Working with backtraces when coroutine is used is pretty hard, it would be
> nice if ./scripts/qemugdb/coroutine.py could work with core files (i.e.
> without a process to debug), and trace back to co->caller automatically.
>
IIRC, this used to work, right?
> It's always useful to dump block graph. Maybe we should add a helper
> function
> in block layer that dumps all node graphs in graphviz DOT format, and even
> make it available in QMP as x-dump-block-graph?
>
> Of course gdb scripts to dump various lists are also nice little things to
> have.
>
> Idea: write more ./scripts/qemugdb/<scriptlet>.py.
More qemugdb macros would be great, especially for dumping the block chain
and making coroutines less opaque.
-Jeff