Re: [Qemu-devel] Block I/O Tracing

[Stefan Hajnoczi]

On Wed, Dec 16, 2015 at 12:37:39PM -0500, Luis Pabón wrote:
>   I am really interested in the following feature: 
> http://wiki.qemu.org/Features/Block/Todo#Trace_guest_block_I.2FO.2C_replay_with_qemu-io
> .  Is there any more information about this feature?

Hi Luis,
It's a useful idea but non-trivial to implement, I think.

Often times when an I/O performance problem is reported:

1. Reproducing the poor performance is difficult due to the requirements
   of setting up the same VM and guest application on comparable storage
   hardware.  Usually developers ask the bug reporter questions until
   the issue is narrowed down to essentially a microbenchmark that can
   easily be reproduced.

   This is really a funnel: only a fraction of cases makes it from the
   original bug report to a solution because the bug reporter or
   developer might give up during the back-and-forth stage of asking
   questions to narrow down the problem.

2. File system, volume manager, and operating system block layer
   developers need straightforward reproducible test cases that do not
   involve setting up a VM.  It's hard to get a file system developer's
   attention if you ask them to set up a VM in the hopes of spotting a
   file system bug.

3. QEMU developers may implement performance optimizations which need to
   be evaluated against real-world I/O patterns.  Again, if it takes too
   much effort setting up various VMs to benchmark against then less
   performance evaluation will end up being done.

So the idea was to record I/O patterns from a real workload and then
share them so others can replay them with qemu-io or fio (without
running a VM).

This is appealing because qemu-io and fio are small userspace programs
that do not run a VM.  They are much easier to run and analyze than
QEMU.

Unfortunately there are a few problems with the approach of replaying
recorded I/O patterns:

1. Eliminating the VM also removes the storage controller emulation
   (e.g. virtio-blk) and its impact on performance.  Some bottlenecks
   are related to the storage controller but they wouldn't be
   reproduced.

2. Eliminating the guest application that generates I/O makes it hard to
   replay the "think time" and dependencies between I/O requests.  For
   example, a database application may submit a read request and a write
   request at the same time.  When the write request completes it does
   some additional processing (the "think time" or delay before the next
   I/O request) and then submits a flush request.

   If you have a recorded I/O pattern it does not include the
   information that the flush request must happen n microseconds after
   the write request completes.  So when you replay the I/O pattern the
   behavior will not be the same as the real application.

Anyway, it's an interesting problem and I'd start with blktrace and fio.
Then look at how to integrate that sort of trace with QEMU.

Stefan

signature.asc
Description: PGP signature