Re: [Qemu-devel] [PATCH v5 0/3] Add common QEMU control functionality to qemu-iotests

[Stefan Hajnoczi]

On Mon, May 05, 2014 at 11:44:55AM -0400, Jeff Cody wrote:
> On Mon, May 05, 2014 at 05:32:09PM +0200, Kevin Wolf wrote:
> > Am 05.05.2014 um 17:21 hat Stefan Hajnoczi geschrieben:
> > > On Wed, Apr 30, 2014 at 10:55:07AM -0400, Jeff Cody wrote:
> > > > This adds some common functionality to control QEMU for qemu-iotests.
> > > > 
> > > > Additionally, test 085 is updated to use this new functionality.
> > > > 
> > > > Some minor fixups along the way, to clear up spaced pathname issues, 
> > > > for common.rc, test 019, and test 086.
> > > > 
> > > > 
> > > > Jeff Cody (3):
> > > >   block: qemu-iotests - add common.qemu, for bash-controlled qemu tests
> > > 
> > > Once a test launches QEMU, it soon needs to parse QMP commands or wait
> > > for QMP events.  That doesn't lend itself to the traditional
> > > qemu-iotests shell model.  That is why iotests.py exists.
> > > 
> > > Shell script is a poor language for test cases that go beyond
> > > pre-defined commands whose output is saved for diffing.  The string
> > > manipulation is clumsy, JSON is not supported, tricks with fifos can
> > > easily deadlock or break when a process terminates unexpectedly, etc.
> > > 
> > > If we go further in the direction of this patch series, we'll duplicate
> > > existing iotests.py code and have complex shell tests that are hard to
> > > extend.  I think it's time to draw the line and convert any test cases
> > > that need to complexity to Python.
> > > 
> > > Why not use iotests.py?
> > 
> > Because it's hard to use. The "compare against reference output" thing
> > is the first thing that you lose with iotests.py, and it's the most
> > useful feature in qemu-iotests.
> > 
> > When a Python test case fails, you get into real debugging. When a shell
> > script test case fails, you usually see immediately from the reference
> > output diff what's wrong.
> > 
> > I accept iotest.py for anything that needs to evaluate QMP return
> > values, reluctantly, because we have nothing better. But that's it, I
> > don't actually _like_ using it.
> >
> 
> I agree with Kevin.  
> 
> Also, for a test framework, having some language duplication is not
> necessarily a *bad* thing.  One, functionality may be tested in
> slightly different ways, due to architectural differences.  Second, it
> gives developers increased flexibility in writing test cases in a
> language they may be more comfortable in - and that hopefully leads to
> an increased number of tests being written.

It's duplicated effort for the test case authors.  It also requires that
people debugging broken tests switch between different frameworks
depending on the test case that's failing.  We already have two types of
test case (simple and QEMU interaction), now a third is being added that
duplicates QEMU interaction.

Regarding the language, you can get comfortable with Python in an
afternoon.  If you already know C++, Java, Perl, or Ruby then it's very
easy to pick up:
https://docs.python.org/2.7/tutorial/index.html

> And if you look at the actual tests (091 and 085, in this case), the
> framework is pretty simple to use. If you run into something
> unsupported (json parsing, etc.), then you can use iotests.py.

You can already do this stuff in iotests.py but you still wrote it in
shell :).  So I'm not sure I buy that.

Stefan