Re: [Qemu-devel] Improving QMP test coverage

[Daniel P. Berrange]

On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote:
> On Thu, 07/27 10:14, Markus Armbruster wrote:
> > This brings some advantages of "verify output with diff" to tests that
> > verify with code.  Improvement if it simplifies the verification code.
> > 
> > I'd still prefer *no* verification code (by delegating the job to diff)
> > for tests where I can get away wit it.
> 
> Python based iotests can be (re)done in such a way that they print actual logs
> (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the
> current dot dot dot summary, then we automatically have diff based 
> verification,
> no?

The python test 149 that I wrote does exactly that. There's no reason why
the others couldn't do the same.

> One thing I feel painful with bash iotests is how harder it is to write
> complicated test scenarios such as migration, incremental backup, etc.

Yes, shell is an awful language if you need non-trivial control
logic or data structures

> On the other hand the iotests are more difficult to debug when things go wrong
> because it eats the output which, if done with shell, should be very easy to
> get.

Even if the python tests are not doing verify-by-diff, it should be fairly
easy to wire up an env variable IOTESTS_DEBUG=1 which would force them
to propagate stdout/err of all commands run (or at least save it to a
log file somewhere).

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|