Re: [Qemu-block] [Qemu-devel] [PATCH 0/3] build configuration query tool and conditional (qemu-io)test skip

[Daniel P. Berrange]

On Tue, Jul 25, 2017 at 12:16:13PM -0400, Cleber Rosa wrote:
> 
> 
> On 07/25/2017 11:49 AM, Stefan Hajnoczi wrote:
> > On Fri, Jul 21, 2017 at 10:21:24AM -0400, Cleber Rosa wrote:
> >> On 07/21/2017 10:01 AM, Daniel P. Berrange wrote:
> >>> On Fri, Jul 21, 2017 at 01:33:25PM +0100, Stefan Hajnoczi wrote:
> >>>> On Thu, Jul 20, 2017 at 11:47:27PM -0400, Cleber Rosa wrote:
> >> Without the static capabilities defined, the dynamic check would be
> >> influenced by the run time environment.  It would really mean "qemu-io
> >> running on this environment (filesystem?) can do native aio".  Again,
> >> that's not the best type of information to depend on when writing tests.
> > 
> > Can you explain this more?
> > 
> > It seems logical to me that if qemu-io in this environment cannot do
> > aio=native then we must skip those tests.
> > 
> > Stefan
> > 
> 
> OK, let's abstract a bit more.  Let's take this part of your statement:
> 
>  "if qemu-io in this environment cannot do aio=native"
> 
> Let's call that a feature check.  Depending on how the *feature check*
> is written, a negative result may hide a test failure, because it would
> now be skipped.
> 
> Suppose that a feature check for "SDL display" is such that you run
> "qemu -display sdl".  A *feature failure* here (SDL init is broken), or
> an environment issue (DISPLAY=), will cause a SDL test skip.

You could have a way to statically define what features any given run
of the test suite should enable, then report failure if they were not
detected.

This is a similar situation to that seen with configure scripts. If invoked
with no --enable-xxx flags, it will probe for features & enable them if
found.  This means you can accidentally build without expected features if
you have a missing -devel package, or a header/library is broken in some
way. This is why configure prints a summary of which features it actually
found. It is also why when building binary packages like RPMs, it is common
to explicitly give --enable-xxx flags for all features you expect to see.
Automatic enablement is none the less still useful for people in general.

So if we applied this kind of approach for testing, then any automated
test systems at least, ought to provide a fixed list of features they
expect to be present for tests. So if any features accidentally broke
the tests would error.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|