[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-block] [Qemu-devel] [PATCH 0/3] build configuration query tool
From: |
Markus Armbruster |
Subject: |
Re: [Qemu-block] [Qemu-devel] [PATCH 0/3] build configuration query tool and conditional (qemu-io)test skip |
Date: |
Tue, 08 Aug 2017 16:52:25 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) |
Stefan Hajnoczi <address@hidden> writes:
> On Tue, Aug 08, 2017 at 10:06:04AM +0200, Markus Armbruster wrote:
>> Stefan Hajnoczi <address@hidden> writes:
>>
>> > On Wed, Jul 26, 2017 at 02:24:02PM -0400, Cleber Rosa wrote:
>> >>
>> >>
>> >> On 07/26/2017 01:58 PM, Stefan Hajnoczi wrote:
>> >> > On Tue, Jul 25, 2017 at 12:16:13PM -0400, Cleber Rosa wrote:
>> >> >> On 07/25/2017 11:49 AM, Stefan Hajnoczi wrote:
>> >> >>> On Fri, Jul 21, 2017 at 10:21:24AM -0400, Cleber Rosa wrote:
>> >> >>>> On 07/21/2017 10:01 AM, Daniel P. Berrange wrote:
>> >> >>>>> On Fri, Jul 21, 2017 at 01:33:25PM +0100, Stefan Hajnoczi wrote:
>> >> >>>>>> On Thu, Jul 20, 2017 at 11:47:27PM -0400, Cleber Rosa wrote:
>> >> >>>> Without the static capabilities defined, the dynamic check would be
>> >> >>>> influenced by the run time environment. It would really mean
>> >> >>>> "qemu-io
>> >> >>>> running on this environment (filesystem?) can do native aio". Again,
>> >> >>>> that's not the best type of information to depend on when writing
>> >> >>>> tests.
>> >> >>>
>> >> >>> Can you explain this more?
>> >> >>>
>> >> >>> It seems logical to me that if qemu-io in this environment cannot do
>> >> >>> aio=native then we must skip those tests.
>> >> >>>
>> >> >>> Stefan
>> >> >>>
>> >> >>
>> >> >> OK, let's abstract a bit more. Let's take this part of your statement:
>> >> >>
>> >> >> "if qemu-io in this environment cannot do aio=native"
>> >> >>
>> >> >> Let's call that a feature check. Depending on how the *feature check*
>> >> >> is written, a negative result may hide a test failure, because it would
>> >> >> now be skipped.
>> >> >
>> >> > You are saying a pass->skip transition can hide a failure but ./check
>> >> > tracks skipped tests. See tests/qemu-iotests/check.log for a
>> >> > pass/fail/skip history.
>> >> >
>> >>
>> >> You're not focusing on the problem here. The problem is that a test
>> >> that *was not* supposed to be skipped, would be skipped.
>> >
>> > As Daniel Berrange mentioned, ./configure has the same problem. You
>> > cannot just run it blindly because it silently disables features.
>> >
>> > What I'm saying is that in addition to watching ./configure closely, you
>> > also need to look at the skipped tests that ./check reports. If you do
>> > that then you can be sure the expected set of tests is passing.
>> >
>> >> > It is the job of the CI system to flag up pass/fail/skip transitions.
>> >> > You're no worse off using feature tests.
>> >> >
>> >> > Stefan
>> >> >
>> >>
>> >> What I'm trying to help us achieve here is a reliable and predictable
>> >> way for the same test job execution to be comparable across
>> >> environments. From the individual developer workstation, CI, QA etc.
>> >
>> > 1. Use ./configure --enable-foo options for all desired features.
>> > 2. Run the ./check command-line and there should be no unexpected skips
>> > like this:
>> >
>> > 087 [not run] missing aio=native support
>> >
>> > To me this seems to address the problem.
>> >
>> > I have mentioned the issues with the build flags solution: it creates a
>> > dependency on the build environment and forces test feature checks to
>> > duplicate build dependency logic. This is why I think feature tests are
>> > a cleaner solution.
>>
>> I suspect the actual problem here is that the qemu-iotests harness is
>> not integrated in the build process. For other tests, we specify the
>> tests to run in a Makefile, and use the same configuration mechanism as
>> for building stuff conditionally.
>
> The ability to run tests against QEMU binaries without a build
> environment is useful though. It would still be possible to symlink to
> external binaries but then the build feature information could be
> incorrect.
I don't dispute it's useful. "make check" doesn't do it, though.
I think we can either have a standalone test suite (introspects the
binaries under test to figure out what to test), or an integrated test
suite (tests exactly what is configured). "make check" is the latter.
qemu-iotests is kind-of-sort-of the former.