qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH for-2.6?] qemu-iotests: iotests: fail hard if no


From: Sascha Silbe
Subject: Re: [Qemu-devel] [PATCH for-2.6?] qemu-iotests: iotests: fail hard if not run via "check"
Date: Tue, 19 Apr 2016 13:59:15 +0200
User-agent: Notmuch/0.19+1~g6b3e223 (http://notmuchmail.org) Emacs/24.3.1 (x86_64-pc-linux-gnu)

Dear Markus,

Markus Armbruster <address@hidden> writes:

>> Running an iotests-based Python test directly might appear to work,
>> but may fail in subtle ways and is insecure:
>>
>> - It creates files with predictable file names in a world-writable
>>   location (/var/tmp).
>>
>> - Tests expect the environment to be set up by check. E.g. 041 and 055
>>   may take the wrong code paths if QEMU_DEFAULT_MACHINE is not
>>   set. This can lead to false negatives.
>>
>> Instead fail hard and tell the user we want to be run via "check".
>
> Are the problems serious enough to warrant rejecting the attempt to run
> the tests directly outright?
>
> Judging from your description, I'm inclined to "no".

Having tests do unexpected things and / or fail silently is "serious" in
my book. After all, what's the value of tests if they don't yell when
something is broken?


[tests/qemu-iotests/iotests.py]
[...]
>> +    if test_dir is None or qemu_default_machine is None:
>> +        sys.stderr.write('Please run this test via ./check\n')
>> +        sys.exit(os.EX_USAGE)
>> +
>
> Aha, you're not actually rejecting the attempt to run the tests
> directly, you're merely requiring two environment variables.  That's
> okay with me. [...]

I should probably add a comment that these two environment variables are
merely used as proxies that indicate we haven't been run via
"check". They are the ones where there isn't any useful default value we
could fall back, but without a full audit we can't tell what else some
of the tests may expect that would normally be set up by "check" (or at
least I can't).

Sascha
-- 
Softwareentwicklung Sascha Silbe, Niederhofenstra├če 5/1, 71229 Leonberg
https://se-silbe.de/
USt-IdNr. DE281696641




reply via email to

[Prev in Thread] Current Thread [Next in Thread]