Re: [Qemu-devel] [PATCH] Acceptance tests: host arch to target arch name mapping

[Cleber Rosa]

On 10/17/18 6:15 PM, Eduardo Habkost wrote:
> On Wed, Oct 17, 2018 at 04:59:25PM -0400, Cleber Rosa wrote:
>> On 10/17/18 4:46 PM, Murilo Opsfelder Araujo wrote:
> [...]
>>> Then avocado could multiplex these variants file and call 
>>> ./tests/acceptance/run
>>> for each value of qemu_bin.  `run` script could skip and return if $qemu_bin
>>> doesn't exist.  This approach also allows user forcing a value of qemu_bin 
>>> when
>>> calling `run` manually, for example:
>>>
>>>     ./tests/acceptance/run --qemu_bin=/path/to/your/qemu-system-blah ...
>>>
>>> This ./tests/acceptance/run can serve as an entry point to run all the 
>>> tests.
>>> If more parameters are considered mandatory in the future, the logic can be
>>> placed there.
>>>
>>
>> I'm completely favorable to having extra scripts or make rules that
>> define a standard "job" behavior.  In fact, I think we'll end up having
>> many of those.  "make check-acceptance" is just the first one.
>>
>> But being able to running individual tests should still be possible, and
>> easy IMO.
> 
> Agreed.  But is there anything that requires "running an
> individual test" to be synonymous to "running a test against only
> one QEMU binary"?
> 

I consider, generally speaking:

 * the test: the (parameterizable) code
 * running an individual test: the execution of that code, once, with
one parameter

There may be tests in which the logic of *one test* may require
executing different QEMU binaries, but that is the exception, not the
rule IMO.

Given that the acceptance tests are Avocado tests, I think it'd be a
mistake (or very hard) to try to make "an individual (Avocado) test" use
all the built QEMU binaries.  Variants (one for each unique set of
parameters) is the natural thing to use here.

> We seem to have a terminology problem here: "I'm running one
> individual test" may mean something to an user, but mean
> something completely different to somebody thinking about Avocado
> test cases.  Is this the source of our disagreement?
> 

It could be, I'm certainly biased by the Avocado definition of tests and
my limited understanding what others here may mean when they say "a
test".  For instance, it may be that, to some, "make check" is
considered "a test", and consequently, needs to run with all built QEMU
binaries.

Regards,
- Cleber.