Re: [Qemu-devel] [PATCH v2 02/20] Acceptance tests: show avocado test execution by default

[Cornelia Huck]

On Wed, 6 Feb 2019 12:02:55 -0500
Cleber Rosa <address@hidden> wrote:

> On 2/6/19 9:36 AM, Cornelia Huck wrote:
> > On Fri,  1 Feb 2019 19:55:52 -0500
> > Cleber Rosa <address@hidden> wrote:
> >   
> >> The current version of the "check-acceptance" target will only show
> >> one line for execution of all tests.  That's probably OK if the tests
> >> to be run are quick enough and they're always the same.
> >>
> >> But, there's already one test alone that takes on average ~5 seconds
> >> to run, we intend to adapt the list of tests to match the user's build
> >> environment (among other choices).  
> > 
> > Btw: What are our expectations regarding execution time for tests?
> > Especially if we continue adding tests, and architecture-specific tests
> > are bound to be slower if run on a foreign architecture via tcg.
> > 
> > Would a make check-acceptance-quick command make sense? ("I only want
> > to verify quickly that I didn't break too much, so run the quicker
> > tests only, probably only for my host architecture")
> >   
> 
> Yes, it definitely makes sense.  Now, let me know if the following also
> makes sense to you:
> 
> 1) Because these tests focus on functional testing, the default
> target/shortcut ("make check-acceptance") should run the complete set of
> test cases (including the slow ones).

Nod.

> 
> 2) Requirements vary greatly from user to user, to while adding a
> "check-acceptance-quick" is fine, you just mentioned one extra test
> execution variation ("for my host architecture").  For those, the idea
> is that:
> 
>  a) "make check-acceptance[-quick]" will adapt to the build environment
> (if you only built s390x targets, that's all it's going to use)

Yes. However, I usually build some extra targets (just to verify that
files that e.g. include headers I modify don't break for other
environments), but don't necessarily want to run all acceptance tests
for them (as I don't expect functional changes for them). That's
probably something I only want to do when I run the full set anyway.

> 
>  b) "avocado" command line interface *should* be easy enough to fulfill
> other requirements, and not necessarily require a "make" target.  For
> instance, if you're only interested in your host arch and one specific
> machine type, a command line such as the following should do the trick:
> 
>   $ make check-venv
>   $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t
> machine:WHAT_I_CARE_ABOUT tests/acceptance/
> 
> How does that sound?

Yes, that looks quite usable (I can easily define aliases locally for
shortcuts) and solves the case I mentioned above.

So, to summarize:
- add 'make check-acceptance-quick' that excludes tests marked as 'slow'
- use the avocado command line interface to further narrow down
  architectures and machines, if wanted
?