Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 05:14 AM, Kevin Wolf wrote:
> Am 09.03.2012 00:51, schrieb Ademar Reis:
> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
> > > > Plus it's not unconditional: the test runner will report tests
> > > > SKIPPED if a dependency is not present.
> > >
> > > But then the tests aren't run so if most developers didn't have it
> > > installed, and most tests were written with it, most developers
> > > wouldn't be running most tests which defeats the purpose, no?
> >
> > Part of a TDD approach is to have build and test bots frequently
> > running tests on multiple platforms with different
> > configurations.
> >
> > You can't expect developers to run all tests all the time.
>
> I think this is one of the most important points: Not all developers
> must run all the tests all the time.
>
> Actually, Anthony agreed with me when I said that developers should run
> some sanity tests for all of qemu and maybe a few more tests for the
> subsystems they're touching.

And a small number of randomly chosen test cases.

We don't want to have test cases that never run under normal circumstances or 
else they're prone to break.  That's why I've talked a lot about keeping 'make 
check' time bound.

> I agree that it would be bad to have a
> autotest dependency for those basic tests that everyone should run and
> that should take a few minutes at most.
>
> For the rest of test cases, however, not everyone needs to run them and
> I think an external dependency (that is reasonably easy to satisfy) is
> not a problem there.

I'd prefer to avoid external dependencies as it just encourages people not to 
test.  There may be exceptions for certain types of tests, but it should be an 
exception, not the rule.

> A test bot would be great, but even if people just
> run them occasionally by hand, that would already detect bugs that are
> currently left in the code for months. If maintainers do it before
> pushing code into master, you'll even catch everything before it goes
> into master. This is as good as it gets.

We'll integrate make check into buildbot which currently does look at 
submaintainer trees.

Regards,

Anthony Liguori

> The important thing is that tests exist in the first place, not who runs
> them.
>
> Kevin