Re: [Qemu-devel] Please read: make check framework

[Anthony Liguori]

On 01/09/2012 05:22 PM, Andreas Färber wrote:
> Am 09.01.2012 22:42, schrieb Anthony Liguori:
> > On 01/09/2012 02:57 PM, Andreas Färber wrote:
> > > Am 09.01.2012 20:28, schrieb Anthony Liguori:
> > > > We're still short on infrastructure right now so I expect that a lot of
> > > > stuff will get merged without tests at first.  As we get more test
> > > > infrastructure like qtest and qemu-test, I expect that to change.
> > > >
> > > > In terms of total coverage for any given feature, I think ultimately the
> > > > goal is to move the responsibility for regression testing from the
> > > > contributors of new features to the subsystems themselves.
> > > >
> > > > IOW, after we get going with our test infrastructure, when we do big
> > > > sweeping changes like the memory API or QOM, I will have much less
> > > > sympathy for breakages that are caused by subsystems with an incomplete
> > > > test suite.
> > > >
> > > > So I think the most important thing for maintainers to start thinking
> > > > about is how they can add the necessary infrastructure for testing their
> > > > subsystems.
> > >
> > > Well, I already tried pointing out that qemu-test in the way previously
> > > proposed does not help catching the recent regressions for PReP.
> >
> > Actually, qemu-test did catch it.  The only reason I didn't see it is
> > that I didn't add qemu-system-powerpc to my regression script that I'm
> > using to drive it.  But simply booting up a Linux guest would have
> > caught it.
>
> Booting up a Linux kernel yes, but not the latest-and-greatest like you
> proposed. Meaning, we need to be able to reference different branches or
> trees instead of a single submodule (cf. qemu-test thread).

I don't understand what you mean.  Given the set of submodules currently used, 
I'm not aware of any target that has issues.  You asserted that it may not be 
the case that a single version of Linux, busybox, uclibc would all work on all 
targets.  But so far, I don't see any evidence to suggest that this isn't, in 
fact, possible.

> > If there's some piece of QEMU you care about, start writing tests and
> > tie it into make check.  It's that simple :-)
>
> Not quite. It would be easier if we just set up some storage space with
> images like the handful of *-test images on qemu.org.

It's not that simple.  We cannot just host an image on qemu.org without getting 
into GPL considerations.  Even beyond the GPL, I don't think having opaque 
images that people don't understand how to recreate if they need to is a good 
strategy.

I've considered trying to do something based on automated installation (using 
kickstart/preseed).  This has limitations though.  It takes an extremely long 
time to generate these having the necessary hooks with preseeds appears to be 
very difficult.

> That way we can
> supply working images for all targets, you can supply build scripts to
> rebuild them and we can all be done quickly and have fun.

That's what qemu-jeos is.

> I was just saying, stop bike-shedding about how we're supposed to deny
> patches without test cases from tomorrow on,

I never said that.

> and instead let's get the
> test infrastructure in place and see how well we get along with it.
> That's even simpler and much more convenient.

That's what I'm proposing.  Specifically:

"I'm going to apply this series quickly and will start running 'make 
check-quick' as part a sniff test before pushing patches.

I'd like to request that all maintainers/submaintainers do the same and that 
everyone contributes unit tests to this target."

> You're building up quite a lot of time pressure on other people lately,
> claiming two days for review were long already, wanting to merge QOM and

The latest QOM series has been sitting on the list for a few weeks and I'm still 
waiting to give people time to review.  I think it's hard to claim I'm rushing it.

> now make check ASAP - despite all inchoate.

Yes, I am going to rush through make check but there's nothing new or exciting 
here.

> Me, I've reviewed most of this series in what I believe is a
> constructive way;

Yes, and I appreciate that :-)

> I'd love to build test cases for my code based on
> qtest - you've made this series a prerequisite, so getting libqtest for
> writing test cases based on it is going to take some more time until we
> can expect contributors to contribute test cases.

The reason I sent this out before qtest (which is my real goal) is based on past 
experience.  We have had two unit test frameworks in use, libcheck and gtester. 
 I'd wager that most developers haven't used either.  The libcheck tests spent 
a while with a small break too that no one noticed.

I don't want to introduce yet another test framework without having a 
simple-to-consume mechanism for our existing test suites while also unifying 
what we already have into a single mechanism.

> If however you're trying to push the work of writing tons of test cases
> for legacy code - be it qtest or qemu-test or a qemu-test alternative -
> upfront to submaintainers, like you seemed to with this thread, then
> you're on the wrong track IMO.

I think you mean 'existing code', not legacy code.  I don't consider all of the 
code we have today as legacy.

> KVM on x86_64 has the biggest user base
> in terms of people testing and potentially contributing test cases; this
> imbalance of manpower is not going to be remedied by any unit test
> frameworks we introduce.

Actually, I think it will.  Why do non-x86 architectures break so often? 
Because they aren't tested.  Why aren't they tested?  Because it's extremely 
difficult.

AFAIK, the majority of folks that do test non-x86 architectures do so by running 
images manually.  Most of them use Aurelien's debian images.

What I'm trying to get at with make check/qemu-test is a simple way that we can 
add test cases for less commonly used features that makes it very easy to test 
these things.  That's the only way we'll stop breaking other targets.

Don't view this as, "here is extra work you have to do to get your feature in", 
but rather as, "here is an opportunity to make sure that no one else does 
something silly and breaks your feature, forcing you to spend your time tracking 
it down."

Regards,

Anthony Liguori

> Regards,
> Andreas