Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Cornelia Huck]

On Thu, 26 Apr 2018 16:25:40 +0200
Thomas Huth <address@hidden> wrote:

> On 26.04.2018 16:22, Eduardo Habkost wrote:
> > On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:  
> >> On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:  
> >>> On 26.04.2018 15:57, Eduardo Habkost wrote:  
> >>>> (Starting a new thread, for more visibility)
> >>>>
> >>>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> >>>> devices with all machines, not only with "none")
> >>>>
> >>>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:  
> >>> [...]  
> >>>>> I don't mind having make check SPEED=slow run more extensive tests.
> >>>>> Assuming we actually run them at least once in a while, which seems
> >>>>> doubtful.  
> >>>>
> >>>> We probably don't do that, but we really must be running a more
> >>>> extensive (and slower) test set at least once before every
> >>>> release.
> >>>>
> >>>> Maybe some people are running SPEED=slow tests, or even more
> >>>> extensive test suites like avocado-vt once in a while, but we
> >>>> need to know who is running them, and when.  
> >>>
> >>> At least I am running "make check SPEED=slow" manually from time to
> >>> time, especially when we enter the hard freeze period.  
> >>
> >> Hmm, we could get this done by travis. It has the concept of "cron jobs"
> >> for scheduling builds separately from pushes.
> >>
> >> So we could keep the current travis jobs unchanged, but then add an
> >> use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
> >> so we can get SPEED=slow run once a day. Just have to be careful which
> >> jobs we make slow so we don't hit the 50 minute timeout.  
> > 
> > I expect SPEED=slow tests to eventually take longer than 50
> > minutes, but we can try this and see what happens.  
> 
> I already had to move some of the slow tests to SPEED=slow mode in the
> past due to the travis timeouts (which are hit when the machines are
> very loaded), so my guess is that SPEED=slow will hardly work with travis.

Do we have another automation infrastructure for that, preferably with
multiple host architectures? I usually run SPEED=slow before I send a
pull request, on s390x and x86 -- unless I forget it. (I really should
script my procedure. Some time.)