Re: "make check-acceptance" takes way too long

[Cleber Rosa]

On Sat, Jul 31, 2021 at 2:40 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 31/07/2021 00.04, Cleber Rosa wrote:
> > On Fri, Jul 30, 2021 at 11:43 AM Peter Maydell <peter.maydell@linaro.org> 
> > wrote:
> >>
> >> On Fri, 30 Jul 2021 at 16:12, Peter Maydell <peter.maydell@linaro.org> 
> >> wrote:
> >>>
> >>> "make check-acceptance" takes way way too long. I just did a run
> >>> on an arm-and-aarch64-targets-only debug build and it took over
> >>> half an hour, and this despite it skipping or cancelling 26 out
> >>> of 58 tests!
> >>>
> >>> I think that ~10 minutes runtime is reasonable. 30 is not;
> >>> ideally no individual test would take more than a minute or so.
> >>
> >> Side note, can check-acceptance run multiple tests in parallel?
> >
> > Yes, it can, but it's not currently enabled to do so, but I'm planning
> > to.  As a matter of fact, Yesterday I was trying out Avocado's
> > parallel capable runner on a GitLab CI pipeline[1] and it went well.
>
> Was this one of the shared gitlab CI runners? ... well, those feature only a
> single CPU, so the run was likely not very different compared to a single run.
>

Yes, the two pipeline executions I referred to were run in the shared
GitLab CI runners.  I was testing two things:

1. Possible caveats/issues with the parallel Avocado runner (AKA
"nrunner") and the Acceptance tests (first pipeline linked, with "max
parallel tasks" set to 1)
2. Any possible gains/losses with running with "max parallel tasks"
set to 2 (second pipeline linked)

> > But the environment on GitLab CI is fluid, and I bet there's already
> > some level of overcommit of (at least) CPUs there.  The only pipeline
> > I ran there with tests running in parallel, resulted in some jobs with
> > improvements, and others with regressions in runtime.  Additionally,
> > lack of adequate resources can make more tests time out, and thus give
> > out false negatives.
>
> It certainly does not make sense to enable parallel tests for the shared
> runners there.
>
>   Thomas
>
>

There could be gains on scenario #2 if there's considerable I/O wait
on some tests.  That's why I mention that previous experiences mixing
the acceptance tests with the iotests were very interesting.  But
you're right, with only acceptance tests, mostly CPU bound, there was
no clear gain.

Best,
- Cleber.