Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 80 minutes

[Thomas Huth]

On 16/11/2021 19.20, Daniel P. Berrangé wrote:
> On Tue, Nov 16, 2021 at 06:36:50PM +0100, Richard Henderson wrote:
> > On 11/16/21 6:22 PM, Thomas Huth wrote:
> > > On 16/11/2021 18.09, Philippe Mathieu-Daudé wrote:
> > > > On 11/16/21 17:49, Daniel P. Berrangé wrote:
> > > > > On Tue, Nov 16, 2021 at 05:33:09PM +0100, Thomas Huth wrote:
> > > > > > The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to
> > > > > > be scheduled, so while the build test itself finishes within 60 minutes,
> > > > > > the total run time of the jobs can be longer due to this waiting time.
> > > > > > Thus let's increase the timeout on the gitlab side a little bit, so
> > > > > > that these jobs are not marked as failing just because of the delay.
> > > ...>>> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > > > > > index e7b25e7427..22d42585e4 100644
> > > > > > --- a/.gitlab-ci.d/cirrus.yml
> > > > > > +++ b/.gitlab-ci.d/cirrus.yml
> > > > > > @@ -14,6 +14,7 @@
> > > > > >      stage: build
> > > > > >      image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > > > > >      needs: []
> > > > > > +  timeout: 80m
> > > > > >      allow_failure: true
> > > > > >      script:
> > > > > >        - source .gitlab-ci.d/cirrus/$NAME.vars
> > > > >
> > > > > Whether 80 or 100 minute, consider it
> > > > >
> > > > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > > >
> > > > This pipeline took 1h51m09s:
> > > > https://gitlab.com/qemu-project/qemu/-/pipelines/409666733/builds
> > > > But Richard restarted unstable jobs, which probably added time
> > > > to the total.
> > > >
> > > > IIRC from a maintainer perspective 1h15 is the upper limit.
> > > > 80m fits, 100m is over.
> > >
> > > I think I agree ... I normally don't want to wait more than a little bit
> > > more than one hour, so 100 minutes feels too long already. We already
> > > have some 70m timeouts in other jobs, and one 80 minute timeout in
> > > .gitlab-ci.d/crossbuild-template.yml, so I'd say 80 minutes are really
> > > the upper boundary that we should use.
> >
> > We are also talking apples and oranges:
> > Gitlab timeouts are on the amount of time the job runs.
> > Cirrus timeouts appear to be on the amount of time the job is queued.
> >
> > If cirrus would just not start accounting until the thing runs we'd be fine.
>
> Unfortunately it isn't that easy. Our cirrus CI jobs are launched using
> the cirrus-run tool, from a gitlab job. The timeouts we're usually
> hitting are from the gitlab job which is sitting around waiting for
> the cirrus job it launched to finish, so it can report back stats.
>
> Cirrus CI does itself have a job timeout, but I'm not aware of us
> hitting that typically, unless i'm misinterpreting something.

Right, the problem is the timeout on the gitlab-CI side, not the timeout on 
the Cirrus-CI side. I've never seen us hitting the timeout on the Cirrus 
side either.

 Thomas