Re: [PATCH v2 0/3] fix build failures from incorrectly skipped container build jobs

[Daniel P . Berrangé]

ping - can we get this series merged so we reduce the chances of
stale container images for people working across multiple branches

On Tue, Feb 16, 2021 at 01:29:51PM +0000, Daniel P. Berrangé wrote:
> This series fixes a problem with our gitlab CI rules that cause
> container builds to be skipped. See the commit description in the
> first patch for the details on this problem.
> 
> The overall result of this series though is a small increase in overall
> pipeline time.
> 
> Previously
> 
>  - When container jobs are skipped: approx 1hr 5 mins
>  - When container jobs are run, cached by docker: approx 1hr 15 mins
>  - When container jobs are run, not cached by docker: approx 1hr 30 mins
> 
> With this series applied the first scenario no longer exists, so
> all piplines are either 1hr 15 or 1hr 30 depending on whether the
> container phase is skipped.
> 
> On the plus side the builds are more reliable as we're actually
> building container images at correct times.
> 
> There is still a race condition though where build jobs can run
> with the wrong containers. This happens if you push two different
> branches to gitlab with different docker file content. If the
> container jobs for the 2nd branch finish before the 1st
> branch runs its build jobs, the 1st branch can end up using
> containers fro the second branch.  The only fix to truely fix
> that would be to stop using "latest" docker tag and always
> use a tag based on the branch name. This would mean we build
> up a growing set of docker images in the gitlab registry.
> 
> At least this series is much more correct that what exists in
> git currently. I'm intending to continue to investigate a solution
> for the remaining race condition, but don't want to block this
> series from merging, since this already solves the problem for the
> majority of contributors' usage.
> 
> Changed in v2:
> 
>  - Set needs rules for cross jobs too
> 
> Daniel P. Berrangé (3):
>   gitlab: always build container images
>   gitlab: add fine grained job deps for all build jobs
>   gitlab: fix inconsistent indentation
> 
>  .gitlab-ci.d/containers.yml  |  7 ----
>  .gitlab-ci.d/crossbuilds.yml | 46 ++++++++++++++++++++++
>  .gitlab-ci.yml               | 74 ++++++++++++++++++++++++++++++++----
>  3 files changed, 112 insertions(+), 15 deletions(-)
> 
> -- 
> 2.29.2
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|