Re: Hundreds of failed build on master following git package update

[Sarah Morgensen]

Hello,

brice@waegenei.re writes:

> Hello Guix,
>
> Today I pushed 2 commits¹ related to the package to master, mainly to update 
> it to 2.33.0. But since then https://data.guix.gnu.org/ went offline and 
> Cuirass' evaluation #4930² has 556 failed build. Newly failed builds have 
> unhelpful logs, ending with "cannot build missing derivation [...]"³.
>
> Before pushing those commits, I build git and git-minimal successfully on 
> x86_64 and assessed the total impacted package for this update to be inferior 
> to 300. Did I do something wrong with the update commit? Or is it an 
> unrelated issue?
>
> I'm currently locally building some packages which have failed on the CI to 
> see if I can reproduce them and to obtain more verbose build log.
>
> ¹ 
> https://git.savannah.gnu.org/cgit/guix.git/log/?id=ea04295256329511b3201feaefb17900b05053b0
> ² https://ci.guix.gnu.org/eval/4930?status=failed
> ³ https://ci.guix.gnu.org/build/396873/log/raw
>
> Cheers,
> - Brice

That sounds awfully familiar.  I wonder if this is related to the
following issue?

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hello Sarah,
>
>> substitute: [Kupdating substitutes from 'http://141.80.167.131'...   0.0%
>> substitute: [Kupdating substitutes from 'http://141.80.167.131'... 100.0%
>> cannot build missing derivation 
>> ?/gnu/store/r1p932bjq21nv4g2qf8fydrphr6fwnw7-go-golang-org-x-lint-0.0.0-0.83fdc39.drv?
>
> Yes, I recently configured Cuirass to use the publish server behind
> ci.guix.gnu.org instead of its own server.
>
> This server uses caching and when a remote worker tries do build a
> derivation that is not available locally, it will also try to fetch the
> input derivations. If one of them is being baked and is above the cache
> bypass limit (50 MiB), the publish server will send a 404 error causing
> the build to stop altogether as you noticed.
>
> This is a clear limitation of this bypass/404 mechanism, that I intend
> to solve with: https://issues.guix.gnu.org/50040.
>
> Thanks,
>
> Mathieu

--
Sarah