Re: 02/03: gnu: openblas: Update architectures we provide substitutes for.

[Christopher Baines]

Efraim Flashner <efraim@flashner.co.il> writes:

> On Sat, Jun 03, 2023 at 08:12:48PM +0100, Christopher Baines wrote:
>> 
>> Efraim Flashner <efraim@flashner.co.il> writes:
>> 
>> > On Fri, Jun 02, 2023 at 11:03:42PM +0100, Christopher Baines wrote:
>> >> 
>> >> guix-commits@gnu.org writes:
>> >> 
>> >> > efraim pushed a commit to branch master
>> >> > in repository guix.
>> >> >
>> >> > commit 076688fa1e41a09f034a80e1a593bac43f1f1482
>> >> > Author: Efraim Flashner <efraim@flashner.co.il>
>> >> > AuthorDate: Thu Jun 1 11:06:00 2023 +0300
>> >> >
>> >> >     gnu: openblas: Update architectures we provide substitutes for.
>> >> >
>> >> >     * gnu/packages/maths.scm (openblas)[arguments]: Adjust the 
>> >> > substitutable?
>> >> >     flag to only not provide substitutes when building for 
>> >> > powerpc-linux.
>> >> >     Adjust the comment accordingly.
>> >> > ---
>> >> >  gnu/packages/maths.scm | 11 ++---------
>> >> >  1 file changed, 2 insertions(+), 9 deletions(-)
>> >> 
>> >> I've been looking at why armhf-linux substitute availability has been
>> >> dropping recently, and I think this change triggered a lot of
>> >> rebuilds. Could this have gone to core-updates?
>> >> 
>> >> → guix refresh -l openblas
>> >> Building the following 2282 packages would ensure 5596 dependent packages 
>> >> are rebuilt: ...
>> >
>> > It's not that it's triggered rebuilds, but that it's triggered builds.
>> > It's also triggered builds on powerpc64le and riscv64. Before any
>> > package which had openblas as a transitive dependency wasn't built by
>> > the CI because it wasn't substitutable¹. People still have the option of
>> > using package transformations to use openblas tuned for the cortex a7 or
>> > a15 on armhf, but in reality this just unlocks substitutes for those
>> > ~5600 packages which wasn't available before.
>> >
>> > ¹ We saw this in the past briefly in the past when openzfs made its way
>> > as a dependency to qemu and through that to Gnome.
>> 
>> Ok, so the documentation does mention "rebuilding", and I do see that
>> indeed ci.guix.gnu.org doesn't build not substitutable things.
>> 
>> Although I think it doesn't apply recursively. Take qjson, guix refresh
>> -l tells me it's dependent on openblas, and looking back at say this
>> output [1] for powerpc64le-linux, that's available from both
>> ci.guix.gnu.org. Which makes sense, as that derivation is substitutable,
>> even though one of it's inputs isn't.
>> 
>> 1: 
>> https://data.guix.gnu.org/gnu/store/fibiwzyz8s899ccpix5zs6r2pcdpxk5b-qjson-0.9.0
>> 
>> Maybe on the client side this works differently, and guix won't
>> substitute things which have a non substitutable input?
>> 
>> Assuming ci.guix.gnu.org was building things for armhf-linux, I think
>> this would have still caused ~5596 rebuilds, and as I say, I think for
>> systems like powerpc64le-linux, I think it did cause ~5596
>> rebuilds.
>
> I looked into it more. First I ran on master:
> './pre-inst-env guix build --no-grafts --system=armhf-linux openblas -d'
> /gnu/store/whi4yhiw2b0c0i3n6l8s0qfcphkvbzg4-openblas-0.3.20.drv
>
> Then I locally reverted the patch expanding the architectures where we
> provided substitutes:
> /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv
>
> Then I locally reverted the patch adjusting the make-flags:
> /gnu/store/1m57z8jkbf6gz7qlbw3ws4ayl0ln9602-openblas-0.3.20.drv
>
> It seems I was wrong, changing the #:substitutable? flag _does_ change
> the derivation of the package. I also checked the substitutes and saw
> that bordeaux did (and does) have substitutes for the non-substitutable
> version, showing that it was built before. I also checked for
> powerpc64le, to see if perhaps cuirass worked a different way and
> honored the #:substitutable flag by not building it there, and it too
> has substitutes for both versions of openblas.
>
> I didn't check for riscv64 but I assume the case is the same with a
> changing derivation.
>
> So obviously if I had realized this would cause ~5596 rebuilds per
> affected arch I wouldn't have pushed the patch. I should've checked the
> derivation before and after to make sure it didn't change.

Thanks for investigating further.

There's maybe a secondary issue here about why changing the
substitutibility of a package affects it's outputs. That doesn't make
much sense to me, but maybe there's a reason.

signature.asc
Description: PGP signature