bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

[zimoun]

Hi,

On Wed, 26 Jan 2022 at 03:53, Leo Famulari <leo@famulari.name> wrote:

> It's not easy to make sure that all packages build successfully. I've
> never seen 100% of packages build successfully since I joined Guix in
> 2015. It's a nice goal but it requires some work... a lot more work.

I think this goal is impossible by design of the current workflow.

Do not take me wrong, I am saying that some clean is not necessary.
My aim with this email is to point that the annoyance of broken
packages comes from our current workflow and our scaling up, yeah! :-)


> Everyone is invited to help. Probably, the first step is to remove
> several hundred packages that don't build; it might even be a couple
> thousand.

We agreed to do that in 2019 with a reminder on June, 20th 2020:

<https://yhetil.org/guix/86y2oisj41.fsf@gmail.com/>

and nothing happened.  Because the issue is the current modelling:
master/staging/core-updates.  Continuing with that workflow, it is not
realistic to purge broken packages.  Here, we have to make a choice:

 a) current workflow with broken packages
 b) adopt another workflow

The pro for a) is that it is already working enough, we have the
habits, etc.  And the cons of a) are some packages break sometimes.
My rough guess is, these days, ~5% of packages are broken per
revision.  And my guess is that this percentage of broken package
stays the same as the number of packages increases; other said, more
packages often means more users, so this ~5% means more "visibility"
to hit a broken package.

For b), the switch comes with a cost and I am not convinced all people
are agreeing if this cost would be counter-balanced.  More details
about such discussion are for instance in this thread:

<https://yhetil.org/guix/86mtv29erk.fsf@gmail.com/>

and the archive guix-devel contains many similar discussions.  Nothing new. ;-)


> > I would like to conclude from the CI is which commit broke how many
> > packages.
>
> Agreed, this is a very important missing feature. Also, we need the
> capability to compare commits in terms of CI results.

The idea was to have this feature provided by the Data Service.
However, AFAIK, many pieces are since then broken between
ci.guix.gnu.org and data.guix.gnu.org.

For instance, using this:

<https://data.guix.gnu.org/repository/1/branch/master/package/rawtherapee/output-history>

you can detect which commit breaks the package.  Well, 2 important notes:

 1. All is Scheduled because data.guix.gnu.org is fetching data from
the other CI managed by the Build Coordinator, not from the CI managed
by Cuirass.
 2. The commits listed there are an approximation.  Basically, there
are based on the mailing list guix-commits and a complete batch pushed
as once is considered as a change, and more than often, this batch
contains irrelevant change, see details there:

<https://yhetil.org/guix/863617oe1h.fsf@gmail.com/>


> > Some missing practice of packaging:
> > - Some essential message of the reason why tests were disabled and any
> > sort of suggestions on how to
> >   make them enabled. Contact upstream if required.
>
> Everyone is supposed to do this when writing packages. It's a failure of
> the code review process if that is not happening.

I would not say it is a failure of the code review process.  Instead,
I would say it is a failure of the current workflow.

On one hand, the project cannot complain that not enough people are
doing reviewing (see this thread [1]), when on the other hand,
potential reviewer would have to build many more than just the change
itself, especially when many potential reviewers do not have enough
power at hand (see this thread [2]).

1: <https://yhetil.org/guix/87mtn56mzg.fsf_-_@inria.fr/>
2: <https://yhetil.org/guix/878rynh0yq.fsf@systemreboot.net/#t>


Note that since I joined the project (around 2018), we are discussing
about more automation using patchwork and other similar tools.  For
instance, Chris started this initiative:

<https://patches.guix-patches.cbaines.net/project/guix-patches/list/>

which would really help to improve the CI process.  However, we, as a
project, are not putting enough love in such initiative.  Well, the
hidden point barely discussed is twofold: necessity of a powerful
infrastructure for building more when the resource are somehow
splitted and strong integration with current CI.  Both require many
more resources: manpower and money.  That's why, I think the current
workflow will make more visible broken packages.


> That doesn't mean that Guix QA is perfect, but rather that your
> measurement is inaccurate and misleading.

Yes, this 1.1% does not make sense.  Firstly, counting the commits
containing "Fix build" completely underestimates the number of real
fixes, because some fixes contains instead "Fix tests", some fixes are
update, etc.  Secondly, the important number is the number of broken
packages at one instant t (revision) and for that, the Git history is
useless.  Basically, the important number is the ratio between the
green vs red bullets [3].  Other said, compare the coverage, say,
using "guix weather" for past revisions.  I guess, ~5% of packages are
broken, on average.

3: <https://ci.guix.gnu.org/eval/63833/dashboard>


My 2 cents.

Cheers,
simon

PS: The discussion usually happens on guix-devel and not in the bug
tracker.  Other said, since it does not appear to me an actionable bug
report, I would like to close it.