Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Ludovic Courtès]

Hello!

Mathieu Othacehe <othacehe@gnu.org> skribis:

> We have already discussed the Cuirass vs Build coordinator situation in
> the past. I haven't changed by mind on that subject. The Guix Build
> Coordinator is more or less the equivalent of the Cuirass remote build
> mechanism[2].

Yes.  So to answer Vincent’s question with my own words and as an
outsider: the Guix Build Coordinator focuses on distributing builds
across several machines that may come and go dynamically.  This is
similar to cuirass-remote-worker.

One difference I see is that cuirass-remote-worker uses ZMQ for
communication while the Coordinator uses HTTP, which makes it easier to
use in a more distributed and dynamic context (HTTP goes through all
firewalls).  Cuirass-remote-worker can discover the central server via
Avahi discovery, which makes it easy to add new workers but is limited
to LANs.  On berlin, a WireGuard VPN was set up to address this (the
non-x86 build nodes are far away from the rest of the build farm).

Another one is that, because the Coordinator focuses on builds, it
brings specific features, such as retrying failed builds.

I like that the Coordinator is quite orthogonal; you can use it together
with the Data Service, or you can use it as an alternative to the
daemon’s offload mechanism.

Conversely, Cuirass is more of an all-in-one solution.  Depending on how
you look at it, it can be a weakness, but it’s also a strength when it
comes to deploying a “build farm” kind of service (I think it’s a good
fit for ci.guix).  Its monitoring features and dashboards have become
very nice and well adapted to someone willing to build packages from a
bunch of channels.

> Integrating the Build coordinator in the Berlin build farm would mean
> hooking in to Cuirass as an alternative to the remote build mechanism. I
> don't think it would be wise because:

[...]

> It really makes me sad that we have two pieces of software that are
> stepping on each other toes. The Build coordinator has a nice concept
> and represents a huge amount of work. However, integrating it to Berlin
> is not an option for me.
>
> I think that the next challenges on the CI front are:
>
> * Increase the number of ARM machines in the build farm.
>
> * Work on substitutes mirrors.
>
> * Find a way to make Cuirass and the Data Service work together.
>
> and we should face those issues together, rather than having competing
> software sharing the few build machines we own.

Right, I think the fruitful way to frame it is not “which one is
better”, but rather how can we take the good things from each approach.

For example, I believe the Guix Data Service relied on the former
Cuirass notification mechanism to learn about build status.  That
mechanism is gone, so “we” (i.e., you :-) Chris & Mathieu) should make
sure the Data Service can take advantage of the new notification
mechanism, possibly adjusting it so there’s a good match.

There are certainly other opportunities for cooperation in this area.
The Data Service does things that Cuirass doesn’t do, such as linting.
It wouldn’t make sense IMO to extend Cuirass to do these things; instead
we should find ways to aggregate and present all this information in the
Data Service.  It already does that to a large extent, but maybe we can
think of tighter integration, such as providing QA-oriented pages
instead of the more generic interface it currently provides.

Thoughts?

Ludo’.