[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Improving CI throughput
From: |
Ludovic Courtès |
Subject: |
Re: Improving CI throughput |
Date: |
Fri, 28 Aug 2020 15:51:28 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) |
Hello!
Mathieu Othacehe <othacehe@gnu.org> skribis:
>> Yeah, this is a ridiculous situation. We should do a hackathon to get
>> better monitoring of useful metrics (machine load,
>> time-of-push-to-time-to-build-completion, etc.), to clearly identify the
>> bottlenecks (crashes? inefficient protocol? scheduling issues? Cuirass
>> or offload or guix-daemon issue?), and to address as many of them as we
>> can.
>>
>> Any volunteers? :-)
>
> I'd really like to improve the situation! A hackathon seems like a
> nice idea.
>
> As a matter of fact, I already spent some times improving the stability
> of Cuirass web interface[1].
Much appreciated!
> Now I can see multiple topics that could be approached in parallel:
>
> * Add metrics to Cuirass as you suggested. There's an open ticket about
> that here[2].
>
> * Investigate offloading issues[3].
>
> * Fix database contention[4].
>
> * Fix guix-daemon deadlocking[5].
>
> * Monitor closely what's happening on Berlin and decide if it is
> opportune to add a build scheduler mechanism somewhere. See what Hydra
> is doing[6] and what Chris is proposing[7].
I’m happy to help tackle daemon/offload issues, but I’ll be more
motivated if others join. :-)
> As most of the issues are only observed on Berlin machines, which access is
> restricted, we will also have to find a way to reproduce them locally.
Yeah, and these are usually non-deterministic issues and not that
frequent.
> Anyway, if some people are motivated, we could try to plan a day or
> week-end to work on those topics :).
I can try and spend some time on it this week-end. I suggest that
people join the IRC channel and shout “CI!” as a way to rally, and then
share what they’re looking at and how they feel. How does that sound?
Thanks for cooking up this list of issues!
Ludo’.