Re: February update on data.guix.gnu.org and the Guix Data Service

[Christopher Baines]

Ludovic Courtès <address@hidden> writes:

> Hi!
>
> Christopher Baines <address@hidden> skribis:
>
>> Derivations for system tests [3], as well as channel instances [4]
>> (which relate to guix pull) are now captured. This is still a work in
>> progress, I think only the x86_64-linux derivations for the system tests
>> are captured, and the systems for the channel instances are limited by
>> what's available in the qemu-binfmt service.
>>
>> 3: 
>> http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/system-tests
>> 4: 
>> http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/channel-instances
>
> Neat!

One small thing to add here is that I've noticed the system test
derivations recorded in the Guix Data Service differ from what Cuirass
is building. I think this is because they're tweaked a bit in
system-test-job (guix ci):

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/ci.scm#n280

>> Related to this, I've also added some code to enable removing data for a
>> branch, and removing unreferenced derivations. This is both for cleaning
>> up the data.guix.gnu.org database now I only want data for master, but
>> it should also be useful when using the Guix Data Service in an
>> environment where branches come and go, or be the basis of setting a
>> retention period for the data.
>
> Nice.  Were you concerned specifically of unbounded growth on the
> data.guix.gnu.org instance?

Partially, although I was more trying to reduce the time between the
master branch changing and the information being available in the
guix-data-service.

Even when the channel-instance derivation was being computed just to
x86-64_linux, it could take up to ~24 hours for some core-updates
changes. Now that the channel-instance derivations are computed for
multiple architectures, I think having those jobs take a long time would
have impacted how fast the jobs for the master branch were processed.

> It would be interesting for data.guix.gnu.org to keep as much data as
> possible, including perhaps for branches that have been deleted.  Then
> the problem becomes more of a UI problem: how to display, for instance,
> only the “active” branches on the main page.
>
> WDYT?

So, after I sent this email, I setup the guix-patches-data.cbaines.net
instance of the Guix Data Service to process data for the main Guix git
repository, but for all branches.

The index page for the Guix Data Service doesn't list deleted branches,
but they are shown on the page for each repository, for example:

  https://guix-patches-data.cbaines.net/repository/2/

>> I forget exactly when, but recently I've been trying to revive the patch
>> review setup I was working on around a year ago [6]. I've setup an
>> instance of the Guix Data Service for this [7] (separate to the
>> data.guix.gnu.org one). I might try and have that instance of the Guix
>> Data Service process all the branches in the Guix git repository, now
>> that data.guix.gnu.org doesn't do that.
>>
>> 6: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html
>> 7: https://guix-patches-data.cbaines.net/
>
> Yay!
>
> At the Guix Days, you also shown very concretely how people could take
> advantage of data.guix.gnu.org in their patch review workflow.  Perhaps
> it’d be useful to share that info here or in the manual, even if it’s
> still evolving.

Indeed, hopefully I can get around to looking at it again soon :)

>> Back to features though, the output from inferior processes used when
>> loading data for a revision is now captured and stored in the
>> database. This means you can see more of what's going on, like the
>> building of libgit2 here for example [8]. Previously you got less
>> information [9].
>>
>> 8: http://data.guix.gnu.org/job/14657
>> 9: http://data.guix.gnu.org/job/14610
>
> Great.
>
> Could you make a support request on Savannah to enable commit
> notifications on the repo?  That’d be awesome.  :-)

Sure, I've created a ticket now:
https://savannah.nongnu.org/support/index.php?110208

> Thanks for the great update, as always!

You're welcome :)

signature.asc
Description: PGP signature