Progress with the Guix Data Service

[Christopher Baines]

So, in my last email featuring the Guix Data Service [1], I focused on
reviewing patches, but since then, I've been focusing on fixing some
issues with the Guix Data Service itself, adding some more
functionality, and making some necessary code improvements.

1: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00010.html

Previously, in the Guix Data Service database, each package had a single
related derivation. This was a bit of an oversight, as it didn't account
for different architectures. I've now fixed this, and I'm now generating
derivations for all architectures, as well as generating cross-built
derivations. I'm not quite sure if the cross-built derivations are a
useful thing to have, but one interesting thing about computing them is
that it's very easy to see what you can cross build, and what you
can't. You can see some counts of derivations for a revision, for
example [2].

2: 
https://prototype-guix-data-service.cbaines.net/revision/54c9d7bb69847c89a7193347f858bb4e9476f7df

Another new feature is following the guix-commits mailing list to find
out about new revisions, and load them in. Following the mailing list
still seems a little crude, but I'm not sure of any other way to get
near real time notifications.

I think I've also made quite a few interface improvements, so maybe
try clicking around if you haven't done recently [3]!

3: https://prototype-guix-data-service.cbaines.net/

Along with these things, I also loaded some information from the
berlin.guixsd.org build farm in to the Guix data service. I queried
berlin for derivations, and stored some information about any builds
that berlin had performed [4]. I was looking at this as I think it
would be a useful addition to the comparison functionally, being able
to compare the build status. I don't think doing the query at
comparison time is going to be quick enough when comparing lots of
packages, which is why the information is fetched and stored in the
database.

4: https://prototype-guix-data-service.cbaines.net/builds

I haven't got very far in terms of using the limited build information
that I fetched, but I'm very interested in the applications of this data
in the future. For example, if in the future, there are multiple build
farms (which would be good along with reproducible builds to provide
confidence in the binary substitutes), then gathering up all the build
information in to one place could help with identifying where there are
reproducibility problems, by comparing the results from different build
farms.

As a proof of concept, querying berlin worked OK, but I don't think
the polling approach is scaleable. I think it would be good if there
was the option to subscribe to updates, builds changing status in this
case, and then get notifications when this happens. This would mean
that polling would be unnecessary, and there would be less risk of
missing messages.

I'm used to custom ways of doing this, but if we were going to go down
this route, it would be good I think if any solution wasn't too specific
to the Guix Data Service, and was open to other consumers as well.

One standardised approach is called WebSub [5]. I haven't tried it
before, but from reading the specification, I think if Cuirass provided
an Atom feed containing the events (when the builds change status), then
somehow sent these to a hub (the service in WebSub responsible for
managing subscriptions and delivering notifications), then the hub would
notify subscribers by sending them the events which they haven't
received yet (in the form of an Atom feed).

5: https://www.w3.org/TR/websub/

I'm still a bit uncertain about whether WebSub is entirely
appropriate. The involvement of Atom feeds seems more related to
traditional publishing of written works, not machine readable messages,
but architecturally, it seems OK, and I don't think there's anything
wrong with implementing something inspired by the specification, but not
fully adhering to it.

I'm still really interested in everyone's thoughts about this,
especially on how to get information about builds in to the Guix data
service.

For the moment though, I've gone back to trying to neaten up the
Patchwork service, and write a Getmail service, as that's currently used
for both Patchwork, and the Guix Data Service.

Thanks,

Chris

signature.asc
Description: PGP signature