bug#32583: Cuirass: some builds are marked as failed, but they were never built

[Ludovic Courtès]

Hi Clément,

Clément Lassieur <address@hidden> skribis:

> This is from the logs:
>
> 2018-08-30T09:14:31 batch of builds (partially) failed:build of
> `/gnu/store/06yy5cl59baa6iczvcrl7gzbs82x5j30-cleanup.drv',
> `/gnu/store/0ilx18r8s4rw00kxynn6nida1grggx46-separate-store-os.drv',
> `/gnu/store/0n887pl0igppd7mr1hh2zrc889sfbh5h-dovecot-test.drv',
> `/gnu/store/0qad25im46dnqwfcd63k2lljgd7h9gy2-dicod.drv',
> `/gnu/store/0y5yb81mx5m98ddmbvhivhyszd5mh1bi-encrypted-root-os.drv',
> `/gnu/store/1af46bw6km0qd6mvm63xsx10f0ipyy2x-nfs.drv',
> `/gnu/store/1b28w2c9bgsjnypmnwyl5bnq7yc8k3pv-btrfs-root-os.drv',
> `/gnu/store/1vv5wp52l1wbfg45b0a44fs9icx31m8f-encrypted-root-os.drv',
> `/gnu/store/2149hi6f7gccn5x8cbgdx8w7sgadp1h0-basic.drv',
> `/gnu/store/21i0ylg5p3q1s6qg395vyr55n64hkbil-separate-home-os.drv',
> `/gnu/store/228h2p10ysd7n4cw3fwg1n7aj0k4xmrr-raid-root-os.drv',
> `/gnu/store/2a4gsnndyslw7cwg9rjc5yhqhkpfag6w-libparserutils-0.2.4.drv',
> `/gnu/store/2i8xq16s6bcwnxfpn7l6yq9jyip3ayxc-installed-extlinux-os.drv',
> `/gnu/store/2lp48f2jdjampri0g9137qcz6ra4v5n3-mongodb-test.drv',
> `/gnu/store/2zvk8ml5ankw0mbq6m693p4a9ppx7x83-exim-test.drv',
> `/gnu/store/3dzssjfjzy5nngfm0cbg747lw2sddfvv-mcron.drv',
> `/gnu/store/43pdwcgm49vs4grfzkw1n4mg7hhc0ih5-libvirt-test.drv',
> `/gnu/store/4li2cm5igrcrkis8b4ilipf9823zy2h1-iso-image-installer.drv',
> `/gnu/store/4w8hbibw77vxr5iwf1avgf0y95l4fwwd-libsvgtiny-0.1.7.drv',
> `/gnu/store/56cz6yfy5s8w95kqxhgrry4iymx9zdbq-hubbub-0.3.5.drv',
>   [...]
> `/gnu/store/w83xmggaj5qxzfafnv1pyhff24s2c40m-installed-os.drv',
> `/gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv',
> `/gnu/store/wqpy6m5kf4wmnjhdqqwfyyx6n0nis06s-libdom-0.3.3.drv',
> `/gnu/store/x2k166lqcmwad0mli8qsrql9a4vsrq45-installed-extlinux-os.drv',
> `/gnu/store/x6v7bdvcvfbny1kdgsq6jilbszrfksbc-libcss-0.8.0.drv',
> `/gnu/store/xg4dzxz54qpf678syvf4379vbps1wqhs-tailon-test.drv',
> `/gnu/store/y08j1qivdlam0kp1w636f4lycnmm866v-disk-image.drv',
> `/gnu/store/ya89cfswkgv7m3az3rzmbggy433lg288-libvirt-test.drv',
> `/gnu/store/ya9q9x208fif1ci4pbsfm7vyk39znzmh-memcached-test.drv',
> `/gnu/store/yhc1liighrcsgl1smpwrdahy3xrj4zd6-libnsutils-0.0.5.drv',
> `/gnu/store/ykhlibxssbg42r2cfyzm65gkkjyqky7h-opensmtpd-test.drv',
> `/gnu/store/yxv7ny3v4ng41rvhd0r245lnr4nwwch4-raid-root-os.drv',
> `/gnu/store/zdakqvwx1na7p8c8ivacz4wlcm6pbdh6-dicod.drv'
> failed (status: 100)
>
> So it seems that the daemon tried to build them, but they all failed.
> Probably at that point the log file weren't created.
>
> I don't really understand why you were able to build
> /gnu/store/wgk46pv09m1y8b7iq5fbkhay5hs6hnmr-libcss-0.8.0.drv manually
> then.

It could be that the failure above is due to an offloading timeout (I’ve
seen ‘guix offload’ hang occasionally before on berlin, and
a708de151c255712071e42e5c8284756b51768cd turned those into timeouts
reported as transient failures.)  So we may have a bug where things get
“stuck” for some reason.

> 'starttime == 0 && stoptime != 0' is explained by the fact the exception
> (from 'build-derivations&'?) was thrown before 'handle-build-event' was
> called (which should have set starttime), and later on
> '(update-build-statuses! store batch)' was called, which only sets
> stoptime (because the output is invalid).

I wonder if we should get rid of ‘update-build-statuses!’: if a build
fails, then we get a ‘build-failure’ event and we already update the
build status at that point, right?

WDYT?

Ludo’.