Re: Proposal: prefetch tarballs in a batch

[Nikita Karetnikov]

I’m attaching the version incorporating your suggestions, which was
tested with the empty store.  There are a bunch of problems:

1. It doesn’t seem to prefetch all the needed dependencies.  ‘guix build
   hello’ (without network access) fails after prefetching the said
   package.

2. The substituter fails from time to time.  Note that eight tests fail
   on the machine I used: ‘builders.scm’, ‘utils.scm’, ‘packages.scm’,
   ‘store.scm’, ‘monads.scm’, ‘gexp.scm’, ‘guix-package.sh’,
   ‘guix-register.sh’.  Perhaps we ought to fix the mentioned failures
   first.  Which log files would you like to see?

   $ ./pre-inst-env guix prefetch -n icecat
   substitute-binary: guile: hashtab.c:137: vacuum_weak_hash_table: Assertion 
`removed <= len' failed.
   Backtrace:
   In ice-9/boot-9.scm:
    157: 15 [catch #t #<catch-closure 9c5bef0> ...]
   In unknown file:
      ?: 14 [apply-smob/1 #<catch-closure 9c5bef0>]
   In ice-9/boot-9.scm:
     63: 13 [call-with-prompt prompt0 ...]
   In ice-9/eval.scm:
    432: 12 [eval # #]
   In ice-9/boot-9.scm:
   2320: 11 [save-module-excursion #<procedure 9b8ae60 at 
ice-9/boot-9.scm:3961:3 ()>]
   3966: 10 [#<procedure 9b8ae60 at ice-9/boot-9.scm:3961:3 ()>]
   1645: 9 [%start-stack load-stack ...]
   1650: 8 [#<procedure 9c5c960 ()>]
   In unknown file:
      ?: 7 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"]
   In guix/ui.scm:
    630: 6 [run-guix-command prefetch "-n" "icecat"]
   In ice-9/eval.scm:
    432: 5 [eval # #]
   In guix/ui.scm:
    265: 4 [show-what-to-build #<build-daemon 256.14 a369660> (# # # ...) ...]
   In guix/utils.scm:
    667: 3 [loop (# # # # ...) () (# # # # ...)]
   In guix/ui.scm:
    267: 2 [#<procedure c6fc288 at guix/ui.scm:265:24 (drv build download)> # # 
()]
   In guix/derivations.scm:
    175: 1 [derivation-prerequisites-to-build # # # ...]
   In guix/store.scm:
    695: 0 [substitutable-paths # #]

   guix/store.scm:695:2: In procedure substitutable-paths:
   guix/store.scm:695:2: Throw to key `srfi-34' with args `(#<condition 
&nix-protocol-error [message: "substituter `substitute-binary' died 
unexpectedly" status: 1] be0d570>)'.

   I’ve also seen this one.  In case it matters, that was before running
   ‘chgrp 1001 /gnu/store; chmod 1775 /gnu/store’.

   $ ./pre-inst-env guix prefetch -n gnunet
   Backtrace:
   In ice-9/boot-9.scm:
    157: 17 [catch #t #<catch-closure 9319ef0> ...]
   In unknown file:
      ?: 16 [apply-smob/1 #<catch-closure 9319ef0>]
   In ice-9/boot-9.scm:
     63: 15 [call-with-prompt prompt0 ...]
   In ice-9/eval.scm:
    432: 14 [eval # #]
   In ice-9/boot-9.scm:
   2320: 13 [save-module-excursion #<procedure 9289e60 at 
ice-9/boot-9.scm:3961:3 ()>]
   3966: 12 [#<procedure 9289e60 at ice-9/boot-9.scm:3961:3 ()>]
   1645: 11 [%start-stack load-stack ...]
   1650: 10 [#<procedure 931a960 ()>]
   In unknown file:
      ?: 9 [primitive-load "/home/nikita/guix/guix-savannah/scripts/guix"]
   In guix/ui.scm:
    630: 8 [run-guix-command prefetch "-n" "gnunet"]
   In ice-9/eval.scm:
    432: 7 [eval # #]
   In guix/ui.scm:
    265: 6 [show-what-to-build #<build-daemon 256.14 9a29660> (# # # ...) ...]
   In guix/utils.scm:
    667: 5 [loop () () ...]
   In guix/ui.scm:
    267: 4 [#<procedure aab63c0 at guix/ui.scm:265:24 (drv build download)> # 
() ()]
   In guix/derivations.scm:
    175: 3 [derivation-prerequisites-to-build # # # ...]
   In guix/store.scm:
    695: 2 [substitutable-paths # #]
    392: 1 [process-stderr #<build-daemon 256.14 9a29660> #f]
   In guix/serialization.scm:
     51: 0 [read-int #<input-output: socket 11>]

   guix/serialization.scm:51:4: In procedure read-int:
   guix/serialization.scm:51:4: In procedure bv-u32-ref: Wrong type argument in 
position 1 (expecting bytevector): #<eof>

3. When using the substituter, the command takes much more time.  Do
   we even need it in this case?  I seem to recall that the GNUnet
   tarball was served by Hydra, but I forgot the details.

   $ time ./pre-inst-env guix prefetch -n icecat
   The following derivations would be built:
      […]
   101987 operations

   real 4m7.477s
   user 0m19.249s
   sys  0m4.848s

   $ time ./pre-inst-env guix prefetch -n --no-substitutes icecat
   The following derivations would be built:
      […]
   101675 operations

   real 0m22.933s
   user 0m14.745s
   sys  0m3.168s

prefetch.patch.8
Description: Text Data

pgpmpd9bzunEN.pgp
Description: PGP signature