Re: guix pull performance

[Simon Tournier]

Hi,

If I read correctly, your machine is:

        a reasonably powerful machine (Core i7, 12 cores, 16G RAM, 1 TB SSD 
drive
        with plenty of available space)

Correct?

On Tue, 28 Nov 2023 at 22:07, Emmanuel Beffara <manu@beffara.org> wrote:

> Now, even if I didn't expect a significant improvement, I am still badly
> surprised by the behaviour of Guix: one month ago, any `guix pull` took 10
> minutes to complete, and these days it takes 40 minutes (the same command, on
> the same machine in the same conditions). When running a second `guix pull`
> while the channels are unchanged, it takes "only" 6 minutes of computation
> before telling me that there is nothing to do.
>
> How is it possible that things are four times slower over just one month ?
> Did the package collection increase that much ?
> Were there structural changes that caused a loss of efficiency ?

Using my desktop machine (i7, 16 cores, 64G RAM), I get:

  + plain “guix pull”: ~10min
  + redo “guix pull” with the same revision: ~2min

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 29   Oct 25 2023 14:42:15    (current)
  guix aabfddb
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: aabfddbe8bc16fa82067e88e7d79c8c1bc802414

$ time guix pull -p /tmp/new
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 842a11f (82 new commits)...
Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git   842a11f
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%

[...]

 config.scm  507B                                                               
                                   174KiB/s 00:00 ▕██████████████████▏ 100.0%
 git.scm  105B                                                                  
                                   191KiB/s 00:00 ▕██████████████████▏ 100.0%
 hash.scm  130B                                                                 
                                   105KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import  2KiB                                                            
                                   335KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import  2KiB                                                            
                                   504KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB                                                 
                                   2.4MiB/s 00:01 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB                                                 
                                   3.2MiB/s 00:00 ▕██████████████████▏ 100.0%
building 
/gnu/store/g83w55vbr1r4p1jbxdxlrd00hsziqyvq-compute-guix-derivation.drv...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
 git-minimal-2.41.0  4.6MiB                                                     
                                   4.9MiB/s 00:01 ▕██████████████████▏ 100.0%
 guix-daemon-1.4.0-15.e0885fc  258KiB                                           
                                 702.4MiB/s 00:00 ▕████▍             ▏  24.8%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/02czx493wwvilvc8yc38insc94s0l5h7-profile.drv
  /gnu/store/j2l9k04mp2wdnkvin6xfqxm9j45i1myd-guix-842a11f1c.drv
  /gnu/store/1jf0d5ch5rpjgbbj7hbh7xwmzbvkp6cm-guix-misc.drv
  /gnu/store/7q0mghfzrpd1kb0ckcc9hg2skigq2d4i-guix-command.drv
  /gnu/store/lvnm196rxcn6d2smz76amj0xj3w73pnk-guix-module-union.drv
  /gnu/store/yc9w06iifsb5l3xikkg7idc4jk9ldsw4-guix-842a11f1c-modules.drv
  /gnu/store/2qqzddfp48inn38zsq2bi9if69aia1dk-guix-home-modules.drv
  /gnu/store/fcrdfvhnm36x3vlh5j87mvl9frwwxxcj-guix-home.drv
  /gnu/store/k2mardcbpcsivc9mygzdgx9xyvvxvq0p-guix-packages-base.drv
  /gnu/store/vfbxyq50j7brlqyai7dr8npbgnivjlnb-guix-system.drv
  /gnu/store/z9acy6mp2zbfvn9jzrqs70f4y7x1im3d-guix-packages.drv
  /gnu/store/crwap92xzmvk3crd2nm3mxv2wfy0b6gg-guix-packages-modules.drv
  /gnu/store/fnpq63yknym9wfg13wxfqfnc0mdd2ynh-guix-extra-modules.drv
  /gnu/store/hbz4z9bq05z8kcmp2j686v76n45l2kfl-guix-system-modules.drv
  /gnu/store/ivfpj6mbx12bizh8v4li6as2xd3sxw0l-guix-packages-base-modules.drv
  /gnu/store/qs9bax2627scfrv9k5q2k8qa9qfri138-guix-cli-core-modules.drv
  /gnu/store/6hb81r74zij9vswbh2zqyy9csifv53cm-guix-cli-core.drv
  /gnu/store/qvf2wkgs2jlqkk983jy7qa7wv9mf4mc2-guix-cli-modules.drv
  /gnu/store/mn5kjnwym7hikdgqkxz5v6q5icr2knp5-guix-cli.drv
  /gnu/store/wjmzwngblydz65ylrh7kkgh0mq62lz0m-guix-config-modules.drv
  /gnu/store/4d1ql296bygb0ph375316ny8i6d5m4va-guix-config-source.drv
  /gnu/store/0lcmrv5mx312w0h80lrbjzbp25xhl8a3-config.scm.drv
  /gnu/store/zvh0ixd433rmganyrcb1prcx4c9q5axb-guix-system-tests-modules.drv
  /gnu/store/cbywx3457flfbf135kzn884ik802q2dz-guix-system-tests.drv
  /gnu/store/qzhy5lyr56qppryvldghl22db5j66qp6-guix-manual.drv
  /gnu/store/wg49p4988b3acgq8mcdrmc7p9gcd7i3y-guix-daemon.drv
  /gnu/store/8jxl2mxyj7cqaqlq7rr4qxy0nrhj1mra-inferior-script.scm.drv
  /gnu/store/zqpnys3yd7gnljv6nan936hzg7j81ygw-profile.drv

4.0 MB will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
 guix-locale-guix                                                               
                                          3.0MiB/s 00:00 | 486KiB transferred
 git-minimal-2.41.0  4.6MiB                                                     
                                   2.8MiB/s 00:02 ▕██████████████████▏ 100.0%
 guix-daemon-1.4.0-15.e0885fc  258KiB                                           
                                   3.3MiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-config  4KiB                                                              
                                   395KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-daemon.cil  3KiB                                                          
                                   4.9MiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-extra  2.8MiB                                                             
                                   6.1MiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-translated-texinfo  1.6MiB                                                
                                   2.4MiB/s 00:01 ▕██████████████████▏ 100.0%
building /gnu/store/0lcmrv5mx312w0h80lrbjzbp25xhl8a3-config.scm.drv...
applying 9 grafts for gd-2.3.2 ...
applying 8 grafts for guile-avahi-0.4.1 ...
applying 19 grafts for graphviz-minimal-7.0.1 ...
applying 7 grafts for guile-ssh-0.16.3 ...
building /gnu/store/4d1ql296bygb0ph375316ny8i6d5m4va-guix-config-source.drv...
building /gnu/store/fnpq63yknym9wfg13wxfqfnc0mdd2ynh-guix-extra-modules.drv...
building /gnu/store/wjmzwngblydz65ylrh7kkgh0mq62lz0m-guix-config-modules.drv...
building /gnu/store/qzhy5lyr56qppryvldghl22db5j66qp6-guix-manual.drv...
building /gnu/store/1jf0d5ch5rpjgbbj7hbh7xwmzbvkp6cm-guix-misc.drv...
building /gnu/store/k2mardcbpcsivc9mygzdgx9xyvvxvq0p-guix-packages-base.drv...
building /gnu/store/6hb81r74zij9vswbh2zqyy9csifv53cm-guix-cli-core.drv...
building 
/gnu/store/ivfpj6mbx12bizh8v4li6as2xd3sxw0l-guix-packages-base-modules.drv...
building 
/gnu/store/qs9bax2627scfrv9k5q2k8qa9qfri138-guix-cli-core-modules.drv...
building /gnu/store/z9acy6mp2zbfvn9jzrqs70f4y7x1im3d-guix-packages.drv...
building 
/gnu/store/crwap92xzmvk3crd2nm3mxv2wfy0b6gg-guix-packages-modules.drv...
building /gnu/store/vfbxyq50j7brlqyai7dr8npbgnivjlnb-guix-system.drv...
building /gnu/store/fcrdfvhnm36x3vlh5j87mvl9frwwxxcj-guix-home.drv...
building /gnu/store/hbz4z9bq05z8kcmp2j686v76n45l2kfl-guix-system-modules.drv...
building /gnu/store/mn5kjnwym7hikdgqkxz5v6q5icr2knp5-guix-cli.drv...
building /gnu/store/2qqzddfp48inn38zsq2bi9if69aia1dk-guix-home-modules.drv...
building /gnu/store/qvf2wkgs2jlqkk983jy7qa7wv9mf4mc2-guix-cli-modules.drv...
building /gnu/store/cbywx3457flfbf135kzn884ik802q2dz-guix-system-tests.drv...
building 
/gnu/store/zvh0ixd433rmganyrcb1prcx4c9q5axb-guix-system-tests-modules.drv...
building 
/gnu/store/yc9w06iifsb5l3xikkg7idc4jk9ldsw4-guix-842a11f1c-modules.drv...
building /gnu/store/lvnm196rxcn6d2smz76amj0xj3w73pnk-guix-module-union.drv...
building /gnu/store/7q0mghfzrpd1kb0ckcc9hg2skigq2d4i-guix-command.drv...
building /gnu/store/wg49p4988b3acgq8mcdrmc7p9gcd7i3y-guix-daemon.drv...
building /gnu/store/j2l9k04mp2wdnkvin6xfqxm9j45i1myd-guix-842a11f1c.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/8jxl2mxyj7cqaqlq7rr4qxy0nrhj1mra-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:

     GUIX_PROFILE="/tmp/new"
     . "$GUIX_PROFILE/etc/profile"

Alternately, see `guix package --search-paths -p "/tmp/new"'.
hint: After setting `PATH', run `hash guix' to make sure your shell refers to 
`/tmp/new/bin/guix'.

real    10m51.145s
user    2m57.646s
sys     0m1.010s

$ /tmp/new/bin/guix describe
Generation 1    Dec 01 2023 16:33:07    (current)
  guix 842a11f
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 842a11f1caa1bb929c427722ad9d7b7c1ff65727

$ time /tmp/new/bin/guix pull --commit=842a11f1caa1bb929c427722ad9d7b7c1ff65727 
-p /tmp/again
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git   842a11f
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%

[...]

building /gnu/store/k99k4dqavfbpk04bahnv9z36374mnjj1-module-import.drv...
 module-import  2KiB                                                            
                                   134KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB                                                 
                                   4.3MiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB                                                 
                                   3.3MiB/s 00:00 ▕██████████████████▏ 100.0%
building 
/gnu/store/y8g46qfpg7yakglc40avby0grgxc2jb6-compute-guix-derivation.drv...
Computing Guix derivation for 'x86_64-linux'... /
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/shrj2p15crrrsj15yd0yancxq0pwi9qi-profile.drv
  /gnu/store/8gwgjd9rq23hpi9hzg05p4fd8lcnyd5y-guix-842a11f1c.drv
  /gnu/store/7n5jbjagkqi38gmvf0sayfxvsw6r4imz-guix-command.drv
  /gnu/store/82f2w8qgliw8db8aaal9hw4v7mgmzgqk-guix-module-union.drv
  /gnu/store/f3zl4nnjdbfj9zl04q1cn7hcrpdgdqvz-guix-842a11f1c-modules.drv
  /gnu/store/q16d608sr9nn2l4vdhkbr0b5fwf88238-guix-config-modules.drv
  /gnu/store/dlk4i3q2x64p4xqi4w628pbx9xir0jha-guix-config.drv
  /gnu/store/hnlw8r17x70wdph7lkmj3jqba9cqn9cg-guix-config-source.drv
  /gnu/store/lcw7jwv1sav314b3pys380p4vyy9vr8c-config.scm.drv
  /gnu/store/m2pba333pg2c1s0rszvzgrfvvqqzbghi-guix-daemon.drv
  /gnu/store/906nmg79jw2jxqdb1sbw77wkm21ddmbd-profile.drv
  /gnu/store/dmakvrb0nvpvf6g959q6zggygc88nj6c-inferior-script.scm.drv

building /gnu/store/lcw7jwv1sav314b3pys380p4vyy9vr8c-config.scm.drv...
building /gnu/store/hnlw8r17x70wdph7lkmj3jqba9cqn9cg-guix-config-source.drv...
building /gnu/store/dlk4i3q2x64p4xqi4w628pbx9xir0jha-guix-config.drv...
building /gnu/store/q16d608sr9nn2l4vdhkbr0b5fwf88238-guix-config-modules.drv...
building 
/gnu/store/f3zl4nnjdbfj9zl04q1cn7hcrpdgdqvz-guix-842a11f1c-modules.drv...
building /gnu/store/82f2w8qgliw8db8aaal9hw4v7mgmzgqk-guix-module-union.drv...
building /gnu/store/7n5jbjagkqi38gmvf0sayfxvsw6r4imz-guix-command.drv...
building /gnu/store/m2pba333pg2c1s0rszvzgrfvvqqzbghi-guix-daemon.drv...
building /gnu/store/8gwgjd9rq23hpi9hzg05p4fd8lcnyd5y-guix-842a11f1c.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/dmakvrb0nvpvf6g959q6zggygc88nj6c-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:

     GUIX_PROFILE="/tmp/again"
     . "$GUIX_PROFILE/etc/profile"

Alternately, see `guix package --search-paths -p "/tmp/again"'.
hint: After setting `PATH', run `hash guix' to make sure your shell refers to 
`/tmp/again/bin/guix'.

real    2m5.273s
user    3m0.652s
sys     0m1.315s

$ time /tmp/new/bin/guix pull --commit=842a11f1caa1bb929c427722ad9d7b7c1ff65727 
-p /tmp/again
Updating channel 'guix' from Git repository at 
'https://git.savannah.gnu.org/git/guix.git'...
Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git   842a11f
Computing Guix derivation for 'x86_64-linux'... \
nothing to be done

hint: After setting `PATH', run `hash guix' to make sure your shell refers to 
`/tmp/again/bin/guix'.

real    1m39.950s
user    2m57.294s
sys     0m1.195s
--8<---------------cut here---------------end--------------->8---


> Or might there actually be someting wrong with my setup, and in this case how
> could I diagnose it ?

Well, ~40min seems something wrong with your setup.  Considering “guix
pull”, what is the part (from a quick look) that appears to you the
longest?


Cheers,
simon