[bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

> This is an attempt to improve the performance of "guix search".  It is still
> half baked but it allows to discuss further the idea about expanding the
> current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-)

Thanks for resuming this discussion.  :-)

> From my understanding, the issue that 'package-relevance' accepts a 'package'
> (and then all the chain until displaying) and 'fold-avaibale-packages' does
> not return a package.  Well, I do not know; especially where to put something
> similiar to 'read-package-from'.

Yeah that’s annoying.  Perhaps we need <proto-package> or
<package-metadata>.  With some trickery we could have record type
inheritance or something, maybe.  Dunno.

It would be good if we could arrange so that ‘fold-available-packages’
doesn’t allocate anything though.

> Let compare only for cold cache and time this cache building (Guix 7db8fd6):
>
>   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
>   time guix build --check $(guix gc --derivers $(readlink -f 
> ~/.config/guix/current/lib/guix/package.cache))
>
>   real        0m28,848s
>   user        0m1,481s
>   sys 0m0,252s
>
>   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
>   time guix build --check $(guix gc --derivers $(readlink -f 
> /tmp/new/lib/guix/package.cache))
>
>   real        0m40,279s
>   user        0m1,582s
>   sys 0m0,232s
>
> It seems longer but compared to the time of "guix pull" completion, it seems
> acceptable.

Both the initial timing and the target are waaay too much.  :-/
On my i7 laptop I have:

--8<---------------cut here---------------start------------->8---
$ time ./pre-inst-env  guile -c '(use-modules (gnu packages)) 
(generate-package-cache "/tmp/t.cache")'

real    0m20.738s
user    0m44.413s
sys     0m0.341s
--8<---------------cut here---------------end--------------->8---

It’s CPU-bound; we should probably start by optimizing that.

In Guile 3.0.7 there was a change that improved this noticeably:

  
https://git.savannah.gnu.org/cgit/guile.git/commit/?id=05614f792bfabbc33798863edd0bb67c744e9299

We should prolly look for similar optimization opportunities in the
assembler…

> Let compare for some queries:
>
>   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
>   time guix search game | recsel -C -P name | wc -l
>   371
>
>   real        0m7,561s
>   user        0m3,525s
>   sys 0m0,391s

I think you should run:

  time guix search game > /dev/null

otherwise Bash’s built-in ‘time’ shows the wall-clock time of the whole
pipeline, and the processing time of ‘recsel’ is probably not negligible
here.

[...]

> Last, two minor remarks about previous comments.
>
> 1. Ludo commented:
>
>         > Therefore the cache '/lib/guix/package.cache' contains more
>         > information.
>
>         This breaks the binary interface, so we’ll have to analyze the impact 
> of
>         such a change and devise a strategy.
>
>         <http://issues.guix.gnu.org/39258#93>
>
> and after some checking, this should be fine, IIUC.  The '--news' is ok
> because of '#:allow-other-keys'.  And other parts are modified accordingly.
> Guix revision N creates a cache that Guix revision N+1 will read but it should
> not be an issue; see 'inferior-available-packages'.
>
> 2. And Ludo wrote:
>
>         I realize the other cache also has that problem, but it would be nice 
> to add a
>         version tag to the cache.  Basically emit something like:
>
>           (package-metadata-cache (version 0) VECTOR …)
>
>         instead of just:
>
>           (VECTOR …)
>
>         <http://issues.guix.gnu.org/39258#93>
>
> which is, after discussions, not necessary.  Versioning does not make sense
> here because the cache is read by the Guix which generates it.  Therefore,
> specify a version is extraneous here.

I confirm!  :-)

Thanks,
Ludo’.