Extension for improving Guix search?

[Simon Tournier]

Hi,

Guile-Xapian@0.4.0 is out!  That’s a fun opportunity for revisiting
“guix search”.  Well, “guix search” is currently an alias of “guix
package --search” and it can be considered as “slow”.  The main reason
is from the lack of a good cache.

Although it is not a fair comparison, “guix package --list-available” is
much faster than “guix package --search”.  Obviously, they do not
process and format the same content.  Well, the point is that
’list-available’ relies on a cache, namely
~/.config/guix/current/lib/guix/package.cache.

It had been discussed to expand this cache (or create another one) for
helping “guix search”.  Instead of walking through Guile modules, just
read a cache.  However, this implies a slower “guix pull” – and “guix
time-machine” too –; the penalty depends on your machine and from my
opinion, I can wait some seconds more after waiting many minutes, but
that had not been the consensus.

All that said, there is a very long thread where different approaches
had been tried; see all the subthreads. ;-)

 1. Rely on SQL [1]
 2. Rely on Xapian [2]
 3. Extend package.cache [3]
 4. Improving string search algorithm [4]
 5. Rely on package.cache and load the required Guile [5]

On the top of that, it is not handy to find which Guix revision contains
which package version.

For a description of the problem [6].  For instance, I do stuff as:

    $ git -C ~/src/guix/guix log --format="%h %s" | grep 'gnu: bowtie:'
    a47a90b900 gnu: bowtie: Remove reference to %outputs.
    f336cc4fe7 gnu: bowtie: Replace invalid characters.
    e5a26a1f02 gnu: bowtie: Remove trailing #T.
    2ec601580b gnu: bowtie: Use TBB 2020.
    21c837405a gnu: bowtie: Update to 2.3.4.3.
    06e372360e gnu: bowtie: Use 'modify-phases'.
    d6e63cf31c gnu: bowtie: Update to 2.3.2.
    2642231b39 gnu: bowtie: Update to 2.2.9.
    0047d26a22 gnu: bowtie: Update to 2.2.6.
    241e122193 gnu: bowtie: fix build errors

which is not super handy.  Well, it was somehow an idea behind this
Outreachy internship [7].  It’s another direction: exploit the format of
Guix commit messages. Anyway.

Therefore, all in all, I think it’s time to have a Guix extension that
improves the situation on both sides:

  a) faster search
  b) search inside all the history

And I think that Xapian could offer a very great framework.  Well, I
have started an embryo months (years?) ago and it is still an embryo.
This issue deserve more love. :-)

Adding Xapian as Guix dependencies is too much.  But it is possible to
extend Guix with GUIX_EXTENSIONS_PATH.  This appears to me the best
solution here: write an extension relying on Xapian for indexing.


Still reading?  I think this fit a very good topic if you are looking
for something to work on.  Maybe it could be turned on some GSoC – hum I
am a bit lost with all the deadlines.

Feel free to ping. :-)

Cheers,
simon



1: [bug#39258] Faster guix search using an sqlite cache
Arun Isaac <arunisaac@systemreboot.net>
Fri, 24 Jan 2020 01:21:57 +0530
id:cu7pnfaar36.fsf@systemreboot.net
https://issues.guix.gnu.org/39258
https://issues.guix.gnu.org/msgid/cu7pnfaar36.fsf@systemreboot.net
https://yhetil.org/guix/cu7pnfaar36.fsf@systemreboot.net

2: [bug#39258] [PATCH v2 0/3] Xapian for Guix package search
Arun Isaac <arunisaac@systemreboot.net>
Sat, 07 Mar 2020 19:01:13 +0530
id:20200307133116.11443-1-arunisaac@systemreboot.net
https://issues.guix.gnu.org/39258
https://issues.guix.gnu.org/msgid/20200307133116.11443-1-arunisaac@systemreboot.net
https://yhetil.org/guix/20200307133116.11443-1-arunisaac@systemreboot.net

3: [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3)
zimoun <zimon.toutoune@gmail.com>
Sun, 03 May 2020 17:01:51 +0200
id:20200503150154.26532-1-zimon.toutoune@gmail.com
https://issues.guix.gnu.org/39258
https://issues.guix.gnu.org/msgid/20200503150154.26532-1-zimon.toutoune@gmail.com
https://yhetil.org/guix/20200503150154.26532-1-zimon.toutoune@gmail.com

4: [bug#39258] KMP string search algorithm?
zimoun <zimon.toutoune@gmail.com>
Mon, 01 Jun 2020 12:11:52 +0200
id:CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@mail.gmail.com
https://issues.guix.gnu.org/39258
https://issues.guix.gnu.org/msgid/CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@mail.gmail.com
https://yhetil.org/guix/CAJ3okZ2ZNYaH=_OjP2Axvq-anY6Q9CkVmjUVLPx-Z6+kZTVsdg@mail.gmail.com

5: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances
zimoun <zimon.toutoune@gmail.com>
Thu, 15 Jul 2021 09:33:26 +0200
id:20210715073328.212123-1-zimon.toutoune@gmail.com
https://issues.guix.gnu.org/39258
https://issues.guix.gnu.org/msgid/20210715073328.212123-1-zimon.toutoune@gmail.com
https://yhetil.org/guix/20210715073328.212123-1-zimon.toutoune@gmail.com

6: Re: List available versions of package.
Philippe Veber <philippe.veber@gmail.com>
Tue, 11 Jun 2019 09:43:08 +0200
id:CAOOOohSzUezKvm=RO0bXRGH3m0eo2x0cOTvd--vARxWoqtceaQ@mail.gmail.com
https://lists.gnu.org/archive/html/help-guix/2019-06
https://yhetil.org/guix/CAOOOohSzUezKvm=RO0bXRGH3m0eo2x0cOTvd--vARxWoqtceaQ@mail.gmail.com

7:
https://guix.gnu.org/en/blog/2021/outreachy-guix-git-log-internship-wrap-up/

8: Re: Mechanism for helping in multi-channels configuration (and Xapian index)
Simon Tournier <zimon.toutoune@gmail.com>
Mon, 06 May 2024 14:05:50 +0200
id:87pltzp2ld.fsf@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2024-05
87pltzp2ld.fsf@gmail.com">https://yhetil.org/guix/87pltzp2ld.fsf@gmail.com