Re: How to search on emacs-devel?

[Eric Abrahamsen]

On 03/10/22 00:49 AM, Kyle Meyer wrote:
> Eric Abrahamsen writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>>
>>>> public-inbox is git under the hood (see
>>>> https://yhetil.org/emacs-devel/_/text/mirror/).
>>>
>>> Maybe local indexing with notmuch or what have you is still the easiest
>>> solution, to.
>
> For those that want to go this route, the main thing is getting the
> Git-stored messages [1] into a format that the indexer understands.
> There are some existing tools (not part of public-inbox) that can
> convert a public-inbox archive into a Maildir.  l2md [2] and impibe [3]
> are the ones listed at
> <https://public-inbox.org/public-inbox.git/plain/Documentation/clients.txt>,
> though I haven't used either of them myself.  (There's also
> public-inbox's lei, which I'll get to below.)
>
> In the case of Notmuch, the idea of making the Maildir-based Notmuch
> understand public-inbox archives was mentioned briefly on the Notmuch
> mailing list, but that hasn't gone anywhere yet:
>
>   https://yhetil.org/notmuch/20210501051612.GB6409@dcvr/

Thanks for all this background! I guess I don't think there's much value
in converting public-inbox formats to something that notmuch can read,
if public-inbox comes with its own search facilities.

>
> [1] Documentation on public-inbox's archive format:
>     
> https://public-inbox.org/public-inbox.git/plain/Documentation/public-inbox-v2-format.pod
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
> [3] https://leahneukirchen.org/dotfiles/bin/impibe
>
>> Oh I see, if you have the public-inbox package installed locally, then
>> it can be used to index the git repo and provide a search interface.
>> Installing public-inbox on Arch has a dependency on approximately 9,000
>> perl packages, so I won't be doing that, but it looks like that would be
>> the easiest solution.
>
> I suspect you're referring to public-inbox's lei (local email interface)
> that was introduced in the most recent release (1.7.0).  One way to use
> it is to wire up some number of archives as "externals".  These can be
> remote, local, or some combination.
>
> For example, below adds emacs-devel as a local external.  This clones
> the corresponding Git repo and indexes it with public-inbox-index.  It
> unsurprisingly takes some time (maybe around 20 minutes on my machine).
>
>   $ mkdir -p inboxes
>   $ lei add-external --mirror https://yhetil.org/emacs-devel 
> inboxes/emacs-devel
>
> Later, you can update the Git repo and index new messages with
>
>   $ make -C inboxes/emacs-devel update
>
> And here's an example of adding a remote external:
>
>   $ lei add-external https://yhetil.org/emacs-bugs
>
> The lei-q command is the main interface for searching:
>
>   $ lei q f:eric d:2.days.ago.. | wc -l
>   8
>   $ lei q f:eric d:2.days.ago.. | head -1
>   
> [{"blob":"91ec6216f8daa3b731c832f5c5de722e891b49b7","dt":"2022-03-09T22:23:54Z","f":[["Eric
>  Abrahamsen" ...
>
> If there are any local externals configured, lei-q won't reach out to
> the remote externals unless explicitly requested with --remote:
>
>   $ lei q --remote f:eric d:2.days.ago..  | wc -l
>   9
>
> You can also limit a search to only a set of externals:
>
>   $ lei q --only emacs-bugs f:eric d:2.days.ago.. | jq '.[0].s'
>   "bug#54324: 29.0.50; Don't error during debbugs-gnu-rescan"
>
> Or specify a one-off remote URL that wasn't added via add-external:
>
>   $ lei q --only https://lore.kernel.org/git userdiff NEAR emacs
>
> All of the above commands show JSON output, but there are several output
> formats available, including various mbox flavors and Maildir.
>
> Anyway, I've already written too much, but, for those interested,
> https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> along with lei-overview(7) and the other lei- manpages may be worth
> checking out.

All this is really interesting, thank you. It looks like one tool (or
tool suite) will handle mail transport, storage, and search -- that's
really nice. I'll read the overview, hold my nose and install all the
dependencies, and play around with it.

Thanks again,
Eric