bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time

[Eli Zaretskii]

> Date: Fri, 24 Sep 2021 13:18:22 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 50733@debbugs.gnu.org, dgutov@yandex.ru, mardani29@yahoo.es
> 
> IMO, the one and only case where a specialized tool beats ripgrep (or just 
> plain grep) is when you just want the place(s) where the identifier is 
> defined.

No, a specialized tool that uses a DB will scale much better than any
tool which searches the filesystem.  _And_ it will be more accurate
(if used correctly).

> That's not correct, mkid only supports a limited number of programming 
> languages. And it's not even precise: rg O_CREAT on the Emacs trunk for 
> example returns 45 matches, gid O_CREAT returns 33 matches.

I'm sorry, but this has NIH written all over it.  Am I right guessing
that you aren't an active user of ID Utils, and perhaps didn't even
know about it before I mentioned it?

More to the point: are you saying that a tool that returns more
matches is necessarily better?  That'd fly in the face of our
switching to the Xref package, whose explicit goal was to provide
_fewer_ matches, the ones that matter.  Look closer at those matches
which gid "missed", and you will see why it didn't show them to you.

Oh, and if ripgrep finds only 45 matches, then something is wrong with
it, because there are actually no less than 119 literal matches for
O_CREAT in the tree (not counting many binary files that also match).
So by this measure, ripgrep is also not the right tool for the job.

> Five seconds to scan the whole Emacs trunk is IMO not fast enough (ripgrep 
> does it in < 0.2 seconds).

Those 5 sec are invested only when needed, while the time it takes
Grep/ripgrep to scan the files is invested every search.  Do this
enough times, and you paid too much time.

> And without incremental updates, updating the database would be
> necessary before each invocation of gid, because what users expect
> when they search for something are accurate results corresponding to
> the current state of the project, not results from, say, an hour
> ago.

It is very easy to trigger a new mkid run from a file-notification
that watches the project tree, so that it runs in the background
without the user noticing when needed.  Puff! the problem's gone.