bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time

[Dmitry Gutov]

On 24.09.2021 15:02, Eli Zaretskii wrote:

> You don't need to convince me, I have all of those installed, and
> couldn't do without them.  The difficulties are practical, not
> principal.

Bundling extra tools affects the size of the distribution, for example. 
I figured that's something you might have an opinion on.

> > What is the concern about the presence of the current maintainer? Would
> > we do something different if he leaves? We'd probably need to find
> > another maintainer, no?
>
> Yes, and the question is: will we succeed, and how quickly?  We
> already had a period of time without one, which means having no one is
> not a theoretical danger.

I'm not sure how that affects our opinion on bundling, though.

> > > Btw, I don't understand why we focus on general-purpose text-searching
> > > tools for these features.  Why not focus on packages like ID Utils
> > > instead, they are so much faster.  Daniel, could you time the same
> > > search in that large tree when xref-search-program is 'gid'?  (You'd
> > > need to run 'mkid' first, to create the ID database, but that is
> > > one-time, and is very fast.)
> >
> > There is no such option in xref-search-program.
>
> Hmm... I'm sure this worked in the past, at least for
> xref-find-references?  Or does that use a different variable?

Different variable, yes: semantic-symref-tool.

It affects xref-find-references.

> > If you can suggest an
> > appropriate invocation for xref-search-program-alist, I can try and add it.
>
> Just "gid -r <R>".  But if this could run in an arbitrary directory,
> there should also be a "-f .../ID" argument, telling it where to find
> the ID database (usually, in the project's root).

Anyway, it seems the most pressing problem is not the performance of the 
external tool (ripgrep is quite fast, for example). But our 
post-processing of the results, when there are a lot of them.

id-tools (or alternatives) could help avoid the "fetch all project 
files" step, though. At the cost of extra manual management, which I 
personally don't like much.

> > But I thought id-tools are for scanning for identifiers, not for
> > arbitrary regexp searches?
>
> They can scan for identifiers that match regular expressions, where
> "identifier" is defined by each file's PL.
>
> >   >  As I told many times, I think this is
> >   > the future: program language sensitive tools that use a precomputed
> >   > DB.
> >
> > xref-find-references implies some language-awareness.
> > project-find-regexp does not.
>
> Are you sure people indeed use project-find-regexp _only_ for
> language-agnostic searches?

Oh, I use it for all kinds of searches. The point is not being limited 
in the search input or the choice of files to search.

Instead, we can and will add UI for specifying extra filters for the 
results.

> Anyway, maybe we need a separate command, or a different (but similar)
> tool, one that indexes the tokens in the project files instead of
> scanning all of the files each time anew.

"Tokens" implies searching only for identifiers, no?