bug#47799: 28.0.50; Default `project-files' implementation doesn't work with quoted filenames

[Dmitry Gutov]

On 16.05.2021 16:37, Philipp wrote:

> One thing that came to my mind is: in general, in Elisp (not just XRef), we spend lots of 
> time parsing filenames to support remote and quoted filenames.  Other languages probably 
> solve this by introducing proper types for filenames (e.g. the Java Path class), which 
> can then hold preprocessed information about the underlying filesystem (or special file 
> name handler, in the case of Elisp).  How about doing similar for Elisp?  For example, 
> introduce a `parsed-file-name' class or structure holding the remote/quoting state, or 
> attach it to string properties?  I haven't tried out that idea, but I think it could 
> significantly speed up the parsing (since we'd only have to do it once and don't have to 
> search for filename handlers all the time), as well as remain backward-compatible to 
> "plain" unparsed filenames by allowing both strings and this new object type.  
> WDYT?

That sounds like an interesting idea to explore.

We create/concatenate those file names inside project-files, and then 
"parse" them again to convert to local names inside 
xref-matches-in-files. Creating such structures might indeed save us on 
some parsing and garbage generation.

Experiments and patches welcome.

What I was also thinking of previously, is some "fileset" data structure 
which could contain a list of local file names and their connection in a 
separate slot. Maybe even separating the parent/root directory into a 
separate slot when feasible, to minimize GC further, though that might 
complicate applications.

A more structured "file" value format might make this stuff easier to 
use indeed, and perhaps the performance difference will be negligible.

The difficulty is having a method like project-files return one format 
for some users, and another for users who want to take advantage of this 
performance improvement. Or we break the compatibility and/or introduce 
a new method with this new behavior.

There is a one in the works already in the 'scratch/etags-regen' branch 
after all.

Or another, more simplistic approach would be to have the method 
project-files-filtered return file names relative to the root (always, 
or when called with a certain argument). And then pass the root (and the 
connection/host) in the default-directory var. Then change 
xref-matches-in-files to use default-directory if the values in FILES 
are not absolute.

The last approach would only work if we decide that a search across 
multiple roots (e.g. project roots together with external roots) can be 
done efficiently enough through multiple calls to xref-matches-in-files 
(and thus using multiple consecutive process calls). Someone should 
benchmark this in a real-world scenario; it might or might not show 
worse performance: OT1H, the potential for parallelism is more limited, 
and there is more overhead on process calls, OTOH, the practical 
parallelism is not infinite anyway, and the process soon bottlenecks on 
CPU and/or disk access throughput.