bug#60601: 29.0.60; project-find-files reports files in seemingly random order

[Dmitry Gutov]

Hi!

On 06/01/2023 11:48, Jostein Kjønigsen via Bug reports for GNU Emacs, 
the Swiss army knife of text editors wrote:
> When working in large projects, and using completion engines like Helm, 
> one often navigates within an Emacs-project by first typing some 
> preliminary filtering-criteria, observing the result, and then further 
> limiting the results until one eventually finds the one file you want to 
> work with.
>
> For this flow to work well, one needs to have related files show up 
> close to one another, and not being listed in "random" order.

One of the problems is, Helm is not a proper/compatible completing-read 
substitute. Just like Ivy, which I myself use.

If you check out the default completion, you'll see the file names are 
sorted, because a completing-read-function is supposed to obey the 
display-sort-function metadata, and when it's absent -- the variable 
completions-sort (in Emacs <29 it was hardcoded alphabetical sort by 
default).

So one solution would be to use a framework-specific way to apply 
sorting, e.g. helm-completing-read-handlers-alist. In Ivy's case, I 
think ivy-sort-functions-alist is supposed to fill that role, but 
apparently it's not used when the collection is a function. I'll 
probably file that as a bug later.

> However, in some projects I've observed that generated files (not part 
> of VC) are shown on top, and VC-files comes later.
>
> When the generated files are supposed to be alongside the checked in 
> files, this makes navigation using project-find-file hard, because of 
> the low data-locality in the presented results.

This never bothered me much personally, but the reason is apparently 
that 'git ls-files' behaves like that. We do apply sorting to find's 
output (the default project-files impl), starting with 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23179#296.

> Example:
>
> - src/someFile.d.ts   # compiled files
> - src/someFile.js
> - tests/someTest.d.ts
> - tests/someTest.js
> - src/someFile.ts     # vc-files
> - tests/someTest.ts
>
> If the results are sorted, the proper files are much easier to find quickly:
>
> - src/someFile.d.ts
> - src/someFile.js
> - src/someFile.ts
> - tests/someTest.d.ts
> - tests/someTest.js
> - tests/someTest.ts
>
> Are there any reason we're currently not doing that?
>
> If we can agree sorting the result is a proper thing to do, attached is 
> a patch which implements this change.

No hard reason not to, but the minor ones are:

- A little extra garbage collection.
- We'll probably apply the new sorting in project-files, keeping the one 
in project--files-in-directory already in place. Then I suppose we'll 
need to document the new restriction on project-files that it returns 
alphabetically sorted files (option A)?

Or if we move the sorting to project-find-file-in, that will keep the 
xref output unsorted. Do we sort the results after the search is done 
instead? That might be an improvement, but it could create a performance 
hit in the pathological case when there are a lot more matches than files.

So we can do that (option A), probably.

Alternatively, this could be a good occasion to file the bug reports to 
Helm and Ivy, to bring their completing-read functions closer to 
conformant behavior.