bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus]

On Tue, Jun 29, 2021 at 04:00:02PM +0300, Dmitry Gutov wrote:
> Hi!
>
> Thanks for the report, I'll follow up on this discussion some more 
> later, but some initial observations:
>
> On 29.06.2021 01:11, Ergus via Bug reports for GNU Emacs, the Swiss 
> army knife of text editors wrote:
> > As a workaround I removed all the uninteresting handlers from
> > vc-handled-backends and I get better times now, but IMHO it is still
> > very inefficient (almost a minute for project-switch-to-buffer is
> > excessive). And make it practically unusable.
>
> Could you evaluate (benchmark 1 '(project-current)) in one of your 
> buffers? That should give us an estimate how long it takes to find the 
> "current project" on your remote system.
Only with git and mercurial it takes:

Elapsed time: 3.018998s (0.109912s in 1 GCs)

With the entire list in vc-handled-backends

Elapsed time: 8.197923s (0.507396s in 6 GCs)

> If I'm right, project-switch-to-buffer should take 25 x (that time).
>
> If you indeed only have 25 buffers (including hidden ones).
>
> > In any case:
> >
> > Maybe (I think I mentioned this before) `project.el` needs a sort of
> > cache to speedup some functions like `project-current` that are called
> > very frequently inside the project.el code.
>
> The difficulty here is probably with the large latency to the remote 
> system. And our current approach calls the "find current project" 
> logic for each open buffer.
>
> Even if we add the "current project" cache, it will only take effect 
> at second and further invocations. Your first project-switch-to-buffer 
> call will still take 3-5 minutes, which is unacceptable.
>
> Please get back to us with the requested measurements, and perhaps 
> other observations (any research initiative is welcome), but if 
> finding the current vc root for each buffer is unavoidably slow, we'll 
> finally need to switch to a "project-contains-buffer-p generic" 
> approach, previously discussed in e.g. "Speed up project-kill-buffers" 
> thread on emacs-devel.

Another (and maybe even simpler) optimization, may be to consider that
all the buffers with the same default-directory should have the same vcs
and vc root (and probably belong to same project). (I think that all vcs
backends only do the search based on default-directory at the end.)

So if the association in the cache is for `default-directory` instead of
individual file names; then, less files will need to evaluate the search
of vc root, and vc-backend only 1/subdir will search the first time. It
is a very good trade of IMO.

This will reduce the time even the first time we iterate project-current
over all opened buffers if multiple files are in the same directory
(example: sources in src, includes in include and so on). And I think it
will cover 99% of normal use cases.

This won't affect nested projects, git-submodules or similar, because
those will be in different subdirs.