File path completions: strange speed discrepancy

[JD Smith]

I’m trying to debug a very strange issue I’ve encountered when using completion 
UI’s like vertico, selectrum, and fido to navigate remote file paths with tramp 
(over SSH).  It also occurs (I believe) during normal tab-driven file 
completion, but isn’t as obvious there.  

Here’s the issue:

When navigating a remote SSH host’s files, starting with find-file from a 
non-remote buffer (like *scratch*), completion is very slow, with each key 
event taking ~1/2s or so.  Very strangely, in exactly the same session, indeed 
with the same (controlmaster’d) SSH connection to the remote, C-x C-f starting 
from a remote file or dired buffer is very fast — new completions come in 
almost instantly.  

Here’s a small movie I made of the difference navigating to the same remote 
file (first fast, from a remote dired buffer; then slow, from *scratch*): 

        https://youtu.be/B_igNkaraoE

I did some investigation from an emacs -Q session (with tramp and vertico 
loaded).  This is v27.2 (emacs-mac port), with tramp 2.4.3 pre, but I also 
tried tramp 2.5.1.2 and found the same issue.  With tramp-verbose 6, and I see 
tramp working as expected, sending new directory listings via 
tramp_perl_file_name_all_completions, or hitting the cache for directories 
which have already been visited.  There is no obvious difference in tramp 
communication at verbosity 6 between the slow (starting from non-remote 
buffer), and fast (starting from a remote buffer) paths.  But they feel like 
night and day.  As long as you start in ANY remote buffer, you can even change 
the host and benefit from the dramatic speedup.  So something gets configured 
quite differently from these two starting points. 

Any ideas to further investigate this strange discrepancy?  I tried some 
profiling but didn’t notice anything noteworthy. It’s a big enough difference 
that I find myself thinking “if you need to load another remote file, be sure 
to START from a remote file.” 

Thanks for tramp.