optimize tramp for for many start-file-process'es in parallel

[Hendrik Tews]

Hi,

I am maintaining an Emacs library (coq-par-compile in Proof
General, details below) that manages a number of parallel running
background jobs, started via `start-file-process'. To decide,
which jobs need to be started, the library also uses
`file-attributes', `file-truename' and maybe other
remote-file-aware functions. When I use the library with a remote
file, even over /ssh:localhost:, it gets totally slow. For local
files Emacs stays responsive, while this library is running. For
remote files Emacs is more or less locked up.

I am not a Tramp expert and I haven't spend much time
investigating, but from what I found, the problem might be that
Tramp opens a new ssh channel (using ControlMaster, but still
new) for each asynchronous (`start-file-process') connection. I
also see several repeating blocks of

    00:03:14.329296 tramp-process-one-action (5) # Looking for regexp "\(\(Are 
you sure you want to continue connecting 
(yes/no\(?:\(?:/\[fingerprint]\)?)\?\)\)\s-*\)\'" from remote shell
    00:03:14.329349 tramp-process-one-action (5) # Looking for regexp 
"\(\(\(?:Store key in cache\? (y/\|Update cached key\? (y/n, Return cancels 
connectio\)n)\)\s-*\)\'" from remote shell
    00:03:14.329399 tramp-process-one-action (5) # Looking for regexp "\(\(TERM 
= (.*)\|Terminal type\? \[.*\]\)\s-*\)\'" from remote shell
    00:03:14.329447 tramp-process-one-action (5) # Looking for regexp "\(Access 
granted\. Press Return to begin session\. \)\'" from remote shell
    00:03:14.329494 tramp-process-one-action (5) # Looking for regexp "\(\)\'" 
from remote shell
    00:03:14.329542 tramp-process-one-action (5) # Call 
‘tramp-action-process-alive’
    00:03:14.351261 tramp-process-one-action (5) # Looking for regexp 
"\(.*\(user\|login\)\( .*\)?: *\)\'" from remote shell
    00:03:14.351405 tramp-process-one-action (5) # Looking for regexp 
"\(^.*\(\(?:adgangskode\|contrase\(?:\(?:ny\|ñ\)a\)\|geslo\|h\(?:\(?:asł\|esl\)o\)\|iphasiwedi\|jelszó\|l\(?:ozinka\|ösenord\)\|m\(?:ot
 de passe\|ật khẩu\)\|p\(?:a\(?:rola\|s\(?:ahitza\|s\(?: 
phrase\|code\|ord\|phrase\|wor[dt]\)\|vorto\)\)\|in\)\|s\(?:alasana\|enha\|laptažodis\)\|wachtwoord\|лозинка\|пароль\|ססמה\|كلمة
 السر\|गुप्तशब्द\|शब्दकूट\|গুপ্তশব্দ\|পাসওয়ার্ড\|ਪਾਸਵਰਡ\|પાસવર્ડ\|ପ୍ରବେଶ 
ସଙ୍କେତ\|கடவுச்சொல்\|సంకేతపదము\|ಗುಪ್ತಪದ\|അടയാളവാക്ക്\|රහස්පදය\|ពាក្យសម្ងាត់\|パスワード\|密[码碼]\|암호\)\).*:\0?
 *\)\'" from remote shell
    00:03:14.351511 tramp-process-one-action (5) # Looking for regexp 
"\(^.*\(Connection \(?:\(?:clo\|refu\)sed\)\|Host key verification 
failed\.\|Login \(?:[Ii]ncorrect\)\|N\(?:ame or service not known\|o supported 
authentication methods left to try!\)\|Permission denied\|\(?:Sorry, try 
again\|Timeout, server not responding\)\.\).*\|^.*\(Received signal 
[0-9]+\).*\)\'" from remote shell
    00:03:14.351577 tramp-process-one-action (5) # Looking for regexp "\(^[^#$%>

which seem to be responsible for the delay of 0.03 sec between

    00:07:26.744746 tramp-process-actions (3) # Waiting for prompts from remote 
shell...
and

    00:07:26.774526 tramp-process-actions (3) # Waiting for prompts from remote 
shell...done

I would be very grateful for any hints about how to increase the
performance of my library for remote files over Tramp. I would
even consider using a lower level interface for accessing the
remote system, where, for instance, I tell Tramp which of the 5
open ssh connection it should use for the next `file-attributes'
call.

Thanks in advance,

Hendrik


Background: Proof General (https://github.com/ProofGeneral/PG) is
a generic Emacs front-end for proof assistants operating via a
read-eval-print loop. Coq is such a proof assistant that has the
feature to load pre-compiled, binary blobs of logical libraries
that one wants to use in the current file. The coq-par-compile
library, which is integrated in Proof General, implements
transparent asynchronous background compilation of such
libraries. For each such library, coq-par-compile runs an
asynchronous "coqdep" command to determine the dependencies.
After recursively dealing with the dependencies, coq-par-compile
compares file modification times (reported by `file-attributes')
and possibly starts an asynchronous compilation command "coqc".
Except from kickoff, initiated by a user action in Proof General,
the coq-par-compile library operates from within process
sentinels of previously started asynchronous background commands.
The user can specify how many background jobs shall be run at
most in parallel in the background. If that limit is reached new
jobs are queued instead of started directly.

Quite a few users are using the coq-par-compile library for
transparent background compilation of logical modules for 10
years now. I now started in Proof General PR #569 to make
coq-par-compile compatible with Tramp, such that the background
compilation does also work transparently for remote files. It was
slightly more work than just replacing `start-process' by
`start-file-process'. Now it runs, but with such a severe
performance degradation that makes the feature basically useless.

I am looking forward for any hints, but also for a discussion
about how to make this programmatic use of Tramp more efficient.