Re: Faster start-file-process?

[Michael Albinus]

Michael Albinus <address@hidden> writes:

Hi Philipp,

>> This uses the current master of Tramp. All these results are
>> statistically significant (on my machine, that is). I think at least
>> start-file-process and make-process should definitely switch to the
>> connectionless approach (or at least offer it as option), that would
>> make language servers or Flymake backends over SSH possible.
>> I'm also happy to contribute patches!
>
> I plan to provide a detailed analysis what happens in Tramp when running
> an asynchronous process. Based on this, we could discuss
> improvements. With commit 4c3c175a63823d851c7e9326f4f70c81b9c0110e I've
> done already a first step this direction.

Well, I have applied (start-file-process "" nil "true") with a current
buffer being already remote. This avoids the initialization of the
connection, and should show us just the commands Tramp has sent. You can
do the same, analyzing the Tramp debug buffer for entries with Tramp
level (6).

--8<---------------cut here---------------start------------->8---
10:04:29.011033 tramp-maybe-open-connection (6) # /bin/sh -i
10:04:29.012136 tramp-wait-for-regexp (6) #
#$
--8<---------------cut here---------------end--------------->8---

This opens a local shell in the connection buffer. Whether this is needed
or not might be discussed, it is just a general approach for *all*
different Tramp methods. However, it takes just 0.05 sec before raising
the next command, the performance gain wouldn't be too much.

--8<---------------cut here---------------start------------->8---
10:04:29.060983 tramp-send-command (6) # exec ssh -q   -o ControlMaster=auto -o 
ControlPath='tramp.%C' -o ControlPersist=no -e none detlef
10:04:29.161427 tramp-process-actions (6) #
Last login: Mon Mar 23 10:02:09 2020 from 192.168.178.30
detlef:~>
--8<---------------cut here---------------end--------------->8---

This starts the remote connection with the home shell, and waiting for
the shell prompt. The command in question is not called, because at this
point, Tramp doesn't know whether it must interact for whatever reason,
for example providing a password. Tramps knows this only once the prompt
has been seen.

--8<---------------cut here---------------start------------->8---
10:04:29.163495 tramp-send-command (6) # rm -f ~/.editrc.tramp
10:04:29.181094 tramp-wait-for-regexp (6) #
detlef:~>
10:04:29.181402 tramp-send-command (6) # test -e ~/.editrc && mv -f ~/.editrc 
~/.editrc.tramp
10:04:29.212769 tramp-wait-for-regexp (6) #
detlef:~>
10:04:29.213056 tramp-send-command (6) # echo 'edit off' >~/.editrc
10:04:29.214819 tramp-wait-for-regexp (6) #
detlef:~>
--8<---------------cut here---------------end--------------->8---

Tramp wants to avoid line editing, so it provides a simple
~/.editrc. Some shells send annoying escape sequences otherwise. If the
remote shell isn't /bin/sh, but .../zsh or .../bash, Tramp doesn't apply
this, because the shell invocation cares already about.

--8<---------------cut here---------------start------------->8---
10:04:29.215306 tramp-send-command (6) # exec env TERM='dumb' 
INSIDE_EMACS='28.0.50,tramp:2.5.0-pre' ENV='' HISTFILE=~/.tramp_history 
PROMPT_COMMAND='' PS1=\#\$\  PS2='' PS3='' /bin/sh
10:04:29.231797 tramp-wait-for-regexp (6) #
#$
--8<---------------cut here---------------end--------------->8---

Now Tramp opens the remote shell, /bin/sh this case.

--8<---------------cut here---------------start------------->8---
10:04:29.232118 tramp-send-command (6) # rm -f ~/.editrc
10:04:29.250077 tramp-wait-for-regexp (6) #
#$
10:04:29.250411 tramp-send-command (6) # test -e ~/.editrc.tramp && mv -f 
~/.editrc.tramp ~/.editrc
10:04:29.264064 tramp-wait-for-regexp (6) #
#$
--8<---------------cut here---------------end--------------->8---

~/.editrc is reset.

--8<---------------cut here---------------start------------->8---
10:04:29.264890 tramp-send-command (6) # (cd ~/) 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.275592 tramp-wait-for-regexp (6) #
tramp_exit_status 0
#$
--8<---------------cut here---------------end--------------->8---

Tramp checks, whether the tilde ~/ is expanded. Maybe this is not needed
for asynchronous processes.

--8<---------------cut here---------------start------------->8---
10:04:29.276401 tramp-send-command (6) # stty -inlcr -onlcr -echo kill '^U' 
erase '^H'
10:04:29.283313 tramp-wait-for-regexp (6) #
#$
10:04:29.285092 tramp-wait-for-regexp (6) #
foo
#$
10:04:29.285543 tramp-send-command (6) # 
PS1=///bbca37f3c9f6555a2319eae4da499ab9\#\$ PS2='' PS3='' PROMPT_COMMAND=''
10:04:29.287075 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.288034 tramp-send-command (6) # echo \"`uname -sr`\" 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.291794 tramp-wait-for-regexp (6) #
"Linux 5.3.0-40-generic"
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.293108 tramp-send-command (6) # (echo foo ; echo bar)
10:04:29.303182 tramp-wait-for-regexp (6) #
foo
bar
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.303632 tramp-send-command (6) # set +o vi +o emacs
10:04:29.305080 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

These are some interactive settings for the shell. Maybe we could get
rid of some.

--8<---------------cut here---------------start------------->8---
10:04:29.306664 tramp-send-command (6) # echo \"`getconf PATH 2>/dev/null`\" 
2>/dev/null; echo tramp_exit_status $?
10:04:29.344451 tramp-wait-for-regexp (6) #
"/bin:/usr/bin"
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.345910 tramp-send-command (6) # /bin/sh -l -c 'echo 
101183ec1d014197807aa97f43793ee7 \"$PATH\"' 2>/dev/null; echo tramp_exit_status 
$?
10:04:29.464168 tramp-wait-for-regexp (6) #
101183ec1d014197807aa97f43793ee7 
"/home/albinus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

Tramp checks for the values of $PATH to be set. Needed for commands
which are not absolute.

--8<---------------cut here---------------start------------->8---
10:04:29.466752 tramp-send-command (6) # test -d /home/albinus/bin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.468028 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.470670 tramp-send-command (6) # test -d /usr/local/sbin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.472209 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.475370 tramp-send-command (6) # test -d /usr/local/bin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.476956 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.480232 tramp-send-command (6) # test -d /usr/sbin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.481974 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.486268 tramp-send-command (6) # test -d /usr/bin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.488413 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.492653 tramp-send-command (6) # test -d /sbin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.494734 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.499274 tramp-send-command (6) # test -d /bin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.501362 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.505625 tramp-send-command (6) # test -d /usr/games 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.508350 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.512642 tramp-send-command (6) # test -d /usr/local/games 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.514700 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.518718 tramp-send-command (6) # test -d /snap/bin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.741115 tramp-wait-for-regexp (6) #
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.744624 tramp-send-command (6) # test -d /local/bin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.746228 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.748851 tramp-send-command (6) # test -d /local/freeware/bin 
2>/dev/null; echo tramp_exit_status $?
10:04:29.753279 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.757459 tramp-send-command (6) # test -d /local/gnu/bin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.764677 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.805413 tramp-send-command (6) # test -d /usr/freeware/bin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.807191 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.843228 tramp-send-command (6) # test -d /usr/pkg/bin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.856355 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.884514 tramp-send-command (6) # test -d /usr/contrib/bin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.887926 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.900189 tramp-send-command (6) # test -d /opt/bin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.901665 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.904255 tramp-send-command (6) # test -d /opt/sbin 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.905553 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.908146 tramp-send-command (6) # test -d /opt/local/bin 2>/dev/null; 
echo tramp_exit_status $?
10:04:29.911510 tramp-wait-for-regexp (6) #
tramp_exit_status 1
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.919287 tramp-send-command (6) # 
PATH=/home/albinus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin;
 export PATH
10:04:29.923720 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

Tramp tests every element of $PATH, whether it exists. I believe we
could get rid of these checks. In this test case, it would save 0.5 sec.

--8<---------------cut here---------------start------------->8---
10:04:29.924141 tramp-send-command (6) # mesg n 2>/dev/null; biff n 2>/dev/null
10:04:29.926596 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.926945 tramp-send-command (6) # stty tab0
10:04:29.929181 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.929508 tramp-send-command (6) # stty iutf8 2>/dev/null
10:04:29.931592 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

Some further interactive shell settings.

--8<---------------cut here---------------start------------->8---
10:04:29.932319 tramp-send-command (6) # echo \"`tty`\" 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.934489 tramp-wait-for-regexp (6) #
"/dev/pts/9"
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

Tramp determines the tty name.

--8<---------------cut here---------------start------------->8---
10:04:29.939209 tramp-send-command (6) # while read var val; do export 
$var="$val"; done <<'101183ec1d014197807aa97f43793ee7'
LC_ALL en_US.utf8
TMPDIR $HOME
ENV ''
TMOUT 0
LC_CTYPE ''
PAGER cat
101183ec1d014197807aa97f43793ee7
10:04:29.940929 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
10:04:29.941354 tramp-send-command (6) # unset CDPATH HISTORY MAIL MAILCHECK 
MAILPATH autocorrect correct
10:04:29.942930 tramp-wait-for-regexp (6) #
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

Tramp sets shel environment variables, as dictated by
tramp-remote-process-environment and process-environment.

--8<---------------cut here---------------start------------->8---
10:04:29.944731 tramp-send-command (6) # echo $$ 2>/dev/null; echo 
tramp_exit_status $?
10:04:29.946021 tramp-wait-for-regexp (6) #
22453
tramp_exit_status 0
///bbca37f3c9f6555a2319eae4da499ab9#$
--8<---------------cut here---------------end--------------->8---

Tramp determines its own pid. Needed for kill-process, if called.

--8<---------------cut here---------------start------------->8---
10:04:29.947139 tramp-send-command (6) # cd /home/albinus/ &&  exec   env 
PS1\=/ssh\:detlef\:/home/albinus/\ \#\$\  true
--8<---------------cut here---------------end--------------->8---

Finally, Tramp changes the default directory to the one the current
buffer keeps (/home/albinus), and calls "true". The setting of $PS1 is
useful for shells called this way; it doesn't hurt for other commands.

Well, some room for improvements. I'll try to change the obvious
candidates. Otherwise, comments welcome.

Best regards, Michael.