Re: Faster start-file-process?

[Philipp Stephani]

Am Mo., 23. März 2020 um 11:14 Uhr schrieb Michael Albinus
<address@hidden>:
>
> Michael Albinus <address@hidden> writes:
>
> Hi Philipp,
>
> >> This uses the current master of Tramp. All these results are
> >> statistically significant (on my machine, that is). I think at least
> >> start-file-process and make-process should definitely switch to the
> >> connectionless approach (or at least offer it as option), that would
> >> make language servers or Flymake backends over SSH possible.
> >> I'm also happy to contribute patches!
> >
> > I plan to provide a detailed analysis what happens in Tramp when running
> > an asynchronous process. Based on this, we could discuss
> > improvements. With commit 4c3c175a63823d851c7e9326f4f70c81b9c0110e I've
> > done already a first step this direction.
>
> Well, I have applied (start-file-process "" nil "true") with a current
> buffer being already remote. This avoids the initialization of the
> connection, and should show us just the commands Tramp has sent. You can
> do the same, analyzing the Tramp debug buffer for entries with Tramp
> level (6).
>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.011033 tramp-maybe-open-connection (6) # /bin/sh -i
> 10:04:29.012136 tramp-wait-for-regexp (6) #
> #$
> --8<---------------cut here---------------end--------------->8---
>
> This opens a local shell in the connection buffer. Whether this is needed
> or not might be discussed, it is just a general approach for *all*
> different Tramp methods. However, it takes just 0.05 sec before raising
> the next command, the performance gain wouldn't be too much.

It might not seem like much, but for the use case of Flymake/LSP we
talk about launching a process after every keystroke. Blocking user
input for longer than a few milliseconds is a non-starter. For
example, according to https://pavelfatin.com/typing-with-pleasure/,
Emacs has an average typing latency of 50 ms. This is already way
slower than e.g. Vim, or IntelliJ. This first step alone would already
double the latency!

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.060983 tramp-send-command (6) # exec ssh -q   -o ControlMaster=auto 
> -o ControlPath='tramp.%C' -o ControlPersist=no -e none detlef
> 10:04:29.161427 tramp-process-actions (6) #
> Last login: Mon Mar 23 10:02:09 2020 from 192.168.178.30
> detlef:~>
> --8<---------------cut here---------------end--------------->8---
>
> This starts the remote connection with the home shell, and waiting for
> the shell prompt. The command in question is not called, because at this
> point, Tramp doesn't know whether it must interact for whatever reason,
> for example providing a password. Tramps knows this only once the prompt
> has been seen.

That's in indeed an issue, for the connectionless approach SSH will
need to be started with -o BatchMode=yes so that it doesn't ask for
passwords.

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.163495 tramp-send-command (6) # rm -f ~/.editrc.tramp
> 10:04:29.181094 tramp-wait-for-regexp (6) #
> detlef:~>
> 10:04:29.181402 tramp-send-command (6) # test -e ~/.editrc && mv -f ~/.editrc 
> ~/.editrc.tramp
> 10:04:29.212769 tramp-wait-for-regexp (6) #
> detlef:~>
> 10:04:29.213056 tramp-send-command (6) # echo 'edit off' >~/.editrc
> 10:04:29.214819 tramp-wait-for-regexp (6) #
> detlef:~>
> --8<---------------cut here---------------end--------------->8---
>
> Tramp wants to avoid line editing, so it provides a simple
> ~/.editrc. Some shells send annoying escape sequences otherwise. If the
> remote shell isn't /bin/sh, but .../zsh or .../bash, Tramp doesn't apply
> this, because the shell invocation cares already about.

That seems like a compatibility measure for "exotic" systems that
could/should be optional.

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.215306 tramp-send-command (6) # exec env TERM='dumb' 
> INSIDE_EMACS='28.0.50,tramp:2.5.0-pre' ENV='' HISTFILE=~/.tramp_history 
> PROMPT_COMMAND='' PS1=\#\$\  PS2='' PS3='' /bin/sh
> 10:04:29.231797 tramp-wait-for-regexp (6) #
> #$
> --8<---------------cut here---------------end--------------->8---
>
> Now Tramp opens the remote shell, /bin/sh this case.

Why does it need a shell, instead of just running the command directly?

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.232118 tramp-send-command (6) # rm -f ~/.editrc
> 10:04:29.250077 tramp-wait-for-regexp (6) #
> #$
> 10:04:29.250411 tramp-send-command (6) # test -e ~/.editrc.tramp && mv -f 
> ~/.editrc.tramp ~/.editrc
> 10:04:29.264064 tramp-wait-for-regexp (6) #
> #$
> --8<---------------cut here---------------end--------------->8---
>
> ~/.editrc is reset.
>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.264890 tramp-send-command (6) # (cd ~/) 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.275592 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> #$
> --8<---------------cut here---------------end--------------->8---
>
> Tramp checks, whether the tilde ~/ is expanded. Maybe this is not needed
> for asynchronous processes.

Yes, it seems all of these checks work around some weird/broken/exotic
behavior, which might be OK compatibility-wise, but should be
optional.

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.276401 tramp-send-command (6) # stty -inlcr -onlcr -echo kill '^U' 
> erase '^H'
> 10:04:29.283313 tramp-wait-for-regexp (6) #
> #$
> 10:04:29.285092 tramp-wait-for-regexp (6) #
> foo
> #$
> 10:04:29.285543 tramp-send-command (6) # 
> PS1=///bbca37f3c9f6555a2319eae4da499ab9\#\$ PS2='' PS3='' PROMPT_COMMAND=''
> 10:04:29.287075 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.288034 tramp-send-command (6) # echo \"`uname -sr`\" 2>/dev/null; 
> echo tramp_exit_status $?
> 10:04:29.291794 tramp-wait-for-regexp (6) #
> "Linux 5.3.0-40-generic"
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.293108 tramp-send-command (6) # (echo foo ; echo bar)
> 10:04:29.303182 tramp-wait-for-regexp (6) #
> foo
> bar
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.303632 tramp-send-command (6) # set +o vi +o emacs
> 10:04:29.305080 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> These are some interactive settings for the shell. Maybe we could get
> rid of some.
>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.306664 tramp-send-command (6) # echo \"`getconf PATH 2>/dev/null`\" 
> 2>/dev/null; echo tramp_exit_status $?
> 10:04:29.344451 tramp-wait-for-regexp (6) #
> "/bin:/usr/bin"
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.345910 tramp-send-command (6) # /bin/sh -l -c 'echo 
> 101183ec1d014197807aa97f43793ee7 \"$PATH\"' 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.464168 tramp-wait-for-regexp (6) #
> 101183ec1d014197807aa97f43793ee7 
> "/home/albinus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> Tramp checks for the values of $PATH to be set. Needed for commands
> which are not absolute.

Why? The remote shell is capable of searching the PATH itself, no?

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.466752 tramp-send-command (6) # test -d /home/albinus/bin 
> 2>/dev/null; echo tramp_exit_status $?
> 10:04:29.468028 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.470670 tramp-send-command (6) # test -d /usr/local/sbin 2>/dev/null; 
> echo tramp_exit_status $?
> 10:04:29.472209 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.475370 tramp-send-command (6) # test -d /usr/local/bin 2>/dev/null; 
> echo tramp_exit_status $?
> 10:04:29.476956 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.480232 tramp-send-command (6) # test -d /usr/sbin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.481974 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.486268 tramp-send-command (6) # test -d /usr/bin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.488413 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.492653 tramp-send-command (6) # test -d /sbin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.494734 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.499274 tramp-send-command (6) # test -d /bin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.501362 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.505625 tramp-send-command (6) # test -d /usr/games 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.508350 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.512642 tramp-send-command (6) # test -d /usr/local/games 
> 2>/dev/null; echo tramp_exit_status $?
> 10:04:29.514700 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.518718 tramp-send-command (6) # test -d /snap/bin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.741115 tramp-wait-for-regexp (6) #
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.744624 tramp-send-command (6) # test -d /local/bin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.746228 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.748851 tramp-send-command (6) # test -d /local/freeware/bin 
> 2>/dev/null; echo tramp_exit_status $?
> 10:04:29.753279 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.757459 tramp-send-command (6) # test -d /local/gnu/bin 2>/dev/null; 
> echo tramp_exit_status $?
> 10:04:29.764677 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.805413 tramp-send-command (6) # test -d /usr/freeware/bin 
> 2>/dev/null; echo tramp_exit_status $?
> 10:04:29.807191 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.843228 tramp-send-command (6) # test -d /usr/pkg/bin 2>/dev/null; 
> echo tramp_exit_status $?
> 10:04:29.856355 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.884514 tramp-send-command (6) # test -d /usr/contrib/bin 
> 2>/dev/null; echo tramp_exit_status $?
> 10:04:29.887926 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.900189 tramp-send-command (6) # test -d /opt/bin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.901665 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.904255 tramp-send-command (6) # test -d /opt/sbin 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.905553 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.908146 tramp-send-command (6) # test -d /opt/local/bin 2>/dev/null; 
> echo tramp_exit_status $?
> 10:04:29.911510 tramp-wait-for-regexp (6) #
> tramp_exit_status 1
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.919287 tramp-send-command (6) # 
> PATH=/home/albinus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin;
>  export PATH
> 10:04:29.923720 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> Tramp tests every element of $PATH, whether it exists. I believe we
> could get rid of these checks. In this test case, it would save 0.5 sec.

That's an enormous amount of time!

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.924141 tramp-send-command (6) # mesg n 2>/dev/null; biff n 
> 2>/dev/null
> 10:04:29.926596 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.926945 tramp-send-command (6) # stty tab0
> 10:04:29.929181 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.929508 tramp-send-command (6) # stty iutf8 2>/dev/null
> 10:04:29.931592 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> Some further interactive shell settings.
>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.932319 tramp-send-command (6) # echo \"`tty`\" 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.934489 tramp-wait-for-regexp (6) #
> "/dev/pts/9"
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> Tramp determines the tty name.
>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.939209 tramp-send-command (6) # while read var val; do export 
> $var="$val"; done <<'101183ec1d014197807aa97f43793ee7'
> LC_ALL en_US.utf8
> TMPDIR $HOME
> ENV ''
> TMOUT 0
> LC_CTYPE ''
> PAGER cat
> 101183ec1d014197807aa97f43793ee7
> 10:04:29.940929 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> 10:04:29.941354 tramp-send-command (6) # unset CDPATH HISTORY MAIL MAILCHECK 
> MAILPATH autocorrect correct
> 10:04:29.942930 tramp-wait-for-regexp (6) #
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> Tramp sets shel environment variables, as dictated by
> tramp-remote-process-environment and process-environment.

This could be done in a single command:

ssh "foo=bar exec cmd..."

>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.944731 tramp-send-command (6) # echo $$ 2>/dev/null; echo 
> tramp_exit_status $?
> 10:04:29.946021 tramp-wait-for-regexp (6) #
> 22453
> tramp_exit_status 0
> ///bbca37f3c9f6555a2319eae4da499ab9#$
> --8<---------------cut here---------------end--------------->8---
>
> Tramp determines its own pid. Needed for kill-process, if called.
>
> --8<---------------cut here---------------start------------->8---
> 10:04:29.947139 tramp-send-command (6) # cd /home/albinus/ &&  exec   env 
> PS1\=/ssh\:detlef\:/home/albinus/\ \#\$\  true
> --8<---------------cut here---------------end--------------->8---
>
> Finally, Tramp changes the default directory to the one the current
> buffer keeps (/home/albinus), and calls "true". The setting of $PS1 is
> useful for shells called this way; it doesn't hurt for other commands.
>
> Well, some room for improvements. I'll try to change the obvious
> candidates. Otherwise, comments welcome.

Thanks, but I can only say that without my approach or a similar one
this can't get significantly faster. The only possible really fast
approach I'm aware of is starting exactly one process asynchronously
and never calling accept-process-output.