Re: Faster start-file-process?

[Michael Albinus]

Philipp Stephani <address@hidden> writes:

Hi Philipp,

> It might not seem like much, but for the use case of Flymake/LSP we
> talk about launching a process after every keystroke. Blocking user
> input for longer than a few milliseconds is a non-starter. For
> example, according to https://pavelfatin.com/typing-with-pleasure/,
> Emacs has an average typing latency of 50 ms. This is already way
> slower than e.g. Vim, or IntelliJ. This first step alone would already
> double the latency!

It sounds to be a problematic approach, to call a remote async process
after every keystroke with such tight response requirements. Couldn't it
be rearranged to call a local async process instead, even if the file
being edited is remote?

>> This starts the remote connection with the home shell, and waiting for
>> the shell prompt. The command in question is not called, because at this
>> point, Tramp doesn't know whether it must interact for whatever reason,
>> for example providing a password. Tramps knows this only once the prompt
>> has been seen.
>
> That's in indeed an issue, for the connectionless approach SSH will
> need to be started with -o BatchMode=yes so that it doesn't ask for
> passwords.

We do not speak only about ssh connections. There's also su/sudo and
whatever, where it is guaranteed to be asked for a password.

>> --8<---------------cut here---------------start------------->8---
>> 10:04:29.163495 tramp-send-command (6) # rm -f ~/.editrc.tramp
>> 10:04:29.181094 tramp-wait-for-regexp (6) #
>> detlef:~>
>> 10:04:29.181402 tramp-send-command (6) # test -e ~/.editrc && mv -f
>> ~/.editrc ~/.editrc.tramp
>> 10:04:29.212769 tramp-wait-for-regexp (6) #
>> detlef:~>
>> 10:04:29.213056 tramp-send-command (6) # echo 'edit off' >~/.editrc
>> 10:04:29.214819 tramp-wait-for-regexp (6) #
>> detlef:~>
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tramp wants to avoid line editing, so it provides a simple
>> ~/.editrc. Some shells send annoying escape sequences otherwise. If the
>> remote shell isn't /bin/sh, but .../zsh or .../bash, Tramp doesn't apply
>> this, because the shell invocation cares already about.
>
> That seems like a compatibility measure for "exotic" systems that
> could/should be optional.

It isn't exotic, but reported be users. And Tramp does not know in
advance, whether it is necessary.

A user option might be possible, but I doubt that the majority of users
understands how to set. Remember, Tramp is intended for a vast number of
different remote systems, and it shall work out of the box.

>> --8<---------------cut here---------------start------------->8---
>> 10:04:29.215306 tramp-send-command (6) # exec env TERM='dumb'
>> INSIDE_EMACS='28.0.50,tramp:2.5.0-pre' ENV=''
>> HISTFILE=~/.tramp_history PROMPT_COMMAND='' PS1=\#\$\ PS2='' PS3=''
>> /bin/sh
>> 10:04:29.231797 tramp-wait-for-regexp (6) #
>> #$
>> --8<---------------cut here---------------end--------------->8---
>>
>> Now Tramp opens the remote shell, /bin/sh this case.
>
> Why does it need a shell, instead of just running the command directly?

Because of setting terminal olptions, and determining information like
remote tty and remote pid etc pp. You don't need it in your use case,
but Tramp will need in general.

>> --8<---------------cut here---------------start------------->8---
>> 10:04:29.264890 tramp-send-command (6) # (cd ~/) 2>/dev/null; echo
>> tramp_exit_status $?
>> 10:04:29.275592 tramp-wait-for-regexp (6) #
>> tramp_exit_status 0
>> #$
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tramp checks, whether the tilde ~/ is expanded. Maybe this is not needed
>> for asynchronous processes.
>
> Yes, it seems all of these checks work around some weird/broken/exotic
> behavior, which might be OK compatibility-wise, but should be
> optional.

Again: if it is optional, there are good chances that the Tramp
maintainer will be flooded by bug reports, because people tend to set
those options wrong.

>> --8<---------------cut here---------------start------------->8---
>> 10:04:29.306664 tramp-send-command (6) # echo \"`getconf PATH
>> 2>/dev/null`\" 2>/dev/null; echo tramp_exit_status $?
>> 10:04:29.344451 tramp-wait-for-regexp (6) #
>> "/bin:/usr/bin"
>> tramp_exit_status 0
>> ///bbca37f3c9f6555a2319eae4da499ab9#$
>> 10:04:29.345910 tramp-send-command (6) # /bin/sh -l -c 'echo
>> 101183ec1d014197807aa97f43793ee7 \"$PATH\"' 2>/dev/null; echo
>> tramp_exit_status $?
>> 10:04:29.464168 tramp-wait-for-regexp (6) #
>> 101183ec1d014197807aa97f43793ee7
>> "/home/albinus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
>> tramp_exit_status 0
>> ///bbca37f3c9f6555a2319eae4da499ab9#$
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tramp checks for the values of $PATH to be set. Needed for commands
>> which are not absolute.
>
> Why? The remote shell is capable of searching the PATH itself, no?

Tramp works hard to use default paths in order to get a chance that the
called utilities are running as expected. People tend to have their
private versions of basic utilities in their own paths.

Of course, people could overwrite the default by tramp-own-remote-path,
but then they are responsible for the results. I will always recommend
to remove this setting, when hunting problems.

>> 10:04:29.908146 tramp-send-command (6) # test -d /opt/local/bin
>> 2>/dev/null; echo tramp_exit_status $?
>> 10:04:29.911510 tramp-wait-for-regexp (6) #
>> tramp_exit_status 1
>> ///bbca37f3c9f6555a2319eae4da499ab9#$
>> 10:04:29.919287 tramp-send-command (6) #
>> PATH=/home/albinus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin;
>> export PATH
>> 10:04:29.923720 tramp-wait-for-regexp (6) #
>> ///bbca37f3c9f6555a2319eae4da499ab9#$
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tramp tests every element of $PATH, whether it exists. I believe we
>> could get rid of these checks. In this test case, it would save 0.5 sec.
>
> That's an enormous amount of time!

Yes. I've removed these checks for asynchronous processes meanwhile.

>> --8<---------------cut here---------------start------------->8---
>> 10:04:29.939209 tramp-send-command (6) # while read var val; do
>> export $var="$val"; done <<'101183ec1d014197807aa97f43793ee7'
>> LC_ALL en_US.utf8
>> TMPDIR $HOME
>> ENV ''
>> TMOUT 0
>> LC_CTYPE ''
>> PAGER cat
>> 101183ec1d014197807aa97f43793ee7
>> 10:04:29.940929 tramp-wait-for-regexp (6) #
>> ///bbca37f3c9f6555a2319eae4da499ab9#$
>> 10:04:29.941354 tramp-send-command (6) # unset CDPATH HISTORY MAIL
>> MAILCHECK MAILPATH autocorrect correct
>> 10:04:29.942930 tramp-wait-for-regexp (6) #
>> ///bbca37f3c9f6555a2319eae4da499ab9#$
>> --8<---------------cut here---------------end--------------->8---
>>
>> Tramp sets shel environment variables, as dictated by
>> tramp-remote-process-environment and process-environment.
>
> This could be done in a single command:
>
> ssh "foo=bar exec cmd..."

The resulting command could be too long (yes, this was reported). That's
why I use a heredoc implementation.

> Thanks, but I can only say that without my approach or a similar one
> this can't get significantly faster. The only possible really fast
> approach I'm aware of is starting exactly one process asynchronously
> and never calling accept-process-output.

That's what my proposal of changing the make-process implementation is
good for. However, I cannot decide myself. I propose we move this
discussion to emacs-devel.

Best regards, Michael.