bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks

[Michael Albinus]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

>> It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must
>> still use "readlink --canonicalize".
>
> According to a quick search, it is possible to merge output of several
> shell commands together. This seems to work even with dumb `sh', it's
> not a Bash extension:
>
>     $ sh -c '{ stat xxx && readlink xxx; }'
>
> I guess TRAMP could just sth. similar, as I understand it doesn't have
> to be strictly one executable call, just one command given to the
> remote shell.

I'll check, perhaps it could be used in this special case.

But Tramp uses already combination of shell commands, and it uses even
private shell functions when applicable. Don't expect too much in
general.

>> Tramps communication with the remote host is like a REPL engine. It
>> sends shell commands to the remote hosts, reads the result, and waits
>> for the shell prompt. If it doesn't wait for the final shell prompt, it
>> is likely that the result or the shell prompt will be seen when reading
>> the output of the next command. This confuses. So no, I don't see a
>> chance to implement this kind of "asynchronity".
>
> I see parameter `nooutput' to `tramp-send-command'. Couldn't that be
> used?

No. This argument tells tramp-send-command not to call the final
accept-process-output. But this is only because on the caller side,
accept-process-output will be called with other arguments but the
default ones.

> Even if not, I could imagine sth. like this:
>
>     (defvar pending-commands nil)
>     (defvar reading-output nil)
>
>     (defun send-command (x output-inessential)
>       (if output-inessential
>           (setf pending-commands (nconc pending-commands (list x)))
>         (while reading-output  ; make sure the connection is free for
> the next essential command
>           (read-next-output-chunk)
>           (when (and (not reading-output) pending-commands)
>             (do-send-command (pop pending-commands))))
>         (do-send-command x)
>         (read-output-now)))
>
>     (defun do-send-command (x)
>       (really-do-send-it x)
>       (setf reading-output t))
>
>     (defun read-output-now ()
>       (while reading-output
>         (read-next-output-chunk))
>       (extract-received-output-from-process-buffer))
>
>     (defun emacs-idling ()  ; hooked up using `run-with-idle-timer' or
> something like that
>       (cond (reading-output
>              (read-next-output-chunk))
>             (pending-commands
>              (do-send-command (pop pending-commands)))))
>
>     (defun read-next-output-chunk ()
>       (when reading-output
>         (do-read-output-chunk)  ; this should be non-blocking
>         (when (end-of-command-output)
>           (setf reading-output nil))))

Something like this could be done, yes. Perhaps not with an (idle)
timer, because they are known to interrupt Tramp's REPL engine at any
point, with (sometimes) desastrous results. In general, one shall try to
avoid Tramp actions in timers, process sentinels and process filters.

So we could use this for deleting temporary files and alike, but there
won't be too many performance boosters I fear.

I will play with this idea, perhaps it helps. But for sure it will be an
opt-in feature only.

> Paul

Best regards, Michael.