bug#47896: 27.2; TRAMP hang/lockup/fault on compile/file access

[David Greene]

I've been trying to track down an issue with TRAMP that has plagued me
for over a year.

In emacs 27, I frequently get "hangs" opening new files. Sometimes emacs
doesn;'t lock up but sits in the "encoding/decoding remote file" stage
for ever and the file never loads. Other times I get a hard lockup at
"Inserting..." A quit does not end the lockup, I have to go to another
terminal and kill -SIGUSR2 many times to get emacs to respond again.

This is especially bad in compile mode where any compiler message
triggers a similar lockup of emacs, presumably when compile is trying to
load the file referred to in the error message.

So I tried emacs 28. No lockups at all. Yay! But every single compile
failes with exit code 9. This isn't an error from the compilation
itself. I can run the compilation under emacs 27 and it completes (that
is, when it doesn't cause a lockup) and I can run it in a terminal and
it completes. I suspect that whatever was causing lockups in emacs 27 is
now causing signal 9 in 28. Perhaps this has something to do with the
new process handling in 28?

I have tried disabling as much of my config as I can but nothing seems
to help.

Here is a backtrace after SIGUSR2 during a compile lockup:

Debugger entered--entering a function:
* tramp-signal-hook-function(quit nil)
  format-time-string("%T." (24701 49656 439373 0))
  tramp-debug-message((tramp-file-name #("ssh" 0 3 (tramp-default t)) "dgreene" 
nil "dgreene-dev" nil "/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." 
nil) "(5) # Looking for regexp \"%s\" from remote shell" "\\(\\(\\(?:Store key 
in cache\\? (y/\\|Update cached ke...")
  apply(tramp-debug-message (tramp-file-name #("ssh" 0 3 (tramp-default t)) 
"dgreene" nil "dgreene-dev" nil 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." nil) "(5) # Looking 
for regexp \"%s\" from remote shell" "\\(\\(\\(?:Store key in cache\\? 
(y/\\|Update cached ke...")
  tramp-process-one-action(#<process *tramp/ssh dgreene@dgreene-dev*> 
(tramp-file-name #("ssh" 0 3 (tramp-default t)) "dgreene" nil "dgreene-dev" nil 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." nil) 
((tramp-login-prompt-regexp tramp-action-login) (tramp-password-prompt-regexp 
tramp-action-password) (tramp-wrong-passwd-regexp 
tramp-action-permission-denied) (shell-prompt-pattern tramp-action-succeed) 
(tramp-shell-prompt-pattern tramp-action-succeed) (tramp-yesno-prompt-regexp 
tramp-action-yesno) (tramp-yn-prompt-regexp tramp-action-yn) 
(tramp-terminal-prompt-regexp tramp-action-terminal) (tramp-antispoof-regexp 
tramp-action-confirm-message) (tramp-process-alive-regexp 
tramp-action-process-alive)))
  tramp-process-actions(#<process *tramp/ssh dgreene@dgreene-dev*> 
(tramp-file-name #("ssh" 0 3 (tramp-default t)) "dgreene" nil "dgreene-dev" nil 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." nil) 1 
((tramp-login-prompt-regexp tramp-action-login) (tramp-password-prompt-regexp 
tramp-action-password) (tramp-wrong-passwd-regexp 
tramp-action-permission-denied) (shell-prompt-pattern tramp-action-succeed) 
(tramp-shell-prompt-pattern tramp-action-succeed) (tramp-yesno-prompt-regexp 
tramp-action-yesno) (tramp-yn-prompt-regexp tramp-action-yn) 
(tramp-terminal-prompt-regexp tramp-action-terminal) (tramp-antispoof-regexp 
tramp-action-confirm-message) (tramp-process-alive-regexp 
tramp-action-process-alive)) 60)
  tramp-maybe-open-connection((tramp-file-name #("ssh" 0 3 (tramp-default t)) 
"dgreene" nil "dgreene-dev" nil 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." nil))
  tramp-send-command((tramp-file-name #("ssh" 0 3 (tramp-default t)) "dgreene" 
nil "dgreene-dev" nil "/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." 
nil) "\\readlink --canonicalize-missing /net/dgreene-dev/...")
  tramp-send-command-and-check((tramp-file-name #("ssh" 0 3 (tramp-default t)) 
"dgreene" nil "dgreene-dev" nil 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/..." nil) "\\readlink 
--canonicalize-missing /net/dgreene-dev/...")
  
tramp-sh-handle-file-truename(#("/ssh:dgreene@dgreene-dev:/net/dgreene-dev/srv/nfs/..."
 1 4 (tramp-default t)))
  apply(tramp-sh-handle-file-truename 
#("/ssh:dgreene@dgreene-dev:/net/dgreene-dev/srv/nfs/..." 1 4 (tramp-default 
t)))
  tramp-sh-file-name-handler(file-truename 
#("/ssh:dgreene@dgreene-dev:/net/dgreene-dev/srv/nfs/..." 1 4 (tramp-default 
t)))
  apply(tramp-sh-file-name-handler file-truename 
#("/ssh:dgreene@dgreene-dev:/net/dgreene-dev/srv/nfs/..." 1 4 (tramp-default 
t)))
  tramp-file-name-handler(file-truename 
#("/ssh:dgreene@dgreene-dev:/net/dgreene-dev/srv/nfs/..." 1 4 (tramp-default 
t)))
  file-truename(#("/ssh:dgreene@dgreene-dev:/net/dgreene-dev/srv/nfs/..." 1 4 
(tramp-default t)))
  compilation-get-file-structure(("src/ws/stack/project.mk" . 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/...") nil)
  compilation-internal-error-properties(("src/ws/stack/project.mk" . 
"/net/dgreene-dev/srv/nfs/dgreene-data/ws/<redacted>/...") 4 nil nil nil 2 nil)
  compilation-error-properties(1 2 4 3 5 (6 . 7) nil)
  compilation-parse-errors(1390442 1391425)
  compilation--parse-region(1390442 #<marker at 1391469 in 
*clang-build-tdcleanup-all-legacy-devrel-local-dgreene-dev*>)
  compilation--ensure-parse(1391469)
  compilation-filter(#<process compilation> "/ws/stack/project.mk:4: x86_64 is 
the build arch\n[...")
  read-from-minibuffer("M-x " nil (keymap keymap (36 . ivy-magic-read-file-env) 
(3 keymap (19 . ivy-rotate-sort) (1 . ivy-toggle-ignore) (15 . ivy-occur)) 
(33554464 . ivy-restrict-to-matches) (15 . hydra-ivy/body) (22 . 
ivy-scroll-up-command) (prior . ivy-scroll-down-command) (next . 
ivy-scroll-up-command) (7 . minibuffer-keyboard-quit) (right . 
ivy-forward-char) (32 . self-insert-command) (18 . ivy-reverse-i-search) (remap 
keymap (describe-mode . ivy-help) (kill-ring-save . ivy-kill-ring-save) 
(kill-whole-line . ivy-kill-whole-line) (kill-line . ivy-kill-line) 
(scroll-down-command . ivy-scroll-down-command) (scroll-up-command . 
ivy-scroll-up-command) (end-of-buffer . ivy-end-of-buffer) (beginning-of-buffer 
. ivy-beginning-of-buffer) (kill-word . ivy-kill-word) (forward-char . 
ivy-forward-char) (delete-char . ivy-delete-char) (backward-kill-word . 
ivy-backward-kill-word) (backward-delete-char-untabify . 
ivy-backward-delete-char) (delete-backward-char . ivy-backward-delete-char) 
(previous-line . ivy-previous-line) (next-line . ivy-next-line)) (9 . 
ivy-partial-or-done) (10 . ivy-alt-done) (27 keymap (1 . ivy-read-action) (15 . 
ivy-dispatching-call) (111 . ivy-dispatching-done) (25 . 
ivy-insert-current-full) (105 . ivy-insert-current) (106 . ivy-yank-word) (114 
. ivy-toggle-regexp-quote) (97 . ivy-toggle-marks) (16 . 
ivy-previous-line-and-call) (14 . ivy-next-line-and-call) (118 . 
ivy-scroll-down-command) (112 . ivy-previous-history-element) (110 . 
ivy-next-history-element) (10 . ivy-immediate-done) (13 . ivy-call)) (mouse-3 . 
ivy-mouse-dispatching-done) (mouse-1 . ivy-mouse-done) (down-mouse-1 . ignore) 
(13 . ivy-done)) nil extended-command-history)
  ivy-read("M-x " #f(compiled-function (string pred action) #<bytecode 
0x40e60ea9>) :predicate commandp :require-match t :initial-input nil :preselect 
nil :def nil :history extended-command-history :keymap nil :dynamic-collection 
nil :extra-props (:caller ivy-completing-read) :caller execute-extended-command)
  ivy-completing-read("M-x " #f(compiled-function (string pred action) 
#<bytecode 0x40e60ea9>) commandp t nil extended-command-history nil nil)
  completing-read("M-x " #f(compiled-function (string pred action) #<bytecode 
0x40e60ea9>) commandp t nil extended-command-history)
  read-extended-command()
  byte-code("\302\30\11\303 \10E)\207" [execute-extended-command--last-typed 
current-prefix-arg nil read-extended-command] 3)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

I've attached TRAMP verbose 6 output.

David

In GNU Emacs 27.2 (build 1, aarch64-apple-darwin20.2.0, NS appkit-2022.20 
Version 11.1 (Build 20C69))
 of 2021-04-01 built on Davids-MacBook-Pro.local
Windowing system distributor 'Apple', version 10.3.2022
System Description:  macOS 11.2.3

Recent messages:
Tramp: Sending command ‘exec ssh -l dgreene   -e none dgreene-dev’
Tramp: Waiting for prompts from remote shell...
Entering debugger...
o is undefined
m is undefined
tramp-signal-hook-function: No previous stack frame
Compilation finished
Saved text from "Debugger entered--entering a function:
*"
Quit: "" [2 times]

Configured using:
 'configure --disable-dependency-tracking --disable-silent-rules
 --enable-locallisppath=/opt/homebrew/share/emacs/site-lisp
 --infodir=/opt/homebrew/Cellar/emacs-plus@27/27.2/share/info/emacs
 --prefix=/opt/homebrew/Cellar/emacs-plus@27/27.2 --with-xml2
 --with-gnutls --without-dbus --with-imagemagick --with-modules
 --with-rsvg --with-ns --disable-ns-self-contained'

Configured features:
RSVG IMAGEMAGICK GLIB NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS XIM NS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: InactiveMinibuffer

Minor modes in effect:
  recentf-mode: t
  ivy-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  display-time-mode: t
  global-so-long-mode: t
  global-company-mode: t
  volatile-highlights-mode: t
  global-hl-todo-mode: t
  electric-pair-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  global-whitespace-mode: t
  whitespace-mode: t
  override-global-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
/Users/dag/.emacs.d/straight/build/boxquote/boxquote hides 
/Users/dag/lib/dotfiles/emacs/lisp/boxquote
/Users/dag/.emacs.d/straight/build/cmake-mode/cmake-mode hides 
/opt/homebrew/share/emacs/site-lisp/cmake/cmake-mode
/Users/dag/.emacs.d/straight/build/password-store/password-store hides 
/opt/homebrew/share/emacs/site-lisp/pass/password-store
/Users/dag/lib/dotfiles/emacs/lisp/completion hides 
/opt/homebrew/Cellar/emacs-plus@27/27.2/share/emacs/27.2/lisp/completion
/Users/dag/.emacs.d/straight/build/auth-source-pass/auth-source-pass hides 
/opt/homebrew/Cellar/emacs-plus@27/27.2/share/emacs/27.2/lisp/auth-source-pass
/Users/dag/lib/dotfiles/emacs/lisp/terminal hides 
/opt/homebrew/Cellar/emacs-plus@27/27.2/share/emacs/27.2/lisp/obsolete/terminal

Features:
(shadow sort mailalias bbdb-message mail-extr emacsbug help-fns
radix-tree cl-print tramp-cmds noutline outline debug backtrace recentf
tree-widget magit-bookmark bookmark swiper ivy delsel ivy-faces
ivy-overlay colir magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func magit-diff smerge-mode diff git-commit log-edit pcvs-util
add-log magit-core magit-autorevert autorevert filenotify magit-margin
magit-transient magit-process magit-mode transient magit-git
magit-section magit-utils tramp-cache vc-git hober2-theme
rainbow-delimiters rainbow-mode llvm-mode tablegen-mode time so-long
quite quite-autoloads smtpmail-multi smtpmail-multi-autoloads smtpmail
sendmail exec-path-from-shell exec-path-from-shell-autoloads vlf-setup
vlf-autoloads magit-autoloads git-commit-autoloads transient-autoloads
company-graphviz-dot graphviz-dot-mode graphviz-dot-mode-autoloads hydra
lv hydra-autoloads lv-autoloads company-oddmuse company-keywords
company-etags company-gtags company-dabbrev-code company-dabbrev
company-files company-clang company-capf company-cmake company-semantic
company-template company-bbdb company company-autoloads swiper-autoloads
ivy-autoloads use-package-bind-key rainbow-mode-autoloads
rainbow-delimiters-autoloads paredit paredit-autoloads etags-select
etags-select-autoloads cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons all-the-icons-autoloads
boxquote rect boxquote-autoloads pass imenu password-store
auth-source-pass with-editor pass-autoloads password-store-otp-autoloads
password-store-autoloads auth-source-pass-autoloads
with-editor-autoloads cmake-font-lock-autoloads cmake-mode rx rst
cmake-mode-autoloads ldap tramp-term tramp-term-autoloads advice
volatile-highlights volatile-highlights-autoloads hl-todo
hl-todo-autoloads markdown-mode-autoloads elisp-slime-nav etags fileloop
generator xref project elisp-slime-nav-autoloads ielm pp dired-x
elec-pair paren notmuch notmuch-tree notmuch-jump notmuch-hello
notmuch-show notmuch-print notmuch-crypto notmuch-mua notmuch-message
notmuch-draft notmuch-maildir-fcc notmuch-address notmuch-company
notmuch-parser notmuch-wash diff-mode coolj notmuch-query goto-addr
thingatpt icalendar diary-lib diary-loaddefs cal-menu calendar
cal-loaddefs notmuch-tag notmuch-lib notmuch-compat hl-line
notmuch-autoloads vterm face-remap color find-func vterm-module
term/xterm xterm vterm-autoloads eterm-256color pcase f dash s term
disp-table ehelp xterm-color eterm-256color-autoloads f-autoloads
dash-autoloads s-autoloads xterm-color-autoloads gnus-demon nntp
gnus-registry registry eieio-base gnus-art mm-uu mml2015 mm-view
mml-smime smime dig gnus-sum url url-proxy url-privacy url-expand
url-methods url-history mailcap shr url-cookie url-domsuf url-util svg
xml dom gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-win gnus-int gnus-range
message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 gmm-utils
mailheader gnus nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045
ietf-drums text-property-search mail-utils mm-util mail-prsvr bbdb-mua
bbdb-com crm mailabbrev bbdb bbdb-site timezone bbdb-autoloads
eudcb-bbdb eudc eudc-vars edmacro kmacro tramp-sh cus-edit cus-start
cus-load wid-edit tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat shell pcomplete parse-time iso8601 time-date
ls-lisp format-spec whitespace finder-inf use-package-diminish
use-package-core server bind-key easy-mmode compile comint ansi-color
ring package browse-url url-handlers url-parse auth-source eieio
eieio-core eieio-loaddefs password-cache json map url-vars
use-package-autoloads bind-key-autoloads straight-autoloads info
cl-extra help-mode easymenu seq byte-opt straight subr-x bytecomp
byte-compile cconv cl-seq cl-macs cl gv cl-loaddefs cl-lib tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win
ns-win ucs-normalize mule-util term/common-win tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue cocoa ns
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 560656 69667)
 (symbols 48 35896 1)
 (strings 32 131508 11154)
 (string-bytes 1 8722894)
 (vectors 16 58823)
 (vector-slots 8 1093862 94648)
 (floats 8 823 387)
 (intervals 56 725406 8696)
 (buffers 1000 29))

tramp-debug.txt
Description: TRAMP debug output