Re: [AUCTeX] files with non-ascii characters and LaTeX as in TeXLive 2018

[Ikumi Keita]

Hi all,

> (1) The output of \message{...} also changed in TeXLive 2018.  If
> non-ascii characters are in \message{...}, they are output in tokenized
> form by default in TL 2018.  This makes partial typset, such as C-c C-r,
> C-c C-s and so on, to fail to recognize the right file name when source
> correlate mode is enabeld, so I inserted another \detokenize{} in
> `TeX-quote-filename'.

It turned out that my patch doesn't work as expected in some cases.
When `TeX-quote-filename' inserts "\string" to escape "#" and "~" before
\detokenize{} takes place, wrapping the file name inside \detokenize{}
neutralizes the "\string" so that it appears as literal 7-character
"\string", not escaping "#" and "~".  In addition, \detokenize{} doubles
the pound sign "#" for some reason unknown to me.

> (1) According to texdoc etex, \detokenize was added in e-TeX extension.
> So if the user has quite old TeX distribution where e-TeX extension was
> not incorporated in the engine (command binary) yet, \detokenize raises
> error.  I hope this is a permissible incompatibility.

I realized that the default command binary "tex" for plain-TeX does not
incorporate e-TeX extension, while the binaries of latex families
("latex", "pdflatex", "lualatex", ...) do.  (The e-TeX enabled binary
producing dvi file for plain-TeX is "etex".)  We only need \detokenize
for latex family because it is LaTeX kernel that the internal UTF-8-nize
was performed on and other families including plain-TeX are not
affected.

In addition, I heard that Omega does not have e-TeX extension, either.
If this is the case, "lambda" is an exception in latex families and does
not accept \detokenize.  (I cannot confirm because Omega binaries no
longer exist in TeX Live these days.)

Summarizing these considerations, it seems that we have to do something
like this:
(defun TeX-quote-filename (file)
  ... (snip) ...
  (if (and (eq major-mode 'latex-mode)
           (not (and (eq TeX-engine 'omega)
                     ;; lamed, Aleph version of lambda, has e-TeX extension.
                     (equal LaTeX-Omega-command "lambda"))))
      (replace-regexp-in-string "[[:multibyte:]]+"
                                "\\\\detokenize{\\&}" file t)
    file))

Some notes.
(a) Should we treat `doctex-mode' in the same way as `latex-mode' in
this case?  I'm not sure since I don't know docTeX.
(b) Note that preview-latex does not work with "lambda" even if
`TeX-quote-filename' is modified as above.  I hope this is not a major
drawback because very few users still use Omega now, IIUC.

> N.B.  The initial release of TeXLive 2018 lacks the file ".tex" [1],
> which makes preview-latex to fail to dump format file when preamble
> caching is enabled.  Running `tlmgr update --all' doesn't recover
> ".tex" at the time of writing this paragraph, thus it seems that the
> latest TL repositry still lacks ".tex".

Now TL repository has ".tex".  Running `tlmgr update --all' will restore
".tex" in the texmf tree of TL 2018.

Regards,
Ikumi Keita