bug#60750: 29.0.60; encode-coding-char fails for utf-8-auto coding system

[Robert Pluim]

src/emacs -Q
M-x toggle-debug-on-error
M-: (setq buffer-file-coding-system 'utf-8-auto)
C-b
C-u C-x =

=>
Debugger entered--Lisp error: (args-out-of-range "))" 3 1)
  encode-coding-char(41 utf-8-auto ascii)
  describe-char(189)
  what-cursor-position((4))

This is because utf-8-auto has a non-nil :bom property:

(define-coding-system 'utf-8-auto
  "UTF-8 (auto-detect signature (BOM))"
  :coding-type 'utf-8
  :mnemonic ?U
  :charset-list '(unicode)
  :bom '(utf-8-with-signature . utf-8))

and `encode-coding-char' does this:

        ;; We also need to exclude the leading 2 or 3 bytes if they
        ;; come from a BOM.
        (setq i0
              (if bom-p
                  (cond
                   ((eq (coding-system-type coding-system) 'utf-8)
                    3)
                   ((eq (coding-system-type coding-system) 'utf-16)
                    2)
                   (t 0))
                0))
        (substring enc2 i0 i2)))))

Iʼm not sure if this needs fixing, but it was surprising, and the
docstring of `define-coding-system' didnʼt make it clear to me whether
a BOM should have been produced here or not. (Iʼm willing to be told
that buffer-file-coding-system shouldnʼt be 'utf-8-auto, but I never
set that explicitly as far as I know 😀)

Thanks

Robert

In GNU Emacs 29.0.60 (build 14, x86_64-pc-linux-gnu, GTK+ Version
 3.24.24, cairo version 1.16.0) of 2023-01-12 built on rltb
Repository revision: f4f30ff4c44dcfdf780f1981aa541af713f2805f
Repository branch: emacs-29
System Description: Debian GNU/Linux 11 (bullseye)

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB