bug#44307: 27.1; UTF-8 parts transferred as 8bit in multipart messages fail to decode

[Alexandre Duret-Lutz]

Alexandre Duret-Lutz <adl@lrde.epita.fr> writes:
> Clicking inside this message on the "Attachement: [2. text/plain]"
> button inserts "\344\344\344\344".   I.e., that's
> the Latin-1 version of "ääää".  (M-x describe-char on these say that they
> are "not encodable by coding system utf-8-unix")

Digging the code, I believe that the unexpected conversion occurs in this macro:

(defmacro mm-with-part (handle &rest forms)
  "Run FORMS in the temp buffer containing the contents of HANDLE."
  ;; The handle-buffer's content is a sequence of bytes, not a sequence of
  ;; chars, so the buffer should be unibyte.  It may happen that the
  ;; handle-buffer is multibyte for some reason, in which case now is a good
  ;; time to adjust it, since we know at this point that it should
  ;; be unibyte.
  `(let* ((handle ,handle))
     (when (and (mm-handle-buffer handle)
                (buffer-name (mm-handle-buffer handle)))
       (with-temp-buffer
         (mm-disable-multibyte)
         (insert-buffer-substring (mm-handle-buffer handle))
         (mm-decode-content-transfer-encoding
          (mm-handle-encoding handle)
          (mm-handle-media-type handle))
         ,@forms))))


In my case the (mm-handle-buffer handle) is multibyte.  This
multibyteness was preserved by mm-copy-to-buffer while creating the
handle buffer, but a did not check the original source of it, since the
comment above the macro suggests that having multibyte parts is OK.

However the 

         (mm-disable-multibyte)
         (insert-buffer-substring (mm-handle-buffer handle))

seems to be doing harm.  The documentation of
insert-buffer-substring/insert notes that multibyte strings will be
converted by taking the lowest 8 bits of each multibyte character, not
by spliting those characters.

Mimicking it with

(let ((utf8string "ääää")) ; typed as utf8
  (with-temp-buffer
    (mm-disable-multibyte)
    (insert utf8string)
    (print (string-bytes utf8string))
    (print (string-bytes (buffer-string)))
    (buffer-string)))

this prints :

8
4
"\344\344\344\344"


So it would seem that (mm-disable-multibyte) should be called *after* the
insertion and not before, in order to perserve all bytes.

Does this make sense?

-- 
Alexandre Duret-Lutz