Re: master ef14acf: Make nnml handle invalid non-ASCII headers more consistently

[Florian Weimer]

* Eli Zaretskii:

>> From: Florian Weimer <fweimer@redhat.com>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>, ding@gnus.org
>> Date: Fri, 16 Dec 2022 23:42:21 +0100
>> 
>> * Lars Ingebrigtsen:
>> 
>> > branch: master
>> > commit ef14acfb68bb5b0ce42221e9681b93562f8085eb
>> > Author: Lars Ingebrigtsen <larsi@gnus.org>
>> > Commit: Lars Ingebrigtsen <larsi@gnus.org>
>> >
>> >     Make nnml handle invalid non-ASCII headers more consistently
>> >     
>> >     * lisp/gnus/nnml.el (nnml--encode-headers): New function to
>> >     RFC2047-encode invalid Subject/From headers (bug#45925).  This
>> >     will make them be displayed more consistently in the Summary
>> >     buffer (but still "wrong" sometimes, since there's not that much
>> >     we can guess at at this stage, charset wise).
>> >     (nnml-parse-head): Use it.
>> > ---
>> >  lisp/gnus/nnml.el | 16 ++++++++++++++++
>> >  1 file changed, 16 insertions(+)
>> >
>> > diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
>> > index ebececa..3cdfc74 100644
>> > --- a/lisp/gnus/nnml.el
>> > +++ b/lisp/gnus/nnml.el
>> > @@ -769,8 +769,24 @@ article number.  This function is called narrowed to 
>> > an article."
>> >        (let ((headers (nnheader-parse-head t)))
>> >    (setf (mail-header-chars  headers) chars)
>> >    (setf (mail-header-number headers) number)
>> > +  ;; If there's non-ASCII raw characters in the data,
>> > +  ;; RFC2047-encode them to avoid having arbitrary data in the
>> > +  ;; .overview file.
>> > +  (nnml--encode-headers headers)
>> >    headers))))
>> 
>> Unfortunately, this change in particular causes Gnus to stops storing
>> messages into nnmail after receiving a message with this header:
>> 
>> From: =?utf-8?b?572X5YuH5YiaKFlvbmdnYW5nIEx1bykgdmlhIEVsZnV0aWxzLWRldmVs?=
>>  <elfutils-devel@sourceware.org>
>> 
>> The logged error message is:
>> 
>> Mail source (maildir :path …) failed: (error Invalid data for rfc2047 
>> encoding: 罗勇刚(Yonggang Luo) via Elfutils-devel 
>> <elfutils-devel@sourceware.org>)
>> 
>> On an older Emacs without this change, it seems that the original header
>> is written to the .overview file, which sidestep the problem that not
>> all strings are encodable by the rfc2047 functions.
>
> Thanks.  I guess this From header is invalid because there's no space
> between the "罗勇刚" and the "(Yonggang Luo)" parts?

Yes, that seems to be what's tripping the encoder.  But I'm not sure if
proper encoding of ( or ) (as =28 or =29 using the Q encoding, or using
the B encoding as in the raw text) is actually invalid.  RFC 2047 only
talks about unencoded ( or ).  In contrast, encoded ( and ) are valid
syntax at the RFC 822 layer because encoding hides them.

> Does the naïve patch below solve the problem?
>
> diff --git a/lisp/gnus/nnml.el b/lisp/gnus/nnml.el
> index 40e4b9e..7aa445e 100644
> --- a/lisp/gnus/nnml.el
> +++ b/lisp/gnus/nnml.el
> @@ -776,17 +776,22 @@ nnml-parse-head
>       (nnml--encode-headers headers)
>       headers))))
>  
> +;; RFC2047-encode Subject and From, but leave invalid headers unencoded.
>  (defun nnml--encode-headers (headers)
>    (let ((subject (mail-header-subject headers))
>       (rfc2047-encoding-type 'mime))
>      (unless (string-match "\\`[[:ascii:]]*\\'" subject)
> -      (setf (mail-header-subject headers)
> -         (mail-encode-encoded-word-string subject t))))
> +      (let ((encoded-subject
> +             (ignore-errors (mail-encode-encoded-word-string subject t))))
> +        (if encoded-subject
> +            (setf (mail-header-subject headers) encoded-subject)))))
>    (let ((from (mail-header-from headers))
>       (rfc2047-encoding-type 'address-mime))
>      (unless (string-match "\\`[[:ascii:]]*\\'" from)
> -      (setf (mail-header-from headers)
> -         (rfc2047-encode-string from t)))))
> +      (let ((encoded-from
> +             (ignore-errors (rfc2047-encode-string from t))))
> +        (if encoded-from
> +            (setf (mail-header-from headers) encoded-from))))))
>  
>  (defun nnml-get-nov-buffer (group &optional incrementalp)
>    (let ((buffer (gnus-get-buffer-create

Thanks!

I somehow can't reproduce the original issue.  I expect more problematic
messages to arrive next week, though, and will report then how it goes.

Florian