bug#20623: XML and HTML files with encoding/charset="utf-8" declaration loose BOM; Coding system is reset from utf-8-with-signature to utf-8 on save

[Simon Ledergerber]

Hello Eli

I have done some more research to answer your questions. You will find 
the details of my statement at the end of this mail.

On 22.05.2015 09:11, Eli Zaretskii wrote:
> [Please don't remove the bug address from the CC list, so that this
> discussion is recorded in the bug data base.]
>
> > Date: Thu, 21 May 2015 22:49:47 +0200
> > From: Simon Ledergerber <sledergerber@gmx.net>
> >
> >   From the documentation I understand that utf-8 is without BOM and
> > utf-8-with-signature is with BOM. Maybe I am wrong and should rather
> > understand that utf-8 is auto-detect. But then there is something like
> > utf-8-without-signature missing to specify explicitly that no BOM is
> > desired.
> >
> > In my opinion, it is correct when Emacs prefers utf-8 over
> > utf-8-with-signature when it opens a file without BOM that can still be
> > recognized as UTF-8.
> >
> > However when a file is opened with a BOM already present, it should
> > stick to the utf-8-with-signature coding system, because the BOM "EF BB
> > BF" unambiguously marks the file as UTF-8. (For UTF-16 for example,
> > there is a different BOM byte pattern. There are other coding systems
> > which do not have a BOM at all.)
> What do you mean by "stick to"?  When I try visiting an XML file that
> is encoded with BOM, Emacs decodes the file correctly, and the value
> of buffer-file-coding-system is utf-8-with-signature.  Isn't that what
> you want?  If that's what you want, but it doesn't happen for you,
> please try in "emacs -Q".  It's possible that the default you set:
>
>    (setq-default buffer-file-coding-system 'utf-8-dos)
>
> is the reason for what you see.  (I don't understand why you need such
> a default, and it sounds like a bad idea to me.)
You're right. When I open a file that was really saved with BOM, Emacs 
detects its encoding correctly, i. e. utf-8-with-signature-dos. But when 
I change the content and save with C-x C-s, the encoding changes to 
utf-8-dos and the BOM gets lost. Even when I start Emacs with -Q. This 
is the actual bug.
> > By doing C-x <RET> f and then saving it with C-x C-s, I expect to be
> > able to change the coding system.  For example, if I specify utf-8-dos,
> > the BOM should be removed, if one was present, and CR LF should be
> > inserted for EOL. On the other side, if I choose
> > utf-8-with-signature-unix, a BOM should be written and LF be taken for
> > EOL. (The conversion between DOS and Unix works, just the BOM is the
> > problem.)
> >
> > I have found a link, where this topic was already discussed, but it
> > didn't help me further:
> > http://superuser.com/questions/41254/make-emacs-not-remove-the-bom-from-xml-files
> >
> > In that post Vebjorn Ljosa asked exactly the question I have. Richard
> > Hoskins replies with the answer to change the coding system with C-x
> > <RET> r utf-8-with-signature. Unfortunately, it didn't work for me -
> > after doing a change in the file and saving, it got back to utf-8
> > automatically - that's why I have filed the bug.
> That's not how you force a file to be saved in a specific encoding.
> You should do this instead:
>
>    C-x RET c utf-8-with-signature RET C-x C-s
>
> The "C-x RET c" prefix forces the next Emacs operation to use the
> specified encoding.  In this case, Emacs will ask for confirmation,
> because the encoding you specified is different from what the XML
> comment says.
This is true and it worked for me. Please see below for further 
explanations.

Summary:
- C-x RET c utf-8-with-signature RET C-x C-s is a good workaround, 
because it really forces the file being written with BOM. In order to 
have an effect however, the file must be dirty, i. e. there must be a 
pending change. But before the command completes in this case, the 
prompt "Selected encoding utf-8-with-signature-dos disagrees with 
utf-8-dos specified by file contents.  Really save (else edit coding 
cookies and try again)? (yes or no)" appears. I think this is what you 
mean with your sentence: "In this case, Emacs will ask for confirmation, 
because the encoding you specified is different from what the XML 
comment says."

- But consider the following: The encoding in the XML declaration or in 
the HTML <meta charset="utf-8"> just specifies UTF-8 (or another 
encoding). It doesn't say anything about the presence or absence of the 
BOM. Therefore an editor detecting and deciding about the file's 
encoding should not rely on this information only.

- When such a file, which was saved successfully with BOM, is closed and 
reopened again, Emacs detects its encoding correctly, say 
utf-8-with-signature-dos.

- However, when I change the file content and save it again just with 
C-x C-s (without C-x RET c ... first!), then it changes back to 
utf-8-dos. Yes, even if I start emacs with -Q! (That's the point.)

- I do not fully understand the criterion for and the magic behind how 
Emacs chooses the file encoding when I do C-x C-s. But I was able to 
reproduce it several times by applying the procedures given in the bug 
report, even when -Q is on. As we already have stated above, this could 
be because Emacs favors (and forces) utf-8 whenever it sees something 
like XML or HTML that might be UTF-8-encoded.

-> Conclusion: C-x RET c utf-8-with-signature RET C-x C-s is a good way 
to force the file being written as I want. But what I still do not 
understand: When I open a file with BOM and Emacs recognizes that, why 
does it change the encoding silently to drop the BOM when I regularly 
save with C-x C-s - and this even without giving me a notice or warning?