bug#44486: 27.1; C-@ chars corrupt elisp buffer

[Stefan Monnier]

>> >> Actually, for prefer-utf-8 files, I think we never want to automatically
>> >> fallback to binary.
>> > I think you are assuming prefer-utf-8 is something other than what it
>> > is.  It is not a variant of UTF-8, it is a variant of 'undecided'
>> > (i.e. it starts by detecting the encoding), which prefers UTF-8 if
>> > that can decode the text.
>> My position is not based on principles but on pragmatic concerns.
>> AFAIK `prefer-utf-8` is only ever used for files which are known to
>> contain text and should almost always contain UTF-8 text.
> For those, we should use utf-8, not prefer-utf-8.

No, `utf-8` should be used when other coding systems should be
considered as errors (i.e. not "almost always" but "always"), whereas
`prefer-utf-8` is for use when utf-8 is the most likely one and other
coding systems should be tried only when there's some evidence that the
file actually doesn't use utf-8.

`prefer-utf-8` was introduced specifically for `.el` files (and I don't
know of any other use of that encoding so far).  If `utf-8` is
preferable over `prefer-utf-8` for this usage I think the problem is in
`prefer-utf-8` since it was introduced specifically for that.

>> I believe if there's a NUL byte in such a files but it otherwise doesn't
>> contain any invalid UTF-8 byte sequence, it will result in better
>> behavior if we treat it as UFT-8 than as binary.
> We treat null bytes as the _single_ telltale sign of a binary file.

A .el file should *never* be a binary file.

> If we disable that in coding-systems that are supposed to _detect_
> encoding, we will never be able to detect binary files.

In which scenario would it be beneficial to detect a `.el` file as being
binary instead of utf-8?


        Stefan


PS: Especially since NUL bytes can and do occur in ELisp code.