bug#57531: 28.1; Character encoding missing for "eo"

[Eli Zaretskii]

> Date: Sat, 03 Sep 2022 20:00:41 +0000
> From: Jonathan Reeve <jonathan@jonreeve.com>
> Cc: 57531@debbugs.gnu.org
> 
> > I believe this is due to the fact that the text was saved in UTF-8,
> > and Emacs was trying to decode it as if it were in Latin-3.
> 
> That’s the problem. Emacs should decode UTF-8 as UTF-8, not Latin-3.

It tries, but it isn't always 100% successful.

> > Using the prefer-coding-system customization should fix that.
> 
> The user shouldn’t have to customize an encoding system to have a UTF-8 
> locale be interpreted as UTF-8. A UTF-8 locale should be encoded as such, 
> without needing to be told otherwise.

Ideally, I agree.  But in practice, we've found that goal unreachable
in some cases.

> > I disagree.  I think your system doesn’t tell Emacs enough to guess
> > correctly.
> 
> It does, though. The locale data is already there, which is why I only have 
> this problem in Emacs, and nowhere else on my whole system. The problem is in 
> this line from `locale-language-names'. Here’s what it says:
> 
> `("eo" . "Esperanto")'
> 
> Here’s what it should say:
> 
> `("eo" "Esperanto" utf-8)'

That's only correct for glibc systems, though, as I already explained.
I found no authoritative place on the Internet which would mandate
that the Esperanto locale should use or prefer UTF-8 as its encoding.
Once again, glibc is just one C library on just one OS.

If you can show me some authoritative source of information about this
locale which says it should use UTF-8, that could be a reason good
enough to make such an incompatible change.  And we need good reasons
for such incompatible changes, because some users out there could have
configurations or applications that depend on previous behavior.

> The system says that it’s a UTF-8 locale.

How does it say so?

> > Please try the solution I proposed, and if it doesn’t work, let’s see
> > what else is needed.  If you keep insisting on defaulting Esperanto to
> > UTF-8, I see now way to make any progress here.
> 
> You’re not proposing a solution, you’re proposing a workaround.

Would you please try it nonetheless?

> There’s no reason at all why Esperanto should be encoded in latin-3. It never 
> has been, as far as I can tell, and it never well be, in latin-3, with the eo 
> locale. If you can find any good reason why it should be in latin-3, I’m all 
> ears, but as far as I can tell, this is a mistake.

No, it isn't a mistake.  Latin-3 was introduced to cover Esperanto
(and some other languages).  That's why the Emacs Esperanto locale
was configured to use it.  It wasn't a random choice.

> Please keep in mind that /I’m trying to help improve emacs,/

Please keep in mind that so am I.  For many years, as a matter of
fact.