bug#50983: 28.0.50; [REGRESSION, BUG] Display bugs with uncommon characters

[Alan Third]

On Sun, Oct 03, 2021 at 03:02:20PM +0300, Eli Zaretskii wrote:
> Can you show the output of "C-x =" on all the characters, one by one,
> starting from "n" in "note" and ending with "t" in "taking" after it?
> Are they all incorrect, i.e. do not correspond to the place the cursor
> is on?  That is, does the corruption start around there or does it
> start much earlier (and if the latter, where does it start)?

n -> i
o -> t
t -> h
e -> SPC
- -> n
_ -> o
t -> t

So it's off-set by some 4 characters.

Looking at the raw file, there are 4 0xAD (SOFT HYPHEN) characters
before "note", and after each one the offset increases by one.

I do not see them displayed in the terminal.

> > I think I agree with your other email that it's down to the terminal
> > doing something strange with characters it doesn't understand.
> 
> If this is the case, the only way to fix the display is to use
> us-ascii as terminal encoding.  Or maybe set up the terminal for a
> "simpler" encoding, like latin-1, and then set up Emacs to that using
> set-terminal-coding-system.

Indeed, changing the "character encoding" setting in iTerm to ASCII
displays the soft-hyphens as a red "A" and everything seems to work
right.

The default is UTF-8 and it reports itself as "xterm-256color". I
suspect most terminal applications on macOS will default to UTF-8
since that's the default everywhere else, which might help explain why
this seems to be limited to macOS.
-- 
Alan Third