bug#46933: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos

[Eli Zaretskii]

> From: handa <handa@gnu.org>
> Cc: gregory@heytings.org, 46933@debbugs.gnu.org
> Date: Sat, 27 Mar 2021 14:38:56 +0900
> 
> In article <83ft0obk7i.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Kenichi, why are these 6 bytes inserted by encode-coding-region, but
> > not when we encode the same text as part of saving the buffer to its
> > file?  And why does it happen near the end of the text, between those
> > 2 particular letters?
> 
> There surely exists a bug.  Could you please try the attached patch?
> 
> The reason why that bug did not happen on file writing is that the code
> in write_region calls encoding routine repeatedly without
> CODING_MODE_LAST_BLOCK flag, and only in the case that flushing is
> required (e.g. the case of iso-2022-jp), just for flushing, it calls
> enoding routine again with CODING_MODE_LAST_BLOCK flag.  In that case,
> carryover does not happen in encode_coding ().

Thanks.  The patch fixes the problem with the extra 6 bytes, so I
installed it.

The results of filepos-to-bufferpos with the file attached by Gregory
are better now, but there are still problems for some values of BYTE
argument.  The problem is that ISO-2022 encoding (and others like it)
include shift-in and shift-out sequences, used to switch between
character sets.  As a trivial example, each CR+LF sequence has the
"ESC ( B" sequence before it and "ESC $ B" sequence after it, to
switch to ASCII before the newline, then switch to Japanese after it.
And likewise whenever there's Latin text within Japanese (there are
quite a lot of them in this particular file).  These shift-in and
shift-out sequences consume bytes, but don't produce any characters.
So if the BYTE argument of filepos-to-bufferpos specifies a byte in
the middle of one of these shift sequences, the result will be
incorrect, because decoding a partial sequence produces the bytes of
that sequence verbatim, and the logic in filepos-to-bufferpos of using
the length of the decoded text breaks.  We need special handling of
this and other similar coding-systems to fix these corner use cases,
similarly to what we do in filepos-to-bufferpos--dos.  Patches
welcome.

I'm leaving this bug open because not all of the problem was fixed.