Re: Improving the handling of system data (env, users, paths, ...)

[Mike Gran]

On Sunday, July 7, 2024 at 08:58:34 AM PDT, Eli Zaretskii <eliz@gnu.org> wrote: 
>> I don't think it's worth it. If anybody's going to work on this problem,
>> I'd recommend simply adding APIs like program-arguments-bytevector,
>> getenv-bytevector and the like, returning raw bytevectors instead of strings,
>> and letting programs which need to be reliable against invalid UTF-8
>> in the environment use these.
> 
>> That is also the approach taken in, e.g., Rust (except that due to the
>> static typing, you are forced to handle the "invalid UTF-8" error case
>> when you use, e.g., std::env::args as opposed to std::env::args_os).

> The Emacs experience shows that (rare) raw bytes as part of otherwise
> completely valid text are a fact of life.  They happen all the time,
> for whatever reasons.  Granted, those reasons are most probably
> something misconfigured somewhere, but as long as that happens in a
> program other than the one you are developing, or even on another
> computer, the ability of the user, let alone the programmer, to fix
> the whole world is, how shall I put it, somewhat limited.  The
> question is what do you when this stuff happens, and how you prepare
> your package for dealing with it as well as reasonably possible?

To halfway follow Emacs's lead, Guild could use some of Unicode's
Private Use Area characters to represent raw bytes.

Raw bytes 0x00 to 0xFF could map to U+100000 to U+1000FF, for example.

We could make an encoding option such that
when converting from locale to internal Guile representation fails,
raw bytes could be transcoded thus for storage, and when outputting that string 
as a locale string
those characters can be output as raw bytes.  When outputting that string as 
UTF-8,
they can remain as PUA characters or be converted to the U+FFFD Replacement 
Character.

It would make corner cases: what if you use more than one non-UTF locale,
what if you actually wanted to use SMP PUA characters... 
And it would not be memory efficient; however, it would be
simple enough given Guile internals.

Regards,
Mike