Re: Emacs Lisp's future

[Florian Weimer]

* Mark H. Weaver:

> To give an example, consider a procedure that needs to pass a string
> from an untrusted source to an SQL query.  To do this safely, it needs
> to quote the string.  I haven't researched how to properly quote SQL
> string literals, but in general, quoting is typically done by
> recognizing some set of special characters that must be escaped, and
> allowing all other characters through unmodified.

For are truly robust solution, you need parameterized queries.  Most
database servers support other encodings besides UTF-8, and the
required quoting logic can be quite complicated.

> However, "raw byte" code points can be used to bypass such a quoting
> mechanism, and thus send an unescaped closing quote to the SQL database
> followed by arbitrary SQL commands.

This can happen with certain multi-byte character sets as well.

> UTF-8 decoders are supposed to detect and reject these "overlong"
> encodings, but it is likely that many programs fail to do this.

That's not very common anymore.

> To cope with this, the Unicode standards require that UTF-8 codecs
> reject overlong encodings and other invalid byte sequences.  This is in
> direct conflict with the idea of "raw byte" code points, whose purpose
> is to be tolerant of arbitrary byte sequences and to propagate them
> unchanged.

The charset conversion functionality could support binary-transparent
UTF-8 and pure UTF-8 at output boundaries.  This way, the application
can make a choice.