[Chicken-hackers] read/write invariance of #!

[Jim Ursetto]

In Chicken 5, read/write invariance of symbols beginning with #! is violated. 

> (with-input-from-string

    (with-output-to-string (lambda () (write (string->symbol "#!abc")))) read)

Error: invalid `#!' token: “abc"

> (with-input-from-string

    (with-output-to-string (lambda () (write (string->symbol "#!")))) read)

Error: invalid `#!' token: “"

This causes a problem with chicken-doc, as there is a syntax entry called “#!” which is written out but cannot be read back in. In Chicken 4, this symbol is escaped when written out and reads back in fine. 

This occurs because in Chicken 5, sym-is-readable? was changed:

commit 05f341e07a179ea95d891e5c55b2fe3d0dbe1ffe

Author: Evan Hanson <address@hidden>

Date:   Wed Mar 14 17:53:00 2018 +1300

    Print #!-style symbols verbatim, without pipes

    These symbols are readable, so should be printed as-is by `##sys#print'

    just like #:keywords or the #!eof token.

-                                ((and (eq? c #\#)

-                                      (not (eq? #\% (##core#inline "C_subchar" str 1))))

-                                 #f)

+                                ((eq? c #\#) ;; #!rest, #!key etc

+                                 (eq? (##core#inline "C_subchar" str 1) #\!))

Unfortunately this assertion is incorrect: not all #! style symbols are readable, only valid #! style symbols such as #!rest. Other symbols are rejected by the reader. Furthermore, certain strange combinations, such as “(#! (foo bar))” result in an unterminated list, no matter how many close parens are added. The latter is the issue I am hitting.

The fix is probably to explicitly test for valid tokens, which appear to be only: “optional”, “rest” and “key”. Probably “eof” is ok as well—not 100% sure. The reader, of course, already tests for these exact tokens, so the writer should too.

Alas, chicken-doc won’t work with Chicken 5 until this is fixed — I worked around the file locking bug in 5.0.0, but the writer bug is fatal.

Jim