bug#10099: Null (begin) blocks - V2.0.3 reports error was OK in V2.0.2

[David Kastrup]

Ian Hulin <address@hidden> writes:

> Hi Andy,
> On 23/11/11 12:27, Andy Wingo wrote:
>> Hi Ian,
>> 
>> Did it really work in v2.0.2, I wonder?
>> 
> Apparently not, see below for explanation.

This mail addresses two different issues, mixing them in somewhat
surprising ways.  I'll try to split the issues again.

>> On Tue 22 Nov 2011 00:02, Andy Wingo <address@hidden> writes:
>>> On Mon 21 Nov 2011 18:25, Ian Hulin <address@hidden> writes:
>>>> (define-public (void? x) (eq? x (begin))) This works in V1.8,
>>>> and apparently used to work in 2.0.2 (no errors),
>>> 
>>> Interesting.  If it changed incompatibly in 2.0.x, that is a
>>> Guile bug. Sorry about that!  We'll fix it.
>> 
>> I cannot reproduce this issue with 2.0.2.  I suspect that this code
>> has been this way since early 1.9 releases, and is a consequence of
>> switching to the new syntax expander.
>> 
> Looks like the change was already in 2.0.2. Another LilyPond developer
> (David Kastrup, address@hidden) noticed the advertised API and decided to
> use it in part of an ongoing clean-up of the Lily parser.  Meanwhile I
> was working on the Guile V2 migration stuff and it broke.

More concretely regarding "the advertised API": the Guilev1.8
documentation does neither mention `*unspecified*', nor `unspecified?'.
On the C side, it documents SCM_UNSPECIFIED.  On the Scheme side, the
only available information is the documentation for begin:

 -- syntax: begin expr1 expr2 ...
     The expression(s) are evaluated in left-to-right order and the
     value of the last expression is returned as the value of the
     `begin'-expression.  This expression type is used when the
     expressions before the last one are evaluated for their side
     effects.

     Guile also allows the expression `(begin)', a `begin' with no
     sub-expressions.  Such an expression returns the `unspecified'
     value.

So it is somewhat disingenuous to laugh off such uses of begin.  At the
very least you should add documentation for *unspecified* and
unspecified? to the documentation of the 1.8 branch.  That way, you at
least can point people to documentation relevant for the migration.

There _are_ a few problems even in Guilev1.8: v1.8 refuses to accept
some forms that should likely be *unspecified*:
guile> (lambda ())

Backtrace:
In standard input:
   1: 0* (lambda ())

standard input:1:1: In procedure memoization in expression (lambda ()):
standard input:1:1: In file "standard input", line 0: Missing expression in 
(lambda ()).
ABORT: (syntax-error)
guile> ((lambda () (begin)))

Backtrace:
In standard input:
   4: 0* [#<procedure #f ()>]

standard input:4:1: In procedure memoization in expression ((lambda () #)):
standard input:4:1: Missing body expression in ((begin)).
ABORT: (syntax-error)
guile> ((lambda () (begin (begin))))

Backtrace:
In standard input:
   6: 0* [#<procedure #f ()>]

standard input:6:1: In procedure memoization in expression ((lambda () #)):
standard input:6:1: Missing body expression in ((begin (begin))).
ABORT: (syntax-error)
guile> ((lambda () #f (begin)))
guile> 

I have switched to using *unspecified* and unspecified? by now, but it
might be worth ironing out the strange behaviors of (begin) nevertheless
in particular where they conflict with the behavior to be expected from
the documentation (preferably both in v1.8 as well as v2), in addition
to mentioning the more straightforward commands.

So much for that.  The next quote is for a totally different issue, the
availability of local environments and evaluation in them.  Lilypond has
an input syntax of its own, and it allows interspersing Scheme code. $
or # switches to the Scheme interpreter (for one sexp) when in Lilypond
syntax, and #{ ... #} switches to Lilypond inside.

We have constructs like

spacingTweaks =
#(define-music-function (parser location parameters) (list?)
   (_i "Set the system stretch, by reading the 'system-stretch property of
the `parameters' assoc list.")
   #{
     \overrideProperty #"Score.NonMusicalPaperColumn"
     #'line-break-system-details
     #(list (cons 'alignment-extra-space (cdr (assoc 'system-stretch 
parameters)))
             (cons 'system-Y-extent (cdr (assoc 'system-Y-extent parameters))))
   #})

As you can see, the music function argument "parameters" is used in
Scheme code embedded in Lilypond code.  Previously, #{ (implemented in
the Scheme reader) captured the enclosed material at read time as a
string for interpretation (using the Scheme reader to skip over any
#sexp or $sexp inside) as well as (procedure-environment (lambda ()
'())), then called the Lilypond parser at execution time, and the parser
executed all embedded Scheme in the procedure-environment captured in
the lexical scope of the define-music-function text.

My _current_ implementation can't capture an execution environment, so I
need to precompile a whole host of closures, and later correlate them
with input source positions.  This is somewhat error-prone since the
full parsing of Lilypond might not agree with the cursory scan of the #{
... #} body.  Since the parser and lexer are context-dependent and can
switch behavior according to variables at execution time, a more
complete parse at compile time is not feasible.

To give an example, the above #{ ... #} nowadays gets translated into

(let* ((clone (ly:parser-clone parser (list (lambda () 
"Score.NonMusicalPaperColumn") (lambda () (quote line-break-system-details)) 
(lambda () (list (cons (quote alignment-extra-space) (cdr (assoc (quote 
system-stretch) parameters))) (cons (quote system-Y-extent) (cdr (assoc (quote 
system-Y-extent) parameters)))))))) (result (ly:parse-string-expression clone "
     \\overrideProperty #\"Score.NonMusicalPaperColumn\"
     #(quote line-break-system-details)
     #(list (cons (quote alignment-extra-space) (cdr (assoc (quote 
system-stretch) parameters))) (cons (quote system-Y-extent) (cdr (assoc (quote 
system-Y-extent) parameters))))
   "))) (if (ly:parser-has-error? clone) (ly:parser-error parser (_ "error in 
#{ ... #}"))) result)

Namely, all contained # expressions are precompiled into
(list (lambda () "Score.NonMusicalPaperColumn")
      (lambda () (quote line-break-system-details))
      (lambda () (list (cons (quote alignment-extra-space)
                             (cdr (assoc (quote system-stretch)
                                         parameters)))
                       (cons (quote system-Y-extent)
                             (cdr (assoc (quote system-Y-extent)
                                         parameters))))))

Previously, all that was needed to record instead was

(procedure-environment (lambda () '()))

(again note the useless '() to bypass the issue of complaints about a
missing body) and just use that for feeding local-eval inside of
ly:parse-string-expression.

This results in quite heavy storage requirements and will cause problem
if the full parsing does not use the same # $ starting points as the
preliminary scanning, like when # or $ are inside of Lilypond comments:
they will get turned into lambdas at compile time, but not regarded at
execution time.

I have a patch in the queue for Lilypond that also records the starting
positions in the string, and skips "obvious" constants, only wrapping
symbols and pairs with a car different from 'quote into lambdas.  That
reduces the stored amounts of data and improves reliability, at the cost
of considerable complexity.

> I'm sure David will let me know if I've over-simplified stuff or got
> things wrong here.
>
> However, it boils down to lousy timing for LilyPond finding a use for
> this bit of the Guile API, which you guys had decided to withdraw
> because no-one apparently had a use for it :-{

The use would be in implementing embedded lexical environments not
exclusively written in Scheme.  The thrust of Guilev2 to completely
abandon support of interpreted and nested environments in favor of
getting speedups for compiled applications casts doubt on its usefulness
as an _extension_ language, and that was the principal aim
distinguishing it from compiled Scheme variants like Chicken which, I
would guess, have enough of a headstart compared to Guile that
abandoning Guile's original goals and trying to compete with the
established compilers on their home ground is not likely to increase the
core audience of Guile.

As I said: for this particular application, I have coded a rather
inelegant and resource-grabbing workaround that really is not going to
help performance since the intertwined Lilypond interpreter does not
benefit from precompilation of mostly trivial lambda functions when the
actual procedure-environment is unlikely to ever reference more than
five variables.

So for smooth integration into an interpretative environment, this is a
definite step for the worse.  It was perhaps a reasonably easy decision
to make since the original functionality was heavily underused, likely
also due to abysmally bad documentation that provides not much more than
the motivation to hunt through source code and search the web for other
people who actually managed to make something work.

Please feel free to split this twofold mail into separate parts and
answer in appropriate lists.  I apologize for not having a clue what
belongs where myself.

-- 
David Kastrup