bug#10132: Help lilypond interleave scheme and lilypond code in guile 2.x

[David Kastrup]

Andy Wingo <address@hidden> writes:

> Hi David,
>
> This bug was forked from bug 10099, where David has a longer
> explanation.
>
> On Fri 25 Nov 2011 11:37, David Kastrup <address@hidden> writes:
>
>> So much for that.  The next quote is for a totally different issue, the
>> availability of local environments and evaluation in them.  Lilypond has
>> an input syntax of its own, and it allows interspersing Scheme code. $
>> or # switches to the Scheme interpreter (for one sexp) when in Lilypond
>> syntax, and #{ ... #} switches to Lilypond inside.
>
> Aaah.  Thanks for this explanation; I had never seen this code before.
>
> Do you use a read-hash-extend reader for #{#} ?

Yes:

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/scm/parser-ly-from-scheme.scm>

This link is likely a bit of a moving target since I have several
patches in the queue to make the behavior more predictable (and produce
better quality error messages when things go wrong).

> What do you use to parse the lilypond code?  What does it parse to?

Classical Bison/Flex parser/scanner.  There is no "what does it parse
to" since the Bison rules execute the actions on the fly: it is a
classical interpreter.  With a number of lexical and semantical tie-ins,
it would be non-trivial to actually create an intermediate
representation.

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/lexer.ll>

The file responsible for reading and evaluating embedded Scheme
expressions is

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parse-scm.cc>

The proposed robustifying changes currently in the queue are in
<URL:http://codereview.appspot.com/5437055>.

The reading is done by the lexer.  Evaluation of $ forms is done in the
lexer, evaluation of # forms is delayed and happens in the parser in
order to avoid timing problems (lookahead tokens should preferably not
be evaluated while they are still lookahead since they might depend on
the actions of the commands before them).

> I agree that the-environment and local-eval were nice solutions for
> this.  In Guile 2.0 it's not as nice for you, because if you implement
> another evaluator, you don't get backtraces that are as nice.

We don't really do anything in the line of backtraces.  Until the
proposed patch gets through, we don't really do anything sensible in the
line of error messages either.

>> As I said: for this particular application, I have coded a rather
>> inelegant and resource-grabbing workaround that really is not going
>> to help performance since the intertwined Lilypond interpreter does
>> not benefit from precompilation of mostly trivial lambda functions
>> when the actual procedure-environment is unlikely to ever reference
>> more than five variables.
>
> Understood.  Let's work to find a good solution in 2.0.

If you follow the history of parser-ly-from-scheme.scm, you'll see that
there has been a flurry of activity recently.  Before I had to cater for
GuileV2, the code just sweeped up the procedure-environment of a
basically empty lambda function and left all rereading and evaluation to
runtime.  Even earlier than that, there was a complicated interpretation
of # and $ where the respective expressions were all evaluated in a
let-form at runtime before the Lilypond parser went to work.

The current approach of wrapping everything in lambda as compared to the
historic implementation of evaluating everything Scheme before starting
the Lilypond interpreter has the advantage that the timing of evaluation
(if any) is determined by the Lilypond interpreter.

The procedure-environment approach was elegant and minimally complex.
The question is how feasible it is for the Guile compiler to capture an
environment in a form that can be used even after compilation.  Like
taking the address of a variable in C, the export of such an environment
interferes with a number of static optimizations.  For our particular
application, readonly access to the symbols in the environment should be
quite sufficient, but of course I can't vouch for other potential uses.

-- 
David Kastrup