The starting point is that it might be good to move to using GLR for
our parser. For instance, it will allow to get rid of the id_colon
trick. But GLR, since it desynchronizes the parser from the scanner,
forbids so-called lexical tie-ins, in other words, the parser and the
scanner can no longer try to shared some evolving variables.
Currently the translation of special symbols such as $$, @2 etc. is
done by the scanner. To do this the scanner needs to know the length
of the rule. It knows it thanks to the id_colon trick. Therefore if
we want to get rid of it, the scanner can no longer perform this
transformation.
I must confess that I was quite pleased with this, since I'm not fond
of modules that do many things at the same time; here the scanner
scans, protects characters from M4, translates user code, computes
rule lengthes etc. Since I would like to enable new means to
designate semantic values (using names instead of $1 etc.), moving
that
part elsewhere seems to provide some fresh air. I also cherish the
possibility to see the values flowing in a "compiler", and allowing a
means to see the action after scanning but before conversion seemed
nice to me. It would certainly also help moving from M4 since the
reader is no longer concerned with it.
So the following patch does just this: extract the action processing
parts from the scanner. But to where? My first desire was to have
another pass, even after parsing, because I'm fond of simple
multi-passes compilers where the scanner and the parser are just
concerned with reading, transformations belong to latter passes. But
I finally chose to do that in the parser, since it required less
changes (for a start?).