Re: Finding out when a token is consumed

[Hans Aberg]

At 20:38 +0200 2003/05/08, Frank Heckenbach wrote:
>The actual case is quite complicated, so I'll try to give a strongly
>simplified example. Let's assume arithmetic expressions, say only
>containing `+', `*' and parentheses.
>
>But there can also be "directives" (here denoted just `#', while in
>the actual case there can be several kinds of directives) which have
>some global effects, e.g., change the number of fractional digits
>used in computations or whatever. The exact effect is immaterial
>here; what's important is that they can occur at any point in the
>input and should affect all computations done after this point (and
>of course none that happens before this point). ("Done" in the sense
>where the last tokens used in that computation appears.)

Your description of something that can be inserted just anywhere goes
against the idea of a language ruled by a grammar. This is in part the
reason why this stuff you are trying to do is so difficult to achieve with
Bison, as it is completely built up around the idea of a grammar to be
parsed. The general tendency is that Bison should become more like that,
because then one can change parsing algorithms.

So one becomes curious why you are facing this situation: Have you decided
it should be so, or who has invented this language you try to parse?

>As a first attempt, I tried to handle them during lexing. But that's
>too early when bison reads look-ahead tokens. E.g., when the input
>is `1+2#+3', bison will read the second `+' before it can do the
>first addition, so the directive will affect the first addition
>which it shouldn't do.

Unless you try integrate this somehow into the grammar, such things are
very difficult to handle with Bison as the generated parser (as you say)
may or may not make use of the lookahead. The problem here is that you say
that this stuff can be inserted anywhere, which is essentially saying that
you do not have a language ruled by a grammar.

You might experiment with grammars like:

%left '+'
%left '*'

%%

expr:
    expr '+' expr
  | expr '*' expr
  | number
;

number:
    NUMBER
  | directive NUMBER
  | NUMBER directive
  | directive NUMBER directive
;

directive:
    '#'
   | ...
;

Here you have control over when your directives enter the scene and should
be able to control its effects via the actions.

But you still have to give the insertion of these directives a grammatic
structure to get it working.

  Hans Aberg