Re: Finding out when a token is consumed

[Hans Aberg]

At 14:45 +0200 2003/05/09, Frank Heckenbach wrote:
>> The unusual is that we do not get to know if this language you are working
>> with has a given grammar or whether it just has the ability to throw in
>> directives anywhere, as you claim. In the latter case, the hope diminishes
>> rapidly.
>
>The latter. Of course, as I said, one can also describe this by a
>grammar, even a context-free one, but it seems an ugly way to do
>it.

My own hunch is that one should try to work with respect to the grammar
directly, and then working out from there trying to simplify whenever
possible. Then you will see what will work and what will not. (See other
suggestion below, though.) If one is using context switches, it will depend
on the parsing algorithm when a lookahead is used or not, or so I think. So
one is down to something more special.

>> The difference here is that these are some localize, well defined, features
>> which can be handled by special tricks.
>
>Localized WRT the grammar rules, not necessarily WRT the input
>(since expressions can be arbitrarily long).

Right. If you have input related directives, then it is hard to get the
Bison parser to work with that.

One idea that strikes me with your example, though, is that you might stamp
your lexemes with types in the lexer already. In your example, you had
    a+b#+c
where # is a directive that takes effect on lexemes after the #. You then
stamp it say as semantics, so that the sequence that the lexer produces
will be
  lexeme  type/effect  semantics
    a     NUMBER       value of a
    +     "+"
    b     NUMBER       value of b
    #     set #
    +     "+"
    c     NUMBER       value of #c
When your "+" action combines the value of a+b with the value of #c, it
should recognize that it is a #c and not a c.

This should then work.

> And since you mentioned
>alternative parsing methods in your previous mail -- it will fail,
>e.g., in a GLR parser.

I do not think of GLR and such in this context. Bison with its M4 approach
now has the capability to more easily be adapted to other parsing
algorithms than LALR(1), which might be LR(1), DR(1) or whatever people
have the effort to write for.

>I'm not designing the language, it was given to me. I'm just trying
>various ways to implement it.

What language is it, and what is it used for (if it is not well known)?

  Hans Aberg