Re: Finding out when a token is consumed

[Hans Aberg]

At 04:51 +0200 2003/05/10, Frank Heckenbach wrote:
>> We are told that there is no specific given grammar for those extra
>> directives, but they can be thrown in just anywhere. So how could those
>> methods work then?
>
>I hope I can clear up this confusion at least. If directives can be
>thrown in anywhere (between tokens), there still is a grammar: Just
>change the original grammar to accept directives before each
>terminal. This grammar is still context-free. However, when I tried
>this in the obvious way, I got R/R conflicts (even with the simple
>example), and I didn't see an easy way to remove them. I'm not even
>sure if the grammar is still LALR(1) or even LR(1) or whatever.
>
>So there *is* a grammar, but it might be quite unpractical (because
>it might not allow for efficient parsing and isn't written in the
>most natural form).

The problem is that you do not provide sufficiently detailed language
information that it can be resolved from where I stand. Perhaps you knock
on an increased lookahead, perhaps there is another problem. If the needed
lookahead increase, LALR(1)/LR(1) and such will leave you out in the cold.

Can directives be followed in sequence without any other tokens? If so, you
may have arbitrary lookahead, which cannot be even handled by LR(k) for any
k. Then the lexer must read all the directives and somehow register them.
Then one lands on the suggestion in my other letter, with the closures.

  Hans Aberg