Re: How to decide what to put in the lexer and the grammar respectively?

[Hans Åberg]

> On 17 Feb 2019, at 23:10, Peng Yu <address@hidden> wrote:
> 
> This lexical tie-in creates feedback from the parser to the lexer. So
> the lexer cannot be tested standalone.
> 
> But the principle of separating lexer and parser is to make parser
> builtin on top of the parser. Is there something that can avoid the
> feedback yet still allow context-dependent parsing? Alternatively, how
> to just testing the lexer without having to get the parser involved?

The LARL(1) that Bison uses is for context free grammars only, so contexts must 
involve switches somehow. Think of a definition which changes a name into an 
identifier, implemented by putting it on a (typically stacked) lookup table, 
which the lexer checks, and returns its token value to the parser. A grammar 
definition alone would require an attribute grammar.