Re: How to decide what to put in the lexer and the grammar respectively?

[Peng Yu]

This lexical tie-in creates feedback from the parser to the lexer. So
the lexer cannot be tested standalone.

But the principle of separating lexer and parser is to make parser
builtin on top of the parser. Is there something that can avoid the
feedback yet still allow context-dependent parsing? Alternatively, how
to just testing the lexer without having to get the parser involved?

> There is one reason that could have driven to this architecture:
> the fact that the lexical structure in strings is very different.
> There are no comments, spaces matter, etc.  So that's a case
> where the parser state controls the type of scanning you need,
> which is called lexical tie-in in Bison's documentation.  It's
> doable, but not very elegant.

-- 
Regards,
Peng