Re: lexer support?

[Kaz Kylheku]

On 2024-02-29 09:07, Lex Spoon wrote:
> Greetings, Bison developers,
> 
> What would the sentiment be to include lexer support in core Bison?

It has "lexer support". You can write a yylex() function that just returns
character codes, and then write Bison grammar rules to recognize tokens.

The grammar language isn't very expressive for that (e.g. for repeat
matching similar to a regex Kleene star, you have to write recursive
rules, and alternation is likewise clumsy). But it has the power.

(If you want the separation of a grammar based on tokens which have
a lexical grammar, you can do that with two Bison parsers. I've not
heard of that being done, but I can't think of any road blocks
against it.)

(Some things you can do in Lex are going to be difficult, namely
those that are made possible by start conditions. You can't easily
put a Bison-generated parser into different states in which
different rules are active or dormant. A yylex function could
switch between calling several different Bison-generated parsers
based on a state variable, though.)

> %%tokens
> 
> DIVIDE: "/"
> 
> LPAREN: "("
> 
> MINUS: "-"
> 
> NL: "\n"
> 
> NUM: [0-9]+ ("." [0-9]+)?

This can all be done in Bison if you make DIVIDE, LPAREN, and the others,
into nonterminal symbols.

  num : digits frac_opt ;

  fract_opt : '.' digits | /*nothing*/ ;

  digits : digit | digits digit ;

  digit : '0' | '1' | '2' | ... | '9' ;

To convince yourself, try to make a version of the calculator program
in which yylex isn't much more than just { return getchar(); }