yylineno

[Bill Fenlason]

My application needs to keep track of both line number and line offset for
tokens.  I know that the use of %option yylineno adds a performance penalty
(documented in the man page).  When it is specified, code is included that
rescans each token to count the \n characters, and that certainly adds
overhead.  I'd like to find out if others have handled this problem in a
different (hopefully better) way than I have.

It seemed to me that if the rescan is to be avoided, the globals for line
number and offset (yylineno and yylineoffset in my case) must be maintained
by each rule that contains a \n.   Also, yylineoffset must be maintained by
every rule for every token, not just those that are returned.  I preferred
not to add code to every rule for this, but adding a bit of code in rules
with \n is not a problem.  This is the piece of code that I chose to be
executed between tokens to maintain the line number and offset:

if (nl_flag) {nl_flag = yylineoffset = 0; ++yylineno;} else yylineoffset +=
yyleng;
/* nl_flag is set by rules ending with \n */

I considered using YY_BREAK and YY_USER_ACTION for this, but neither seemed
to work.  YY_BREAK code is not executed if a return is done, and
YY_USER_ACTION is done after yyleng has been set up for the current token.
And both of them generate redundant code for each rule.

The YY_DO_BEFORE_ACTION macro (which sets up yytext and yyleng after a match
is made) cannot be overridden, and that seemed to be the logical place to
execute the code.

Finally I added a YY_USER_BEFORE_ACTION macro to the skeleton just before
the YY_DO_BEFORE_ACTION macro.  This approach works as far as I can tell.  I
do not use either "input" or "unput" so I did not deal with that part of the
problem.

I think that something along these lines, or some other solution, could be
added to FLEX to improve upon the current yylineno implementation.

If anyone has thoughts, experiences or recommendations regarding the
yylineno and offset issue, I hope you will share them.

Thanks.

Bill Fenlason