[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Bison/Flex interface generation for re-entrant parsers is broken

From: Eric S. Raymond
Subject: Bison/Flex interface generation for re-entrant parsers is broken
Date: Thu, 9 Oct 2014 06:47:06 -0400
User-agent: Mutt/1.5.21 (2010-09-15)

The combination of the options %reentrant and %bison-bridge with
%lex-param fails to generate correct code.  The files included with
this report demonstrate the problem and constitute the smallest
possible test case for it.

While the test case uses vacuous grammars, it preserves the essential
features of a real case in the program cvs-fast-export.  That program
implements a parser for CVS master files which must be re-entrant in
order to allow the most compute-intensive phase of its analysis to be

Environment: bison 3.0.2, flex 2.5.35.  OS is Ubuntu 14.04 and
compiler is gcc 4.8.2 but I am pretty sure the bug will reproduce

The main bug is in bison, but this also concerns flex interfacing and
exposes a flex annoyance that should be fixed. Accordingly, I'm
copying it to the flex development list.

To begin, drop the include files (gram.y, lex.l, yyscan.h, and
Makefile) into an empty directory. Run make and observe successful
code generation and compilation, as in this session transcript.

bison --defines=gram.h --output-file=gram.c -d gram.y
flex --header-file=lex.h --outfile=lex.c lex.l
cc -DYY_DECL="int yylex (YYSTYPE * yylval_param , yyscan_t yyscanner, char 
*cvs)"   -c -o gram.o gram.c
cc -DYY_DECL="int yylex (YYSTYPE * yylval_param , yyscan_t yyscanner, char 
*cvs)"   -c -o lex.o lex.c

Now comment out the CFLAGS line in Makefile and try again:

make -k
cc    -c -o gram.o gram.c
gram.c: In function ‘yyparse’:
gram.c:1120:7: error: too many arguments to function ‘yylex’
       yychar = yylex (&yylval, scanner, cvsfile);
In file included from gram.y:4:0:
lex.h:326:12: note: declared here
 extern int yylex \
make: *** [gram.o] Error 1
cc    -c -o lex.o lex.c

The failure occurred because, having been given this:

%lex-param {yyscan_t *scanner} {char *cvsfile}

bison is generating a call to yylex() correctly corresponding to it.
In lex.c and lex.h, meanwhile, this piece of boilerplate code defines
the actual signature of yylex():

/* Default declaration of generated scanner - a define so the user can
 * easily add parameters.
#ifndef YY_DECL
#define YY_DECL_IS_OURS 1

extern int yylex \
               (YYSTYPE * yylval_param ,yyscan_t yyscanner);

#define YY_DECL int yylex \
               (YYSTYPE * yylval_param , yyscan_t yyscanner)
#endif /* !YY_DECL */

Observe that while the %reentrant option of flex has correctly caused the
prototype to be generated with YYSTYPE* and yyscan_t arguments, the 
additional char *cvsfile argument specified by %lex-params is not there.

The macro definition

-DYY_DECL="int yylex (YYSTYPE * yylval_param , yyscan_t yyscanner, char *cvs)

works around this problem by specifying the correct prototype manually.

To fufill the promises implied by its documentation, bison should generate 
a YY_DECL from its %lex-params and %parse-params options and include
that definition in its header file where the scanner generated code
will see it.

I also believe that in this case flex should generate lex.h to include

typedef void *yyscan_t;

It is odd and in my opinion broken that flex generates a prototype that uses
this type without including the definition before it.
                <a href="http://www.catb.org/~esr/";>Eric S. Raymond</a>

Attachment: Makefile
Description: Text document

Attachment: gram.y
Description: Text document

Attachment: lex.l
Description: Text document

Attachment: yyscan.h
Description: Text Data

reply via email to

[Prev in Thread] Current Thread [Next in Thread]