flex: unput(char) not valid outside body of grammar

[Paul Eggert]

This email is following up to a Debian bug report about flex
<http://bugs.debian.org/189332>, which reports that flex test version
2.5.31 breaks Bison 1.875.

> From: Manoj Srivastava <address@hidden>
> Date: Date: Wed, 16 Apr 2003 17:40:13 -0500

> This is a new feature of flex. It cleans up after it self; there are
> a number of assets present in the rules section that are not present
> outside it. This is helpful since flex nmow no longer pollutes the
> user namespace as much, and helps in allowing for multiple scanners,
> with potentialy different options, in the same program.

This change means that flex no longer conforms to POSIX 1003.1-2001.
The POSIX specification for lex
<http://www.opengroup.org/onlinepubs/007904975/utilities/lex.html>
says that the functions yylex, yymore, yyless, input, and unput are
all accessible to user code included in the lex input.  As far as I
can tell, POSIX imposes no restriction that the functions are
accessible only to user actions.

The change to flex may be necessary for reentrant scanners, but it
shouldn't be necessary for traditional scanners such as those
specified by POSIX.


> Please consider the following modification to the flex file
> you submitted in th patch

I tried doing something like that for Bison (along the lines suggested
by Peter Muir in
<http://mail.gnu.org/archive/html/bison-patches/2003-04/msg00002.html>),
but this had other problems: it called yyunput before it was declared,
and this violates C99.

I shall rework Bison so that it doesn't use unput at all.  This will
avoid the portability hassles, at the cost of making the Bison scanner
a bit less elegant.  However, I suggest that the flex maintainers fix
the incompatibility with POSIX, as people will be using flex to build
older versions of Bison for quite some time.  Also, people will be
using flex to build other scanners that may assume the POSIX-required
behavior.