Re: Flex vs. POSIX 1003.2-1992 repeat operator {} precedence

[Bruce Lilly]

Casey Leedom wrote:
[...]

First, I'd like to thank Casey for bringing the issue to notice.

Second, everything that Casey says sounds reasonable and well-thought-out.

On to the questions:

>   The questions are:
> 
>  1. Should we do anything at all about this?  We could simply correct the
>     mis-statement in the manual page and note that flex and lex use
>     different repeat operator precedence and lex is POSIX-conformant while
>     flex is not.

There are two issues:

1. lex/flex compatibility
2. POSIX conformance

flex has a -l lex compatibility option.  The manual has an "INCOMPATIBILITIES
WITH LEX AND POSIX" section.  This specific issue is mentioned there, albeit
with the incorrect interpretation of POSIX.  It appears that Vern's intent was
to conform to POSIX, giving priority to POSIX conformance in cases where lex
and POSIX diverge.

If flex is modified to conform to the correct POSIX interpretation for
repetition, it will also be compatible with lex.  And the part in the
aforementioned section of the manual addressing repetition can simply
be elided.

Fixing the bug has benefits (POSIX conformance and lex compatibility)
and is easily documented (by eliding text).  The alternative means
that flex has to be documented as not being POSIX conformant, it would
remain incompatible (in this respect) with lex, even with -l, and
the documentation would have to be revised to provide the correct
interpretation of POSIX and some rationale for non-conformance and
incompatibility.  Sounds like a no-brainer.

>  2. If we change flex's repeat operator precedence to be conformant with
>     POSIX, do we want to offer any compatibility capability for the old
>     non-POSIX-conformant and flex-unique repeat operator precedence?  Note
>     that it's a simple matter of adding parenthesis to a lexer specification
>     to get any precedence you desire.

Certainly the change should be clearly announced in the distribution. This
change by itself probably wouldn't warrant a major version increment (to 3.x),
but in conjunction with some of the other changes being incorporated, such
an increment may be warranted.  That ought to prompt users to check for
compatibility issues w.r.t. earlier releases.

Given the facts that:

1. the current implementation (contrary to the man page) is NOT
   POSIX-conformant
2. the current implementation is not compatible with lex
3. changing the behavior will make flex simultaneously POSIX-conformant
   AND lex-compatible (w.r.t. this issue)
4. parenthesization can be used to force the old (broken) flex behavior
5. such parenthesization will work with modified flex and with lex

there's probably no need to introduce yet another option to support
what is essentially a bug.