[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
should we deprecate %token-table? (was: Re: [bug-bison] Bug in string-va
Joel E. Denny
should we deprecate %token-table? (was: Re: [bug-bison] Bug in string-valued terminals)
Sun, 9 Jan 2011 10:18:45 -0500 (EST)
Alpine 2.00 (DEB 1167 2008-08-23)
On Fri, 31 Dec 2010, twlevo wrote:
> > On Thu, 30 Dec 2010 18:20:09 -0500 (EST), Joel E. Denny wrote:
> > > Oldest found doc about a option --token-table and %token-table is
> > > in gcc-1.22 or gcc-1.24 in bison.info.2 or .4 without a note how to use
> > > it.
> > Could you explain a little more about this?
> No, Not in GCC but only about GNU BISON and --token-table option
> is first found in bison-1.25 may 1996,
Ok, thanks for clarifying.
> maybe there are maillist
> archives from 1995 / 1996 with discussions about this option.
I wish. Those years don't exist in the GNU archives, and a quick google
search didn't find them elsewhere.
> checked the oldbison archive for token-table documentation at
> There is also a patched bison version A2.3 and A2.6 (Andrew)
> which does say it is based on bison-1.22 and bison-1.24 but
So A2.3 and A2.6 have the token table functionality too?
> bison-1.25 is the first GNU official version with --token-table.
> The -k, --token-table commandline option depends on a
> variable toknumflag in bison-1.25 and is implemented:
> Tue Aug 1 12:30:38 EDT 1995 Wilfred J. Hansen <address@hidden>
> * bison.cld, getargs.c, vmsgetargs.c: Added -n, -k, and -raw switches.
> (noparserflag, toknumflag, rawtoknumflag): New variables.
Ah, thanks, that helped. I missed this when I searched ChangeLog before.
Here's what I found....
yytname and yytranslate exist in 1.24. However, there appears to be no
directive to control whether they are generated, and they are not
mentioned in the manual.
yytranslate is still not mentioned in the manual in 1.25, and I see no
evidence to suggest it's meant to be associated somehow with yytname
(other than the fact that it can be useful for yytname).
yytoknum and %token_table are both introduced in 1.25. There, yytname and
yytoknum are generated unconditionally if %token_table (or -k) is
specified. That is, there's no dependence on YYPRINT as there is in
recent releases. %token_table and yytname are documented in the manual.
Unfortunately, yytoknum is not documented in the manual, but a comment in
The parser tables consist of these tables.
Starred ones needed only for the semantic parser.
Double starred are output only if switches are set.
yytranslate = vector mapping yylex's token numbers into bison's token numbers.
** yytname = vector of string-names indexed by bison token number
** yytoknum = vector of yylex token numbers corresponding to entries in
Finally, I don't see any code that accesses yytoknum in the 1.25 source.
Thus, I'm assuming that it's not useful internally and that it's meant
solely for Bison users.
My conclusion is that the original developers meant for users to use
yytname with yytoknum even though they forgot to document yytoknum. Of
course, it's possible that they meant to allow users to choose either
yytranslate or yytoknum, but so far there's little evidence to suggest
In any case, yytname has changed since 1.25 in a way that the original
developers may not have intended. The 1.25 manual says this about yytname
All the characters of the literal string token appear verbatim in the
string found in the table; even double-quote characters are not escaped.
The 2.4.3 manual (I didn't check other versions) says:
The name in the table includes all the characters needed to represent
the token in Bison. For single-character literals and literal strings,
this includes the surrounding quoting characters and any escape
Both manuals provide examples that make the escaping differences clear.
I see two possible ways forward:
1. We document yytoknum, yytranslate, or both. If we choose to document
yytoknum, we make sure that it's generated when %token-table is specified
regardless of whether YYPRINT is defined. We assume users do not care
about the escaped characters. Has anyone ever complained?
2. Because neither yytoknum nor yytranslate was originally documented, we
assume that %token-table was originally a broken feature. Moreover, it
may be broken further now that escaping has been introduced into the
yytname entries. Thus, we document %token-table and yytname as deprecated
features and make Bison warn when %token-table is specified. We leave
yytoknum and yytranslate alone. Of course, like all deprecated features,
we won't remove %token-table until it really starts to get in the way of
> Maybe Wilfred Hansen can remember about it and how it was intended to use.
I hate to bother him with a personal email if he no longer follows Bison,
but I'd be glad to hear his opinion if he's listening.
- should we deprecate %token-table? (was: Re: [bug-bison] Bug in string-valued terminals),
Joel E. Denny <=