Re: RIP: c++: merge symbol_type and token

[Akim Demaille]

> Le 23 déc. 2018 à 12:16, Frank Heckenbach <address@hidden> a écrit :
> 
> Akim Demaille wrote:
> 
> I see. So struct token only exists to put the yytokentype values in
> their own namespace to avoid those clashes. An enum class would do
> the same trick (and provide for stricter type-checking), but of
> course requires C++11.

Actually, it would not help either.  We don't need enum class,
embedding the enum in a class gives exactly the same scoping
behavior.  So if I were to define

struct token
{
  enum class token_type { foo, bar };
};

the users would have to write yy::parser::token::token_type::foo,
which is a breaking change, and is too long imho.  The only
real alternative would be to rename all the routines of `token`
as yy_ something.  Including `token` itself.  Then provide a type
alias for backward compatibility.  That should work.



>>>> - In the signature of make_symbol, I've had to use int for the
>>>> token type, instead of the enum token_type, and then
>>>> to convert the int into token_type.  I don't like that, but I
>>>> probably don't have much of a choice.  (A template would be
>>>> overkill IMHO).
>>> 
>>> Why, is it because of char tokens (like '+' in your example)?
>> 
>> Yes, exactly.  I don't like that we accept ASCII, but we have
>> to.
> 
> I see your point, but as a user of Bison I like not having to
> declare tokens for syntactic fluff like ',' and '.'.

I understand this from the user side point of view.

> Would it help if Bison (internally) generates enum values for the
> chars used, such as "TOK_46 = '.'" (with some non-conflicting
> prefix), so token_type would cover all valid values and could be
> used as the parameter type? (This might also be required for the
> stricter type-checking of an enum class.)

Yes, we can do that.  However, I don't think there's a real
value for it: the user cannot guess the name, she would have
to read the code, and it's of course fragile, that number can
change easily.

The only alternative I see would be to teach Bison the name
of these characters: '+' => PLUS, etc.  That would allow to
generate not only a better enum, but also the token constructors
(make_PLUS).