Re: Dynamic token kinds

[Hans Åberg]

> On 16 Dec 2018, at 10:02, Frank Heckenbach <address@hidden> wrote:
> 
> Hans Åberg wrote:
> 
>>> On 30 Nov 2018, at 00:40, Frank Heckenbach <address@hidden> wrote:
>>> 
>>> Hans Åberg wrote:
>>> 
>>>> It seems pretty standard to have lookup tokens with different
>>>> syntactic behavior, for example when they are declared of
>>>> different type in a language. So it is worrisome that the typed
>>>> parser deems the use unsafe.
>>> 
>>> What is potentially unsafe is that the actual type may not match the
>>> declared type in the grammar. With std::variant, a mismatch would
>>> cause an exception to be thrown. With Bison's static variant, a
>>> mismatch might lead to UB.
>>> 
>>> So perhaps this function could actually do a type check (which
>>> probably requires another auto-generated switch) and also throw or
>>> (if this is not desired) call std::terminate or so on mismatch,
>>> Akim?
>> 
>> The C++17 std::variant stores the type as an index. So perhaps
>> there should be and additional table storing the type, with a
>> symbol constructor that constructs the right value from the token.
> 
> Unlike std::variant, Bison's variant does not store the type at all.
> So the type check I suggested isn't actually that easy. By the time
> the symbol_type constructor is called, the variant value has already
> been constructed and type information lost.
> 
> So to make it safe, we might need something like this:
> 
>  static inline
>  symbol_type
>  make_symbol (token_type type, b4_locations_if([const location_type& l, ])T&& 
> v);
> 
> auto-generated for each semantic type T of any token (plus one
> without the "v" parameter for untyped tokens) that checks (at
> runtime) the "type" parameter against the (statically known) valid
> token types for T.

I felt that one should just return the token value, in part because otherwise 
the lookup data must be changed, and in part because the type is internal to 
the parser. Then the constructor must be able to add the type, and so arrived 
at the suggestion of an additional table, for use with std::variant.

Perhaps an entirely static approach can be achieved by the type being a part of 
the token_type. On the other hand, the use is for dynamic token lookup, so it 
may be too much of an effort for that.

Personally, I am already at C++17, so I have no reason for using a simpler 
variant. Having the type stored just adds a size_t, and that is used a lot in 
other circumstances, so no overhead to worry about.