Re: Dynamic token kinds

[Frank Heckenbach]

Hans Åberg wrote:

> > On 30 Nov 2018, at 00:40, Frank Heckenbach <address@hidden> wrote:
> > 
> > Hans Åberg wrote:
> > 
> >>> Sure, though for my taste "unsafe" sounds a bit harsh, perhaps
> >>> "unchecked"? If you put in the next release, I'll change my code to
> >>> use it.
> >> 
> >> It seems pretty standard to have lookup tokens with different
> >> syntactic behavior, for example when they are declared of
> >> different type in a language. So it is worrisome that the typed
> >> parser deems the use unsafe.
> > 
> > What is potentially unsafe is that the actual type may not match the
> > declared type in the grammar. With std::variant, a mismatch would
> > cause an exception to be thrown. With Bison's static variant, a
> > mismatch might lead to UB.
> > 
> > So perhaps this function could actually do a type check (which
> > probably requires another auto-generated switch) and also throw or
> > (if this is not desired) call std::terminate or so on mismatch,
> > Akim?
> 
> The C++17 std::variant stores the type as an index. So perhaps
> there should be and additional table storing the type, with a
> symbol constructor that constructs the right value from the token.

Unlike std::variant, Bison's variant does not store the type at all.
So the type check I suggested isn't actually that easy. By the time
the symbol_type constructor is called, the variant value has already
been constructed and type information lost.

So to make it safe, we might need something like this:

  static inline
  symbol_type
  make_symbol (token_type type, b4_locations_if([const location_type& l, ])T&& 
v);

auto-generated for each semantic type T of any token (plus one
without the "v" parameter for untyped tokens) that checks (at
runtime) the "type" parameter against the (statically known) valid
token types for T.

Regards,
Frank