Re: Dynamic token kinds

[Hans Åberg]

> On 17 Dec 2018, at 10:48, Frank Heckenbach <address@hidden> wrote:
> 
> Hans Åberg wrote:
> 
>>> On 16 Dec 2018, at 15:48, Frank Heckenbach <address@hidden> wrote:
>>> 
>>> Hans Åberg wrote:
>>> 
>>>> The idea would be that rather than returning the token value,
>>>> something that does not need translation. I don't know if that
>>>> helps up the static approach, though.
>>> 
>>> Not sure what you mean here. Please elaborate.
>> 
>> I couldn't see the details when I looked at it. I don't use the typed 
>> parser, but might if it is safe.
> 
> The parser is type safe. This is only about an alternative way of
> creating tokens by the lexer, which is also type safe when used
> properly (as mine does). It's only about adding an additional safety
> net.

Right.

>>> I think Akim made it clear enough that he doesn't like the overhead.
>>> (I don't mind so much, as I used std::variant in my implementation,
>>> but I only have a few parsers to care about.)
>> 
>> In that case, my impression was that he thought he could do without it.
> 
> Well, he can. :)

In that case.

>>> One might validly say that preventing it is the job of the lexer
>>> (and my lexer does so), not Bison's, just like other kinds of
>>> undefined or wrong behaviour outside of the generated code, also
>>> dynamic token types are a somewhat special usage anyway, so Bison
>>> can just do nothing about it, that's fine.
>> 
>> I use the same thing, returning the token value found on a lookup
>> table, but I would not want to use the typed parser if I would
>> have to add translations for every possibility. The information
>> about it is in Bison, therefore it should not be put on the
>> writing of the lexer.
> 
> I think we agree here, and that was actually my concern when I
> started this thread. I don't want to have to write a separate case
> for each token kind in my lexer. Of course, we need a separate case
> for each semantic type because that involves a different type in the
> constructor/builder call already, but these are relatively few,
> compared to token kinds, in my lexers.

Might Bison generate a function with a switch statement, generate the right 
return for the lexer to use?

>>> I also suggested an approach in my previous mail with a few more
>>> generated functions that help runtime checking. I'd prefer Bison to
>>> add them, and then we'd have runtime checking as good as we'd have
>>> with std::variant in this respect.
>> 
>> Maybe an option. Akim perhaps haven't used this dynamic token
>> lookup.
> 
> I guess he hasn't. But I don't think we need an option. These would
> just be additional functions that one can use or not.

The with an option would be that those that do not need this feature could use 
a more optimal variant.

>> Those that do might prefer not risking the program to bomb.
> 
> It's not that bad actually. Again, my lexers work fine as is.
> I just brought this up because Akim proposed to call the function
> "unsafe_..." which I thought was too harsh and proposed
> "unchecked_..." -- but by adding the checks, it would be neither
> unsafe nor unchecked. :)

This worries me. But also having having to use something more complex to be 
returned by the lexer than a value on the lookup table .