[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

symbol_type::token () removed?

From: Wolfgang Thaller
Subject: symbol_type::token () removed?
Date: Wed, 01 Jan 2020 23:51:44 +0100
User-agent: Evolution 3.34.1-2

Commit 8c87a623 says:

> c++: get rid of symbol_type::token ()
> It is not used.  And its implementation was wrong when api.token.raw
> was defined, as it was still mapping to the external token numbers,
> instead of the internal ones.  Besides it was provided only when
> api.token.constructor is defined, yet always declared.
> * data/skeletons/c++.m4 (by_type::token): Remove, useless.

Well, I disagree. It's not useless, I'm using it, and I think my use
case is legitimate.

Low priority, I have an easy workaround.

My parser uses the C++ parser template with

%define api.token.constructor
%define api.value.type variant

I've got a hand-written lexer which returns symbol_type.
And because I'm a good boy, I have unit tests for my lexer. And in my
unit tests, I call my lexer on small bits of input, and then I need to
inspect the resulting symbol_type and make sure the right kind of token
is returned:

So I have things like:

        Lexer lex("12345");
        symbol_type t = lex.nextToken();
        assert(t.token() == token::INTLIT);
        assert(t.value.as<int>() == 12345);

... which suddenly stopped working.

Fortunately, I can instead do...

        assert(t.type_get() == by_type(token::INTLIT).type_get());

... but that's a bit ugly.

In general, I think symbol_type should always provide enough public API
in order to allow inspecting it. In my case, I only need it for
implementing tests, but it general it might be useful when writing the
lexer, as well.

So I request to reinstate symbol_type::token().



P.S.: Keep up the good work, bison is my favorite way of parsing things
using C++....

reply via email to

[Prev in Thread] Current Thread [Next in Thread]