Re: Token types with constructor

[Hans Aberg]

At 22:03 +0200 2004/08/31, Martin Trautmann wrote:
>I think I understood the main problem. But can you tell me what a bison
>parser may do with tokens? Am I right with:
>1) flex creates the token object (no problem: constructor is called)
>2) bison stores it in a stack *++yyvsp = yylval;
>Both variables are of type YYSTYPE, so operator=() is called

Right.

>3) when a stack overflow occurs: bison copies the stack
>(!!! no copy Constructor / operator= is called !!!)

The C-stack just makes a memcpy(), so under C++, no user copy constructors
are invoked. (Strictly speaking, POD's have a built in, default copy
constructor which both C and C++ invoke.) The get that, one needs to use a
proper C++ stack, such as std::deque, std::vector or something.

>4) bison calls a rule function that gets a stack element as parameter
>This access is done using the correct type and shouldn't do any harm.

There should be no access problems, if only the object exists (which it may
not do, if not copied properly).

>And there is one version of YYCOPY:
>      do                                       \
>       {                                       \
>         register YYSIZE_T yyi;                \
>         for (yyi = 0; yyi < (Count); yyi++)   \
>           (To)[yyi] = (From)[yyi];            \
>       }                                       \
>      while (0)
>That would have no problems using operator=() as well. So the only
>additional thing to do would be to force the usage of that.
>
>But I think in my small parser with the token type I showed you, stack
>overflows didn't occur during my tests.

You have to ensure that it does not. Somebody else will have to reply how
to do that, as I am currently happy with my C++ parser that does that
automatically. Essentially, with the current setup, and if you are using
nontrivial copy constructors with the C-parser, you need to ensure that the
stack is fixed in size, and that the parser aborts when the stack overflows.

>And in my other parser I do a special version of reference counting:
>"object there or not there":
>class Token
>{
>  int type;
>  u{
>    T1* t1;
>    T2* t2;
>       ...
>  };
>  T1& t1();
>  T2& t2();
>       ...
>  Token() : type(no_token) {}
>};
>
>The only problem is a memory leak, that I don't free any Tokens finally
>on the stack.

When such an object is memcpy()'ied, you will get two object pointing at
the same object without the refcount being increased. So eventually, you
will end up with an object that will be deleted twice -- a segfault.

>Oh, Finally I found a real problem:
>Token::operator=() can't be sure whether the destination object was
>already initialized by a constructor or not.
>
>So there are to ways:
>1) YYSTACK_ALLOC constructs all tokens on the stack
>2) back to the roots: how does plain C work? ;-)
>
>Do you know any parser that stores more than just some integers and is
>implemented in a clean way?

If you by "more than just some integers" mean a C++ non-POD, then I  gave
example of that -- my parser, used with refcount objects. It is posted in
the Bison-Patches list with instructions on how to use it.

  Hans Aberg