Re: Finding out when a token is consumed

[Hans Aberg]

At 04:18 +0200 2003/05/10, Frank Heckenbach wrote:
>> One idea that strikes me with your example, though, is that you might stamp
>> your lexemes with types in the lexer already. In your example, you had
>>     a+b#+c
>> where # is a directive that takes effect on lexemes after the #. You then
>> stamp it say as semantics, so that the sequence that the lexer produces
>> will be
>>   lexeme  type/effect  semantics
>>     a     NUMBER       value of a
>>     +     "+"
>>     b     NUMBER       value of b
>>     #     set #
>>     +     "+"
>>     c     NUMBER       value of #c
>> When your "+" action combines the value of a+b with the value of #c, it
>> should recognize that it is a #c and not a c.
>>
>> This should then work.
>
>I think it would work in theory. But there are practical problems.
>I'd have to attach the state of all directives (there are quite a
>few of them) with each token, as opposed to the (genrally rather
>rare) changes of directives. And pass all this to the code that does
>the real work (which is, for the most part, in other source files
>called by the parser). All in all, I'd have to pass much information
>around all the time which is both inefficient and, again, extra code
>to write in many places ...

I usually program using C++ around a polymorphic variable (by a virtual
class hierarchy) using a reference count. Under such circumstances, there
is no problem building suitable closures. It is easy to create complicated
structures.

In your example, each type of directive would get a C++ class. If you admit
a list of them in a row, add a list object. Your actions need to recognize
these objects and know how to execute them. But clearly, you must treat
your directives somewhere.

  Hans Aberg