Re: Featuritism

[Hans Aberg]

At 10:24 +0100 2002/01/10, Akim Demaille wrote:
>Hans> The location should only be that of the token returned, that is,
>Hans> [yytext, yytext+yyleng).
>
>What tokens of that are wider than a single yylex invocation.

I only have the case of giving the location of yytext in my mind. So you
will have to give some examples on what you have in your mind.

>Hans> The Flex scanner should avoid that, instead provide a pointer to
>Hans> the line, if that is needed, for further computations, if the
>Hans> use is for errors alone. Anything else will generate unnecessary
>Hans> overhead.
>
>This is my point.  Thanks Hans, it seems that you're getting it.

I suggested that from the beginning, before your inputs. :-)

>Oh, BTW, the day you'll implement something like a *scanner* tracking
>*location* (Gosh, how can you answer `yes, I did implement a
>*backtracking* *parser*?  Do you read me?),

Sometimes the parsing is difficult. :-)

> you'll see that your
>*parser* needs the location of more than a single token.  For instance
>in semantic checking.

I am not sure what type of semantic checking you have in your mind. For
example, errors in C++ that depend on semantic problems are quite difficult
to understand, with or without token locations. I think one will have to
live with that for now.

I think that is though is beyond the scope of the current Flex.

But if the Flex generated scanner simply provides the locations of the
tokens, that will give the Flex/Bison user the opportunity to write more
complicated error generating schemes. One will then have to make a scheme
recording the token locations at the point the scanner generates them for
later use.

>>> Worse yet: in multibyte characters, do you count bytes or
>>> characters?
>
>Hans> All points towards that for efficent programming with Unicode,
>Hans> one should use single width characters.
...
>Great, you finally understood me.  So why didn't you remove the `this
>becomes a non-issue' since apparently you just understood it is one?

I do not understand all this polemics. :-) I did not understand you.

Multibyte characters are not an issue if one translates them into single
width before they arrive in the scanner. Current software does not seem to
be made to handle variable width characters, and computers works most
efficient with a single width.

>>> 2. computation of something that can ease the computation of the
>>> locations (typically the current position).
>>>
>>> But then, the author of the scanner will have to put some operation
>>> to compute the locations from the position at many different
>>> places.
>
>Hans> I am not sure what you mean here: If the buffer keeps track of
>Hans> the stream (ftell) position, it can be used to tell the location
>Hans> of the token found itself.
>
>
>Well, I stop answering, I'm wasting my time.

You are wasting your time not explaining carefully what you mean.

>  Please, proceed,
>implement whatever you want.  And some day, try to use it.  There is
>no use in talking to someone who answers without even having tried
>what he says.

The idea is simply that the scanner helps finding the locations of yytext.
More complicated location schemes, the Flex/Bison programmer will have to
implement by themselves.

So what do you have in your mind?

-- It seems that you have experimented with more complicated location
recording schemes that was not discussed here.

  Hans Aberg