Re: Featuritism

[Akim Demaille]

>>>>> "Hans" == Hans Aberg <address@hidden> writes:

Hans> At 14:51 +0100 2002/01/09, Akim Demaille wrote:
>> 1. computation of the location.
>> 
>> I.e., completely help the beginner.  But then, as I explained, you
>> have to explain to Flex what is skipped (again, think of the
>> difference between a start condition for strings, and another for
>> comments).

Hans> The location should only be that of the token returned, that is,
Hans> [yytext, yytext+yyleng).

What tokens of that are wider than a single yylex invocation.


>> You also need to get into gory details, such as how tabulations are
>> understood by the application.

Hans> The Flex scanner should avoid that, instead provide a pointer to
Hans> the line, if that is needed, for further computations, if the
Hans> use is for errors alone. Anything else will generate unnecessary
Hans> overhead.

This is my point.  Thanks Hans, it seems that you're getting it.

Oh, BTW, the day you'll implement something like a *scanner* tracking
*location* (Gosh, how can you answer `yes, I did implement a
*backtracking* *parser*?  Do you read me?), you'll see that your
*parser* needs the location of more than a single token.  For instance
in semantic checking.

And _of course_ the main point of location is accurate error message.



>> Worse yet: in multibyte characters, do you count bytes or
>> characters?

Hans> All points towards that for efficent programming with Unicode,
Hans> one should use single width characters. As Unicode uses 21 bits
Hans> (0x000000..10FFFF I think), that will most likely be 32-bit
Hans> characters (UTF-32), the first typical alignment.

Hans> So, then this becomes a non-issue.

Hans> But the stream position (ftell) numbers are independent of this.

Great, you finally understood me.  So why didn't you remove the `this
becomes a non-issue' since apparently you just understood it is one?



>> 2. computation of something that can ease the computation of the
>> locations (typically the current position).
>> 
>> But then, the author of the scanner will have to put some operation
>> to compute the locations from the position at many different
>> places.

Hans> I am not sure what you mean here: If the buffer keeps track of
Hans> the stream (ftell) position, it can be used to tell the location
Hans> of the token found itself.


Well, I stop answering, I'm wasting my time.  Please, proceed,
implement whatever you want.  And some day, try to use it.  There is
no use in talking to someone who answers without even having tried
what he says.