Re: Locations suggest

[Akim Demaille]

>>>>> "Hans" == Hans Aberg <address@hidden> writes:

>> I implemented many scanners with location tracking, and they all
>> have different needs, especially because they have different
>> ``blanks''.

Hans> The main point is that the current Flex developers are evidently
Hans> looking into this matter.

Hans> I do not know what "blanks" your are speaking about; please
Hans> explain. If it is different types of whitespaces, my idea is
Hans> that Flex should not support any of that stuff, but that should
Hans> be computable at error time. This will cut down overhead during
Hans> scanning.

I don't think we understood each other.  Let me ask again: have you
implemented a scanner equipped with location tracking?  Then you know
that yylex does not always return, and when you parse

        -   2

Then you the second invocation of yylex will point to the blanks I
left betwen the - and the 2.  You must *skip* these blanks: they are
_not_ part of the location of `2'.  Idem with

        -/* The heck of a
        comment

        that makes
        stupid scanners
        think that the following token started
        at line 1. */2


Hans> The problem appears to be that Flex has some quick fixes (as the
Hans> REJECT code implementation) which slows it down.

Aside from the REJECT issue, I guess you agree the only valid
implementation from Flex is the current one (i.e., basically a
YY_USER_ACTION that count the \n in yytext).  And this is less
efficient that if I do it myself in my very rules that capture the
newlines.

>> That's another issue, buggy IMO.  I'm talking about the
>> implementation of option yylineno itself.

Hans> In view of that is how Flex implements it, your earlier posted
Hans> timings do not apply to the implementation of yylineno itself.

Err?  What timings?  What other implementation could you imagine?
You're not thinking of having Flex decompose the user actions matching
newlines are you?  You're not thinking about adding hooks inside Flex'
DFA to match \n, aren't you?

The current implementation is slow, granted, but the only valid IMHO.


>>>> In addition, it is more expensive to have Flex do it for us: it
>>>> duplicates the walks through yytext.
>>
Hans> What do you have in your mind here? No such backtracking is
Hans> needed in any of the discussion that we have had here.
>> You wrote backtracking, I did not.
>> 
>> Just read the code, you'll see what I mean.  If you never read the
>> code, then there is no point in debating.

Hans> I was going to study that, but I was diverted by a flood of
Hans> emails.

Hans> It would be better if you explained what you have in your mind,
Hans> so people can see if this whatever you want to call it multiple
Hans> walk through yytext is necessary: Then it might be changed.

You saw the code, since you pasted it yourself since.

Hans> Perhaps you should address the issue so that the current Flex
Hans> developers, which are at work on the problem, can follow your
Hans> thought, rather than addressing it to me.

I'm talking to the ML Hans :)




Hans> The cases mentioned where Flex does yytext manipulations
Hans> suggested that this would not be necessary. If one records
Hans> [begin, end) location information, that can be used to avoid
Hans> re-consulting yytext; have you considered that?

Hans> Clearly, if Flex implements it properly, that will be the most
Hans> efficient variation, as Flex has access to things that the user
Hans> does not.
>> No, the user knows better than Flex the lexical structure of the
>> file it parses.

Hans> So do you claim that humans knows some mysterious mathematics
Hans> that is not implementable in Flex? Clearly, if you want to go to
Hans> the extreme, and make a complete static analysis of the problem,
Hans> that is implementable. So you mast have something else in your
Hans> mind.

Of course I do claim that: I know the semantics.  Flex has no cue!!!