Re: Locations suggest

[Hans Aberg]

At 15:12 +0100 2002/01/09, Akim Demaille wrote:
>Hans> I do not know what "blanks" your are speaking about; please
>Hans> explain. If it is different types of whitespaces, my idea is
>Hans> that Flex should not support any of that stuff, but that should
>Hans> be computable at error time. This will cut down overhead during
>Hans> scanning.
>
>I don't think we understood each other.  Let me ask again: have you
>implemented a scanner equipped with location tracking?

I did implement a non-deterministic parser once, so I know about keeping
track of branching points.

>  Then you know
>that yylex does not always return,

A function that does not return is called non-termination; so I figure you
need to use a more explanatory wording:

> and when you parse
>
>        -   2
>
>Then you the second invocation of yylex will point to the blanks I
>left betwen the - and the 2.  You must *skip* these blanks: they are
>_not_ part of the location of `2'.

You probably mean that one may use a rule:
[[:space:]]+     { /* Skip white-space. */ }

I figure that as yytext in this case will reset to produce only the token
"2" in that case, that behavior should be copied over to the location
tracking:

One merely indicates the location of the token that the scanner detects.
Are you saying that something more complicated is needed?

>Hans> The problem appears to be that Flex has some quick fixes (as the
>Hans> REJECT code implementation) which slows it down.
>
>Aside from the REJECT issue, I guess you agree the only valid
>implementation from Flex is the current one (i.e., basically a
>YY_USER_ACTION that count the \n in yytext).  And this is less
>efficient that if I do it myself in my very rules that capture the
>newlines.

That would be the immediate implementation: Clearly, a conditional on the
every character takes time, but the question is if it really makes any
diufference, in view of that each caharcter is already examnined quite a
bit before it ends up in the Flex scanner. Only proper rpofileing, with the
REJECT code first removed can tell that.

Another would be to implement special states that are called for when the
\n token is implemented (using a modifed DFA).

>...You're not thinking about adding hooks inside Flex'
>DFA to match \n, aren't you?

If it is worth it, one can think of it.

>The current implementation is slow, granted, but the only valid IMHO.

But only after ones knows whether the overhead of the current
implementation, without the REJECT code, is significant.

>I'm talking to the ML Hans :)

ML usually stands for the Meta-Language; what do you use it for?

>Hans> So do you claim that humans knows some mysterious mathematics
>Hans> that is not implementable in Flex? Clearly, if you want to go to
>Hans> the extreme, and make a complete static analysis of the problem,
>Hans> that is implementable. So you mast have something else in your
>Hans> mind.
>
>Of course I do claim that: I know the semantics.  Flex has no cue!!!

I doubt that the \n problem is so deep it cannot be analysed properly from
the theoretical point of view, and implemented.

  Hans Aberg