Re: Locations suggest

[Hans Aberg]

At 15:21 +0100 2002/01/07, Akim Demaille wrote:
>>> Some people will tell you they need (c0, l0) + wdith, not (c0, l0)
>>> + (c1, l1).  And you cannot convert from one to the other without
>>> having the text handy.
>
>Hans> I am not sure what you mean here: The width is just the
>Hans> difference between the position numbers; see the thread
>Hans> "Flex/Bison standardized diagnostics".
>
>Wrong, you're thinking only of the case where everything is on a
>single line.

I am not thinking on tokens on a single line. Please define the width
number you want to get hold of.

>Hans> I think that it would be interesting with an option that allows
>Hans> any of \n, \r, \r\n to be read as a newline, so that streams can
>Hans> be read as binary.
>
>Hans> Unicode has a special line separator symbol, so then tha issue
>Hans> may go away.
>
>No, it's not another issue: it's the same.  This idea, IMO, is fine,
>but cannot face the difference between uses.

Under Unicode, I figure one is expected to not use \n, \r, etc. But Unicdoe
is several years away.

>...  Still, I'm OK with some sugar, but
>definitely having Flex imagining it can perform the job for me is
>wrong.

I think you have misread my suggest: I do not want to automate or
standardize the diagnostics itself, but to find the minimum information
that is needed to be passed between the Flex lexer and the Bison parser, so
that the appropriate error messages can be computed.

For example, with the position number recorded, one does not need not
record the column number as it can be computed.

>>> All the rest is pushing an additional burden onto you, for
>>> extremely limited added value.
>
>Hans> Isn't that for the Flex developers remaining to see? If one is
>Hans> curious about a problem, perhaps giving it an attempt, then that
>Hans> will tell exactly what is required to implement it.
>
>Have _you_ tried?  I did.  I implemented many scanners with location
>tracking, and they all have different needs, especially because they
>have different ``blanks''.

The main point is that the current Flex developers are evidently looking
into this matter.

I do not know what "blanks" your are speaking about; please explain. If it
is different types of whitespaces, my idea is that Flex should not support
any of that stuff, but that should be computable at error time. This will
cut down overhead during scanning.

>Hans> The problem appears to be that Flex has some quick fixes (as the
>Hans> REJECT code implementation) which slows it down.
>
>That's another issue, buggy IMO.  I'm talking about the implementation
>of option yylineno itself.

In view of that is how Flex implements it, your earlier posted timings do
not apply to the implementation of yylineno itself.

>>> In addition, it is more expensive to have Flex do it for us: it
>>> duplicates the walks through yytext.
>
>Hans> What do you have in your mind here? No such backtracking is
>Hans> needed in any of the discussion that we have had here.
>
>You wrote backtracking, I did not.
>
>Just read the code, you'll see what I mean.  If you never read the
>code, then there is no point in debating.

I was going to study that, but I was diverted by a flood of emails.

It would be better if you explained what you have in your mind, so people
can see if this whatever you want to call it multiple walk through yytext
is necessary: Then it might be changed.

Perhaps you should address the issue so that the current Flex developers,
which are at work on the problem, can follow your thought, rather than
addressing it to me.

The cases mentioned where Flex does yytext manipulations suggested that
this would not be necessary. If one records [begin, end) location
information, that can be used to avoid re-consulting yytext; have you
considered that?

>Hans> Clearly, if Flex implements it properly, that will be the most
>Hans> efficient variation, as Flex has access to things that the user
>Hans> does not.
>
>No, the user knows better than Flex the lexical structure of the file
>it parses.

So do you claim that humans knows some mysterious mathematics that is not
implementable in Flex? Clearly, if you want to go to the extreme, and make
a complete static analysis of the problem, that is implementable. So you
mast have something else in your mind.

  Hans Aberg