Re: why are locations dictated by bison?

[Akim Demaille]

Hi John,

| I had to bend over backwards to get the latest flex code to talk
| nicely with bison (the next release will have a bison option), since
| bison will expect any one of several possible declarations of 'yylex',
| depending upon user options -- No, not even dependent upon user
| options, but whether parser actions use @N, in which case yylloc is
| passed to yylex.

Maybe this is something we should have discussed together.  Have you
considered the case where YYLEX_PARAM is used?  I'm sure you did, but
it's just safer to ask.


| As maintainer of Bison, you have strongly opposed implementing
| location tracking in flex, and you believe this should be left to the
| user to define.

I'm not strongly opposed to it, I explained the reasons why I believe
it is not going to help anybody.  Up to now, *no one* showed me a
sample .l file which would demonstrate what you have in mind.  I have
showed several examples of what are the problems to solve, and I have
implemented several prototypes.  I think it would be fair from the
defendant of flex-location-tracking to show at least what kind of user
interface they have in mind.

| What business, then, does Bison have dictating the
| format of the location tracking?

Bison does not impose any kind of structure at all.  There is a
default definition, just as but default yylval is int, but that's only
a default.

| Why is location tracking a parser
| issue?

You seem to have missed some of the mails I sent on Flex Help.

The parser propagates the locations computed by the scanner.  That's
why, as I already explained, it is easy for the parser to provide
quite generic location support: because it's only propagating values,
which can be done without any knowledge of the structure of the data,
or, if you need to do something finer grained, then the user can
explicitly tell how to propagate the values.

In other words, before locations were unusable in Bison because there
was no equivalent of YY_USER_ACTION.  Now there is one, and the user
can tell Bison what to do, just as the user can tell Flex what to do.

| The result of Bison's YYLTYPE mechansim is that the scanner needs to
| know about parser details. This is arguably bad design -- The scanner
| should be completely separate from the parser -- according to
| some. With that in mind, bison should simply leave out location
| tracking altogether, shouldn't it?

This is my point from the beginning: neither Bison nor Flex have to
decide anything: the master is the user, period.  What matters is that
Bison and Flex provide the user with means to compute what she needs.
Flex has it since long ago thanks to YY_USER_ACTION, and the yylex
prologue.  Nothing else is needed.  Worse, some well meaning
extensions cannot only, IMNSHO, make things worse: higher impedance
between Flex and Bison in particular.

Which is just what you are discovering.

| If I may greatly simplifiy your case (over the past week) against
| location tracking in flex, it rests on two arguments. (1) Location
| tracking is a performance problem, and (2) there's no simple general
| implementation that we could provide anyway.

To make (1) more precise, I should say that location tracking by the
user will always be able to be faster than by Flex itself.  But I
personally don't think this is an issue anyway.  The real issue is
(2), indeed.  Please see my post were I explained the two levels of
help you could provide.  Or better: show me a Flex file showing the
interface we have in mind.  We don't seem to understand each other
because we don't have code to comment.  I'm not asking for an
implementation, just a theoretic foo.l.  _Then_ it will be easier to
discuss.

| To address the performance point, the following is from the bison
| manual, discussing YYLTYPE:
|
|      In order for the members of this structure to contain valid
|      information, you must make `yylex' supply this information about
|      each token.  If you need only certain members, then `yylex' need
|      only fill in those members.  The use of this feature makes the
|      parser noticeably slower.
|
| So here we have a performance problem, caused by bison, and
| independent of the scanner's implementation of location tracking!

What is your point?  Of course there is a performance issue?  What is
it that you want to demonstrate?  The problem for the scanner is
_much_ different than for the parser.

Have you ever tried?

| I
| wonder how this bison problem compares to rescanning yytext (i.e.,
| looking for '\n' chars) in flex. It may be that so much overhead is
| imposed in the layers above the scanner, that improving yylineno
| detection will go unnoticed.

I agree, and I never said I cared that much about the performance
issue.  Some people might, and I can tell you than Flex will be slower
than me, but I don't care much if it really helps me.

The point is I doubt very much it will significantly help me.

Prove me wrong: show me a scanner.

| To address point (2): If there is no simple, general implementation we
| could provide at the scanner level, why, then does Bison dictate both
| the structure AND the scanner function prototype for location tracking
| (i.e., at the parser level)?

Because you wrong on the two points: Bison provides default, but the
user does whatever she wants.  That's the only viable choice, we're
making tool for maintainers: they know better than we do.  What they
need is to be able to write their location tracking actions in the
scanner, and their location propagation actions in the parser.
Period.

| You can answer all of these questions by defending the decision to
| implement location tracking as you have done, and at all, in Bison.

I'm sorry, but up to now, I have the feeling that you are referring to
a feature that looks nice on the paper, but until I see a prototype I
won't believe you have really examined the problem in length.

I shall repeat it: up to now, I am the only one to have implemented
all this.  Show me what you have in mind.  We should stop discussing
`in the air', let's talk about concrete material.