Re: common per-token info

[Hans Aberg]

At 22:51 +0100 2002/01/07, Hans Aberg wrote:
>At 11:08 -0800 2002/01/07, John W. Millaway wrote:
>>Everyone: What are some example parsers/grammars that need per-token
>>information not covered by the list [file,line,column, USER_DEFINED_MEMBERS].
>
>The only missing part is the stream position number:
...
>Since the column number can be computed if one has the position number of
>the >beginning of the line, plus of the character itself, but that one
>cannot find >the position number from the line and column numbers, I think
>one should settle >for the former, and compute the column number at need.

Or one can just as well compute the column number the way you already do,
perhaps adding position number recording by some option, which then will be
wholly independent of the line/column number issue.

The list of independent options would (roughly) then be:
- line number only, or line number plus column number; both with or without
pointer to beginning of line (if buffer question can be solved).
- stream position number (ftell).
- Record just the beginning of token or locations in the form of [begin,
end) semi-open intervals. (Unless it is easier to always record locations
because of backtracking to the start of the token.)
- Some other stuff, like file descriptors/structures/names.
- Additonal user stuff.

  Hans Aberg