Re: Locations suggest -- we're stupid

[Hans Aberg]

At 14:35 +0200 2002/01/08, Nikos Balkanas wrote:
>> You do not know that the scan over yytext to compute yylineno causes
>> heavier overhead than the REJECT code, as Flex always imposes the latter,
>> and nobody seems to have profiled without. So let's stick to facts.
>>
>
>Yes I have done tests. Duplicating the yytext scan has an overhead. Used
>yy_scanbuffer and scanned input from the buffer twice or once.

Did you first, in the Flex source file main.c, check_options(), zip out
  if ( do_yylineno )
    /* This should really be "maintain_backup_tables = true" */
    reject_really_used = true;
recompile, and then do timings on a test with and without the %option yylineno?

If so, what did the test show, on which computer?

>> Then, the loop already has conditionals and some in it, so the question is
>> how much overhead is caused relative to that.
>
>The overhead is with rescanning yytext, not with the overall loop.

It is possible to put this yylineno count test inside or outside the
overall loop.

>> And for the efficiency of the code itself, the only thing that matter is
>> how much time the computer spends it relative to the other code it does,
>> and most parsers spend most time in their actions, not the parsing.
>
>Processing time adds up. A slow lexxer can hog down the fastest parser.
>One of the main advantages of flex is its speed. Let's keep it that way.

The thing is that for the processing of the characters, the main bottleneck
of a stream is to get it off the hard disk. There are several buffers,
possible involving character translation with conditionals before the
character even enters the Flex generated lexer. But that probably does not
affect the overall speed, as the CPU's are much faster than the hard disk
can read the file.

So therefore, it is good to have some profiling on modern computers done
(which will behave differently than old ones).

>> So one should not wasting time on trying to write efficient code, but to
>> write structured code, and optimize it wherever needed, if possible: Much
>> more precious than computer time is programmer time, which forces this
>> development.
>
>Programming style. I like to write optimized code from the start, so that I
>avoid going back at it later on. Development is very subjective. One person
>can write code in 1/4 the time other people do.

The thing is that computer are changing, so it is not worth doing
optimizations, if the code is not executed often. Structured code is easier
to maintain, which is waht counts if programmer time is expensive.

> It is a question of how used
>one is to the particular task. If you maintain lineno or the other error
>tracking features, after a while you do it subconsciously.

But you are still free to do that, by turning the line number feature off,
in which case it will not affect the generated lexer at all.

>... I used Mac until OS 8.0. Pitty they used such a old
>kernel.

MacOS pre-X is an exercise in low mem programming on an underpowered CPU,
which was interesting at the time, but should avoided at this point in time.

> AIX used to use it until v 4.3.3, now they switched to a Linux
>kernel.

If you mean the Mach kernel, there are later versions which have been
rewritten and are more efficient. There is a tradeoff between being
structured and slow on the one hand, and monolithic and fast on the other
hand. Structured is better for a number of issues (like security), but
CPU's cannot cope with it just yet.

> What is MSOS?

MicroSoft OS, as opposed to DOS; often called Windows something.

> Do you mean that the Codewarrior IDE exists in other Unix platforms,
>too?

Yes, but I do know exactly which platforms. It is besides the point for
this discussion, because HP has their own, based on GCC, and I think this
is fairly common.

>> I already made Flex a plugin for my IDE. :-)
>
>That's nice. I actually meant why stop at these features? Add a few menu
>items, buttons and an interactive pattern generator to make development even
>faster! :-)
>
>So, basically you want these features to work better with your IDE. Any idea
>how many people use flex that way? Should you wait until the specifications
>for the new GNU IDE you mentioned are in?

As for this discussion on error messages, I by that got aware of the
information that it could absorb and make use of. So the error message
information that looked as though being of general use, I fed back into the
Flex/Bison lists.

But it was not exactly what is used in that IDE: Actually, it uses MacOS
file descriptors, and some other stuff I did not mention.

I have had conversations with other people, and the reason one is using an
IDE and not GNU is the convenience. SO I want to feed the information back
into GNU/Flex, so that people do not feel there is any advantage of that
other stuff anymore.

  Hans Aberg