Re: threaded I/O

[Nikos Balkanas]

----- Original Message -----
From: John W. Millaway <address@hidden>
To: Nikos Balkanas <address@hidden>; Help Flex <address@hidden>
Sent: Friday, January 11, 2002 11:03 PM
Subject: threaded I/O


> > There are some things I wish I could see instead (egcs 2.91.66 -
> > wrong pattern generation, I/O on a seperate thread).
>
> Please elaborate on both of these. Recall that flex reads in large chunks,
not
> char by char, as does the libc stream library. Without any tweaks, flex
calls
> fread/read only twice for a 32k file. I'm not convinced that a separate
I/O
> thread would improve overall performance *at all*.
>
>
> __________________________________________________
> Do You Yahoo!?
> Send FREE video emails in Yahoo! Mail!
> http://promo.yahoo.com/videomail/
>
> _______________________________________________
> Help-flex mailing list
> address@hidden
> http://mail.gnu.org/mailman/listinfo/help-flex
>
>

Hi John,

Let me elaborate:

1) egcs: You remember my previous email where I posted some code with a
newline pattern. You took the time and tried it on your system, where it
worked. I know it works in other systems (with different compilers).

The problem is quite insidious. No warnings, no errors whatsoever. Just
wrong value in flex tables. I have searched suse bugs and gcc-bugs list,
without any luck. Haven't found anything on web search either. Problem is
with the whole distribution (SUSE 6.2, egcs 2.91.66) because both
distribution flex and flex compiled sources display the same problem. One
could say why bother? just get an update. Suse is quite popular in Europe
and a lot of people have this particular distribution. In addition I don't
plan to update my linux, before they include Kyro II video drivers (I'll
propably wait for some time).

What can be done? Maybe if one could send me a linux i386 flex binary that
works, I could diff the binaries to see if I can get anywhere. If a
conclusion is reached maybe a test can be included in the make test option.

2) threaded I/O: As Aberg has correctly pointed out, most of the time in
plain lexxers is waisted on disk I/O. Tests that I have done in my system
indicate that writing to disk is roughly 3x slower than reading. In addition
input during development is most of the time cached and read from memory.

Problem: Flex does blocking I/O. In other words every time the buffer fills
on read, it stops processing until the buffer fills again from the hard
disk. Worse even on writing it will try to write to disk on every new line.

Let's focuse on write. Instead on writing to stdout, one could write to a
buffer. A separate thread could maintain and write from that buffer. It
would require changes to ECHO and aliasing printf to sprintf. (Incidentally
a cosmetic glitch - the definition to ECHO has arguments 2 & 3 to fread
reversed, according to man fread.). Granted a lot of the time flex just
returns in memory values to bison. A lot
of other times though (95% of my time) is used as a standalone. That's why
ECHO
was developed. One could also use non-blocking I/O, but it's a hassle, and I
think interrupts are more expensive.

I have timed tests where I took standalone lexxers and just removed the
ECHOs from the actions (simple lexxers - no printfs). The difference in
speed was quite significant. With printfs one would have to evaluate time
spent to format the output versus writing it to disk. If there is interest I
can do some benchmarks.

Take care,
Nikos

P.S. Normal discussion groups reply -only- to the list. I have tried writing
to the maintainer, but no answer. Getting twice the messages can be a
burden.