Re: Parsing input from a stream... [Was: Re: Parsing input from a string...]

[Nikos Balkanas]

----- Original Message -----
From: Dave Trombley <address@hidden>
To: John W. Millaway <address@hidden>
Cc: Help Flex <address@hidden>
Sent: Thursday, January 31, 2002 9:47 PM
Subject: Parsing input from a stream... [Was: Re: Parsing input from a
string...]
[...]
>     I suppose what I'm interested in doing is to understand how the
> yy_*buffer* functions work.  Correct me if I'm mistaken, but they seem
> to assume to a large degree that files will be the underlying data
> source for the buffers (although there are functions for specifically
> copying strings into a new buffer),

Not quite. Most often it is assumed it is a stream.

> and more broadly, that all of the
> data will be avaialable by the time the lexer entry point is reached.

Nope. As with streams nothing is assumed until EOF.

>  What I'd really like to be able to do is to have a parser/lexer pair
> which is fully reentrant, and have the lexer drain a stream until either
> the parse terminates, or the stream is empty.  In the latter case, I'd
> like the parser/lexer to block on stream input until more is available.

Coincidence! That's exactly flex's behaviour! If you try it on a console,
you can just keep feeding it fresh data, and it won't quit until <ctrl>-d
(EOF) is pressed :-)

Reentrant stuff coming up soon.

>  It seems I could implement this in 2.5.6,

You still can. Taken from experience on a SUSE 6.2 linux.

> especially given the fact
> that you can pass extra data along in a reentrant lexer, but I'm having
> trouble because I don't know the exact contract for the buffer
> functions.  (For example, should I assume that YY_INPUT will only ever
> be called from a single place?  How can I access the extra data from
> that place?  Is that data placed into a flex buffer?  Is there a more
> low level way of getting input to the lexer, since my MT buffers will be
> around anyway?)
>
>     Are there any plans/thoughts about making the buffer system
> extensible and abstract?  Do you think it would be possible/desirable
> for me to attempt this, and could it be done without sacrificing
> performance?

Well, I am not the maintainer, but I don't think that's a good idea. First
of all flex works and works well. Given that, it is bad style to mix flex
internals into programming. Consider a black box of abstraction around it.
Having said that since flex prereads input in chunks of 16k(?) you can
access the buffer with the extra future input. You can copy it from:

yy_current_buffer->yy_ch_buf + yy_buf_pos

Be careful, though, yy_current_buffer->yy_buf_pos + yylen = '\0'. The value
of that byte is stored in yy_hold_char. You may want to put it back in to be
able to read the whole buffer. You need to restore it to '\0' so that you do
not mess with yytext. Remember, that's poor style and you didn't hear it
from me.

However, I am a buddinsky. I am neither the maintainer nor the addressee of
this letter. But what the heck?

Cheers,
Nikos.
>     Cheers,
>
>     -dj
>
>
>
>
> _______________________________________________
> Help-flex mailing list
> address@hidden
> http://mail.gnu.org/mailman/listinfo/help-flex
>
>