Re: flex -- input buffer overflow problem

[W. L. Estes]

Anyone have any thoughts on this?

> Date:   Mon, 22 Oct 2001 10:31:31 +0200
> From: Nils Nordman <address@hidden>
> User-Agent: Mutt/1.3.21i
> 
> Hi,
> 
> I have encountered a problem with flex that I hoped you perhaps could
> shed some light over.
> 
> First off, I'm using flex as found in the current development version
> of OpenBSD, flex version 2.5.4. I've encountered the problem while
> developing a simple c-code to html converter. What that does is 
> basically to recurse through a directory, scanning all files and 
> match keywords, function declarations/definitions, constants,
> etc etc.
> 
> Sequentially scanning lots of files produces the input buffer overflow
> error:
> 
> input buffer overflow, can't enlarge buffer because scanner uses REJECT
> 
> Now, I do use some really long trailing contexts. That, however, is not
> the problem. This problem will only occur if you've scanned lots of 
> files previously and will not occur otherwise. In this case I use the
> scanner for the OpenBSD kernel source which is composed of 8654 files,
> although only the c and header files are scanned. With approximately 30
> files left to be scanned the scanner will bail out with the above 
> error message. Just scanning the subdirectory that contains the 
> offending file will not cause any problems however and replacing the 
> files in that directory will still produce the same result, proving 
> without doubt that the error is not a result of the trailing context.
> 
> What I've done so far is that I've tried to somehow reinitialize the 
> buffer before scanning each file, trying yy_flush_buffer, creating 
> and using a new buffer for each file to be scanned, etc etc. All to
> no avail.
> 
> I also did a debug run(with the -d flag during scanner generation) and 
> noted that the scanner does behave rather strange after scanning many 
> files. It accepts the default rule for things that never should be 
> matched by that(one character at the time), and prints out 
> 
> --(end of buffer or a NUL)
> 
> every other line. Then at the file for which it bails out, the only info 
> printed after starting to scan it is:
> 
> --(end of buffer or a NUL)
> --(end of buffer or a NUL)
> --(end of buffer or a NUL)
> input buffer overflow, can't enlarge buffer because scanner uses REJECT
> 
> I've put the flex file online at http://nforced.com/users/nino/c_scanner.l
> in case you want to look at it. If you want to I can provide the debug
> output as well(although that's pretty big).
> 
> I realize that this might not be enough information for giving 
> any clues, but I will gladly provide any other information at your 
> request.
> 
> Best regards,
> 
> -- 
> Nils Nordman <address@hidden>
>