Re: Memory allocation problem with latest gawk - new testcase

[Stepan Kasal]

Hello,

[Arnold, it seems there is a bug that needs to be fixed after all.
Plese skip to the word "Luckily" to see exact description of the bug.]

On Thu, Apr 03, 2003 at 03:00:21PM +0400, Stanislav Ievlev wrote:
> > $(ulimit -Sv 16384 && time gawk '/.*/ {;}' /dev/zero)
> 
> Note:
> this testcase works as expected (endless loop) with gawk 3.1.1.

Arnold has already mentioned this; please forgive me that I'll repeat it:

The expected behaviour is not endless loop in awk, as in
        yes |gawk '{}'

The expected behaviour is, in fact, to grow the memory until the
process/system crashes.

But if the input file has finite size, gawk should allocate roughly the
same amount of RAM, put the "record" there and then exit.

On your "2.2" configuration, gawk allocates insane amount of memory.
(I cannot reproduce it with my configuration, kernel 2.2.x + glibc 2.2.4.)

Luckily,
the straces you mailed explain the bug:  gawk tries to fill the buffer
but read() reads only 4kB, which is less then gawk required.
Before calling read() again, gawk calls grow_iop_buffer() again,
even though it is not necessary.
That means that for big files, grow_iop_buffer() is called size/4kB times,
so the memory is about 4kB*2^{size/4kB}.

Even though the behaviour of your "2.2" system is weird, you are right,
gawk should behave better and understand the consequences of the fact
that read() reads only 4kB.

I'm not familiar with the new *_get_a_record() code yet.  Arnold, would
you be so kind to look at this?  I'm sure it'll take you much less time
than it would take me.  (And I'm currently quite busy, sorry.)

Thank you very much,
        Stepan

PS: Thank you, Stanislav, that you kept working, until we hunted the bug!