Re: tail and grep

[Bob Proulx]

Tillman, Don wrote:
> I'm trying to redirect output of tail that is grep'd to a
> file. However, no data is ever redirected. If I leave out the
> redirect the desired output is displayed on the screen.

But you are observing that 'tail -f file.txt' is producing output to
your terminal though, right?  Which basically means that you just
aren't being patient enough to wait for enough output to be generated
from tail as input to grep so that grep would then have something to
write.  If you give grep enough input it will write it out as output.
Or if you terminate the input then it will exit and flush all of the
pending output at that time.

> Example: tail -f file.txt | grep ??? > file.txt

Note: You don't want the output file to be the same as the input file
here.  That wouldn't work.  You would want a different file on output
then is used on input.  But I know this was just a typo while
generating the example so no worries.

> Is this a bug or is there a better way to accomplish?

It isn't a bug.  Just a misunderstanding of libc's stdio buffering.
Because the output of grep is a file, not a terminal, grep will buffer
the output.  This is done by the stdio library.  It will buffer the
data until a full block write can be performed.  If that much data
hasn't been generated then it won't have written the data yet.  Grep
is waiting for input, buffering it up into a full block and waiting
for the buffer block to become full before writing it out.  On various
systems the stdio buffer size might be 4k or larger or smaller, as a
system specific libc behavior.  If you wait until enough data is
produced from tail as input to fill an output buffer block then grep
should write the output.  But because grep's input source is 'tail -f'
which could run forever, at least until interrupted, the amount of
time that grep might run is indefinite but grep doesn't know that.

If you want grep to write out data line by line then you should add
the 'grep --line-buffered' option to the command line.

  tail -f file1.txt | grep --line-buffered PATTERN > file2.txt

Bob