Proposed new feature for bash: unbuffered pipes, part 1: overview

[Dale R. Worley]

It may seem intemperate to propose that a program with an 82-page
manual "page" could benefit from yet another feature, but that is what
this is doing:  Bash's users would benefit from a feature that
suppresses buffering of output sent to particular pipes (designated by
the user).

At the bottom of this message is a list of a dozen references spanning
20 years requesting a general mechanism to obtain this behavior.

At least the following programs have options or default behaviors to
suppress buffering stdout in situations where a generic program would
buffer:
    python -u (or env. var. PHTHONUNBUFFERED)
    grep --line-buffered
    sed -u
    cat -u (unbuffered output by default in Gnu)
    tcpdump -l
    tee (unbuffered output by default)
    tail -f
    awk/mawk (fflush function or unbuffered output by default)

The following commands have been devised to suppress buffering:
    unbuffer (probably == expect_unbuffer)
    stdbuf -oL
    script -q
    env. var. NSUnbufferedIO=YES (on MacOS)
Each runs into one or more of these issues:
    does not work with statically-linked executables
    security issues involving env. var. LD_PRELOAD
    allocation of a pty
    combines stderr and stdout
    the effect does not apply to subprocesses of command, particularly
        commands within shell scripts, or applies to subprocesses
        whose output is not the pipe

This message describes a proposed solution to this problem which is
easier to invoke and works in a much broader range of circumstances
than the previous solutions.  The associated messages provide a
proof-of-concept demonstration implementation of this proposal.

This command demonstrates the issue:

    for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A | cat

Currently, when this command is run, after 10 seconds it outputs 10
lines reading "ABCDE".  That is because the "grep" command's stdout is
to a pipe, and so is buffered.  The buffer accumulates all 70
characters of output and outputs them to the pipe when "grep" exits.

The demo proposed change defines:

    for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A >|> cat

The "unbuffered pipe" symbol ">|>" causes Bash to set in the
environment of the "grep" process a variable "STDOUT_UNBUFFERED" with
a value that contains the dev and ino values for the pipe which the
"grep" process sees as fd 1.

The stdio library of the "grep" process, when it initializes
processing of stdout (== fd 1), observes that the fd has the dev and
ino values specified in the env. var. STDOUT_UNBUFFERED, and sets
stdout's buffering state to unbuffered.  (This can be overridden later
if the process calls setvbuf.)

The result is that the above command outputs one line "ABCDE" each
second for 10 seconds.

In addition to ">|>" for an unbuffered pipe, the demo also defines
">|>&", which is the unbuffered parallel of "|&", i.e., it is
equivalent to "2>&1 >|>" and redirects both stdout and stderr from one
command into an unbuffered pipe.

There are a large number of open issues regarding this proposal, and
in particular, the code only works for the simplest usages.  But the
first issue to be addressed is whether this functionality is worth
pursuing.

Dale
----------------------------------------------------------------------
References

14 Sep 1999
https://marc.info/?l=glibc-bug&m=98313957306295&w=4
"[REMINDER] stdio buffer flushing control environment variable"

26 May 2006
http://www.pixelbeat.org/programming/stdio_buffering/
"buffering in standard streams"

19 May 2009
https://stackoverflow.com/questions/881696/unbuffered-stdout-in-python-as-in-python-u-from-within-the-program
"unbuffered stdout in python (as in python -u) from within the program 
[duplicate]"
upvoted 57 times

16 Jun 2009
https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe
"Turn off buffering in pipe"
upvoted 415 times

20 Jan 2010
https://lists.gnu.org/archive/html/bug-bash/2010-01/msg00069.html
"weird bash pipe behavior"

26 Jul 2010
https://stackoverflow.com/questions/3332045/bash-force-execd-process-to-have-unbuffered-stdout
"bash: force exec'd process to have unbuffered stdout"
upvoted 21 times

12 Aug 2010
https://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered
"How to make output of any shell command unbuffered?"
upvoted 66 times

23 Jun 2011
https://perkin.org.uk/posts/how-to-fix-stdio-buffering.html
"How to fix stdio buffering"

26 Jul 2011
https://serverfault.com/questions/294218/is-there-a-way-to-redirect-output-to-a-file-without-buffering-on-unix-linux/589614#589614
"Is there a way to redirect output to a file without buffering on unix/linux?"
upvoted 50 times

30 Nov 2012
https://stackoverflow.com/questions/13644024/stdbuf-with-setuid-capabilities/18624182
"stdbuf with setuid/capabilities"
upvoted 3 times

2 Nov 2013
https://blog.jpalardy.com/posts/grep-and-output-buffering/
"Grep and Output Buffering"

31 Jan 2015
https://unix.stackexchange.com/questions/182232/pipes-how-do-data-flow-in-a-pipeline
"Pipes, how do data flow in a pipeline?"
upvoted 22 times

4 May 2015
https://unix.stackexchange.com/questions/200235/how-to-use-sed-to-manipulate-continuously-streaming-output
"How to use sed to manipulate continuously streaming output?"
upvoted 13 times