[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Feature request: Support Linux packetized pipes

From: sbaugh
Subject: Feature request: Support Linux packetized pipes
Date: Thu, 16 Jun 2016 23:18:29 -0400
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux)

Hi bug-bash,

The Linux kernel supports creating pipes in a mode (enabled by passing
the O_DIRECT flag, and known sometimes as a "packetized pipe") that
respects message boundaries, to an extent.

An excerpt from the Linux manpage pipe(2):

        O_DIRECT (since Linux 3.4)
                Create a pipe that performs I/O in "packet" mode.  Each
                write(2)  to  the pipe  is  dealt  with as  a  separate
                packet, and read(2)s from the pipe will read one packet
                at a time.  Note the following points:

                *  Writes of greater than  PIPE_BUF bytes (see pipe(7))
                   will be  split into multiple packets.   The constant
                   PIPE_BUF is defined in <limits.h>.

                *  If a read(2) specifies a buffer size that is smaller
                   than the  next packet, then the  requested number of
                   bytes are read,  and the excess bytes  in the packet
                   are discarded.  Specifying a buffer size of PIPE_BUF
                   will  be sufficient  to  read  the largest  possible
                   packets (see the previous point).

                *  Zero-length packets  are not supported.   (A read(2)
                   that specifies a buffer size of zero is a no-op, and
                   returns 0.)

                Older  kernels  that  do  not support  this  flag  will
                indicate this via an EINVAL error.

I would like to use a packetized pipe in a conventional shell pipeline
to support parallelism. Multiple processes can read from the same pipe
at the same time and not receive any partial lines, as long as the pipe
is being written to one-line-at-a-time, with lines under PIPE_BUF size.
(Parallelism in shell pipelines is an interest of mine; I wrote about
one technique here: https://catern.com/posts/pipes.html but bash support
of packetized pipes would make this much less painful)

So, it would be nice if it was possible in bash to somehow explicitly
request that a pipe between two processes be packetized (on systems that
support this behavior).

I can go into more detail about my use cases, and I can work on adding
such support. Before I do so, though, does this sound like a good
feature to add to bash?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]