Re: reading all data from FIFO with 'dd' without blocking

[Pádraig Brady]

On 19/11/14 00:40, Assaf Gordon wrote:
> Hello,
> 
> I'm looking for a way to read all available data from a FIFO without 
> blocking, using coreutils (or other unix shell commands?).
> 
> The use-case:
> 1. A program (inotifywait) waits for a file-system event, then writes several 
> lines to a pipe.
> 2. Another script waits for those events, and runs a program upon changes.
> 
> A simple method would be:
> ===
>    SRC=/my/data/directory
> 
>    mkfifo myfifo
>       ### InotifyWait will write several lines to "myfifo" on every file 
> change in '$SRC'
>    inotifywait --outfile myfifo --monitor "$SRC" &
> 
>    ### Wait for events, and run a program after each file change
>    while read A ; do
>       ## call external program
>       my-program
>    done < myfifo
> ===
> 
> I want to minimize the amount of times I run 'my-program'.
> That is: if 'inotifywait' reported 10 changed files in ~20 lines to 'myfifo' 
> (in one quick burst),
> I want to group them together.
> 
> A better solution, using 'timeout' and 'cat' is:
> ===
>    SRC=/my/data/directory
> 
>    mkfifo myfifo
>       ### InotifyWait will write several lines to "myfifo" on every file 
> change in '$SRC'
>    inotifywait --outfile myfifo --monitor "$SRC" &
> 
>    ### Wait for events, and run a program after each file change
>    while read A ; do
>       ## Consume any events that happen in the next 1 second
>       timeout 1 cat > /dev/null
> 
>       ## call external program
>       my-program
>    done < myfifo
> ===
> 
> But this results in a 1 second delay.
> 
> What I tried (but failed) to use is "dd iflag=nonblock":
> 
> ===
>    SRC=/my/data/directory
> 
>    mkfifo myfifo
>       ### InotifyWait will write several lines to "myfifo" on every file 
> change in '$SRC'
>    inotifywait --outfile myfifo --monitor "$SRC" &
> 
>    ### Wait for events, and run a program after each file change
>    ( while read A ; do
>       ## Use DD to consume all available data from STDIN without blocking
>       dd of=/dev/null iflag=nonblock
> 
>       ## call external program
>       my-program
>    done ) < myfifo
> ===
> 
> With the above script, 'dd' prints the following:
>     dd: error reading ‘standard input’: Resource temporarily unavailable
>     0+0 records in
>     0+0 records out
>     0 bytes (0 B) copied, 0.00012379 s, 0.0 kB/s
> 

That error is fine in itself and is just dd indicating that there is no data 
available.
I.E. the shell may have read all the data from the fifo and is splitting it up 
with
the read builtin. So it would probably be better to avoid the read built in and 
use
a loop like:

  while true; do
      stdbuf -i0 head -n1 >/dev/null
      dd iflag=nonblock >/dev/null 2>&1
      my_program
  done

> Is there a way to achieve this with 'dd' ?

However dd's setting of O_NONBLOCK on the _shared_ descriptor is problematic
as it persists across the invocation of dd:

http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/dont-set-shared-file-descriptors-to-non-blocking-mode.html

That's a very non obvious gotcha.
I wonder should we have an [io]flag=block to allow resetting this in dd?

> Or with another program using the shell ?
> 
> (Note that for this application, I do not care about message-boundaries in 
> the fifo - if a line gets truncated that's fine).

thanks,
Pádraig