AW: Re: read command sometimes misses newline on timeout

[Thomas Oettli]

I guess I will wait for an answer from Martin D Kealey. His answer tells me 
that he understood what issue I have.
But thank you a lot, nevertheless.
BTW: I have already tried all kinds of combinations including read -t 0 and 
could not find a working solution, that is why I filled a bug report.
________________________________
Von: alex xmb sw ratchev <fxmbsw7@gmail.com>
Gesendet: Montag, 7. Oktober 2024 21:50
An: Thomas Oettli <thomas.oettli@sfs.com>; bash list <help-bash@gnu.org>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com>        Report as Spam / Check 
Mail<https://mail2.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007215024_7A5FA2A9F1493>

plz write to bash list

about ur case , i dont much get it
async read ?

just small question .. why do u ' read again , to get full line '

and a small note
there is -t 0 , that returns when data'd be available
u maybe can test this approach

On Monday, October 7, 2024, Thomas Oettli 
<thomas.oettli@sfs.com<mailto:thomas.oettli@sfs.com>> wrote:
Sorry, English is not my mother tongue, so let me try to clear this.

I have a script that asynchronously reads from a pipe by using read with 
timeout (basically the "reader" function in the example). The separator could 
be any character, I just assume that it is "\n" in this case. I do not control 
the writer to the pipe, lags may occur at any moment.

Now this happens:


  1.
the script calls "read" on pipe
  2.
"read" hits timeout just after it has read "\n" from the pipe
  3.
"read" returns the full line, but exits with rc > 128
  4.
the script assumes a partial line, appends received data to a buffer and calls 
"read" again, but without timeout to get the rest of the line
  5.
because "\n" was already read at step 2, "read" returns the next line
  6.
script appends received data to the buffer to complete the line (due to wrong 
rc from step 3)

The result is two lines in the buffer (without "\n" in between). Do you 
understand now what I mean?
________________________________
Von: alex xmb sw ratchev <fxmbsw7@gmail.com<mailto:fxmbsw7@gmail.com>>
Gesendet: Montag, 7. Oktober 2024 20:02
An: Thomas Oettli <thomas.oettli@sfs.com<mailto:thomas.oettli@sfs.com>>
Cc: bug-bash@gnu.org<mailto:bug-bash@gnu.org> 
<bug-bash@gnu.org<mailto:bug-bash@gnu.org>>; 
chet.ramey@case.edu<mailto:chet.ramey@case.edu> 
<chet.ramey@case.edu<mailto:chet.ramey@case.edu>>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com>        Report as Spam / Check 
Mail<https://mail1.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007200237_2002029C296CC>

to know if read returned anything , even if its exit code is not 0 , check 
using [[ $reply ]]

i really didnt get ur english or issue

On Monday, October 7, 2024, Thomas Oettli 
<thomas.oettli@sfs.com<mailto:thomas.oettli@sfs.com>> wrote:

I know that it works in that case.
This is just an example that tries to force read into the same situatiation 
that I hit in a real world example in which I don't control the input.
________________________________
Von: alex xmb sw ratchev <fxmbsw7@gmail.com<mailto:fxmbsw7@gmail.com>>
Gesendet: Montag, 7. Oktober 2024 19:17
An: Thomas Oettli <thomas.oettli@sfs.com<mailto:thomas.oettli@sfs.com>>
Cc: bug-bash@gnu.org<mailto:bug-bash@gnu.org> 
<bug-bash@gnu.org<mailto:bug-bash@gnu.org>>; 
chet.ramey@case.edu<mailto:chet.ramey@case.edu> 
<chet.ramey@case.edu<mailto:chet.ramey@case.edu>>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com>        Report as Spam / Check 
Mail<https://mail2.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007191742_006EB2A9EAD22>

well try remove the -n in echo

On Monday, October 7, 2024, Thomas Oettli 
<thomas.oettli@sfs.com<mailto:thomas.oettli@sfs.com>> wrote:

Sorry Alex, I don't understand exactly what you mean.
Here is the Test-Script again with some comments:

    function reader() {
      local buf line
      while :; do
        read -t .01 buf                  # try to read line to $buf with timeout
        rc=$?
        if (( rc == 0 )); then           # got a full line or the rest of a 
partial line, append $buf to $line
          line+=$buf
        elif (( rc > 128 )); then        # ran into timeout
          line+=$buf                     # maybe received partial line, append 
$buf to $line and continue reading
          continue
        fi

        # at this point, the content of $line should always be "TEST"

        [[ $line != TEST ]] && echo Invalid line: $line && exit
        echo OK
        line=""                          # prepare to read next line, set $line 
to empty string
      done
    }
    reader < <(
      while :; do
        echo -n TEST
        sleep .00$(($RANDOM%10))
        echo
      done
    )

Could you please explain to me, where exactly the mistake is? I don't know if 
you ran the script by yourself, but on all of my machines it exits after just a 
few seconds
with the following output:

Invalid line: TESTTEST
________________________________
Von: alex xmb sw ratchev <fxmbsw7@gmail.com<mailto:fxmbsw7@gmail.com>>
Gesendet: Montag, 7. Oktober 2024 16:41
An: Thomas Oettli <thomas.oettli@sfs.com<mailto:thomas.oettli@sfs.com>>
Cc: bug-bash@gnu.org<mailto:bug-bash@gnu.org> 
<bug-bash@gnu.org<mailto:bug-bash@gnu.org>>; 
chet.ramey@case.edu<mailto:chet.ramey@case.edu> 
<chet.ramey@case.edu<mailto:chet.ramey@case.edu>>
Betreff: [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not follow 
guidance, click links or open attachments unless you recognize the sender and 
know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com>        Report as Spam / Check 
Mail<https://mail1.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007164140_2F94A29C1A462>

there is a case , u [[ $readreply ]] after read

On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU Bourne 
Again SHell <bug-bash@gnu.org<mailto:bug-bash@gnu.org>> wrote:
I agree with you, but it should never happen that read returns timeout, also 
returns the full line and has already read the newline character.
If that happens, there is no way for the script to decide what to do. Please 
see the provided test script, it showcases the error.

If I did a mistake there, I would gladly change it. But I currently don't see 
any way how to handle this properly from the script side.

Please also see the answer from Martin D Kealey, I think he is on to something:
https://lists.gnu.org/archive/html/bug-bash/2024-10/msg00007.html




On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
SHell wrote:

> Bash Version: 5.2
> Patch Level: 26
> Release Status: release
>
> Description:
> I have tried to write a bash script that asynchronously reads from a pipe 
> (line by line) with the help of "read -t".
> If the timeout occurs in just the right moment, read returns the full line, 
> but the return code says timeout (rc > 128).

If the read command times out, it always returns > 128, so if you have a
return code in that range, you can assume read timed out and react
accordingly.

> Therefor it is not possible to know if a full line was returned or not.

When read times out, it always returns what it read before the timeout in
the buffer, so you don't lose any data. Whether or not that's a `full line'
is up to timing, and it's up to the script to decide how to cope with it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet@case.edu<mailto:chet@case.edu> 
http://tiswww.cwru.edu/~chet/