af_alg: Fail in continuable manner on Linux/powerpc64le

[Bruno Haible]

This fixes the error message reported by Assaf Gordon on non-seekable
input.

Four approaches come to mind for fixing this:
 1) Change the afalg_stream signature, adding an argument where the function
    can store read but not yet processed input.
 2) Change the afalg_stream signature, passing it the fallback computation
    function as argument. So that afalg_stream can invoke this function on
    read input that it could not process itself.
 3) Try the first send() with 1 byte only, because upon failure it can be
    pushed back onto the stream.
 4) Try the first send() with the number of bytes returned by freadptr,
    instead of fread().

The first two approaches would be a major code change, for just a little
particular case. I prefer the third approach.


2018-06-24  Bruno Haible  <address@hidden>

        af_alg: Fail in continuable manner on Linux/powerpc64le.
        Reported by Assaf Gordon <address@hidden>
        in <https://lists.gnu.org/archive/html/coreutils/2018-06/msg00034.html>.
        * lib/af_alg.c (afalg_stream): On non-seekable streams, try a single-
        byte send() as the first round.

diff --git a/lib/af_alg.c b/lib/af_alg.c
index 2753ab9..549cdcf 100644
--- a/lib/af_alg.c
+++ b/lib/af_alg.c
@@ -157,7 +157,10 @@ afalg_stream (FILE *stream, const char *alg,
       for (;;)
         {
           char buf[BLOCKSIZE];
-          ssize_t size = fread (buf, 1, sizeof buf, stream);
+          /* When the stream is not seekable, start with a single-byte block,
+             so that we can use ungetc() in the case that send() fails.  */
+          size_t blocksize = (nseek == 0 && off < 0 ? 1 : BLOCKSIZE);
+          ssize_t size = fread (buf, 1, blocksize, stream);
           if (size == 0)
             {
               /* On Linux < 4.9, the value for an empty stream is wrong (all 
0).
@@ -170,8 +173,18 @@ afalg_stream (FILE *stream, const char *alg,
           nseek -= size;
           if (send (ofd, buf, size, MSG_MORE) != size)
             {
-              result = (fseeko (stream, nseek, SEEK_CUR) == 0
-                        ? -EAFNOSUPPORT : -EIO);
+              if (nseek == -1)
+                {
+                  /* 1 byte of pushback buffer is guaranteed on stream, even
+                     if stream is not seekable.  */
+                  ungetc ((unsigned char) buf[0], stream);
+                  result = -EAFNOSUPPORT;
+                }
+              else if (fseeko (stream, nseek, SEEK_CUR) == 0)
+                /* The position of stream has been restored.  */
+                result = -EAFNOSUPPORT;
+              else
+                result = -EIO;
               break;
             }
 
@@ -187,6 +200,7 @@ afalg_stream (FILE *stream, const char *alg,
       if (result == 0 && read (ofd, resblock, hashlen) != hashlen)
         {
           if (nseek == 0 || fseeko (stream, nseek, SEEK_CUR) == 0)
+            /* The position of stream has been restored.  */
             result = -EAFNOSUPPORT;
           else
             result = -EIO;