bug#13530: head: memory exhausted when printing all from stdin but last P/E bytes

[Pádraig Brady]

On 05/28/2013 02:38 AM, Jim Meyering wrote:
> Paul Eggert wrote:
>> On 05/27/2013 06:04 PM, Jim Meyering wrote:
>>> +lim=$(echo $SIZE_MAX | subtract_one_)
>>> +lim=$(expr $lim - $max_BUFSIZ)
>>
>> Sorry, I don't see how this will work either.
>> It's common for a GMP-less expr to handle values
>> only up to SIZE_MAX / 2, and subtracting just 1
>> won't work around that problem.
> 
> I was concentrating on the 32-bit case, where GMP-less expr
> works fine on $SIZE_MAX.  It's on 64-bit that it's a problem, so...
> 
>> Maybe divide by 10 instead?  That's easy to do
>> textually.  (I don't know what the test is about
>> so I'm not sure what to suggest.)
> 
> I'd rather not divide by 10 all around.
> That'd decrease it too much in the 32-bit case.
> I took a different tack:
> 
> 
>>From 0d81799b18bef79b49d9042111f9dee3312796a7 Mon Sep 17 00:00:00 2001
> From: Jim Meyering <address@hidden>
> Date: Mon, 27 May 2013 17:01:14 -0700
> Subject: [PATCH] tests: head-c: avoid spurious failure with a 32-bit SIZE_MAX
> 
> * tests/misc/head-c.sh: When eliding N bytes from a non-seekable
> input, N must be slightly smaller than SIZE_MAX in order to handle
> input longer than N bytes, since the current implementation buffers
> N bytes in memory.  This command would fail on 32-bit systems,
> where SIZE_MAX < 1E:
>   head --bytes=-E < /dev/null
> Instead of "E", use a value slightly smaller than SIZE_MAX.
> ---
>  tests/misc/head-c.sh | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/misc/head-c.sh b/tests/misc/head-c.sh
> index 37a86ce..a81754e 100755
> --- a/tests/misc/head-c.sh
> +++ b/tests/misc/head-c.sh
> @@ -19,6 +19,7 @@
>  . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src
>  print_ver_ head
>  require_ulimit_v_
> +getlimits_
> 
>  # exercise the fix of 2001-08-18, based on test case from Ian Bruce
>  echo abc > in || framework_failure_
> @@ -28,9 +29,27 @@ case "$(cat out)" in
>    *) fail=1 ;;
>  esac
> 
> +# Use a limit of N = SIZE_MAX - max_BUFSIZ
> +# The "- max_BUFSIZ" term is because head must be able to add BUFSIZ
> +# to the selected value of N without exceeding SIZE_MAX.
> +# Since we've seen BUFSIZ up to 128K, use 256K to be safe.
> +max_BUFSIZ=$(expr 256 '*' 1024)
> +
> +# It's ok to use a 10-digit $SIZE_MAX, because expr uses wider intmax_t.
> +# However, when $SIZE_MAX is longer, it's a 20-digit quantity that is
> +# too large for GMP-disabled expr to handle, so use $SSIZE_MAX there.
> +# We'd use $SSIZE_MAX in both cases, but want to keep the number as
> +# large as possible for the 32-bit case, since there, it may barely
> +# exceed the size of available RAM, while SSIZE_MAX probably will not.
> +test $(printf $SIZE_MAX|wc -c) -lt 11 \
> +  && big=$SIZE_MAX \
> +  || big=$SSIZE_MAX
> +
> +lim=$(expr $big - $max_BUFSIZ) \

trailing \ is not needed.

The above assumes intmax_t is always wider than size_t,
which is might be OK?
Also since OFF_T_MAX is checked earlier in the code,
it assumes that off_t is wider than size_t which isn't
the case if _FILE_OFFSET_BITS is not 64.

You could take the approach to just use $SSIZE_MAX,
and assume 64 bit testing covers the 32 bit case implicitly?
That would also assume that off_t was never narrower than size_t,
but that's probably OK.

These limits are painful. Sorry for the churn :(

thanks,
Pádraig.