Re: [PATCH] Speedup wc -l

[Pádraig Brady]

On 19/03/15 23:30, Bernhard Voelker wrote:
> On 03/19/2015 11:18 AM, Pádraig Brady wrote:
>> Yes 30 was a bit aggressive.
>> I had already changed it to 15 (150) locally here after more testing.
> 
> Here I'm getting best results when using 100 as the limit, i.e., with
> an average of 9 characters in the first 10 lines; 100 because of:
>   (9 chars + 1 '\n') * 10 = 100
> 
> This is the script I was using (openSUSE-13.2, i5-4570 CPU, 20GB RAM):
> 
>   #!/bin/sh
>   file='file.txt'
>   # Test with various line lengths.
>   for len in 0 1 2 5 6 7 8 9 10 11 15 20 50 100 1000; do
>     printf "\n====== %s ======\n" "$len"
>     # Prepare a 500M file with LEN characters per line.
>     yes "$(seq -s '' 1000 | head -c $len)" \
>       | src/dd iflag=fullblock status=none bs=1M count=500 of="$file"
>     # Do 5 test runs with the old/new 'wc'.
>     for i in $(seq 3); do
>       # dummy run
>       wc -l "$file" > /dev/null
>       for wc in src/wc-before src/wc; do
>         printf "\n= %s =\n" "$wc"
>         time "$wc" -l "$file"
>       done
>     done
>   done

Thanks for this. I'm still getting 150 for the limit with this on my i3-2310M
I guess either the function call overhead is slightly lower or
the memchr code is slightly more efficient on your newer processor.

> You are of course encouraged to play with/reduce the list for 'len'
> in that script.
> 
> Other than that, the patch looks great per se. I couldn't find a way
> to make it faster than that.
> 
> One thing: do you think that the compiler will get better with
> memchr() for short lines some day?  Then we should maybe add a
> "FIXME: re-evaluate magic number (100 or 150) in 2017".

I'll add the comment.

> Finally: can we get some performance test results from some other
> platforms (ARM, PPC, 32 vs 64 bit)? I mean, this patch helps on
> Linux/x86_64, but if it makes things worse on other platforms, then
> I don't think complicating the code is worth it.

Cut-off is around 15 also on PPC (IBM,8231-E2B)

cheers,
Pádraig.