Re: parallel sort at fault? [Re: [PATCH] tests: avoid gross inefficiency...

[Pádraig Brady]

On 16/03/11 15:32, Jim Meyering wrote:
> Pádraig Brady wrote:
>>
>> With SUBTHREAD_LINES_HEURISTIC=128k and -S1M option to sort we get no 
>> threads as
>> nlines never gets above 12787 (there looks to be around 80 bytes overhead 
>> per line?).
>> Only when -S >= 12M do we get nlines high enough to create threads.
> 
> Thanks for pursuing this.
> Here's a proposed patch to address the other problem.
> It doesn't have much of an effect (any?) on your
> issue when using very little memory, but when a sort user
> specifies -S1M, I think they probably want to avoid the
> expense (memory) of going multi-threaded.
> 
> What do you think?
> 
> -#define INPUT_FILE_SIZE_GUESS (1024 * 1024)
> +#define INPUT_FILE_SIZE_GUESS (128 * 1024)

This does seem a bit like whack-a-mole
but at least we're lining them up better.

The above gives reasonable threading by default,
while reducing the large upfront malloc.

$ for len in 1 79; do
    for i in $(seq 22); do
      lines=$((2<<$i))
      yes "$(printf %${len}s)"| head -n$lines > t.sort
      strace -f -c -e clone ./sort --parallel=16 t.sort -o /dev/null 2>&1 |
      join --nocheck-order -a1 -o1.4,1.5 - /dev/null |
      sed -n "s/\([0-9]*\) clone/$lines\t\1/p"
    done
  done

#lines  threads (2 byte lines)
------------------------------
131072  1
262144  3
524288  7
1048576 15
2097152 15
4194304 15
8388608 15

#lines  threads (80 byte lines)
------------------------------
131072  1
262144  3
524288  7
1048576 15
2097152 15
4194304 22
8388608 60

cheers,
Pádraig.