bug-coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] sort: use posix_fadvise to announce access patterns on file


From: Joey Degges
Subject: Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading
Date: Tue, 2 Mar 2010 10:35:27 -0800

2010/3/2 Pádraig Brady <address@hidden>

> On 02/03/10 06:32, Joey Degges wrote:
>
>> 2010/3/1 Joey Degges<address@hidden>
>>
>>  Thanks for testing, these results are interesting. I will run some tests
>>> on
>>> various systems tonight and report back. Maybe my earlier results were
>>> caused by peculiar hardware.
>>>
>>>
>>>  Results differ once again, this time I've tested on a different system.
>>
>> Here is pastebin of the code I used to spin. Its some math in a large
>> loop,
>> should run for ~5 minutes (enough time to do most tests):
>> http://codepad.org/tCtjLw7H
>>
>> CPU: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
>> RAM: 2G
>> kernel: 2.6.31-19-generic #56-Ubuntu SMP x86_64
>> input file: 97M, gensort -a 1000000 file
>> cmd: umount /mnt&&  mount /dev/foo /mnt&&  /usr/bin/time -v sort<  file>
>> /dev/null
>>
>> mechanical disk @ 80MB/s, ext3 - 4x spinning processes
>> mode      :   real, %CPU, voluntary/involuntary context switches
>> NORMAL    : 14.73s,  10%, 408/34
>> SEQUENTIAL: 10.95s,  13%, 220/52
>> WILLNEED  : 05.22s,  35%, 026/57
>>
>> flash key @ 20MB/s, ext4 - 4x spinning processes
>> NORMAL    : 88.49s,  01%, 1521/42
>> SEQUENTIAL: 69.76s,  02%, 1131/57
>> WILLNEED  : 50.29s,  03%, 0174/31
>>
>> flash key @ 11MB/s, FAT32 - 4x spinning processes
>> NORMAL    : 106.5s,  01%, 1334/32
>> SEQUENTIAL: 85.63s,  01%, 0765/53
>> WILLNEED  : 68.70s,  02%, 0193/59
>>
>> mechanical disk @ 80MB/s, ext3
>> NORMAL    : 02.70s,  64%, 748/9
>> SEQUENTIAL: 02.71s,  61%, 374/8
>> WILLNEED  : 02.79s,  57%, 026/5
>>
>> flash key @ 20MB/s, ext4
>> NORMAL    : 06.25s,  25%, 1505/8
>> SEQUENTIAL: 06.18s,  26%, 1136/11
>> WILLNEED  : 06.10s,  26%, 0156/8
>>
>> flash key @ 11MB/s, FAT32
>> NORMAL    : 10.23s,  15%, 1320/7
>> SEQUENTIAL: 10.12s,  16%, 0762/7
>> WILLNEED  : 10.14s,  16%, 0193/4
>>
>>
>> Note the large difference in the number of voluntary context switches
>> between the different modes. This number appears to only be dependent on
>> the
>> device being read from and the advice mode, not the system load. When the
>> system is heavily loaded the process that context switches the least will
>> be
>> able to consume more resources. From these results you can see that
>> SEQUENTIAL reduces the voluntary context switching from 25-46% and
>> WILLNEED
>> reduces the voluntary context switching from 85-96%. This also explains
>> why
>> you won't notice the improvements due to SEQUENTIAL or WILLNEED very much
>> in
>> unburdened systems.
>>
>
> Thanks for the thorough testing.
> That confirms the context switch is done when the read()
> needs data to be retrieved from disk, and that it has a
> large affect on your sort on multicore, whereas it has
> no affect on my single core.
>
> I'm still unsure as to whether we should enable WILLNEED
> as I've got a counter point in another email I'll send in a while.
> Also I'm worried in the low RAM case that WILLNEED will thrash
> other caches, perhaps useful to sort itself if it's processing
> multiple files or running in merge mode. I'll do some more
> testing with a view to enabling WILLNEED, but in any case
> SEQUENTIAL should also be enabled so that readahead is done
> on any portion of the file that it's not possible to
> prepopulate the cache with.
>
>
Agreed, you have convinced me :]

In the future when/if the WILLNEED implementation becomes asynchronous this
might be worth investigating once again.

Thanks,
Joey


reply via email to

[Prev in Thread] Current Thread [Next in Thread]