[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] md5: accepts a new --threads option
From: |
Pádraig Brady |
Subject: |
Re: [PATCH] md5: accepts a new --threads option |
Date: |
Wed, 21 Oct 2009 11:41:16 +0100 |
User-agent: |
Thunderbird 2.0.0.6 (X11/20071008) |
Jim Meyering wrote:
> Pádraig Brady wrote:
>> Pádraig Brady wrote:
>>> You wouldn't want multiple threads/processes fighting over
>>> the disk head so you would do something like:
>>>
>>> find /disk1 | xargs md5sum & find /disk2 | xargs md5sum
>>>
>>> Note if we're piping/redirecting the output of the above
>>> then we must be careful to line buffer the output from md5sum
>>> so that it's not interspersed. Hmm I wonder should
>>> we linebuffer the output from *sum by default.
>> In the attached patch, I've changed the default buffering
>> to line buffered to address the above issue. For standard
>> size files there is a 2% performance drop.
>
> Good catch.
> It sounds like this fixes a real (albeit obscure) bug, so this
> might deserve a NEWS item, though I admit it is borderline.
Well it would easily be hit when one tries to parallelize the processes.
So I'll add a NEWS item and a test along the lines of:
(mkdir t && cd t && seq 100 | xargs touch)
(find t t t t -type f | xargs -n100 -P4 md5sum) | sed -n '/[0-9a-f]\{32\} /!p'
|
grep . >/dev/null && fail=1
cheers,
Pádraig.
Re: [PATCH] md5: accepts a new --threads option, Giuseppe Scrivano, 2009/10/17
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/20
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/20
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option,
Pádraig Brady <=
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
RE: [PATCH] md5: accepts a new --threads option, Voelker, Bernhard, 2009/10/21
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Eric Blake, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Paolo Bonzini, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/23