bug-coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Feature request for md5sum


From: Pádraig Brady
Subject: Re: Feature request for md5sum
Date: Fri, 19 Feb 2010 22:41:33 +0000
User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

On 19/02/10 17:55, Scott Nichol wrote:
I use md5sum a lot, often on network-filesystem-accessed files so that
readers can be sure they are reading files after writers have completed.
The performance of md5sum is generally acceptable, but for large
(gigabyte+) files even on a fast (1 Gbps) LAN, the performance would be
considerably better if I could specify on the command line a buffer size
to use.  I imagine the md5sum code using this value in a setvbuf call.
I use this technique (setting larger buffer sizes for network-accessed
files) frequently and know md5sum and other utilities would benefit from
it.

Yes they would benefit, however it would be nice
not to push this complexity into the tools.

Note the buffer size has changed in v8.1 from 4KiB to 32KiB,
which might help.

Also since v7.5 there is a stdbuf utility available
which can be used to change the buffer size for stdin at least:
  stdbuf -i1M md5sum ...
Note there were issues with glibc when I was testing that,
so don't be surprised if the setting ignored.

Note the linux nfs client should be doing adaptive
read ahead to abstract you from this latency issue.
Check the rsize nfs setting as that may limit the
size of read ahead done.

cheers,
Pádraig.




reply via email to

[Prev in Thread] Current Thread [Next in Thread]