[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Threaded versions of cp, mv, ls for high latency / parallel filesyst
Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?
Wed, 12 Nov 2008 11:01:31 -0500
Thunderbird 184.108.40.206 (Windows/20080914)
James Youngman wrote:
This version should be race-free:
find -type f -print0 |
xargs -0 -n 8 --max-procs=16 md5sum >> ~/md5sums 2>&1
I think that writing into a pipe should be OK, since pipes are
non-seekable. However, with pipes in this situation you still have a
problem if processes try to write more than PIPE_BUF bytes.
You aren't using a pipe there. What you are doing is having the shell
open the file, then the md5sum processes all inherit that fd so they all
share the same offset. As long as they write() the entire line at once,
the file pointer will be updated atomically for all processes and the
lines from each process won't clobber each other.