Re: [coreutils] How do you copy 60 million files?

[Pádraig Brady]

On 24/09/10 19:02, Paul Eggert wrote:
> OK, OK, so I shouldn't waste time reading The Register, but I do,
> so I can't resist sharing a pointer to this story:
> 
> Pott T. How do you copy 60m files? The Register (2010-09-24)
> <http://www.theregister.co.uk/2010/09/24/sysadmin_file_tools/>
> 
> Basically, Pott's problem was that he had to copy 60 million
> files from one Windows file server to another.  He tried all
> sorts of Windowish tools, from Microsoft and from other suppliers.
> 
> Then the light bulb hit: he booted CentOS on a Linux virtual
> machine, mounted the source and destinations as SMB shares,
> and used coreutils' "cp".  Voilà.  His conclusion:
> 
> "So the best way to move 60 million files from one Windows server
> to another turns out to be: use Linux."

Cool, though he was stretching a bit to say cp was
better than richcopy given the latency of cp doing
a read then write over the network. For 60m files, ouch!

This would have been better, with 2 processes
reading and writing asynchronously, with a
little buffering in between.

(cd /server1 && tar -c) | (cd /server2 && tar -xp)

One could bump up the buffering by placing
`pv` in the pipe between the 2, and get a
progress bar also.

That reminds me of this post:
http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html

cheers,
Pádraig.