Re: Pair-wise file operation (copy, link)

[Glenn Golden]

Yair Lenga <yair.lenga@gmail.com> [1970-01-01 00:00:00 +0000]:
> 
> In my case, I have to bulk-move about 2500 files. This is part of a
> recurring sync job that has to mirror an existing hierarchy into a new
> hierarchy with different naming rules.
> 
> It takes no time to create the mapping (even in bash script, case
> statement). When I "pipe" the mapping into "ln" (with xargs) it takes >2
> min to create the symlinks. Practically, all the time is spent on launch=
> ing "ln". With a custom perl script - it's 3 seconds.
> 

2c observation:

Years ago I had a similar weekly need at work, except for an even larger
number of files (10k - 20k or so iirc), and always used a one-liner xargs
script to do the copy.  My recollection is that it would complete in "a few"
seconds (maybe 10s or so). I couldn't find that script, but I just tried
it now manually: Created 2500 randomly named files, each comprising 4kB
random data, and then copied them to new names like this

    $ cat fmap | xargs -L1 cp

where fmap is the name-mapping file, comprising 2500 lines like

        oldname0   newname0
        oldname1   newname1
        oldname2   newname2
           .          .
           .          .
           .          .

It took under 4 seconds, plus another 1-2 seconds for the sync.  This was on
a commodity x86_64 laptop. The target filesystem was the same as original.
Device is a slow 20-year old HDD.

Since you were reporting 2 min, was wondering what your platform is and
whether there might be something else involved eating the 2 min realtime?

Glenn