[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Pair-wise file operation (copy, link)
From: |
Yair Lenga |
Subject: |
Re: Pair-wise file operation (copy, link) |
Date: |
Mon, 26 Aug 2024 08:03:34 -0400 |
good point. I'm running in a VM that is mounting NFS volume. may be part of
the problems (slow exec) are related to search path/slow exec. I'll check
and revert.
On Sun, Aug 25, 2024, 19:34 Glenn Golden <gdg@zplane.com> wrote:
> Yair Lenga <yair.lenga@gmail.com> [1970-01-01 00:00:00 +0000]:
> >
> > In my case, I have to bulk-move about 2500 files. This is part of a
> > recurring sync job that has to mirror an existing hierarchy into a new
> > hierarchy with different naming rules.
> >
> > It takes no time to create the mapping (even in bash script, case
> > statement). When I "pipe" the mapping into "ln" (with xargs) it takes >2
> > min to create the symlinks. Practically, all the time is spent on launch=
> > ing "ln". With a custom perl script - it's 3 seconds.
> >
>
> 2c observation:
>
> Years ago I had a similar weekly need at work, except for an even larger
> number of files (10k - 20k or so iirc), and always used a one-liner xargs
> script to do the copy. My recollection is that it would complete in "a
> few"
> seconds (maybe 10s or so). I couldn't find that script, but I just tried
> it now manually: Created 2500 randomly named files, each comprising 4kB
> random data, and then copied them to new names like this
>
> $ cat fmap | xargs -L1 cp
>
> where fmap is the name-mapping file, comprising 2500 lines like
>
> oldname0 newname0
> oldname1 newname1
> oldname2 newname2
> . .
> . .
> . .
>
> It took under 4 seconds, plus another 1-2 seconds for the sync. This was
> on
> a commodity x86_64 laptop. The target filesystem was the same as original.
> Device is a slow 20-year old HDD.
>
> Since you were reporting 2 min, was wondering what your platform is and
> whether there might be something else involved eating the 2 min realtime?
>
> Glenn
>