coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pair-wise file operation (copy, link)


From: Yair Lenga
Subject: Re: Pair-wise file operation (copy, link)
Date: Mon, 26 Aug 2024 08:03:34 -0400

good point. I'm running in a VM that is mounting NFS volume. may be part of
the problems (slow exec) are related to search path/slow exec. I'll check
and revert.

On Sun, Aug 25, 2024, 19:34 Glenn Golden <gdg@zplane.com> wrote:

> Yair Lenga <yair.lenga@gmail.com> [1970-01-01 00:00:00 +0000]:
> >
> > In my case, I have to bulk-move about 2500 files. This is part of a
> > recurring sync job that has to mirror an existing hierarchy into a new
> > hierarchy with different naming rules.
> >
> > It takes no time to create the mapping (even in bash script, case
> > statement). When I "pipe" the mapping into "ln" (with xargs) it takes >2
> > min to create the symlinks. Practically, all the time is spent on launch=
> > ing "ln". With a custom perl script - it's 3 seconds.
> >
>
> 2c observation:
>
> Years ago I had a similar weekly need at work, except for an even larger
> number of files (10k - 20k or so iirc), and always used a one-liner xargs
> script to do the copy.  My recollection is that it would complete in "a
> few"
> seconds (maybe 10s or so). I couldn't find that script, but I just tried
> it now manually: Created 2500 randomly named files, each comprising 4kB
> random data, and then copied them to new names like this
>
>     $ cat fmap | xargs -L1 cp
>
> where fmap is the name-mapping file, comprising 2500 lines like
>
>         oldname0   newname0
>         oldname1   newname1
>         oldname2   newname2
>            .          .
>            .          .
>            .          .
>
> It took under 4 seconds, plus another 1-2 seconds for the sync.  This was
> on
> a commodity x86_64 laptop. The target filesystem was the same as original.
> Device is a slow 20-year old HDD.
>
> Since you were reporting 2 min, was wondering what your platform is and
> whether there might be something else involved eating the 2 min realtime?
>
> Glenn
>


reply via email to

[Prev in Thread] Current Thread [Next in Thread]