Re: Pair-wise file operation (copy, link)

[Yair Lenga]

Sorry for delay. See below for my results (WSL/Ubunto):
1000 files: ln -sf
* using xargs: clock: 1.58s, cpu: 0.27u, 0.30s
* using pair-wise (Perl) clock: 0.10, cpu: 0.04u ,0.09s

10000 files: ln -sf
* Using xargs: clock:16.7s, cpu: 3.07u, 3.38s
* Using pair-wise (Perl): clock: 0.59, cpu: 0.13u, 0.46s

Estimating overhead of using xargs is 10X - about 16 milliseconds per file,
vs 1.6 milliseconds vs using single process.

The performance difference is 10X, and for many use case, it's far below
SLA, which will necessitate a performance driven solution, where I think
built-in 'pair-wise' option for ln & mv is much better vs. "do it yourself"
ad-hoc script.

I also did the same measurement for 'cp' - with a file size of 1K:
1000 files: cp -f
* Using xargs: clock: 2.26, cpu: 0.30u, 0.67s
* Using pair-wise cp (Perl) clock: 0.15, cpu: 0.05u, 0.10s

And repeated for 10k. It actually show that cp (of small file), the
pair-wise copy will be more performant vs, xargs cp solution. I assume this
has to do with buffering/caching/... For my use case, suppoting just
pair-wise on 'ln' and 'mv' will address my problem. Will be happy to see
this imprvoemtn coming.

Thanks in advance.

Yair


On Tue, Aug 27, 2024 at 1:01 AM Rob Landley <rob@landley.net> wrote:

> On 8/25/24 18:34, Glenn Golden wrote:
> > Since you were reporting 2 min, was wondering what your platform is and
> > whether there might be something else involved eating the 2 min realtime?
>
> Slowest exec I've had to deal with on a modern still-active system is mac
> homebrew, which was about 1/3 of a second each exec() on fairly powerful
> hardware. (It did not LIKE running ELF binaries on a mach-o kernel.)
>
> Of course "this posix system can't really do shell scripts" means your OS
> is
> kinda conceptually broken...
>
> Rob
>