Re: Pair-wise file operation (copy, link)

[Yair Lenga]

Hi Padraig,

After thinking more about my use case - I can narrow it further to linking.
If the 'ln' command will support pair-wise linking - it's
relatively trivial to implement the cp from that point:
mkdir tree
ln --pair src1/file1 tree/dest1/name1 src2/file2 tree/dest2/name2
src3/file3 tree/dest3/name3 ...
cp -r tree final-destination
rm -f tree

Even better, the '--pairs' mode can be set to take input from stdin, which
will remove limits on the number of files.

Hope the reduced scope will make it more likely to get approved.

Yair

On Sun, Aug 25, 2024 at 9:18 AM Yair Lenga <yair.lenga@gmail.com> wrote:

> Hi. Thanks for looking into this my request.
>
> In my case, I have to bulk-move about 2500 files. This is part of a
> recurring sync job that has to mirror an existing hierarchy into a new
> hierarchy with different naming rules.
>
> It takes no time to create the mapping (even in bash script, case
> statement). When I "pipe" the mapping into "ln" (with xargs) it takes >2
> min to create the symlinks. Practically, all the time is spent on launching
> "ln". With a custom perl script - it's 3 seconds.
>
> While technically a performance issue, the 2 minute is beyond my SLA (1
> minute to do the sync, including ). With the xargs approach, the solution
> does not scale well.
>
> I believe this function will have broader use cases: I researched
> Stack-Overflow before posting, and could not find any good Linux solution
> (other than custom scripts). I believe effort is relatively small.
>
> Yair
>
> On Sun, Aug 25, 2024 at 8:22 AM Pádraig Brady <P@draigbrady.com> wrote:
>
>> On 25/08/2024 12:39, Yair Lenga wrote:
>> > Greetings!,
>> >
>> > The 'cp' and 'ln' command provides the ability to perform 'bulk'
>> operation,
>> > by specifying multiple source files and a target destination. In
>> > addition to convenience, this approach provides significant
>> > performance benefits, compared with running multiple cp/ln commands, one
>> > for each file.
>> >
>> > One limit of the bulk approach is that the destination file name must
>> match
>> > the source files name. There are multiple cases where this assumption is
>> > not true. Common example include copying of source files into a new
>> > directory layout, or renaming of files that can not be expressed based
>> on
>> > rules.
>> >
>> > One common solution is to use a scripting engine (Python, Perl, Node),
>> > which can perform the operation without spinning a process per file.
>> Simple
>> > for basic operation (especially, bulk renames), but those tools do not
>> have
>> > the power/flexibility of cp,
>> >
>> > My suggestion: Add '--pairs' option to those tools, which will allow the
>> > tool to work on pairs of arguments, within the same invocation.
>> >
>> > cp --pairs ... -- source1 dest1 source2 dest2 source3 dest3
>> > mv --pairs ... -- source1 dest1 source2 dest2 source3 dest3
>> >
>> > Hope that development team will consider adding this feature into future
>> > release of cp/mv
>>
>> An interesting proposal (for cp,ln,mv,install).
>> It is a bit of an edge case though, so I'm inclined to think
>> a performance only interface change is not worth it here?
>>
>> thank you,
>> Pádraig
>>
>