[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: GNU Parallel Bug Reports A suggestion: --shuf and -k

From: Rob Sargent
Subject: Re: GNU Parallel Bug Reports A suggestion: --shuf and -k
Date: Fri, 30 Jun 2017 15:46:54 -0600

(If jobs can be started independent of order, so too is the analysis of the 
output. From your description, the problem is solved with a call to sort.)
> On Jun 30, 2017, at 3:11 PM, paralleluser <address@hidden> wrote:
> Friends
> A suggestion that merits your comments and review:
> --shuf does exactly what the man page says it does, but when you combine 
> --shuf and -k, the -k does nothing, --shuf rules over -k
> I'm going to propose that combining --shuf and -k that this happens:
>       the jobs are still processed randomly
>       but the output be in order as the true input
> When do you use this?  Assume your input to parallel process is:
>       server1/resource1
>       server1/resource2
>       server1/resource3
>       ...etc...
>       server2/resource1
>       server2/resource2
>       ...etc...
>       ...up to server50
> For human processing reasons, it is easier to keep all the server/resource 
> input lines in ASCII sort order
> For computer processing reasons, server 1 is going to hate you if you are 
> hitting it with a lot of requests all at the same time
> Thus with the "--shuf -k" combo, the sever loads will be spread around, but 
> you will get your data back in the same order.
> Comments welcome........thanks

reply via email to

[Prev in Thread] Current Thread [Next in Thread]