Question: --limit tool?

[lists . gnu . org+parallel]

Hello Parallelizers!

I'm using GNU Parallel to run the AWS CLI utility, and I want to use Parallel's --limit option to avoid being throttled.

Before I sit down and write something, has anybody already invented this wheel?

Essentially, AWS has a token-bucket rate-limiting system:

* Every API has two numbers associated with it: a bucket size (integer) and a refill rate (integer). 

* Every second, the bucket's value is incremented by the second number, until it reaches the first number. 

* Every unthrottled request decrements the bucket's value by one. 

* If the bucket's value is zero, requests are denied with a throttling error.

The script would emulate this logic, and given the two numbers (bucket size and rate) return a '1' if a throttling error would occur and a '0' otherwise.

It would be nice if the script could auto-detect the two numbers, but it's unclear how that could be done without being throttled at least twice.  Maybe it could analyze the job log of a previous run?

Thanks!

--

CH (lists.gnu.org+parallel@ch.pkts.ca)