Re: Feature request: --joblog .json

[CH]

I'm sure Ole would make it optional, e.g. checking if --joblog argument 
ends with .json.

One more feature request: skipping a job if the results file exists or 
is younger than X minutes.

Use case: A cache directory of job outputs, where old files can be 
overwritten.

Currently:
* --resume will work out where to resume by looking in the --result 
directory
* --retry-failed will rerun failed jobs in the --joblog file
* --resume-failed will do both

None of these are perfect, although --resume is close.

Initial effort:

  FILES="cloudformation-{1}-{2}.json"
  COMMAND="aws --profile {1} --region {2} cloudformation 
describe-stacks"
  ARGS=("${PROFILES[@]}" ::: "${REGIONS[@]}")

  parallel --result "$FILES" "$COMMAND" ::: "${ARGS[@]}" > /dev/null

Our current workaround:

  # Delete if older than 12 hours
  find . -maxdepth 1 -name 'cloudformation-*.json' -mmin +1440 -type f 
-delete

  parallel "F='$FILES' ; [ -e \"\$F\" ] || ( $COMMAND ) > \"\$F\"" ::: 
"${ARGS[@]}"

Maybe '--retry-older 1440'?

Thanks

On 2020-12-21 06:56 PM, Rob Sargent wrote:

> Yes, that'll do nicely, thank you!
>
> > Next thing you know, you'll be making the joblog into json format!  
> > :-)
>
> Please do NOT do that
--
Penelope Fudd (parallel@ch.pkts.ca)