SQL save mode for GNU Parallel?

[Ole Tange]

I just discovered a fork of GNU Parallel:
https://github.com/stephen-fralich/parallel-sql/

It saves into PostgreSQL.

If GNU Parallel should have an --sql option, it should be more general
than that. It would be obvious to use a DBURL to specify which driver,
username, password, and database to use.

The most obvious would be having a table containing the columns from
--joblog and the arguments. For some uses it would also make sense to
have the stderr+stdout.

So I am thinking of:

  --sql mysql://user:pass@host/db/table

If the table does not exist: Create it.

But should there be an option to not store stderr+stdout? And if so:
Should that be default? If saving is forced, then you can always just
>/dev/null the output from the job.

I can definitely see uses of being able to run 1000000 simulations
with 10 different variables and then be able to easily get the output
of the jobs where variable A is odd and > variable B (or similar).

What should happen if the user uses variable names that are the same
as the header of --joblog (e.g. Seq or stdout)?

It would also be handy if you could change the status of a job to
'not-run' (which could be represented with exit status -2), so you
could change this while GNU Parallel was running or add new jobs.

You could then have workers that did took jobs out of a database table:

  forever parallel --sql mysql://user:pass@host/db/table

And a master node that submitted jobs to the table:

  parallel --dry-run --sql mysql://user:pass@host/db/table the_job ::: the args

--dry-run with --sql should put status to 'not-run'.

But that would also require some sort of handling of timeout: worker-2
has started job-seq-4 3 seconds ago, and should not be considered
timed out, thus no other worker should take that job.

GNU Parallel will not depend on DBD-packages installed, but will only
used these when the user asks for the driver. So in package speak it
should probably 'suggest' the DBD-packages.

Ideas? Suggestions? Observations?


/Ole