So i think tools like Slurm have more of an emphasis on system resource management. You manage system resources using job queues, time, etc. I think my use-case is related, hence somebody though of Slurm and likes. However, the management i need here is data oriented: ie i want to use data to determine the jobs, as opposed to using job queues, quotas, and other such system environment resources. That is, i want to have jobs determined by what kind of data i have, and not so much jobs determined or directly limited by the kind of system resources(CPUs, GPUs, time/disk quota, etc). The system resources are used indirectly. The tool i need here is a tool to help me synthesize the jobs based on rules i describe using the tool. Once the jobs are described, then the system resources begin to matter. The jobs i describe here are dynamic- they can change depending on what data is coming into this tool. As data comes into the tool, jobs are created according to the provided to the tool, but some data can come in that causes existing jobs to be cancelled, or changed to be processed differently than was planned at initial setting,etc.
It sounds like something you could probably do with the combination of BAsh and Parallel, but expressing this with BAsh is can be very hard to get right- in the same sense that writing your code in 0s and 1s is hard even though not impossible.