[Documentation] Worker for executing multiple Python functions

[jan-janssen]

There are basically three ways to use a Python Task Managers:

• Submit each Python Function as an individual job to the HPC cluster - supported in executorlib by the SlurmClusterExecutor and the FluxClusterExecutor.
• Submit multiple Python Functions within one HPC job allocation - supported in executorlib by the SlurmJobExecutor and the FluxJobExecutor.
• Submit multiple Python Functions to the same resources on a given HPC cluster - similar to the block_allocation=True for the SingleNodeExecutor, the SlurmJobExecutor and the FluxJobExecutor but for the SlurmClusterExecutor and the FluxClusterExecutor. This is the standard use case for solutions like https://www.dask.org and https://parsl.readthedocs.io/ so it might be an interesting addition to executorlib.

[jan-janssen]

It is basically similar to the MPIExecSpawner just using pysqa for the submission of the new process via sbatch rather than srun. Still the challenging part is the submission of new workers or the handling of the case when workers are removed from the pool and do not complete a given task.

[jan-janssen]

Current challenges would be:

• keep track of the running workers - I can validate those via the queuing system of the cluster
• I could start an executor with zero workers and then shut it down when all the futures are evaluated
• If a worker gets killed then it is important that the task is returned to the pool and another worker can start processing it - I have to understand how the queues in Python work

[jan-janssen]

This functionality was implemented in #763