[Feature] How to set `cache_directory` at submission time?

[liamhuber]

I'm still working on getting executorlib integrated into pyiron_workflow by wrapping the executors so that the cache directory/key is automatically set using lexical information in the workflow -- i.e. to set the path to the cache according to the labels/parent labels of nodes in the graph. This is working perfectly for SingleNodeExecutor, but SlurmClusterExecutor is locking in the cache_directory at initialization (and indeed creating the folder then too #704).

This is not an absolute show-stopper -- I could leverage the cache_key exclusively, or set certain expectations/demands on how a SlurmClusterExecutor is accessed by pyiron_workflow users -- but the most convenient and intuitive for me would be to simply have it behave as like the SingleNodeExecutor.

Concretely, when I run this:

import os
import time

from executorlib import SingleNodeExecutor

def foo(x):
    time.sleep(3)
    return x + 1

with SingleNodeExecutor(
    cache_directory="not_this_dir",
) as exe:
    future = exe.submit(
        foo, 
        1, 
        resource_dict={
            "cache_directory": "rather_this_dir", 
            "cache_key": "foo",
        },
    )
    print(future.result())

It is nicely putting all the caching in ./rather_this_dir. But with SLURM:

import os
import time

from executorlib import SlurmClusterExecutor

def foo(x):
    time.sleep(3)
    return x + 1

with SlurmClusterExecutor(
    cache_directory="not_this_dir",
    resource_dict={"partition": "s.cmfe"}
) as exe:
    future = exe.submit(
        foo, 
        1, 
        resource_dict={
            "cache_directory": "rather_this_dir", 
            "cache_key": "foo",
        },
    )
    print(future.result())

It runs OK and recognizes my "cache_key", but puts everything in ./not_this_dir.

I can see where the file-based system writes the undesirable directory too early (#704) but when I tried to figure out how to make this modification myself I wound up getting very lost (#703). Is such a modification possible?

EDIT: completed hyperlinks

[jan-janssen]

I created a separate issue to document the local debugging of the SlurmClusterExecutor and FluxClusterExecutor. #708