[Bug] `SlurmClusterExecutor` resubmitting jobs that have an existing cache

[liamhuber]

I'm running the following notebook cell on MPIsusmat's CM cluster:

import os
import time

from executorlib import SlurmClusterExecutor

def foo(x):
    time.sleep(10)
    return x + 1

with SlurmClusterExecutor(
    cache_directory=os.path.join(os.getcwd(), "foo_dir"),
    resource_dict={
        "partition": "s.cmfe", 
        "cores": 1,
    }
) as exe:
    future = exe.submit(foo, 1)
    print(future.result())

Initial execution is fine: everything takes a few seconds, because of the sleep command I have a good chance to squeue | grep $USER and see my submit job, and the foo_dir directory gets created and I can see run_queue.sh and time.out and am able to watch foo..._i.h5 appear and disappear and am nicely left with foo..._o.h5.

If I re-execute the cell, everything looks fine from the perspective of my notebook: it prints my result (2) rather quickly. Certainly much more quickly than my sleep(10), so it is definitely using the cache.

But, if I am watching my directory, I can again see foo..._i.h5 get written (with the same key), and I can squeue | grep $USER and sacct -u $USER --start=2025-06-20 | wc -l to see that SLURM is re-running the job.

Since the cache is nicely leveraged in-notebook for my return, I am interpreting this re-submission as a bug. The quick return value implies to me that there is somewhere there is an effective if cache_hit() check, and probably this is as simple as some submit getting a safety valve like if cache_hit: return get_cache(); else: actuall_submit(), but I skimmed through the def submit( and couldn't see where myself.

[jan-janssen]

@liamhuber Sorry for the delay - but I am not able to reproduce the error. Is it possible that you had a different job running before?

[jan-janssen]

Maybe the issue was resolved with the changes in #681 ?

EDIT: I tried it with 1.4.0 and 1.5.0 and it seems to be working with both.

[liamhuber]

@jan-janssen, thanks for checking. I am indeed using something with #681 because I get the flat directory with ..._i/o.h5 files as described above. Unfortunately, I am 100% still encountering this error.

• Verify that I'm using the right library (1)
  • import executorlib; print("Executorlib @", executorlib.__file__) --> Executorlib @ /u/lhuber/greyhaven/executorlib/executorlib/__init__.py
  • cd /u/lhuber/greyhaven/executorlib/; git branch --> * (HEAD detached at executorlib-1.5.0)
• Verify I have no cache files lying around
  • cd ~/greyhaven/scratch/; ls --> tmp.ipynb
• Verify that I have no jobs
  • squeue | grep $USER --> nothing
• Run the first time (everything as expected) (2)
  • Restart kernel and execute cells -- operational cell thinks
  • Verify that the job is working
    • squeue | grep $USER --> 17065601 s.cmfe scratch lhuber R 0:01 1 cmti308
  • Go back and watch for the cell to finish
    • Yes, it prints 2
  • Verify that the job is finished
    • squeue | grep $USER --> nothing
  • Verify the cache existence
    • ls -> foo_dir tmp.ipynb
    • ls foo_dir -> foo3792495021fbb9b7b242427d6f3b99f8_o.h5 run_queue.sh time.out
• Restart the kernel and run again (3)
  • The operational cell is not instantaneous, but much faster than the 10s sleep
    • it prints 2
  • Quickly check slurm, then again, then wait a while and again
    • squeue | grep $USER --> 17066762 s.cmfe scratch lhuber PD 0:00 1 (None)
    • squeue | grep $USER --> 17066762 s.cmfe scratch lhuber R 0:10 1 cmti308
    • squeue | grep $USER --> Nothing
  • Re-verify cache -- am I getting a new hash key or something? No
    • ls -> foo_dir tmp.ipynb
    • ls foo_dir -> foo3792495021fbb9b7b242427d6f3b99f8_o.h5 run_queue.sh time.out

I forgot to check this time to see that foo3792495021fbb9b7b242427d6f3b99f8_i.h5 is getting re-written, but IMO the smoking gun is the SLURM job re-running. This is also the dangerous part -- from the perspective of the notebook everything looks totally fine and I get my cached result very quickly. It is silently consuming the extra slurm resources. Unless you go look at your slurm queue, you wouldn't know. This is why I feel it's critical to nail down how this error occurs; failing to do so could have real resource consumption implications.

If you are really unable to reproduce the error following my steps above, then this is great news -- it means I've probably hit some environmental edge-case that other users don't have set up. How I'm doing this is that I am running a standard CMTI conda environment, and then locally cloned executorlib and put it earlier in my python path. So I'm confident I'm running executorlib-0.5.0 (cf. the git branch check above), but maybe it's interacting poorly with my other dependencies? The only other player I can imagine here is pysqa, but maybe it's a less obvious interaction

• module list --> 1) git/2.48 2) pyiron/pyiron_mpie_cmti_2025-06-02
• conda list | grep pysqa --> pysqa 0.2.4 pyhd8ed1ab_0 conda-forge

1

2

3

[liamhuber]

I just fetched, checked out main, pulled, and did it all again. I get a different hash now, but otherwise the behaviour is all this same and the problem has not been resolved between 1.5.0 and HEAD. I also checked for the _i.h5 file, and indeed it transiently exists while the SLURM job re-runs with the exact same hash as the _o.h5 file generated on the first run.

I.e.

• run
  • runs
  • generates a new hash and therefore a new slurm job
  • after waiting, cell outputs correct result
• run again
  • cell outputs cached result very quickly
  • _i.h5 (re)written with the same hash as the existing _o.h5
  • Another SLURM job runs
  • SLURM job concludes, _i.h5 file is removed
  • _o.h5 is presumably being overwritten, but I'd only be able to tell by digging into timestamps because it has the same name and presumably the same content

[liamhuber]

I tried with pyiron/pyiron_mpie_cmti_2025-06-30 and adjusted my path to ensure that I'm using /cmmc/ptmp/pyironhb/mambaforge/envs/pyiron_mpie_cmti_2025-06-30/lib/python3.11/site-packages/executorlib, which gets me 1.4.0. The example submits a slurm job and generates input and then output cache files, but the notebook cell hangs indefinitely. I tried re-starting the notebook and re-running, but the cell still hangs and SLURM still gets a new job, even though the cache directory still shows the same hash and the same files. So at least in a pure 1.4.0-environment, this is still not working for me. Here the example doesn't even work though, so IMO it's not so much a bug as my example maybe being wrong -- did you get my example to run completely on 1.4.0??

[jan-janssen]

@liamhuber I had a different branch sorry for the confusing - can you check the fix in #697 - for me this resolves the issue.

[liamhuber]

@jan-janssen, yes, when I check out #697 the example is running as expected and I don't observe duplicate slurm submission.

[jan-janssen]

@liamhuber thanks for the confirmation, I merged the changes and I am going to release an updated 1.5.1 version today.