Skip to content

Commit b5d66fd

Browse files
committed
feat: warning if run in job (#78)
Already, two issues (#75 and #22) seem to result from running Snakemake in a SLURM job context and using this executor plugin. This PR introduces detecting if triggered within a SLURM job and issuing a warning accordingly. In principle, the plugin may work in job context. Submitting jobs from jobs has always been a highlight of SLURM. However, settings may lead to unintended behaviour (and would do so without Snakemake, presumably). Hence, we can only warn from the executor.
1 parent ffff12f commit b5d66fd

File tree

2 files changed

+19
-0
lines changed

2 files changed

+19
-0
lines changed

docs/further.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,12 @@ set-resources:
291291
Be sure to use sensible settings for your cluster and make use of parallel execution (e.g. threads) and [global profiles](#using-profiles) to avoid I/O contention.
292292

293293

294+
## Nesting Jobs (or Running this Plugin within a Job)
295+
296+
Some environments provide a shell within a SLURM job, for instance, IDEs started in on-demand context. If Snakemake attempts to use this plugin to spawn jobs on the cluster, this may work just as intended. Or it might not: depending on cluster settings or individual settings, submitted jobs may be ill-parameterized or will not find the right environment.
297+
298+
If the plugin detects to be running within a job, it will therefore issue a warning and stop for 5 seconds.
299+
294300
## Summary:
295301

296302
When put together, a frequent command line looks like:

snakemake_executor_plugin_slurm/__init__.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,11 +51,24 @@
5151
# Implementation of your executor
5252
class Executor(RemoteExecutor):
5353
def __post_init__(self):
54+
# run check whether we are running in a SLURM job context
55+
self.warn_on_jobcontext()
5456
self.run_uuid = str(uuid.uuid4())
5557
self.logger.info(f"SLURM run ID: {self.run_uuid}")
5658
self._fallback_account_arg = None
5759
self._fallback_partition = None
5860

61+
def warn_on_jobcontext(self, done=None):
62+
if not done:
63+
if "SLURM_JOB_ID" in os.environ:
64+
self.logger.warning(
65+
"Running Snakemake in a SLURM within a job may lead"
66+
" to unexpected behavior. Please run Snakemake directly"
67+
" on the head node."
68+
)
69+
time.sleep(5)
70+
done = True
71+
5972
def additional_general_args(self):
6073
return "--executor slurm-jobstep --jobs 1"
6174

0 commit comments

Comments
 (0)