Skip to content

feat: warning if run in job #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/further.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,12 @@ set-resources:
Be sure to use sensible settings for your cluster and make use of parallel execution (e.g. threads) and [global profiles](#using-profiles) to avoid I/O contention.


## Nesting Jobs (or Running this Plugin within a Job)

Some environments provide a shell within a SLURM job, for instance, IDEs started in on-demand context. If Snakemake attempts to use this plugin to spawn jobs on the cluster, this may work just as intended. Or it might not: depending on cluster settings or individual settings, submitted jobs may be ill-parameterized or will not find the right environment.

If the plugin detects to be running within a job, it will therefore issue a warning and stop for 5 seconds.

## Summary:

When put together, a frequent command line looks like:
Expand Down
13 changes: 13 additions & 0 deletions snakemake_executor_plugin_slurm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,24 @@
# Implementation of your executor
class Executor(RemoteExecutor):
def __post_init__(self):
# run check whether we are running in a SLURM job context
self.warn_on_jobcontext()
self.run_uuid = str(uuid.uuid4())
self.logger.info(f"SLURM run ID: {self.run_uuid}")
self._fallback_account_arg = None
self._fallback_partition = None

def warn_on_jobcontext(self, done=None):
if not done:
if "SLURM_JOB_ID" in os.environ:
self.logger.warning(
"Running Snakemake in a SLURM within a job may lead"
" to unexpected behavior. Please run Snakemake directly"
" on the head node."
)
time.sleep(5)
done = True

def additional_general_args(self):
return "--executor slurm-jobstep --jobs 1"

Expand Down
Loading