Skip to content

Commit d91ee5f

Browse files
docs: requeue (#153)
only documenting what is new in v0.11.0 (see #152 ) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced new command line flags for the SLURM executor plugin: `--slurm_init_seconds_before_status_checks` and `--slurm_requeue`. - **Documentation** - Expanded guidance on job resubmission and retries, enhancing clarity on the use of `--retries` and `--slurm-requeue`. - Improved overall structure of the documentation for better understanding of job configuration and resource specifications. - Minor formatting corrections made for clarity. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: David Laehnemann <david.laehnemann@hhu.de>
1 parent 0817ec0 commit d91ee5f

File tree

1 file changed

+26
-2
lines changed

1 file changed

+26
-2
lines changed

docs/further.md

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,16 @@ set-resources:
160160
cpus_per_task: 40
161161
```
162162
163+
### Additional Command Line Flags
164+
165+
This plugin defines additional command line flags.
166+
As always, these can be set on the command line or in a profile.
167+
168+
| Flag | Meaning |
169+
|-------------|----------|
170+
| `--slurm_init_seconds_before_status_checks`| modify time before initial job status check; the default of 40 seconds avoids load on querying slurm databases, but shorter wait times are for example useful during workflow development |
171+
| `--slurm_requeue` | allows jobs to be resubmitted automatically if they fail or are preempted. See the [section "retries" for details](#retries)|
172+
163173
## Multicluster Support
164174

165175
For reasons of scheduling multicluster support is provided by the `clusters` flag in resources sections. Note, that you have to write `clusters`, not `cluster`!
@@ -203,7 +213,7 @@ rule ...:
203213
"bio/VinaLC"
204214
```
205215

206-
This will, internally, trigger a `module load bio`/VinaLC` immediately prior to execution.
216+
This will, internally, trigger a `module load bio VinaLC` immediately prior to execution.
207217

208218
Note, that
209219
- environment modules are best specified in a configuration file.
@@ -282,7 +292,21 @@ snakemake --retries=3
282292
If a workflow fails entirely (e.g. when there are cluster failures), it can be resumed as any other Snakemake workflow:
283293

284294
```console
285-
snakemake --rerun-incomplete
295+
snakemake ... --rerun-incomplete
296+
# or the short-hand version
297+
snakemake ... --ri
298+
```
299+
300+
The "requeue" option allows jobs to be resubmitted automatically if they fail or are preempted. This is similar to Snakemake's `--retries`, except a SLURM job will not be considered failed and priority may be accumulated during pending. This might be the default on your cluster, already. You can check your cluster's requeue settings with
301+
302+
```console
303+
scontrol show config | grep Requeue
304+
```
305+
306+
This requeue feature is integrated into the SLURM submission command, adding the --requeue parameter to allow requeuing after node failure or preemption using:
307+
308+
```console
309+
snakemake --slurm-requeue ...
286310
```
287311

288312
To prevent failures due to faulty parameterization, we can dynamically adjust the runtime behaviour:

0 commit comments

Comments
 (0)