Skip to content

Commit 7e19560

Browse files
authored
docs: storage update (#80)
reflects update in storage plugin fs
1 parent 06a1555 commit 7e19560

File tree

1 file changed

+20
-6
lines changed

1 file changed

+20
-6
lines changed

docs/further.md

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -198,8 +198,7 @@ Snakemake will check the status of your jobs 40 seconds after submission. Anothe
198198

199199
When using [profiles](https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles), a command line may become shorter. A sample profile could look like this:
200200

201-
```console
202-
__use_yte__: true
201+
```YAML
203202
executor: slurm
204203
latency-wait: 60
205204
default-storage-provider: fs
@@ -208,18 +207,33 @@ shared-fs-usage:
208207
- software-deployment
209208
- sources
210209
- source-cache
211-
local-storage-prefix: "<your node local storage prefix>"
210+
remote-job-local-storage-prefix: "<your node local storage prefix>"
211+
local-storage-prefix: "<your local storage prefix, e.g. on login nodes>"
212+
```
213+
214+
The entire configuration will set the executor to SLURM executor, ensures sufficient file system latency and allow automatic stage-in of files using the [file system storage plugin](https://github.com/snakemake/snakemake-storage-plugin-fs).
215+
216+
On a cluster with a scratch directory per job id, the prefix within jobs might be:
217+
218+
```YAML
219+
remote-job-local-storage-prefix: "<scratch>/$SLURM_JOB_ID"
220+
```
221+
222+
On a cluster with a scratch directory per user, the prefix within jobs might be:
223+
224+
```YAML
225+
remote-job-local-storage-prefix: "<scratch>/$USER"
212226
```
213227

214-
It will set the executor to be this SLURM executor, ensure sufficient file system latency, and allow automatic stage-in of files using the [file system storage plugin](https://github.com/snakemake/snakemake-storage-plugin-fs).
228+
Note, that the path `<scratch>` needs to be taken from a specific cluster documentation.
215229

216-
Note, that you need to set the `SNAKEMAKE_PROFILE` environment variable in your `~/.bashrc` file, e.g.:
230+
Further note, that you need to set the `SNAKEMAKE_PROFILE` environment variable in your `~/.bashrc` file, e.g.:
217231

218232
```console
219233
export SNAKEMAKE_PROFILE="$HOME/.config/snakemake"
220234
```
221235

222-
Further note, that there is further development ongoing to enable differentiation of file access patterns.
236+
==This is ongoing development. Eventually you will be able to annotate different file access patterns.==
223237

224238
## Retries - Or Trying again when a Job failed
225239

0 commit comments

Comments
 (0)