Skip to content

Commit ffff12f

Browse files
committed
docs: storage update (#80)
reflects update in storage plugin fs
1 parent c6ed8a8 commit ffff12f

File tree

1 file changed

+20
-6
lines changed

1 file changed

+20
-6
lines changed

docs/further.md

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -203,8 +203,7 @@ Snakemake will check the status of your jobs 40 seconds after submission. Anothe
203203

204204
When using [profiles](https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles), a command line may become shorter. A sample profile could look like this:
205205

206-
```console
207-
__use_yte__: true
206+
```YAML
208207
executor: slurm
209208
latency-wait: 60
210209
default-storage-provider: fs
@@ -213,18 +212,33 @@ shared-fs-usage:
213212
- software-deployment
214213
- sources
215214
- source-cache
216-
local-storage-prefix: "<your node local storage prefix>"
215+
remote-job-local-storage-prefix: "<your node local storage prefix>"
216+
local-storage-prefix: "<your local storage prefix, e.g. on login nodes>"
217+
```
218+
219+
The entire configuration will set the executor to SLURM executor, ensures sufficient file system latency and allow automatic stage-in of files using the [file system storage plugin](https://github.com/snakemake/snakemake-storage-plugin-fs).
220+
221+
On a cluster with a scratch directory per job id, the prefix within jobs might be:
222+
223+
```YAML
224+
remote-job-local-storage-prefix: "<scratch>/$SLURM_JOB_ID"
225+
```
226+
227+
On a cluster with a scratch directory per user, the prefix within jobs might be:
228+
229+
```YAML
230+
remote-job-local-storage-prefix: "<scratch>/$USER"
217231
```
218232

219-
It will set the executor to be this SLURM executor, ensure sufficient file system latency, and allow automatic stage-in of files using the [file system storage plugin](https://github.com/snakemake/snakemake-storage-plugin-fs).
233+
Note, that the path `<scratch>` needs to be taken from a specific cluster documentation.
220234

221-
Note, that you need to set the `SNAKEMAKE_PROFILE` environment variable in your `~/.bashrc` file, e.g.:
235+
Further note, that you need to set the `SNAKEMAKE_PROFILE` environment variable in your `~/.bashrc` file, e.g.:
222236

223237
```console
224238
export SNAKEMAKE_PROFILE="$HOME/.config/snakemake"
225239
```
226240

227-
Further note, that there is further development ongoing to enable differentiation of file access patterns.
241+
==This is ongoing development. Eventually you will be able to annotate different file access patterns.==
228242

229243
## Retries - Or Trying again when a Job failed
230244

0 commit comments

Comments
 (0)