diff --git a/docs/further.md b/docs/further.md index 03205000..264466e8 100644 --- a/docs/further.md +++ b/docs/further.md @@ -1,16 +1,30 @@ -### The general Idea +### How this Plugin works -To use this plugin, log in to your cluster's head node (sometimes called the "login" node), activate your environment as usual, and start Snakemake. Snakemake will then submit your jobs as cluster jobs. +In this plugin, Snakemake submits itself as a job script when operating on an HPC cluster using the SLURM batch system. +Consequently, the SLURM log file will duplicate the output of the corresponding rule. +To avoid redundancy, the plugin deletes the SLURM log file for successful jobs, relying instead on the rule-specific logs. + +Remote executors submit Snakemake jobs to ensure unique functionalities — such as piped group jobs and rule wrappers — are available on cluster nodes. +The memory footprint varies based on these functionalities; for instance, rules with a run directive that import modules and read data may require more memory. + +#### Usage Hints + +Install this plugin into your Snakemake base environment using conda. +This process also installs the 'jobstep' plugin, utilized on cluster nodes. +Additionally, we recommend installing the `snakemake-storage-plugin-fs` for automated stage-in and stage-out procedures. + +#### Reporting Bugs and Feature Requests + +We welcome bug reports and feature requests! +Please report issues specific to this plugin [in the plugin's GitHub repository](https://github.com/snakemake/snakemake-executor-plugin-slurm/issue). +For other concerns, refer to the [Snakemake main repository](https://github.com/snakemake/snakemake/issues) or the relevant Snakemake plugin repository. +Cluster-related issues should be directed to your cluster administrator. ### Specifying Account and Partition -Most SLURM clusters have two mandatory resource indicators for -accounting and scheduling, the account and a -partition, respectively. These resources are usually -omitted from Snakemake workflows in order to keep the workflow -definition independent of the platform. However, it is also possible -to specify them inside of the workflow as resources in the rule -definition. +In SLURM, an **account** is used for resource accounting and allocation, while a **partition** designates a subset of compute nodes grouped for specific purposes, such as high-memory or GPU tasks. + +These resources are typically omitted from Snakemake workflows to maintain platform independence, allowing the same workflow to run on different systems without modification. To specify them at the command line, define them as default resources: @@ -21,25 +35,22 @@ $ snakemake --executor slurm --default-resources slurm_account= slurm_partition= --set-resources :slurm_partition= ``` -Usually, it is advisable to persist such settings via a -[configuration profile](https://snakemake.readthedocs.io/en/latest/executing/cli.html#profiles), which -can be provided system-wide, per user, and in addition per workflow. +To ensure consistency and ease of management, it's advisable to persist such settings via a [configuration profile](https://snakemake.readthedocs.io/en/latest/executing/cli.html#profiles), which can be provided system-wide, per user, or per workflow. -The executor waits per default 40 seconds for its first check of the job status. Using `--slurm-init-seconds-before-status-checks=