Skip to content

Commit 9e60004

Browse files
committed
Update documentation with publishOffload options
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
1 parent 8d07890 commit 9e60004

File tree

3 files changed

+34
-3
lines changed

3 files changed

+34
-3
lines changed

docs/reference/config.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1300,6 +1300,37 @@ The following settings are available:
13001300

13011301
Read the {ref}`container-podman` page to learn more about how to use Podman containers with Nextflow.
13021302

1303+
(config-publishOffload)=
1304+
1305+
## `publishOffload`
1306+
1307+
The `publishOffload` scope allows you to configure the offload of the publication of workflow output as remote tasks. An internal `offload:publish_process` is created to offload the file operation (copies and moves) required to publish the output files. This functionality is currently available for 's3' files with the 'awsbatch' executor. File operations can be maneged with `s5cmd` (default option) of Fusion.
1308+
1309+
The following settings are available:
1310+
1311+
`publishOffload.enable`
1312+
: If `true` enables the offload of workflow outputs publication. (Default: `false`)
1313+
1314+
`publishOffload.batchSize`
1315+
: Indicates the number of files operations in each `publish_process` execution. (Default: 10)
1316+
1317+
`publishOffload.maxParallel`
1318+
: Indicates the maximum number of concurrent files operations in each `publish_process` execution. If `maxParallel` is not specified, it gets the value of `batchSize`.
1319+
1320+
`publishOffload.useFusion`
1321+
: Enables the use of Fusion to manage the file operations. (Default: `false`)
1322+
1323+
You can also control the resources assigned to the `publish_process` defining a process selector such as the following:
1324+
1325+
```groovy
1326+
process {
1327+
withName: publish_process {
1328+
cpus = 4
1329+
memory = 8.GB
1330+
}
1331+
}
1332+
```
1333+
13031334
(config-report)=
13041335

13051336
## `report`

modules/nextflow/src/main/groovy/nextflow/processor/PublishOffloadManager.groovy

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ class PublishOffloadManager {
6868
engine.eval(file.text, delegate)
6969
return engine.result
7070
},'publish file process', 'script')
71-
this.publishProcessor = createProcessor( "publish_process", body )
71+
this.publishProcessor = createProcessor( "offload:publish_process", body )
7272
log.debug("Publish Offload Manager initialized.")
7373
}
7474

@@ -156,7 +156,7 @@ class PublishOffloadManager {
156156
throw new IllegalArgumentException("Missing script in the specified process block -- make sure it terminates with the script string to be executed")
157157

158158
// -- apply settings from config file to process config
159-
processConfig.applyConfig((Map)session.config.process, name, name, name)
159+
processConfig.applyConfig((Map)session.config.process, 'publish_process', name, name)
160160

161161
// -- get the executor for the given process config
162162
final execObj = session.executorFactory.getExecutor(name, processConfig, body, session)

tests/checks/publish-offload.nf/.checks

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,5 @@ cp .config nextflow.config
33

44
$NXF_RUN > stdout
55
[[ `grep 'INFO' .nextflow.log | grep -c 'Submitted process > gen_data'` == 1 ]] || false
6-
[[ `grep 'INFO' .nextflow.log | grep -c 'Submitted process > publish_process'` == 1 ]] || false
6+
[[ `grep 'INFO' .nextflow.log | grep -c 'Submitted process > offload:publish_process'` == 1 ]] || false
77
[[ -f publishDir/chunk_1 ]] || false

0 commit comments

Comments
 (0)