diff --git a/content/posts/2024/reana-0.9.3.md b/content/posts/2024/reana-0.9.3.md index c8c50ff..3f0f94e 100644 --- a/content/posts/2024/reana-0.9.3.md +++ b/content/posts/2024/reana-0.9.3.md @@ -1,56 +1,85 @@ --- title: "REANA 0.9.3 is released" -date: 2024-03-13T09:00:00+01:00 +date: 2024-03-13T06:00:00+01:00 --- -The new 0.9.3 version of REANA is now available! This is a minor update that upgrades Snakemake to version 7, better handles stopped and failed workflows, and brings many more performance improvements and bug fixes. +REANA 0.9.3 has just been released. This is a minor update that upgrades +Snakemake workflow engine to version 7, improves job submission performance for +massively-parallel workflows, improves the clean up of stopped and failed +workflows, and brings other minor improvements and bug fixes. - + ## What's new for the users? -### Snakemake upgrade to v7 +### Snakemake version 7 -The support for Snakemake has been improved by upgrading from Snakemake v6.8.0 to v7.32.4. -First of all, this allows you to submit Snakemake workflows with `reana-client` when using Python 3.11 or newer. Furthermore, you can now make use of many new features, including: +The support of Snakemake workflows has been improved by upgrading from +Snakemake 6.8.0 to 7.32.4 which brings compatibility with `reana-client` users +using Python 3.11 or newer versions. Furthermore, you can now make use of the +many new Snakemake 7 features, including: -- Zenodo remote provider to download and upload files to Zenodo ([docs](https://snakemake.readthedocs.io/en/v7.32.3/snakefiles/remote_files.html#zenodo)). -- Improved report capabilities. -- Ability to access Snakemake rule variables from Bash scripts ([docs](https://snakemake.readthedocs.io/en/v7.32.3/snakefiles/rules.html#bash)). -- Improved wildcard matching that reduces ambiguity problems. +- ability to access Snakemake rule variables from Bash scripts + ([docs](https://snakemake.readthedocs.io/en/v7.32.3/snakefiles/rules.html#bash)); +- ability to download and upload files to Zenodo + ([docs](https://snakemake.readthedocs.io/en/v7.32.3/snakefiles/remote_files.html#zenodo)); +- improved wildcard matching that reduces rule ambiguity problems; +- improved workflow execution reporting capabilities. {{< screenshot-browser-mockup src="/images/reana-0.9.3-snakemake-report-output.png" alt="Snakemake report - Workflow output" >}} -### Performance improvements to workflows executing many jobs +### Performance improvements for massively-parallel workflows -Continuing the trend of the last few releases, some additional performance improvements are also present in REANA 0.9.3. -In particular, the submission and creation of jobs has been optimised by avoiding some disk operations that under some circumstances could take quite a long time to complete. +Continuing the trend of the last few releases, REANA 0.9.3 brings further +performance improvements. In particular, the submission and creation of jobs +orchestrated by massively-parallel workflows has been optimised by avoiding +unnecessary disk operations that could take a long time in case of workspaces +containing many files. -The difference is especially noticeable for workflows that generate many hundreds of jobs at the same time, as this performance improvement reduces the time needed to submit all of these jobs, thus reducing the total workflow execution time. +The difference is especially noticeable for workflows that generate many +hundreds of jobs at the same time, such as a scattering process over many +individual files of a dataset. As an example, a workflow step launching about +750 fast parallel jobs could have been executed in about half the time. ### ... and more! -This new REANA release also brings many small improvements and bug fixes. +The new REANA 0.9.3 release brings other small improvements and bug fixes. -The submission of jobs has been improved to avoid situations where it would fail reporting a `FileNotFoundError`. -Not only that, the workflow engine logs of stopped workflows are now correctly stored in REANA. +On the REANA cluster side, the submission of Snakemake jobs has been improved +to avoid situations where a job could previously fail with a +`FileNotFoundError` due to the unsynchronized disk file access when spawning +asynchronous jobs. Moreover, the workflow engine logs for stopped workflows are +now correctly captured and exposed in logs. -Regarding the web interface, the launcher badge creator now creates valid URLs also in the generated markdown snippet. -Furthermore, the CLI client and the web interface now report the correct duration of stopped and failed workflows, respectively. +On the REANA web interface side, the launcher badge creator fixes URLs in the +generated Markdown snippets. Furthermore, the web interface and the +command-line client now report the correct workflow duration for stopped and +failed workflows. -Please see the detailed [REANA 0.9.3 release notes](https://github.com/reanahub/reana/releases/tag/0.9.3) for the complete list of all changes. +Please see the detailed [REANA 0.9.3 release +notes](https://github.com/reanahub/reana/releases/tag/0.9.3) for the complete +list of all changes. ## What's new for the administrators? ### Better handling of stopped workflows -It could happen in the past that some jobs would not be stopped and cleaned up when a workflow failed or was stopped by the user. This meant that jobs would continue running until they either failed or finished, thus utilising resources that could have been given to other workflows. Furthermore, jobs needed to be cleaned up manually from the Kubernetes cluster. +When a workflow failed or was stopped by the user, it could previously happen +that some jobs would not be stopped and cleaned up automatically. If this +occurred, the jobs would continue running until they either finished or failed, +thus utilising computing resources unnecessarily. Furthermore, these jobs were +needed to be cleaned up manually from the Kubernetes cluster by the REANA +administrator. -REANA 0.9.3 has improved the situation by making sure that all jobs are cleaned up before stopping the execution of a workflow. This should result in less cluster maintenance needed and a better utilisation of the available resources. +REANA 0.9.3 improves the situation by making sure that all jobs are cleaned up +before stopping the execution of a workflow. This should result in less cluster +maintenance and a better utilisation of the available computing resources. ### Customising environment of job controller and workflow engines -REANA 0.9.3 also allows to specify custom environment variables to be passed to reana-job-controller or to the workflow engines. This can be done via the new Helm values: +REANA 0.9.3 brings a new possibility to customise environment variables that +are to be passed to REANA workflow engine and job controller components when +running user jobs. This can be achieved via the following new Helm values: - `components.reana_job_controller.environment` - `components.reana_workflow_engine_cwl.environment` @@ -58,7 +87,10 @@ REANA 0.9.3 also allows to specify custom environment variables to be passed to - `components.reana_workflow_engine_snakemake.environment` - `components.reana_workflow_engine_yadage.environment` -These new Helm values can be used to modify some configuration values that were not easily customisable before. As an example, it is now possible to set the maximum number of jobs that Snakemake can run in parallel: +As an example, it is now possible to configure a custom limit for the maximum +number of parallel jobs that Snakemake is allowed to run in parallel to modify +the default hard-coded value. Here is an example on how to allow for up to 1000 +parallel Snakemake jobs in your `values.yaml`: ```yaml components: @@ -67,32 +99,66 @@ components: SNAKEMAKE_MAX_PARALLEL_JOBS: 1000 ``` +This could be suitable for massively-parallel workflows if your cluster +consists of sufficient amount of computing nodes to allow such a workload. + ### Customising PostgreSQL Docker image -REANA 0.9.3 also provides a new Helm value that can be used to choose the image (and thus the version) of the internal PostgreSQL database. Up until now, the default database provided by REANA has been PostgreSQL 12.13 with the official Docker image `docker.io/library/postgres:12.13`. +REANA 0.9.3 also provides a new Helm value that can be used to customise the +image (and thus the version) of the internal PostgreSQL database. The default +version of database provided by REANA has been PostgreSQL 12.13 with the +official Docker image `docker.io/library/postgres:12.13`. + +This will change in the next REANA major release series where the default will +become PostgreSQL 14. If you are using the internal REANA database deployed +inside the cluster, you will eventually have to migrate to the new version of +PostgreSQL sooner or later in the future. The new Helm value allows you to set +the desired PostgreSQL version, for example as a temporary measure to change +the image back to the one you use currently: + +```yaml +components: + reana_db: + image: docker.io/library/postgres:12.13 +``` -This will change in the next major release series, as the default will become PostgreSQL 14.10. -If you are using the internal REANA database, you will have to migrate it to the new version of PostgreSQL. -As a temporary measure, you will also be able to change the image back to the previous one, but incompatibilities and issues might arise in this case. +We shall provide detailed example of the database upgrade in due time as part +of the next REANA release series. Until then, you may skip this possibility or +you may forward-configure the desired database version by means of the new Helm +variable. ### How to upgrade existing REANA 0.9.2 clusters - +If you are a REANA cluster administrator and you would like to upgrade from +REANA 0.9.2 to REANA 0.9.3, you can proceed as follows. -If you are a REANA cluster administrator and you would like to upgrade from REANA 0.9.2 to REANA 0.9.3, you can proceed as follows. +Firstly, please note that you can optionally define some of the new [Helm +values](https://github.com/reanahub/reana/blob/0.9.3/helm/reana/README.md), +notably: -First of all, you can optionally define some of the new [Helm values](https://github.com/reanahub/reana/blob/0.9.3/helm/reana/README.md) in your Helm values file: +- `components.reana_db.image` to choose the PostgreSQL Docker image; +- `components.reana_job_controller.environment` to add additional environment + variables to reana-job-controller's container; +- `components.reana_ui.privacy_notice_url` to add a link to your privacy notice + in the web interface; +- `components.reana_workflow_engine_cwl.environment`, + `components.reana_workflow_engine_serial.environment`, + `components.reana_workflow_engine_snakemake.environment`, and + `components.reana_workflow_engine_yadage.environment` to customise the + environment variables of workflow engine's containers. -- `components.reana_db.image` to choose the PostgreSQL Docker image. -- `components.reana_job_controller.environment` to add additional environment variables to reana-job-controller's container. -- `components.reana_ui.privacy_notice_url` to add a link to your privacy notice in the web interface. -- `components.reana_workflow_engine_cwl.environment`, `components.reana_workflow_engine_serial.environment`, `components.reana_workflow_engine_snakemake.environment`, and `components.reana_workflow_engine_yadage.environment` to customise the environment variables of workflow engine's containers. +You can edit your Helm values file to add any desired new values: ```console $ vim myvalues.yaml ``` -You can then use the [Helm diff plugin](https://github.com/databus23/helm-diff) to inspect the forthcoming change and then perform the upgrade using the standard Helm commands: +This is however fully optional and you can simply reuse your current +`myvalues.yaml` file without any changes to perform the 0.9.2 to 0.9.3 upgrade. + +Secondly, you can use the [Helm diff +plugin](https://github.com/databus23/helm-diff) to inspect the forthcoming +changes and then perform the upgrade using the standard Helm commands: ```console $ helm repo update @@ -100,6 +166,10 @@ $ helm diff upgrade reana reanahub/reana --version 0.9.3 --values myvalues.yaml $ helm upgrade reana reanahub/reana --version 0.9.3 --values myvalues.yaml ``` -Please see the detailed [REANA 0.9.3 release notes](https://github.com/reanahub/reana/releases/tag/0.9.3) for the complete list of all changes. +### More information + +Please see the detailed [REANA 0.9.3 release +notes](https://github.com/reanahub/reana/releases/tag/0.9.3) for the complete +list of all changes. Enjoy!