Adding Support for Metadata #4

emarinier · 2024-05-31T19:17:28Z

This is a basic first pass of adding support for metadata.

There are other things that need to be updated in the pipeline, but this PR tries to tackle the metadata-related things and only changes other things if there very important (although such related things might need another PR).

Command

rm -rf results; nextflow run main.nf -profile docker --input tests/data/samplesheets/samplesheet.csv --outdir results -params-file assets/parameters.yaml

Input

samplesheet.csv:

sample,mlst_alleles,metadata_1,metadata_2,metadata_3,metadata_4,metadata_5,metadata_6,metadata_7,metadata_8
S1,https://raw.githubusercontent.com/phac-nml/clustersplitter/dev/tests/data/profiles/S1.mlst.json,1,"Escherichia coli","EHEC/STEC","Canada","O157:H7",21,"2024/05/30","beef"
S2,https://raw.githubusercontent.com/phac-nml/clustersplitter/dev/tests/data/profiles/S2.mlst.json,1,"Escherichia coli","EHEC/STEC","The United States","O157:H7",55,"2024/05/21","milk"
S3,https://raw.githubusercontent.com/phac-nml/clustersplitter/dev/tests/data/profiles/S3.mlst.json,2,"Escherichia coli","EPEC","France","O125",14,"2024/04/30","cheese"
S4,https://raw.githubusercontent.com/phac-nml/clustersplitter/dev/tests/data/profiles/S4.mlst.json,2,"Escherichia coli","EPEC","France","O125",35,"2024/04/22","cheese"
S5,https://raw.githubusercontent.com/phac-nml/clustersplitter/dev/tests/data/profiles/S5.mlst.json,3,"Escherichia coli","EAEC","Canada","O126:H27",61,"2012/09/01","milk"
S6,https://raw.githubusercontent.com/phac-nml/clustersplitter/dev/tests/data/profiles/S6.mlst.json,unassociated,"Escherichia coli","EAEC","Canada","O111:H21",43,"2011/12/25","fruit"

config.json

    {
        "outlier_thresh": "25",
        "clustering_method": "average",
        "clustering_threshold": "500,100,75,50,25,15,10,5,2,1,0",
        "min_cluster_members": 2,
        "partition_column_name": "outbreak",
        "id_column_name": "sample_id",
        "only_report_labeled_columns": "False",
        "skip_qa": "False",
        
        "grouped_metadata_columns":{ 
            "outbreak":{"data_type": "None","label":"National Outbreak Code","default":"","display":"True"},
            "organism":{"data_type": "None","label":"Organism","default":"","display":"True"},
            "subtype":{"data_type": "None","label":"Subtype","default":"","display":"True"},
            "country":{"data_type": "categorical","label":"Country of Collection","default":"","display":"True"},
            "serovar":{"data_type": "categorical","label":"Serovar","default":"","display":"True"},
            "age":{"data_type": "desc_stats","label":"Patient Age (years)","default":"","display":"True"},
            "date":{"data_type": "min_max","label":"Date","default":"","display":"True"},
            "source":{"data_type": "categorical","label":"Source Type","default":"","display":"True"}
        },

        "linelist_columns":{
            "sample":{"data_type": "None","label":"Sample","default":"","display":"True"},
            "outbreak":{"data_type": "None","label":"National Outbreak Code","default":"","display":"True"},
            "organism":{"data_type": "None","label":"Organism","default":"","display":"True"},
            "subtype":{"data_type": "None","label":"Subtype","default":"","display":"True"},
            "country":{"data_type": "categorical","label":"Country of Collection","default":"","display":"True"},
            "serovar":{"data_type": "categorical","label":"Serovar","default":"","display":"True"},
            "age":{"data_type": "desc_stats","label":"Patient Age (years)","default":"","display":"True"},
            "date":{"data_type": "min_max","label":"Date","default":"","display":"True"},
            "source":{"data_type": "categorical","label":"Source Type","default":"","display":"True"}
        }    
    }

parameters.yaml

partition_column: "outbreak"
metadata_1_header : "outbreak"
metadata_2_header : "organism"
metadata_3_header : "subtype"
metadata_4_header : "country"
metadata_5_header : "serovar"
metadata_6_header : "age"
metadata_7_header : "date"
metadata_8_header : "source"

Output

cluster_summary.tsv

National Outbreak Code	Organism	Subtype	Country of Collection	Serovar	Patient Age (years)	Date	Source Type	date_max_value	count_age_43	count_age_21	count_serovar_O125	count_serovar_O157:H7	count_age_55	date_min_value	count_country_The United States	max_dist	count_age_14	count_outliers	count_source_fruit	count_subtype_EAEC	min_dist	age_max_value	count_source_beef	count_serovar_O126:H27	median_dist	count_subtype_EHEC/STEC	age_min_value	count_country_France	age_mean_value	count_serovar_O111:H21	mean_dist	count_organism_Escherichia coli	count_source_milk	count_subtype_EPEC	count_members	count_age_35	count_age_61	age_median_value	count_source_cheese	count_country_Canada
1	Escherichia coli	EHEC/STEC	Canada,The United States	O157:H7	21,55	2024/05/21,2024/05/30	beef,milk	55.0	0	0	0	2	0	21.0	1	3.0	0	0	0	0	3.0	55.0	1	0	3.0	0	21.0	0	38.0	0	3.0	0	1	0	2	0	0	38.0	0	1
2	Escherichia coli	EPEC	France	O125	14,35	2024/04/22,2024/04/30	cheese	35.0	0	0	2	0	0	14.0	0	2.0	0	0	0	0	2.0	35.0	0	0	2.0	0	14.0	2	24.5	0	2.0	0	0	0	2	0	0	24.5	2	0
3	Escherichia coli	EAEC	Canada	O126:H27	61	2012/09/01	milk	61.0	0	0	0	0	0	61.0	0	0	0	0	0	0	0	61.0	0	1	0	0	61.0	0	61.0	0	0	0	1	0	1	0	0	61.0	0	1
unassociated	Escherichia coli	EAEC	Canada	O111:H21	43	2011/12/25	fruit	43.0	0	0	0	0	0	43.0	0	0	0	0	1	0	0	43.0	0	0	0	0	43.0	0	43.0	1	0	0	0	0	1	0	0	43.0	0	1

metadata.included.tsv

sample	outbreak	organism	subtype	country	serovar	age	date	source
S1	1	Escherichia coli	EHEC/STEC	Canada	O157:H7	21	2024/05/30	beef
S2	1	Escherichia coli	EHEC/STEC	The United States	O157:H7	55	2024/05/21	milk
S3	2	Escherichia coli	EPEC	France	O125	14	2024/04/30	cheese
S4	2	Escherichia coli	EPEC	France	O125	35	2024/04/22	cheese
S5	3	Escherichia coli	EAEC	Canada	O126:H27	61	2012/09/01	milk
S6	unassociated	Escherichia coli	EAEC	Canada	O111:H21	43	2011/12/25	fruit

github-actions · 2024-05-31T19:18:49Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 0b818ab

+| ✅ 147 tests passed       |+
#| ❔  27 tests were ignored |#
!| ❗   6 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 0.1.0
schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/clustersplitter/master/nextflow_schema.json
Found https://raw.githubusercontent.com/phac-nml/iridanextexample/main/nextflow_schema.json
schema_description - No description provided in schema for parameter: av_html
schema_description - No description provided in schema for parameter: ar_config
schema_description - No description provided in schema for parameter: ar_thresholds
nfcore_yml - nf-core version not set in .nf-core.yml

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-clustersplitter_logo_light.png
files_exist - File is ignored: docs/images/nf-core-clustersplitter_logo_light.png
files_exist - File is ignored: docs/images/nf-core-clustersplitter_logo_dark.png
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/WorkflowSnvphylnfc.groovy
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-clustersplitter_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-clustersplitter_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-clustersplitter_logo_dark.png
files_unchanged - File ignored due to lint config: docs/README.md
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/clustersplitter/clustersplitter/.github/workflows/awstest.yml
actions_awsfulltest - actions_awsfulltest
pipeline_name_conventions - pipeline_name_conventions

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-clustersplitter_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowClustersplitter.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.av_html= assets/ArborView.html
nextflow_config - Config default value correct: params.ar_config= tests/data/configs/config.json
nextflow_config - Config default value correct: params.ar_thresholds= 10,9,8,7,6,5,4,3,2,1
nextflow_config - Config default value correct: params.project_name= assembly
nextflow_config - Config default value correct: params.assembler= stub
nextflow_config - Config default value correct: params.random_seed= 1
nextflow_config - Config default value correct: params.metadata_partition_name= outbreak
nextflow_config - Config default value correct: params.metadata_1_header= metadata_1
nextflow_config - Config default value correct: params.metadata_2_header= metadata_2
nextflow_config - Config default value correct: params.metadata_3_header= metadata_3
nextflow_config - Config default value correct: params.metadata_4_header= metadata_4
nextflow_config - Config default value correct: params.metadata_5_header= metadata_5
nextflow_config - Config default value correct: params.metadata_6_header= metadata_6
nextflow_config - Config default value correct: params.metadata_7_header= metadata_7
nextflow_config - Config default value correct: params.metadata_8_header= metadata_8
nextflow_config - Config default value correct: params.max_cpus= 4
nextflow_config - Config default value correct: params.max_memory= 2.GB
nextflow_config - Config default value correct: params.max_time= 1.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
template_strings - Did not find any Jinja template strings (114 files)
schema_lint - Schema lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - LOCIDEX_MERGE found in conf/modules.config and Nextflow scripts.
modules_config - MAP_TO_TSV found in conf/modules.config and Nextflow scripts.
modules_config - ARBORATOR found in conf/modules.config and Nextflow scripts.
modules_config - ARBOR_VIEW found in conf/modules.config and Nextflow scripts.
modules_config - IRIDA_NEXT_OUTPUT found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline

Run details

nf-core/tools version 2.14.1
Run at 2024-06-10 17:51:41

mattheww95

Looks good to me!

kylacochrane

Everything looks good Eric 👍

apetkau

Thanks so much @emarinier . This is great 😄 . Comments below

tests/data/arborview/expected_1.html

workflows/cluster_splitter.nf

…ta_1 and metadata_1_name.

apetkau

This looks great to me. Thanks so much @emarinier 😄

emarinier added 7 commits May 30, 2024 09:40

First pass of switching to 8-column metadata format.

608caa2

First pass of pushing metadata through the pipeline.

ecc08ba

Specifying sample in linelist config so it appears as first column.

ead753e

Adding iridanext output plugin.

9737576

Updating some pipeline info.

3f82e13

Updating tests for metadata.

655e792

Linting

7006f2f

emarinier self-assigned this May 31, 2024

emarinier added 3 commits May 31, 2024 14:19

Removing duplicate parameters file.

24539b9

prettier

110e810

Updating test profile

916ce9f

emarinier requested review from mattheww95 and apetkau May 31, 2024 19:46

mattheww95 approved these changes Jun 3, 2024

View reviewed changes

emarinier requested a review from kylacochrane June 4, 2024 16:14

kylacochrane approved these changes Jun 5, 2024

View reviewed changes

apetkau requested changes Jun 6, 2024

View reviewed changes

tests/data/arborview/expected_1.html Outdated Show resolved Hide resolved

workflows/cluster_splitter.nf Outdated Show resolved Hide resolved

emarinier added 3 commits June 10, 2024 10:01

Changing Arborview tests to compare line rather than whole HTML file.

02df592

Adding metadata_partition, metadata_partition_name, instead of metada…

916ac01

…ta_1 and metadata_1_name.

Merge branch 'dev' into metadata

0b818ab

apetkau approved these changes Jun 12, 2024

View reviewed changes

emarinier merged commit 2ed8f1d into dev Jun 12, 2024
4 checks passed

apetkau deleted the metadata branch August 19, 2024 17:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Support for Metadata #4

Adding Support for Metadata #4

emarinier commented May 31, 2024

github-actions bot commented May 31, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

mattheww95 left a comment

kylacochrane left a comment

apetkau left a comment

apetkau left a comment

Adding Support for Metadata #4

Adding Support for Metadata #4

Conversation

emarinier commented May 31, 2024

Command

Input

Output

github-actions bot commented May 31, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

mattheww95 left a comment

Choose a reason for hiding this comment

kylacochrane left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

github-actions bot commented May 31, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️