Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Outputs Arborator and Arborview Files Associated with Each Cluster #10

Merged
merged 9 commits into from
Aug 12, 2024
Merged

Outputs Arborator and Arborview Files Associated with Each Cluster #10

merged 9 commits into from
Aug 12, 2024

Conversation

emarinier
Copy link
Member

@emarinier emarinier commented Jul 31, 2024 •
edited
Loading

Since there are a dynamic number of clusters, which are determined by sample metadata, this pull request configures the outputs associated with each cluster such that they are provided to the user. The Arborator output directory looks something like this:

1/
2/
3/
cluster_summary.tsv
metadata.excluded.tsv
metadata.included.tsv
run.json
threshold_map.json
unassociated/

and a specific cluster directory ("1/") looks something like this:

clusters.tsv
loci.summary.tsv
matrix.pq
matrix.tsv
metadata.tsv
outliers.tsv
profile.tsv
tree.nwk

Now, the publishDir saveAs closure will check for file paths that contain exactly two file separators and modify their file name so that they become flattened (1_profile.tsv).

The resulting output will look something like this:

1_clusters.tsv      1_outliers.tsv      2_matrix.pq     2_tree.nwk             metadata.included.tsv
1_loci.summary.tsv  1_profile.tsv       2_matrix.tsv    3_metadata.tsv         run.json
1_matrix.pq         1_tree.nwk          2_metadata.tsv  3_profile.tsv          threshold_map.json
1_matrix.tsv        2_clusters.tsv      2_outliers.tsv  cluster_summary.tsv    unassociated_metadata.tsv
1_metadata.tsv      2_loci.summary.tsv  2_profile.tsv   metadata.excluded.tsv  unassociated_profile.tsv

The iridanext.config file has been updated to capture the files we're interested in (profile.tsv and metadata.tsv) and has been updated to capture the Arborview output as well.

Additionally, this pull request contains the small fix related to nf-test being updated, which hasn't been merged into dev yet and is repeated here so that the tests don't fail. This is related fix is related to the recent changes made for nf-test (askimed/nf-test#226). Otherwise, the tests will fail.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 31, 2024 •
edited
Loading

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit c4d5861

+| ✅ 141 tests passed       |+
#| ❔  27 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

  • nextflow_config - Config manifest.version should end in dev: 0.1.0
  • schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/clustersplitter/master/nextflow_schema.json
    Found https://raw.githubusercontent.com/phac-nml/clustersplitter/main/nextflow_schema.json
  • nfcore_yml - nf-core version not set in .nf-core.yml

❔ Tests ignored:

  • files_exist - File is ignored: assets/nf-core-clustersplitter_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-clustersplitter_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-clustersplitter_logo_dark.png
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: lib/Utils.groovy
  • files_exist - File is ignored: lib/WorkflowMain.groovy
  • files_exist - File is ignored: lib/NfcoreTemplate.groovy
  • files_exist - File is ignored: lib/WorkflowSnvphylnfc.groovy
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File does not exist: assets/nf-core-clustersplitter_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-clustersplitter_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-clustersplitter_logo_dark.png
  • files_unchanged - File ignored due to lint config: docs/README.md
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/clustersplitter/clustersplitter/.github/workflows/awstest.yml
  • actions_awsfulltest - actions_awsfulltest
  • pipeline_name_conventions - pipeline_name_conventions

✅ Tests passed:

Run details

  • nf-core/tools version 2.14.1
  • Run at 2024-08-09 14:19:13

mattheww95
mattheww95 reviewed Jul 31, 2024
View reviewed changes
kylacochrane
kylacochrane approved these changes Jul 31, 2024
View reviewed changes
Copy link
Contributor

@kylacochrane kylacochrane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great Eric 😺 ! Love the readability of the saveAs closure now!

@sgsutcliffe
Copy link
Contributor

Looks good! No notes

apetkau
apetkau requested changes Aug 6, 2024
View reviewed changes
Copy link
Member

@apetkau apetkau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much Eric. This looks great 😄

Could you add some additional test cases or assert statements to verify the appropriate files get created for each partition?

@emarinier
Copy link
Member Author

Thanks so much Eric. This looks great 😄

Could you add some additional test cases or assert statements to verify the appropriate files get created for each partition?

Updated in 8465136

mattheww95
mattheww95 reviewed Aug 8, 2024
View reviewed changes
mattheww95
mattheww95 approved these changes Aug 8, 2024
View reviewed changes
Copy link
Collaborator

@mattheww95 mattheww95 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just made some comments for your consideration.

apetkau
apetkau approved these changes Aug 12, 2024
View reviewed changes
Copy link
Member

@apetkau apetkau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks so much Eric 😄

@emarinier emarinier merged commit 727fba7 into dev Aug 12, 2024
4 checks passed
@apetkau apetkau deleted the outputs branch August 19, 2024 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mattheww95 mattheww95 mattheww95 approved these changes

@apetkau apetkau apetkau approved these changes

@kylacochrane kylacochrane kylacochrane approved these changes

@sgsutcliffe sgsutcliffe sgsutcliffe approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@emarinier @sgsutcliffe @apetkau @kylacochrane @mattheww95