Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix SQL for groupings and csv downloads #448

Merged
merged 3 commits into from
Feb 26, 2025
Merged

Conversation

aswallace
Copy link
Contributor

@aswallace aswallace commented Feb 25, 2025

Context

Addresses two issues:

  • Grouping includes incorrect files/values #442 : Groupings include incorrect values when filters are also applied (see this query on staging for an example; note that AICS-11 appears in every group)
  • For csv metadata downloads, the wrong file(s) are occasionally processed; e.g., if I selected files from the AICS-16 group, files in AICS-7 might be processed instead (can replicate by going here and downloading the metadata csv for File ID 1. Metadata for File ID 5226 is downloaded instead)

closes #442

Changes

In both cases, the issue was with "OR" syntax

  • For the groupings one: Using ") OR (" made the query over-select
  • For downloads: "OR" was applied between different filter types instead of within the same filter type

Testing

  • Tested manually on both a smaller local testing dataset & on the online Variance dataset
  • Added a unit test

@aswallace aswallace linked an issue Feb 25, 2025 that may be closed by this pull request
@aswallace aswallace added the bug label Feb 25, 2025
@SeanLeRoy SeanLeRoy removed the bug label Feb 25, 2025

// Assert
// Uses OR within single filter type
expect(modifiedSQLWithOR).to.include("OR");

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this test detect the "()" issue the change is fixing? Would expect(modifiedSQLWithOR).to.include(" OR ")
allow detection of the old code? If we can match more of the generated SQL for correctness, that might make both the test and the intent of the code more easily readable to future workers.

Copy link
Contributor Author

@aswallace aswallace Feb 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This particular test was intended to catch the other bug, where OR was being applied where AND should've been applied. I'll see if I can add a test for the other change as well and possibly clarify this test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remembering now that the other unit test I fixed actually does address the ) OR ( change. The problem was that while we did have a unit test that matched more of the generated SQL code, it was matching the wrong thing (our logic and expectation was wrong, not the code)

@aswallace aswallace merged commit 1678935 into main Feb 26, 2025
7 checks passed
@aswallace aswallace deleted the bugfix/wrong-values-in-group branch February 26, 2025 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Grouping includes incorrect files/values
3 participants