Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete unused snapshots on workorders retention cleanup #2931

Merged
merged 3 commits into from
Feb 18, 2025
Merged

Conversation

jyeshe
Copy link
Member

@jyeshe jyeshe commented Feb 12, 2025

Description

This PR adds an deletion for unused snapshots (those are referenced by no work orders anymore).

There are two scenarios when snapshots needs to be deleted:
a) on project deletion, when workflows deletion takes care of it
b) on data retention cleanup, when much work orders are left after deleting the old ones.

The filtering of the snapshots to delete come from the set difference between snapshots from the deleted work orders and snapshots that are still in use (referenced by recent work orders that are still in the retention period).

Closes #1832

Validation steps

  1. Setup some retention period.
  2. Insert 10k work orders, stop Lightning and update their date to be out of the retention period (older). Also update the state of both the work orders and runs to success.
  3. Take note of the snapshot id used.
  4. Edit the workflow.
  5. Insert 2k work orders and update their date to be out of the retention period (older).
  6. Insert more 2k work orders.
  7. Call Lightning.Projects.perform(%Oban.Job{args: %{"project_id" => "4adf2644-ed4e-4f97-a24c-ab35b3cb1efa", "type" => "data_retention"}}) using your project_id
  8. It shall delete all work orders from steps 1 and 4 in less than 3 seconds. The snapshot of step 2 is also deleted.
image and only one snapshot is left image

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

  • Code generation (copilot but not intellisense)
  • Learning or fact checking
  • Strategy / design
  • Optimisation / refactoring
  • Translation / spellchecking / doc gen
  • Other
  • I have not used AI

You can read more details in our Responsible AI Policy

Pre-submission checklist

  • I have performed a self-review of my code.
  • I have implemented and tested all related authorization policies. (e.g., :owner, :admin, :editor, :viewer)
  • I have updated the changelog.
  • I have ticked a box in "AI usage" in this PR

@jyeshe jyeshe force-pushed the snaphots_cleanup branch 3 times, most recently from a4dd66a to 8731b6a Compare February 13, 2025 10:38
@jyeshe jyeshe self-assigned this Feb 13, 2025
@jyeshe
Copy link
Member Author

jyeshe commented Feb 13, 2025

Unrelated flaky:

  1. test Allow low priority access users to retry steps and create workorders Users with low priority access to the workflow canvas will automatically be locked in a snapshot when the high prior uses saves the workflow (LightningWeb.WorkflowLive.EditTest)
    test/lightning_web/live/workflow_live/edit_test.exs:3654
    Expected false or nil, last got true
    code: refute_eventually(low_priority_view |> has_element?("#inspector-workflow-version", "latest"))
    stacktrace:
    test/lightning_web/live/workflow_live/edit_test.exs:3693: (test)

@jyeshe jyeshe marked this pull request as ready for review February 14, 2025 06:18
Copy link
Collaborator

@midigofrank midigofrank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done @jyeshe

Copy link
Member

@stuartc stuartc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jyeshe this is great! Thanks! I just merged main in, and did a small (uneventful) change for the project_files test, changing all the DateTime.utc_now() to Lightning.current_time(), it doesn't affect the tests at all but nice to default to since we can freeze/stub that call.

@stuartc stuartc merged commit 172aead into main Feb 18, 2025
5 of 6 checks passed
@stuartc stuartc deleted the snaphots_cleanup branch February 18, 2025 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Delete Snapshots when an associated workorder is deleted
3 participants