fix(hybrid-cloud): Adds support for cross database tombstones #69480
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Scope: Backend
GabeVillalobos
force-pushed
the
gv/adds-support-for-cross-db-tombstoning
branch
from
dce5a74 to
f1835b9
Compare
GabeVillalobos
commented
Apr 23, 2024
markstory
reviewed
Apr 23, 2024
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #69480 +/- ##
=======================================
Coverage 79.84% 79.84%
=======================================
Files 6466 6466
Lines 287413 287435 +22
Branches 49535 49538 +3
=======================================
+ Hits 229483 229507 +24
+ Misses 57508 57506 -2
Partials 422 422
|
GabeVillalobos
force-pushed
the
gv/adds-support-for-cross-db-tombstoning
branch
from
0f318b8 to
5658dfa
Compare
markstory
approved these changes
Apr 24, 2024
| watermark_batch: WatermarkBatch, | ||
| ) -> tuple[list[int], datetime.datetime]: | ||
| """ | ||
| Queries the database or databases if spanning multiple), and returns |
There was a problem hiding this comment.
Suggested change
| Queries the database or databases if spanning multiple), and returns | |
| Queries the database or databases if spanning multiple, and returns |
Comment on lines
+381
to
+385
| [cascade_data[0]["monitor"].id, cascade_data[1]["monitor"].id], | ||
| ), | ||
| ( | ||
| {"low": in_order_tombstones[1].id, "up": in_order_tombstones[2].id}, | ||
| [cascade_data[2]["monitor"].id], |
There was a problem hiding this comment.
If cascade_data has four sets of data in it, and all four users were deleted why are only three monitors being deleted?
There was a problem hiding this comment.
Oh that's a good point, let me fix this to span 3 instead. I'll also add a case that tests that the full set is deleted if the watermark low is set to -1
GabeVillalobos
force-pushed
the
gv/adds-support-for-cross-db-tombstoning
branch
from
5658dfa to
0b6ecdc
Compare
|
Some manual testing notes:
Going to merge this change tomorrow now that the tombstone behavior has been tested. |
GabeVillalobos
added
the
Trigger: Revert
|
PR reverted: 6e7147f |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds cross-database tombstone deletion support
The current tombstone code performs a join between the model and
tombstone tables to construct a minimal set of IDs to update or delete
from the target table.
Because this isn't possible when the target model lives in a different
database (e.g. crons monitors), we have to take a 2 step approach:
watermarking batch.
target field
There are possible performance concerns with the model-side query, so
I've placed this behind a new option named
hyrid_cloud.allow_cross_db_tombstones. If the flag is disabled and across db tombstone cleanup is attempted, an explicit exception is raised
noting why the cleanup failed.