Implement mirror_file, mirror_part & finalize_file (#6862) #7043
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nadove-ucsc
added
the
chained
nadove-ucsc
changed the base branch from
develop
to
issues/nadove-ucsc/6861-implement-mirror-partition
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7043 +/- ##
===========================================
- Coverage 85.47% 85.27% -0.20%
===========================================
Files 150 151 +1
Lines 21678 21846 +168
===========================================
+ Hits 18529 18629 +100
- Misses 3149 3217 +68 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6861-implement-mirror-partition
branch
4 times, most recently
from
aa9f3d5 to
0171653
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
7bec7cb to
9f01c2a
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6861-implement-mirror-partition
branch
8 times, most recently
from
bf67c7d to
5847dcd
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
ff6f582 to
788dda8
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6861-implement-mirror-partition
branch
from
6191b82 to
f09c3cb
Compare
dsotirho-ucsc
force-pushed
the
issues/nadove-ucsc/6861-implement-mirror-partition
branch
from
91f35b8 to
78469d7
Compare
dsotirho-ucsc
changed the base branch from
issues/nadove-ucsc/6861-implement-mirror-partition
to
develop
dsotirho-ucsc
removed
the
chained
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
2 times, most recently
from
9dc1566 to
2c91700
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
2c91700 to
a2f9717
Compare
|
Notes on test coverage: since most files are > 10 MB, it's common for no files at all to be mirrored during the IT, so we effectively have no IT coverage for the |
|
Note on scale/cost/efficiency: on my personal deployment, running |
hannes-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
33d63d0 to
50f39fb
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
2 times, most recently
from
e4b935c to
37c4900
Compare
hannes-ucsc
requested changes
May 6, 2025
There was a problem hiding this comment.
Subject: [PATCH] REVIEW 1
---
Index: src/azul/indexer/mirror_service.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/indexer/mirror_service.py b/src/azul/indexer/mirror_service.py
--- a/src/azul/indexer/mirror_service.py (revision 0e764360a78161c7bb8d54e7e8cde365f2052d34)
+++ b/src/azul/indexer/mirror_service.py (date 1746498183684)
@@ -52,8 +52,8 @@
"""
A part of a mirrored file
"""
- #: The part number, starting at 0 for the first part. Note that the S3 API
- #: numbers parts starting at 1.
+ #: The part number, starting at 0 for the first part, unlike S3 API part
+ #: numbers, which start at 1.
#:
index: int
@@ -64,7 +64,7 @@
#:
size: int
- #: Various quotas related to parts and part sizes
+ #: Various S3 quotas related to parts and part sizes
#: https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
#:
min_size: ClassVar[int] = 5 * 1024 ** 2
@@ -102,14 +102,13 @@
last part.
"""
assert file.size is not None, R('File size unknown', file)
- stop = self.offset + self.size
- if stop == file.size:
+ next_offset = self.offset + self.size
+ if next_offset == file.size:
return None
- elif 0 < stop < file.size:
- return attr.evolve(self,
- index=self.index + 1,
- offset=stop,
- size=min(self.size, file.size - stop))
+ elif 0 < next_offset < file.size:
+ next_index = self.index + 1
+ next_size = min(self.size, file.size - next_offset)
+ return attr.evolve(self, index=next_index, offset=next_offset, size=next_size)
else:
assert False, R('Part range exceeds file size', self, file)
Index: src/azul/azulclient.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/azulclient.py b/src/azul/azulclient.py
--- a/src/azul/azulclient.py (revision 0e764360a78161c7bb8d54e7e8cde365f2052d34)
+++ b/src/azul/azulclient.py (date 1746509333358)
@@ -759,7 +759,7 @@
etags: Sequence[str]
):
file = self.load_file(catalog, file_json)
- assert etags
+ assert len(etags) > 0
self.mirror_service.finish_mirroring_file(file, upload_id, etags)
log.info('Successfully mirrored file %r via multi-part upload', file.uuid)
Comment on lines
+56
to
+60
| self.client.mirror_file_part(message['catalog'], | ||
| message['file'], | ||
| message['part'], | ||
| message['upload_id'], | ||
| message['etags']) |
There was a problem hiding this comment.
This is getting unwieldy. It is essentially the deserialization counterpart to mirror_part_message in AzulClient. So the two things that should be really close to each other, since they are so tightly coupled, are actually far apart. So that's brittle. Then again, we should stop writing this type of code and use the serialization framework in azul.attrs instead. The bodies of the SQS messages really want to be represented by a class.
The use of positional arguments is also brittle. There are even two neighboring arguments of the same type so there is real danger of accidentally swapping them. On top of that, I am really surprised that this even gets past the type checker. Each value in the message dict is of type AnyJSON but mypy silently narrows them to the more specific types of the arguments declared in the mirror_file_part signature. Something must be wrong with mypy or how mypy analyzes the code in azulclient.
This is all somewhat worrying and we need to improve this as quickly as possible. I will take a stab at it as soon as we have a functional version.
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
0071471 to
fe9a0b3
Compare
hannes-ucsc
added
1 review
hannes-ucsc
approved these changes
May 6, 2025
Security design review
|
hannes-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
6cdc15d to
cd69dc8
Compare
hannes-ucsc
added
the
sandbox
achave11-ucsc
removed
the
base
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Connected issues: #6862
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all connected issuespartiallabelAuthor (chains)
baseor this PR is not chained to another PRchainedor is not chained to another PRAuthor (reindex, API changes)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAPIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues connected to this PRAuthor (before every review)
develop, squashed old fixupsmake requirements_updateor this PR does not modifyrequirements*.txt,common.mk,MakefileandDockerfileRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
System administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator (before pushing merge the commit)
reindex:…labels andrcommit title tagno demodevelop_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator
dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (before pushing merge the commit)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnersandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices indevanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevpif the PR is also labeledpartialOperator (chain shortening)
developor this PR is not labeledbasechainedlabel from the blocked PR or this PR is not labeledbasebasebaselabel from this PR or this PR is not labeledbaseOperator (after pushing the merge commit)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevdevor this PR does not require reindexingdevdeploy_browserjob in the GitLab pipeline for this PR indevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdeploy_browserjob in the GitLab pipeline for this PR inanvildevor this PR does not require reindexinganvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem