Verify digest of file objects in mirror (#7069, #7089) #7090
+277
−71
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nadove-ucsc
added
reqs
nadove-ucsc
changed the base branch from
develop
to
issues/nadove-ucsc/6862-implement-mirror-file
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
2 times, most recently
from
5675607 to
8ac6802
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7090 +/- ##
===========================================
- Coverage 85.27% 85.18% -0.10%
===========================================
Files 151 152 +1
Lines 21846 21948 +102
===========================================
+ Hits 18629 18696 +67
- Misses 3217 3252 +35 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
hannes-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
33d63d0 to
50f39fb
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
50f39fb to
37c4900
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
8ac6802 to
797754e
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
0071471 to
fe9a0b3
Compare
hannes-ucsc
force-pushed
the
issues/nadove-ucsc/6862-implement-mirror-file
branch
from
6cdc15d to
cd69dc8
Compare
achave11-ucsc
changed the base branch from
issues/nadove-ucsc/6862-implement-mirror-file
to
develop
achave11-ucsc
removed
the
chained
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
797754e to
4e36a22
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
4e36a22 to
e084cdf
Compare
nadove-ucsc
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
fcef109 to
8a18e23
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
2 times, most recently
from
febc6b6 to
e6143f6
Compare
hannes-ucsc
requested changes
May 7, 2025
There was a problem hiding this comment.
Subject: [PATCH] REVIEW 1
---
Index: src/azul/digests.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/digests.py b/src/azul/digests.py
--- a/src/azul/digests.py (revision 7e9ef6fe283c02343f85699fb2ea0e5c286f5ed1)
+++ b/src/azul/digests.py (date 1746601204964)
@@ -2,6 +2,7 @@
import pickle
from typing import (
Any,
+ TYPE_CHECKING,
)
import resumablesha256
@@ -10,8 +11,13 @@
R,
)
-# TODO: add type stub
-Hasher = Any
+if TYPE_CHECKING:
+ class Hasher:
+ def hexdigest(self) -> str: ...
+
+ def update(self, data: bytes, /) -> None: ...
+else:
+ Hasher = Any # REVIEW: without the else branch we would need to put any mention of Hasher in quotes, and the import would also need to be conditional on TYPE_CHECKING
def get_resumable_hasher(digest_type: str) -> Hasher:
Index: src/azul/azulclient.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/azulclient.py b/src/azul/azulclient.py
--- a/src/azul/azulclient.py (revision 7e9ef6fe283c02343f85699fb2ea0e5c286f5ed1)
+++ b/src/azul/azulclient.py (date 1746600449726)
@@ -708,7 +708,7 @@
for file in plugin.list_files(source, prefix):
info_key = self.mirror_service.info_object_key(file)
if info_key in already_mirrored:
- log.info('Not mirroring file %r because an info object is already present at %r',
+ log.info('Not mirroring file %r because info object already exists at %r',
file.uuid, info_key)
else:
log.info('Mirroring file %r', file.uuid)
Index: src/azul/service/storage_service.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/service/storage_service.py b/src/azul/service/storage_service.py
--- a/src/azul/service/storage_service.py (revision 7e9ef6fe283c02343f85699fb2ea0e5c286f5ed1)
+++ b/src/azul/service/storage_service.py (date 1746601204969)
@@ -37,9 +37,11 @@
)
from azul.collections import (
+ OrderedSet,
adict,
)
from azul.deployment import (
+ R,
aws,
)
@@ -114,14 +116,19 @@
**self._object_creation_kwargs(content_type=content_type, tagging=tagging),
**kwargs)
- def list(self, prefix: str) -> list[str]:
- keys = []
+ # REVIEW: Consider OrderedSet. It better reflects S3's ordering and uniqueness guarantees
+
+ def list(self, prefix: str) -> OrderedSet[str]:
+ keys, num_keys = OrderedSet(), 0
token = None
- while True:
+ while True: # REVIEW: Consider using a paginator (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/paginator/ListObjectsV2.html#listobjectsv2)
response = self._s3.list_objects_v2(Bucket=self.bucket_name,
Prefix=prefix,
**adict(ContinuationToken=token))
- keys.extend(object['Key'] for object in response.get('Contents', ()))
+ contents = response.get('Contents', ())
+ num_keys += len(contents)
+ keys.update(object['Key'] for object in contents)
+ assert len(keys) == num_keys, R('Got duplicate keys from S3')
token = response.get('NextContinuationToken')
if token is None:
break
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
7e9ef6f to
3716614
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
1d26528 to
16a4215
Compare
hannes-ucsc
previously approved these changes
May 7, 2025
Security design review
|
achave11-ucsc
force-pushed
the
issues/nadove-ucsc/7069-verify-digest-mirror
branch
from
608521c to
a007756
Compare
achave11-ucsc
added
the
sandbox
hannes-ucsc
approved these changes
May 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Connected issues: #7069
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all connected issuespartiallabelAuthor (chains)
baseor this PR is not chained to another PRchainedor is not chained to another PRAuthor (reindex, API changes)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAPIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues connected to this PRAuthor (before every review)
develop, squashed old fixupsmake requirements_updateor this PR does not modifyrequirements*.txt,common.mk,MakefileandDockerfileRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
PR is marked as approvedSystem administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator (before pushing merge the commit)
reindex:…labels andrcommit title tagno demodevelop_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator
dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (before pushing merge the commit)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnersandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices indevanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevpif the PR is also labeledpartialOperator (chain shortening)
developor this PR is not labeledbasechainedlabel from the blocked PR or this PR is not labeledbasebasebaselabel from this PR or this PR is not labeledbaseOperator (after pushing the merge commit)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevdevor this PR does not require reindexingdevdeploy_browserjob in the GitLab pipeline for this PR indevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdeploy_browserjob in the GitLab pipeline for this PR inanvildevor this PR does not require reindexinganvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem