-
Notifications
You must be signed in to change notification settings - Fork 230
Duplicate RVId handling #1803
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/dcache
Are you sure you want to change the base?
Duplicate RVId handling #1803
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR integrates the new distributed_cache
component end-to-end and refactors key storage and logging APIs to support option structs and better diagnostics.
- Introduce
StorageCallbackImpl
and wire up CLI support fordistributed_cache
. - Refactor all
WriteFromBuffer
andCreateDirectory
calls to use aninternal.WriteFromBufferOptions
struct and add aSetMetadata
path. - Add new common utilities (
GetNodeUUID
, FS metrics, pipeline validation) and enhance logging with goroutine IDs and error injection.
Reviewed Changes
Copilot reviewed 76 out of 76 changed files in this pull request and generated 3 comments.
Show a summary per file
File | Description |
---|---|
component/distributed_cache/storage_callback_impl.go | New callbacks implementation for distributed-cache storage |
component/distributed_cache/distributed_cache_test.go | Tests covering basic configuration of the distributed cache |
component/azstorage/utils.go | Respect BLOBFUSE_DISABLE_SDK_LOG env var; map CannotVerifyCopySource to ErrFileNotFound |
component/azstorage/datalake.go & datalake_test.go | Refactor CreateDirectory and WriteFromBuffer signatures; update tests to use WriteFromBufferOptions |
component/azstorage/block_blob.go & block_blob_test.go | Same WriteFromBuffer and SetMetadata refactor for block blobs; update tests |
component/azstorage/connection.go | Update AzConnection interface signatures to match refactored methods |
common/util.go | Add GetDiskSpaceMetricsFromStatfs , GetNodeUUID , UUID/IP validation, and distributed-cache pipeline rules |
common/config/keys_tree.go & keys_tree_test.go | Extend config parsing to support []string ; update tests |
common/config/config_parser.go | Add AddStringSliceFlag for comma-separated string list flags |
common/assert.go | New Assert utility for debug builds |
common/log/logger.go & base_logger.go | Initialize error injection; include goroutine IDs in logs |
cmd/mount.go & cmd/imports.go | Add hidden --dcache flag and import for the distributed-cache component |
Comments suppressed due to low confidence (2)
common/util.go:579
- New pipeline validation rules for
distributed_cache
should have unit tests covering combinations withfile_cache
andxload
to ensure the correct errors are returned.
if isDCachePresent && isFileCachePresent {
common/util.go:465
- availableSpace is undefined here; you need to compute it (e.g.,
availableSpace := stat.Bavail * uint64(stat.Frsize)
) before returning.
return totalSpace, availableSpace, nil
This change will block the user to start blobfuse with distributed_cache in case the different cache-dir have the same fileSYstem UUID.