Skip to content

Support objectstores other than amazon #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dominikl opened this issue Apr 16, 2025 · 1 comment
Open

Support objectstores other than amazon #17

dominikl opened this issue Apr 16, 2025 · 1 comment

Comments

@dominikl
Copy link

dominikl commented Apr 16, 2025

It looks like at the moment it only works with amazon objectstore, but not for example EBI embassy object store.
In particular public accessible ones, without need for credentials.

@dominikl dominikl changed the title Support objectstores others than amazon Support objectstores other than amazon Apr 16, 2025
@sbesson
Copy link
Member

sbesson commented Apr 22, 2025

Cross linking to https://forum.image.sc/t/compatibility-of-ome-zarr-stored-on-s3-and-omero/111380 which also includes discussions around OMERO, OME-Zarr and object store support

As detailed in the README, the current version of the pixel buffer either works with the filesystem or Amazon S3 buckets under authentication. These have been our primary use cases for Zarr images/labels. Expanding this to other types of storage/authentication is doable given the appropriate development and testing.

Capturing a few technical notes that can serve as starting point:

  • re storage backend, the current implementation uses the AWS SDK (v1) with the lasersonlab fork of s3fs and jzarr. On paper, this set of libraries should be compatible with any so-called "S3 compatible storage" i.e. a storage which supports the AWS S3 API. At minimum, the URI stored in external info and the implementation would need to be updated to store/set the endpoint URL and someAWS specifics like bucket regions might need to be reviewed
  • re authentication, a common pattern for AWS environments is to use EC2 instances with a profile allowing to access a particular S3 bucket. Alternatively, named profile i.e. long-lived access key/secret access keys can be configured like for the AWS CLI. The relevant logic is in
    return new AWSCredentialsProviderChain(
    new ProfileCredentialsProvider(profileName),
    new EC2ContainerCredentialsProviderWrapper()
    . We have no experience authenticating against object storage other than AWS S3 with these. For anonymous access, the AWS SDK offers the AnonymousAWSCredentials. This raises the question of how to specify that a Zarr dataset should be access anonymously, in an analogous manner to setting aws s3 --no-sign-request
  • some of the dependencies will need some maintenance including the AWS SDK for Java 1.x which coming EOL at the end of 2025, s3fs which has been largely unmaintained and jzarr which might need to transition to zarr-java to support Zarr v3 in addition to Zarr v2. None of these are a priori requirements but updates might offer better compatibility and working on alternate storage might provide a good opportunity to evaluate thoese.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants