Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS Template S3 bucket prefixes might not match defaults in metaflow configure aws #2247

Open
dennismoe opened this issue Feb 5, 2025 · 0 comments

Comments

@dennismoe
Copy link

I encountered a small issue.
The default S3 prefixes in the AWS Template might not match with the suggested S3 prefixes in metaflow configure aws

https://github.com/outerbounds/metaflow-tools/blob/b76986369a8a44ce6186743c95c7e65143335982/aws/cloudformation/metaflow-cfn-template.yml#L1835-L1840

  MetaflowDataStoreS3Url:
    Description: Amazon S3 URL for Metaflow DataStore [METAFLOW_DATASTORE_SYSROOT_S3]
    Value: !Sub "s3://${MetaflowS3Bucket}/metaflow"
  MetaflowDataToolsS3Url:
    Description: Amazon S3 URL for Metaflow DataTools [METAFLOW_DATATOOLS_S3ROOT]
    Value: !Sub "s3://${MetaflowS3Bucket}/data"

metaflow configure aws suggests metaflow/data as default prefix for METAFLOW_DATATOOLS_S3ROOT if the environment variable METAFLOW_DATATOOLS_S3ROOT isn't set already - what should not be assumed in my opinion.

default=existing_env.get(
"METAFLOW_DATATOOLS_S3ROOT",
os.path.join(env["METAFLOW_DATASTORE_SYSROOT_S3"], "data"),
),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant