Skip to content

Commit

Permalink
Updates from Kabi's most recent updates
Browse files Browse the repository at this point in the history
  • Loading branch information
Aaron Kanzer authored and Aaron Kanzer committed Feb 18, 2025
1 parent 4e75f38 commit 3dc1823
Show file tree
Hide file tree
Showing 4 changed files with 36 additions and 41 deletions.
41 changes: 19 additions & 22 deletions docs/60_initialize_vendors.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,21 @@

The DANDI ecosystem relies on vendor services to operate. So first you will need to set up accounts with the following vendors:

**Heroku**: Provisions the API componenents
**AWS**: Provides storage buckets, as well as domain management, for resources across the DANDI ecosystem. As well as the services (Kubernetes, etc.) for deploying the JupyterHub.
**GitHub**: Serves as the authentication provider for accounts across the DANDI ecosystem
**Terraform Cloud**: Manages provisioned resources across cloud vendors in a version-controlled manner.
**Netlify**: Deploys production frontend build, as well as staging previews to assist with frontend development
**Sentry**: Provides observability and monitoring for API events
- **Heroku**: Provisions the API components

Some services are not yet integrated within the main infrastructure:
- **AWS**: Provides storage buckets, as well as domain management, for resources across the DANDI ecosystem. As well as the services (Kubernetes, etc.) for deploying the JupyterHub.

https://github.com/dandi/backups2datalad - to populate/update https://github.com/dandi/dandisets, https://github.com/dandisets, and https://github.com/dandizarrs/
- **GitHub**: Serves as the authentication provider for accounts across the DANDI ecosystem

• TODO: heroku logs
- **Terraform Cloud**: Manages provisioned resources across cloud vendors in a version-controlled manner.

• TODO: aws s3 access stats dump
- **Netlify**: Deploys production frontend build, as well as staging previews to assist with frontend development

• TODO: con/tinuous dumps of CI logs
- **Sentry**: Provides observability and monitoring for API events

• TODO: zarr manifests generation (ATM not on drogon even)
Some services are not yet integrated within the main infrastructure:

TODO: access stats analysis/plots (yet to be finished/cron deployed)
https://github.com/dandi/backups2datalad - to populate/update https://github.com/dandi/dandisets, https://github.com/dandisets, and https://github.com/dandizarrs/

## Heroku

Expand Down Expand Up @@ -233,8 +228,7 @@ style="width: 60%; height: auto; display: block; margin-left: auto; margin-righ

##### Obtaining your OAuth App Credentials

After creating your OAuth App, you'll lastly want to obtain a client secret key and your client ID -- make sure to note these values for further steps when creating our API

After creating your OAuth App, you'll lastly want to obtain a client secret key and your client ID -- make sure to note these values for further steps when creating your API.
<br/><br/>
<img
src="../img/client_values.jpg"
Expand Down Expand Up @@ -278,7 +272,7 @@ style="width: 60%; height: auto; display: block; margin-left: auto; margin-righ
Terraform is configuration tool for managing "infrastructure-as-code" -- meaning that we can programmatically manage infrastructure in
a traceable, version-controlled form.

The Terraform ecosystem provides a UI tool called **Terraform Cloud**
The Terraform ecosystem provides a UI tool called **Terraform Cloud**.

Start by visiting `https://app.terraform.io/` and making an account.

Expand Down Expand Up @@ -380,11 +374,10 @@ alt="retrieve"
style="width: 60%; height: auto; display: block; margin-left: auto; margin-right: auto;"/>
<br/><br/>

Your frontend should be able to deploy to an auto-generated URL via Netlify now! Steps for domain management and configuration are described further in the [Frontend Deployment](../64_dandi_archive/#frontend-deployment) section of these within the DANDI Archive setup.

Your frontend should be able to deploy to an auto-generated URL via Netlify now! Steps for domain management and configuration are described further in the [Frontend Deployment](../64_dandi_archive/#frontend-deployment) section within the DANDI Archive setup.
## Sentry

[Sentry](https://sentry.io/) is a monitoring tool used for the DANDI Archive API. It is integral in order to notify engineers if a system is down, experiencing poor performance, or may have unwanted users
[Sentry](https://sentry.io/) is a monitoring tool used for the DANDI Archive API. It is integral in order to notify engineers if a system is down, experiencing poor performance, or may have unwanted users.

Begin by creating a Sentry account -- once successful, you'll start by creating a new Project:

Expand All @@ -395,7 +388,7 @@ alt="retrieve"
style="width: 60%; height: auto; display: block; margin-left: auto; margin-right: auto;"/>
<br/><br/>

For your account, the free `Developer` account should be accurate, as define in [Sentry's pricing structure](https://sentry.io/pricing/)
For your account, the free `Developer` account should be sufficient, as defined in [Sentry's pricing structure](https://sentry.io/pricing/).

##### Select Django as an App Type

Expand All @@ -420,6 +413,10 @@ alt="retrieve"
style="width: 60%; height: auto; display: block; margin-left: auto; margin-right: auto;"/>
<br/><br/>

## datalad (TBD)
## datalad

We will add docs in the future (https://github.com/dandi/dandi-docs/issues/132).

## git-annex

## git-annex (TBD)
We will add docs in the future (https://github.com/dandi/dandi-docs/issues/132).
6 changes: 3 additions & 3 deletions docs/61_dandi_authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ created during [Obtaining your Oauth App creds](..60_initialize_vendors/#obtaini

### Creating "Sites" and "Social App" Objects

**In order to complete this step, you will need to have deployed an initial DANDI Archive API** -- see here for more details [Creating the DANDI Archive API](../64_dandi_archive)
**In order to complete this step, you will need to have deployed an initial DANDI Archive API**. See [Creating the DANDI Archive API](../64_dandi_archive) for more details.

First, sign in via the Django admin panel with your credentials created via the [Creating an admin user account for the DANDI Archive API step](../64_dandi_archive/#creating-a-django-superuser-admin-account)

Expand Down Expand Up @@ -73,9 +73,9 @@ There are two places to update values relevant to the frontend.
- `netlify.toml`
- `.env.production`

For `netlify.toml`: This should be located in the `web/` sub-directory -- [see DANDI Archive web/netlify.toml](https://github.com/dandi/dandi-archive/blob/master/web/netlify.toml). This is a file in which you can configure different settings for different environments in Netlify
For `netlify.toml`: This should be located in the `web/` sub-directory -- [see DANDI Archive web/netlify.toml](https://github.com/dandi/dandi-archive/blob/master/web/netlify.toml). This is a file in which you can configure different settings for different environments in Netlify.

For `.env.production`: This should also be located in the `web/` sub-directory -- [see DANDI Archive web/.env.production](https://github.com/dandi/dandi-archive/blob/master/web/.env.production). This is a file that will inject environment variables into the frontend upon build (e.g. `vite build`)
For `.env.production`: This should also be located in the `web/` sub-directory -- [see DANDI Archive web/.env.production](https://github.com/dandi/dandi-archive/blob/master/web/.env.production). This is a file that will inject environment variables into the frontend upon build (e.g. `vite build`).

Update the relevant values that reflect what was in your `Social Application` object for both files. You will also notice an environment variable related to `Sentry` -- this value can be obtained from your Sentry account.

8 changes: 4 additions & 4 deletions docs/62_dandi_cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,12 @@ Once your DANDI clone is added to the list of available `DandiInstance` objects,
Users will be prompted for a `DANDI_API_KEY`
environment variable. This variable does not need to be unique to your DANDI clone. A user can just set their `DANDI_API_KEY` to the value that your DANDI API clone issues. See docs on [storing access credentials](https://www.dandiarchive.org/handbook/13_upload/#storing-access-credentials).

## Handling Versioning
## Versioning

DANDI CLI leverages a tool called [versioneer](https://pypi.org/project/versioneer/) for semantic versioning in PyPI.
The DANDI Client leverages a tool called [versioneer](https://pypi.org/project/versioneer/) for semantic versioning in PyPI.

Upon merging of a PR into `main`, if a given GitHub label is attached to the PR (`major`, `minor` or `patch` specifically)
`versioneer` will generate a human-readable CHANGELOG entry, and then push to PyPI the proper new semantic version.
Upon merging of a PR into `master`, if the `release` label is attached to the PR
`versioneer` will generate a human-readable CHANGELOG entry, and then push to PyPI the new semantic version. For more details on labeling `dandi-cli` pull requests, see [here](https://github.com/dandi/dandi-cli/blob/master/DEVELOPMENT.md#releasing-with-github-actions-auto-and-pull-requests).

### Versioning-Realted Resources
- [PyPI package](https://pypi.org/project/dandi/)
Expand Down
22 changes: 10 additions & 12 deletions docs/63_dandi_infrastructure.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
## Configuring Terraform

[First, make sure you have properly set up your Terraform Cloud account and linked the account to the proper GitHub repository](../60_initialize_vendors/#terraform-cloud)
[First, make sure you have set up your Terraform Cloud account and linked the account to the GitHub repository](../60_initialize_vendors/#terraform-cloud)

To double-check whether your GitHub repository is properly linked, proceed to the `Version Control` tab.
To double-check whether your GitHub repository is linked, proceed to the `Version Control` tab.

The `Version Control` Repository value should point to the repository, and the `Terraform Working Directory` should point to `terraform`

Expand All @@ -13,8 +13,7 @@ alt="terraform_config"
style="width: 60%; height: auto; display: block; margin-left: auto; margin-right: auto;"/>
<br/><br/>

As described in #Understanding-the-DANDI-Infrastructure,
the terraform code in the [dandi-infrastructure](https://github.com/dandi/dandi-infrastructure)
As described in [Understanding-the-DANDI-Infrastructure](https://github.com/dandi/dandi-infrastructure)
repository includes many components that may not be needed for your use case. You will need to define the infrastructure components in stepwise fashion, starting with the `api.tf` and `sponsored_bucket.tf`.

## Applying Terraform
Expand Down Expand Up @@ -50,14 +49,13 @@ style="width: 60%; height: auto; display: block; margin-left: auto; margin-righ

## Understanding the DANDI Infrastructure

### Girder
### Resonant

In the [api.tf definition](https://github.com/dandi/dandi-infrastructure/blob/master/terraform/api.tf), there is reference
to a `source` keyword, where a [Terraform module called girder is defined](https://github.com/girder/terraform-heroku-girder4).
to a `source` keyword, where a Terraform module called [Resonant](https://github.com/kitware-resonant/terraform-heroku-resonant) is defined.

In the Girder submodule, AWS and Heroku resources are defined that facilitate base resources for compute and networking to work with DANDI Archive.

Within the DANDI Infrastructure downstream, Girder is used by declaring values that the Terraform module expects. **Resources declared by Girder cannot be overwritten**
In the Resonant submodule, AWS and Heroku resources are defined that facilitate base resources for compute and networking to work with DANDI Archive.
Within the DANDI Infrastructure downstream, Resonant is used by declaring values that the Terraform module expects. **Resources declared by Resonant cannot be overwritten**

#### Sponsored Bucket

Expand All @@ -70,6 +68,7 @@ A `sponsored bucket` is also declared in the `main.tf`, with downstream, related
**DANDI Infrastructure assumes that you 1. own a domain, and 2. have purchased that domain (or have that domain managed) via AWS Route 53**

DANDI Infrastructure connects domains from three different vendors:

- **Netlify**: Manages load balancer IPs and custom domains for the UI.
- **AWS Route 53**: Manages CNAME records for SSL certificates and links Heroku API URLs to domains.
- **Heroku**: Provides domains for API services, which are aliased via AWS Route 53.
Expand Down Expand Up @@ -178,8 +177,7 @@ As long as the CNAME is covered by a valid SSL certificate, should be fully set

## AWS Buckets

While [Girder](https://github.com/girder/terraform-heroku-girder4) does declare S3-based resources, configuration is still needed within DANDI Infrastructure.

While [Resonant](https://github.com/kitware-resonant/terraform-heroku-resonant) does declare S3-based resources, configuration is still needed within DANDI Infrastructure.
Find your AWS Account ID. This value will be referenced in the `main.tf` Terraform template

<br/><br/>
Expand All @@ -202,4 +200,4 @@ Setting up staging will require unique AWS Route 53 Domains, as well a different

### Email Setup

TBD
We will add docs in the future (https://github.com/dandi/dandi-docs/issues/177).

0 comments on commit 3dc1823

Please sign in to comment.