Update Docker Pipeline, Dependencies, and Config for Migration to UVic's Arbutus Cloud #1002
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Breakdown of changes by file:
- `.dockerignore`
- Added entries for non-production-related files to reduce image size
- `.github/workflows/docker-push-image.yml` and `.github/workflows/docker-push-uat-image.yml`
- Added `docker/setup-buildx-action@v2` for advanced Docker build features like multi-platform builds and layer caching
- Enabled caching of gem dependencies to speed up builds
- Configured build cache to push to / fetch from Docker Hub to speed up image builds
- NOTE: To adhere to DRY principles, these two files should be combined into one
- `Dockerfile.production`
- Implemented multi-stage build for improved efficiency and reduced image size
- Switched to an Alpine base image for a smaller and more secure container
- Added a healthcheck using the `/api/v1/heartbeat` path
- Configured BUNDLE_WITHOUT to exclude non-production groups during `bundle install`, keeping the image lean
- NOTE: Production environment setup requires the following environmental variables:
- BUNDLE_PATH: 'vendor/bundle'
- WICKED_PDF_PATH: '/usr/bin/wkhtmltopdf'
- BUNDLE_WITHOUT: 'development:test:mysql:aws:sandbox:ci:wkhtmltopdf'
- RAILS_ENV: production
- NODE_ENV: production
- `Gemfile`
- Added `, group: :wkhtmltopdf` to `gem 'wkhtmltopdf-binary'`. This gem is not compatible with Alpine Linux. To address that, 'wkhtmltopdf' has been added to `BUNDLE_WITHOUT`.
`bundle lock --add-platform x86_64-linux-musl` adds platform for compatibility with alpine precompiled binaries
This change aligns with best practices for containerised applications.
Configured to simulate a production environment for local testing.
Prior to this commit, errors were encountered when executing `node -v`. This commit copies the required `Node.js` libraries from the builder stage, thereby ensuring they are available within the production image.
Replaced the `COPY --from=wkhtmltopdf /usr/lib/ /usr/lib/` approach with direct installation of necessary dependencies in the Dockerfile's production stage. The specified dependencies are copied directly from https://github.com/Surnet/docker-wkhtmltopdf/pkgs/container/alpine-wkhtmltopdf#other-images . This change resolves a text styling issue where text was failing to render in bold.
- Directly install Node.js within the Dockerfile production stage. - Because Node.js is also installed within the builder stage, we previously tried copying the binary and its libraries from that stage into the production stage. However, directly installing it again within the production stage simplifies the setup and ensures all dependencies are installed correctly. (The commits 809c832 and cff19b2 fix issues encountered when applying this COPY approach to wkhtmltopdf and Node.js).
- Added `apk update` to address warnings related to outdated package lists. - The chosen nodejs and postgresql versions are the current default ones, which will help in maintaining stability across builds.
Bumps [rack](https://github.com/rack/rack) from 2.2.10 to 2.2.11. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@v2.2.10...v2.2.11) --- updated-dependencies: - dependency-name: rack dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
- Split the Dockerfile's `production` stage into separate `api` and `web` stages - `web` handles static assets via Nginx and proxies dynamic requests to `api` (Rails app) - Created `nginx.conf` to configure Nginx to serve static assets and proxy dynamic requests to the Rails API. - Updated GitHub Actions workflows to build and push both the Web (Nginx) and API (Rails) Docker images to DockerHub.
- Commented out Docker healthcheck to avoid unnecessary retries. - Set Nginx server_name to wildcard (_) to allow all incoming requests. - Commented out build args for simplicity in initial push. This commit will be reverted once the relevant build args are finalised and handled properly.
- Re-enabled Docker Healthcheck, now using dynamic runtime `BASE_URL` environmental var - Replaced wildcard(_) in nginx.conf with dynamic runtime `SERVER_NAME` environmental var - Removed unnecessary build-args from Docker-related .github/workflows files. These changes revert the temporary changes made in commit d19ab7e, ensuring a more secure and flexible setup.
Removed the "Cache Gem dependencies" step from both Docker image workflows. - Bundler dependencies (`vendor/bundle`) were previously cached using GitHub Actions, but since Docker’s `cache-to` and `cache-from` already handle these layers, the step appears to be redundant.
Removed `latest` and `production` tag generation for both the "-api" and "-web" tags Implemented the following tagging logic: - if release_type == prerelease && Git tag matches `.+portage-(\d+\.\d+\.\d+-alpha-\d+)$` pattern, then generate the following tags: - `x.y.z-alpha-n-api` and `x.y.z-alpha-n-web` - else if release_type == latest && Git tag matches `.+portage-(\d+\.\d+\.\d+)$` pattern, then generate the following tags: - `x.y.z-api` and `x.y.z-web`
Thanks to Shaun Bathgate for these code change recommendations: - Set numeric UID (1000) and GID (3000) for dmpuser and dmpgroup to avoid collisions in Kubernetes. - Ensure all necessary directories have the correct ownership and permissions. - Adjust permissions to avoid the need for root or escalation. - Update nginx configuration to run as non-root user (dmpuser). - Ensure nginx.pid is writable by dmpuser. - Use non-root user in builder stage and precompile assets as non-root user.
- Uses `envsubst` at runtime to replace `${API_URL}` in `nginx.conf`, allowing different values for `proxy_pass` in Kubernetes vs Docker environments.
- Ensures `dmpuser` has the necessary permissions to modify `nginx.conf` by setting correct ownership and write access.
- After substitution, updates permissions to 644, making `nginx.conf` read-only to prevent further modifications.
- Avoids permission issues by writing directly to `/etc/nginx/nginx.conf` using `tee` instead of `mv`.
- Keeps Nginx running in the foreground with `exec nginx -g 'daemon off;'`.
Thanks to Shaun Bathgate for these code change recommendations: - NOTE: This will have to be tested within the Kubernetes cluster. - NOTE: `proxy_set_header X-Forwarded-Proto http_x_forwarded_proto;` may need to be `proxy_set_header X-Forwarded-Proto $scheme;` - NOTE: `user dmpuser;` is absent from the file because including it results in the following warning: - `[warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2`
- Updated Content-Security-Policy (CSP) to allow Google Tag Manager (GTM) and Google Analytics (GA). - Added https://www.googletagmanager.com and https://www.google-analytics.com to script-src. - Allowed Google Fonts by adding https://fonts.googleapis.com to style-src and https://fonts.gstatic.com to font-src. - Ensured connect-src includes https://www.google-analytics.com for analytics data tracking. - Added img-src 'self' https://orcid.org; to permit ORCID images
- Added https://info.orcid.org to img-src to account for ORCID’s domain redirection - Prior to this change, the ORCID logo would not render on the `/users/edit` page
…iri-1.18.3 Bump nokogiri from 1.16.8 to 1.18.3
Bumps [uri](https://github.com/ruby/uri) from 0.13.0 to 0.13.2. - [Release notes](https://github.com/ruby/uri/releases) - [Commits](ruby/uri@v0.13.0...v0.13.2) --- updated-dependencies: - dependency-name: uri dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
….13.2 Bump uri from 0.13.0 to 0.13.2
Generated by 🚫 Danger |
Removed proxy configuration, delegating traffic routing to Traefik.
- Note, whatever value is assigned here will likely have to be assigned within `DMPROADMAP_HOST` as well
- Updated Rails host auth config to exclude the "/healthz" endpoint from DNS rebinding checks. This ensures Kubernetes readiness checks can pass successfully when pods are coming online.
Changes provided by Shaun Bathgate along with commit 76b373a
`api/v1/heartbeat` is what we want for the healthcheck request path
This reverts commit 122f927.
Changes and notes provided by Shaun Bathgate: • ^/dmp-pgd-staging/assets/(.*)$ captures everything after /assets/ into $1. • alias /usr/share/nginx/html/assets/; means Nginx maps $1 → /usr/share/nginx/html/assets/<$1>. • try_files $1 =404; ensures that if <filename> doesn’t exist, you get a 404 – and if it does exist, you get the file directly.
Might be best to add these files to assets pipeline in the future.
By using the RAILS_RELATIVE_URL_ROOT argument at build time, the asset pipeline precompiles with an embedded base URL. This approach is to ensure that all asset references are aligned with the required subdirectory.
Bumps [rack](https://github.com/rack/rack) from 2.2.12 to 2.2.13. - [Release notes](https://github.com/rack/rack/releases) - [Changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md) - [Commits](rack/rack@v2.2.12...v2.2.13) --- updated-dependencies: - dependency-name: rack dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…2.2.13 Bump rack from 2.2.12 to 2.2.13
Explicitly set `redirect_uri` in OmniAuth ORCID configuration to prevent duplication of the relative URL root (`RAILS_RELATIVE_URL_ROOT`) in OAuth callback URLs. Previously, relying on OmniAuth's automatic URL construction caused repeated path segments (e.g., `/RAILS_RELATIVE_URL_ROOT/RAILS_RELATIVE_URL_ROOT/...`), leading to OAuth failures due to mismatched registered callback URLs.
This change ensures that the referrer path includes the correct relative root URL when signing in via CILogon. Previously, signing in through CILogon was causing a 404 error due to a missing relative root in the referrer path.
This temporary patch will have to be revisited/improved in the near future.
Renamed api to app Renamed web to assets
Renamed api to app Renamed web to assets
aaronskiba
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes proposed in this PR:
Rework Docker image pipeline b80dc01 & Copy required
Nodelibraries to prod image 809c832.dockerignore.github/workflows/docker-push-image.ymland.github/workflows/docker-push-uat-image.ymldocker/setup-buildx-action@v2for advanced Docker build features like multi-platform builds and layer cachingDockerfile.production/api/v1/heartbeatpathbundle install, keeping the image leanRAILS_ENV: stagingwhen we are in the staging environment):Gemfile, group: :wkhtmltopdftogem 'wkhtmltopdf-binary'. This gem is not compatible with Alpine Linux. To address that, 'wkhtmltopdf' has been added toBUNDLE_WITHOUT.add
x86_64-linux-muslplatform to Gemfile.lock 1ad1de4bundle lock --add-platform x86_64-linux-musladds platform for compatibility with alpine precompiled binariesRedirect prod/staging/sandbox logs to $stdout 893b5c4
gem 'syslog-logger'in the processCopy docker-compose-development to docker-compose d9e52c2 & Add web service to
docker-compose.yml69029ccdocker-compose.ymlto simulate a production environment for local testing.