Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LGA-3086: Setup CI/CD #3

Merged
merged 98 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
8d89aad
Add build-and-push workflow
said-moj May 14, 2024
ec55a4f
Add ruff code formatting and linting job
said-moj May 14, 2024
6c265b2
Added inital helm chart
BenMillar-MOJ May 14, 2024
d04397a
Give the Workflow a name
BenMillar-MOJ May 15, 2024
bb03f14
Gave the jobs names
BenMillar-MOJ May 15, 2024
762cf7c
Only build if linting and formatting has passed
BenMillar-MOJ May 15, 2024
b5acb24
Renamed jobs
BenMillar-MOJ May 15, 2024
251ef9e
Update the GitHub actions in the build and push workflow
BenMillar-MOJ May 15, 2024
f54f732
Add deploy step
BenMillar-MOJ May 15, 2024
8ba8d63
Modified cluster authentication step
BenMillar-MOJ May 15, 2024
b06f2f3
Add secrets to deploy call
BenMillar-MOJ May 15, 2024
efcaff2
Wrap env var in curly braces
BenMillar-MOJ May 15, 2024
3ac0bc4
Change repo path
BenMillar-MOJ May 15, 2024
4608507
Uses values directory
BenMillar-MOJ May 15, 2024
b133cdd
Set nameOverride and fullnameOverride to nothing
BenMillar-MOJ May 15, 2024
f9f6f20
Moved values.yaml
BenMillar-MOJ May 15, 2024
d1e2744
Disable autoscaling
BenMillar-MOJ May 15, 2024
812c4dd
Disable creating a service account
BenMillar-MOJ May 15, 2024
66d2b10
Updated helm upgrade step for other environments
BenMillar-MOJ May 15, 2024
d132408
Added additional environment tags
BenMillar-MOJ May 15, 2024
04c8e9e
Only run staging and production jobs on main
BenMillar-MOJ May 15, 2024
25f09d8
Updated job dependencies
BenMillar-MOJ May 15, 2024
7b0ef10
Remove dependency on dev from UAT
BenMillar-MOJ May 15, 2024
6dd87f3
Add back uat environment requirement
BenMillar-MOJ May 15, 2024
31d14b9
Make deploy workflow modular
BenMillar-MOJ May 15, 2024
03ba3cf
Revert "Make deploy workflow modular"
BenMillar-MOJ May 15, 2024
15c3992
Broke deploy steps into different files
BenMillar-MOJ May 15, 2024
6f07112
Refactored deploy into a reusable step
BenMillar-MOJ May 15, 2024
66f29e3
Added ingres annotations
BenMillar-MOJ May 15, 2024
3b66c37
Set cluster name and weight
BenMillar-MOJ May 15, 2024
afa60c8
Name the workflow after the branch
BenMillar-MOJ May 15, 2024
09d2db4
Revert "Name the workflow after the branch"
BenMillar-MOJ May 15, 2024
a1d7c30
Renamed run name to the name of the branch
BenMillar-MOJ May 15, 2024
02d0698
Add two workflows depending on branch
BenMillar-MOJ May 16, 2024
c8c6628
Fixed typo
BenMillar-MOJ May 16, 2024
4485215
Fixed typo
BenMillar-MOJ May 16, 2024
1fda27f
Add testing workflow
BenMillar-MOJ May 16, 2024
1d8f521
Fixed typo
BenMillar-MOJ May 16, 2024
accd7ab
Adds runs-on
BenMillar-MOJ May 16, 2024
150211e
Removed requirements
BenMillar-MOJ May 16, 2024
c74561e
Add test test
BenMillar-MOJ May 16, 2024
9d8e890
Renamed tests
BenMillar-MOJ May 16, 2024
7dc459c
Renamed Build and Push to ECR
BenMillar-MOJ May 16, 2024
d44c1c9
Upload coverage results as an artifact
BenMillar-MOJ May 16, 2024
2d1c0b4
Write coverage as comment
BenMillar-MOJ May 16, 2024
bc3e34f
Removed coverage report
BenMillar-MOJ May 16, 2024
fab6c1b
Revert back to using artifacts
BenMillar-MOJ May 16, 2024
8c4e4b3
Updated deploy step
BenMillar-MOJ May 16, 2024
cfb2cba
Fixed typo
BenMillar-MOJ May 16, 2024
dad9f07
Add names to Build steps
BenMillar-MOJ May 16, 2024
f2c827f
Log into ECR during deploy step
BenMillar-MOJ May 16, 2024
2d47a39
Use registry as the image repository
BenMillar-MOJ May 16, 2024
e95d8e5
Adds repository to the end of the registry URL
BenMillar-MOJ May 16, 2024
8985404
Remove pod security context
BenMillar-MOJ May 16, 2024
0173c59
Make image run as non-root user
BenMillar-MOJ May 16, 2024
1da0b1d
Added Hello World
BenMillar-MOJ May 16, 2024
76ed496
Added podSecurityContext
BenMillar-MOJ May 16, 2024
007cb3f
Fixed typo
BenMillar-MOJ May 16, 2024
f5d698d
Added additional required securityContext
BenMillar-MOJ May 16, 2024
2feefe5
Add a simple flask app
BenMillar-MOJ May 16, 2024
11413aa
Create user as root
BenMillar-MOJ May 16, 2024
b59dad1
Add simple flask app
BenMillar-MOJ May 16, 2024
96e8f3c
Use user 1000
BenMillar-MOJ May 16, 2024
f739a92
Re-add test.py
BenMillar-MOJ May 16, 2024
acdc0ed
Use numerical user ID
BenMillar-MOJ May 16, 2024
bfe4fc9
Enabled dev ingress
BenMillar-MOJ May 16, 2024
8ebf773
Always register service backend
BenMillar-MOJ May 16, 2024
7ebf11c
Always set path type
BenMillar-MOJ May 16, 2024
248bed2
Changed service to use ClusterIP and added TLS host
BenMillar-MOJ May 17, 2024
8b68078
Set a default ingress ClassName
BenMillar-MOJ May 17, 2024
14568ba
Changed service targetPort to 8000
BenMillar-MOJ May 17, 2024
9061261
Reverted targetPort
BenMillar-MOJ May 17, 2024
f268e76
Change container port to 8000
BenMillar-MOJ May 17, 2024
221f8b1
containerPort is now set in values
BenMillar-MOJ May 17, 2024
1711aea
Added dev deploy
BenMillar-MOJ May 17, 2024
c66678d
Fixed typo
BenMillar-MOJ May 17, 2024
0a67b8d
Added additional \
BenMillar-MOJ May 17, 2024
8036236
Shortened URL octet
BenMillar-MOJ May 17, 2024
5121374
Added TLS to dev
BenMillar-MOJ May 17, 2024
49e5d3d
Added clean up release workflow
BenMillar-MOJ May 17, 2024
9eaf671
Use head_ref rather than ref_name
BenMillar-MOJ May 17, 2024
6f82c9b
Redeploy to dev when PR is reopened
BenMillar-MOJ May 17, 2024
90323d1
Improved method of getting branch name
BenMillar-MOJ May 17, 2024
9695f3c
Use self-hosted runner
BenMillar-MOJ May 17, 2024
86000e3
Runs static analysis on ubuntu-latest
BenMillar-MOJ May 17, 2024
09ff300
Revert "Use self-hosted runner"
BenMillar-MOJ May 17, 2024
51fca4e
Set correct dev TLS URL
BenMillar-MOJ May 17, 2024
a8fcaa3
Change how TLS host is set
BenMillar-MOJ May 17, 2024
31ba414
Use single octet for dev URL
BenMillar-MOJ May 18, 2024
4269d52
Remove unnecessary version checks
BenMillar-MOJ May 19, 2024
34f67e1
Temporarily allow main branch workflow to run from this feature branch
BenMillar-MOJ May 19, 2024
ae24ac1
Added test step to main branch workflow
BenMillar-MOJ May 19, 2024
8bf9cb8
Added Release/ Feature tag to run names
BenMillar-MOJ May 19, 2024
0cdff1c
Add VPN whitelist to ingress
BenMillar-MOJ May 19, 2024
ef19c78
Removed temporary ability for the feature branch to push to staging a…
BenMillar-MOJ May 19, 2024
debea38
Added trailing blank line to EOF
BenMillar-MOJ May 19, 2024
9024ed5
Uses environment variable to get the ECR Registry
BenMillar-MOJ May 21, 2024
daebd24
Removed unused deployment boiler plate
BenMillar-MOJ May 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: Build and Push image
on:
workflow_call:
inputs:
ECR_REGION:
required: true
type: string
ECR_REPOSITORY:
required: true
type: string
secrets:
ECR_ROLE_TO_ASSUME:
required: true


jobs:
build-and-push-to-ecr:
name: Build and Push
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
steps:
- name: Checkout GitHub repository
uses: actions/checkout@v4

- name: Assume role in Cloud Platform
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.ECR_ROLE_TO_ASSUME }}
aws-region: ${{ inputs.ECR_REGION }}

- name: Login to container repository
uses: aws-actions/amazon-ecr-login@v2
id: login-ecr
with:
mask-password: true

- name: Build and push a Docker image to the container repository
id: docker-build
run: |
docker build -t $REGISTRY/$REPOSITORY:$IMAGE_TAG .
docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: ${{ inputs.ECR_REPOSITORY }}
IMAGE_TAG: ${{ github.sha }}
37 changes: 37 additions & 0 deletions .github/workflows/cleanup-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Uninstalls the dev helm chart when a PR is merged, or closed
name: Clean up the dev release

run-name: Clean up ${{ github.head_ref || github.ref_name }}

on:
pull_request:
types:
- closed

jobs:
clean-up-release:
name: Clean up release
environment: dev
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
steps:
- name: Authenticate to the cluster
env:
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
run: |
echo "${{ secrets.KUBE_CERT }}" > ca.crt
kubectl config set-cluster ${KUBE_CLUSTER} --certificate-authority=./ca.crt --server=https://${KUBE_CLUSTER}
kubectl config set-credentials deploy-user --token=${{ secrets.KUBE_TOKEN }}
kubectl config set-context ${KUBE_CLUSTER} --cluster=${KUBE_CLUSTER} --user=deploy-user --namespace=${KUBE_NAMESPACE}
kubectl config use-context ${KUBE_CLUSTER}

- name: Uninstall the helm chart
env:
# head_ref is set if the workflow was triggered by a PR, ref_name is used if the workflow was trigged by a push.
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
run: |
export CLEANED_BRANCH_NAME=$(echo ${BRANCH_NAME} | sed 's/^feature[-/]//' | sed 's:^\w*\/::' | tr -s ' _/[]().' '-' | tr '[:upper:]' '[:lower:]' | cut -c1-28 | sed 's/-$//')
helm uninstall ${CLEANED_BRANCH_NAME}
73 changes: 73 additions & 0 deletions .github/workflows/deploy-dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Deploy image to the dev environment
on:
workflow_call:
inputs:
environment:
required: true
type: string
ECR_REGION:
required: true
type: string
ECR_REPOSITORY:
required: true
type: string
secrets:
ECR_ROLE_TO_ASSUME:
required: true
KUBE_CERT:
required: true
KUBE_CLUSTER:
required: true
KUBE_NAMESPACE:
required: true
KUBE_TOKEN:
required: true


jobs:
deploy:
name: Deploy
environment: ${{ inputs.environment }}
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
steps:
- name: Checkout GitHub repository
uses: actions/checkout@v4

- name: Authenticate to the cluster
env:
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
run: |
echo "${{ secrets.KUBE_CERT }}" > ca.crt
kubectl config set-cluster ${KUBE_CLUSTER} --certificate-authority=./ca.crt --server=https://${KUBE_CLUSTER}
kubectl config set-credentials deploy-user --token=${{ secrets.KUBE_TOKEN }}
kubectl config set-context ${KUBE_CLUSTER} --cluster=${KUBE_CLUSTER} --user=deploy-user --namespace=${KUBE_NAMESPACE}
kubectl config use-context ${KUBE_CLUSTER}

- name: Upgrade the Helm chart
env:
IMAGE_TAG: ${{ github.sha }}
REGISTRY: ${{ secrets.ECR_REGISTRY }}
REPOSITORY: ${{ inputs.ECR_REPOSITORY }}
HELM_DIR: "helm_deploy/laa-access-civil-legal-aid"
DEV_HOST: "access-cla.cloud-platform.service.justice.gov.uk"
# head_ref is set if the workflow was triggered by a PR, ref_name is used if the workflow was trigged by a push.
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
run: |
export CLEANED_BRANCH_NAME=$(echo ${BRANCH_NAME} | sed 's/^feature[-/]//' | sed 's:^\w*\/::' | tr -s ' _/[]().' '-' | tr '[:upper:]' '[:lower:]' | cut -c1-28 | sed 's/-$//')
export HOST_NAME=${CLEANED_BRANCH_NAME}-${DEV_HOST}

helm upgrade ${CLEANED_BRANCH_NAME} \
${HELM_DIR} \
--namespace=${{ secrets.KUBE_NAMESPACE }} \
--values ${HELM_DIR}/values/values-${{ inputs.environment }}.yaml \
--set image.repository=${REGISTRY}/${REPOSITORY} \
--set image.tag=${IMAGE_TAG} \
--set fullnameOverride=${CLEANED_BRANCH_NAME} \
--set ingress.hosts[0].host=${HOST_NAME} \
--set ingress.tls[0].hosts[0]=${HOST_NAME} \
--force \
--install
64 changes: 64 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Deploy to a Cloud Platform environment
on:
workflow_call:
inputs:
environment:
required: true
type: string
ECR_REGION:
required: true
type: string
ECR_REPOSITORY:
required: true
type: string
secrets:
ECR_ROLE_TO_ASSUME:
required: true
KUBE_CERT:
required: true
KUBE_CLUSTER:
required: true
KUBE_NAMESPACE:
required: true
KUBE_TOKEN:
required: true


jobs:
deploy:
name: Deploy
environment: ${{ inputs.environment }}
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
steps:
- name: Checkout GitHub repository
uses: actions/checkout@v4

- name: Authenticate to the cluster
env:
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
run: |
echo "${{ secrets.KUBE_CERT }}" > ca.crt
kubectl config set-cluster ${KUBE_CLUSTER} --certificate-authority=./ca.crt --server=https://${KUBE_CLUSTER}
kubectl config set-credentials deploy-user --token=${{ secrets.KUBE_TOKEN }}
kubectl config set-context ${KUBE_CLUSTER} --cluster=${KUBE_CLUSTER} --user=deploy-user --namespace=${KUBE_NAMESPACE}
kubectl config use-context ${KUBE_CLUSTER}

- name: Upgrade the Helm chart
env:
IMAGE_TAG: ${{ github.sha }}
REGISTRY: ${{ secrets.ECR_REGISTRY }}
REPOSITORY: ${{ inputs.ECR_REPOSITORY }}
HELM_DIR: "helm_deploy/laa-access-civil-legal-aid"
run: |
helm upgrade laa-access-civil-legal-aid \
${HELM_DIR} \
--namespace=${{ secrets.KUBE_NAMESPACE }} \
--values ${HELM_DIR}/values/values-${{ inputs.environment }}.yaml \
--set image.repository=${REGISTRY}/${REPOSITORY} \
--set image.tag=${IMAGE_TAG} \
--force \
--install
60 changes: 60 additions & 0 deletions .github/workflows/feature-branch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Feature Branch

run-name: Feature - ${{ github.head_ref || github.ref_name }}

on:
push:
branches-ignore:
- main
pull_request:
types:
- reopened

jobs:
static-analysis:
name: Static Analysis
uses: ./.github/workflows/static-analysis.yml

test:
name: Test
uses: ./.github/workflows/test.yml

build-and-push:
name: Build
uses: ./.github/workflows/build.yml
needs: static-analysis
with:
ECR_REGION: ${{vars.ECR_REGION}}
ECR_REPOSITORY: ${{vars.ECR_REPOSITORY}}
secrets:
ECR_ROLE_TO_ASSUME: ${{ secrets.ECR_ROLE_TO_ASSUME }}

deploy-dev:
name: Dev
uses: ./.github/workflows/deploy-dev.yml
needs: build-and-push
with:
environment: dev
ECR_REGION: ${{vars.ECR_REGION}}
ECR_REPOSITORY: ${{vars.ECR_REPOSITORY}}
secrets:
ECR_ROLE_TO_ASSUME: ${{ secrets.ECR_ROLE_TO_ASSUME }}
KUBE_CERT: ${{ secrets.KUBE_CERT }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_TOKEN: ${{ secrets.KUBE_TOKEN }}

deploy-uat:
name: UAT
uses: ./.github/workflows/deploy.yml
needs: [build-and-push, test]
with:
environment: uat
ECR_REGION: ${{vars.ECR_REGION}}
ECR_REPOSITORY: ${{vars.ECR_REPOSITORY}}
secrets:
ECR_ROLE_TO_ASSUME: ${{ secrets.ECR_ROLE_TO_ASSUME }}
KUBE_CERT: ${{ secrets.KUBE_CERT }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_TOKEN: ${{ secrets.KUBE_TOKEN }}
57 changes: 57 additions & 0 deletions .github/workflows/main-branch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Release

run-name: Release - ${{ github.ref_name }}

on:
push:
branches:
- main

jobs:
static-analysis:
name: Static Analysis
uses: ./.github/workflows/static-analysis.yml

test:
name: Test
uses: ./.github/workflows/test.yml

build-and-push:
name: Build
uses: ./.github/workflows/build.yml
needs: [static-analysis, test]
with:
ECR_REGION: ${{vars.ECR_REGION}}
ECR_REPOSITORY: ${{vars.ECR_REPOSITORY}}
secrets:
ECR_ROLE_TO_ASSUME: ${{ secrets.ECR_ROLE_TO_ASSUME }}

deploy-staging:
name: Staging
uses: ./.github/workflows/deploy.yml
needs: build-and-push
with:
environment: staging
ECR_REGION: ${{vars.ECR_REGION}}
ECR_REPOSITORY: ${{vars.ECR_REPOSITORY}}
secrets:
ECR_ROLE_TO_ASSUME: ${{ secrets.ECR_ROLE_TO_ASSUME }}
KUBE_CERT: ${{ secrets.KUBE_CERT }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_TOKEN: ${{ secrets.KUBE_TOKEN }}

deploy-production:
name: Production
uses: ./.github/workflows/deploy.yml
needs: deploy-staging
with:
environment: production
ECR_REGION: ${{vars.ECR_REGION}}
ECR_REPOSITORY: ${{vars.ECR_REPOSITORY}}
secrets:
ECR_ROLE_TO_ASSUME: ${{ secrets.ECR_ROLE_TO_ASSUME }}
KUBE_CERT: ${{ secrets.KUBE_CERT }}
KUBE_CLUSTER: ${{ secrets.KUBE_CLUSTER }}
KUBE_NAMESPACE: ${{ secrets.KUBE_NAMESPACE }}
KUBE_TOKEN: ${{ secrets.KUBE_TOKEN }}
13 changes: 13 additions & 0 deletions .github/workflows/static-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: Static Analysis
on: workflow_call

jobs:
lint:
name: Ruff
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: chartboost/ruff-action@v1
with:
args: check --output-format=github
src: './src'
33 changes: 33 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Test

on: workflow_call

jobs:
test:
name: Pytest
runs-on: ubuntu-latest
steps:
- name: Checkout the repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install dependencies
run: |
python -m pip install --upgrade pip

- name: Test with pytest
run: |
pip install pytest pytest-cov
pytest tests.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html

- name: Upload pytest test results
uses: actions/upload-artifact@v4
with:
name: pytest-results
path: junit/test-results.xml
# Use always() to always run this step to publish test results when there are test failures
if: ${{ always() }}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ env/
*.code-workspace
*.sha256
terraform.tfstate
.idea
Loading