Rework Kubernetes deployment to use EKS+fargate for everything

[irvingpop]

Problem: Our current Kubernetes cluster is too expensive to run and also has manageability challenges.

This PR's goal is to fully document the new operationcode-backend EKS cluster, so that it can be rebuilt if there's ever a need. Nearly everything runs "serverless" via Fargate, which should save a considerable amount of money. The only exception is ArgoCD, which requires persistent volumes, must run on an EC2 node.

Add configs for:

• external-dns controller
• ALB ingress controller
• vertical pod autoscaler

Remove:

• default http backend
• nginx ingress controller

Rework:

• python backend ingress
• ArgoCD
• resource-api ingress

Fixes: #138
Fixes: #139

Signed-off-by: Irving Popovetsky irving@honeycomb.io

[kylemh]

i am out of my depth on this one, @irvingpop - sorry!

[irvingpop]

one thought, regarding the infrastructure node. Because of CNI limitations a t3a.small is limited to 8 pods total and is already at 7 out of 8 just running ArgoCD.

a t3.small is slightly more expensive but can run 12 pods if we ever need to run another persistent service. There doesn't appear to be a way to split a namespace between fargate and EC2 nodes, it's all-or-nothing

[irvingpop]

Update: So that the kops cluster isn't impacted, I've created a kops_stable branch in this repo and pointed all the deployments in the ArgoCD instance on kops at that branch. Pretty sure this should be safe to merge now.

[AllenAnthes]

one thought, regarding the infrastructure node. Because of CNI limitations a t3a.small is limited to 8 pods total and is already at 7 out of 8 just running ArgoCD.

a t3.small is slightly more expensive but can run 12 pods if we ever need to run another persistent service. There doesn't appear to be a way to split a namespace between fargate and EC2 nodes, it's all-or-nothing

It's a difference of like $2 a month. I say just go for it.