Skip to content

Rework Kubernetes deployment to use EKS+fargate for everything #143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Mar 1, 2020

Conversation

irvingpop
Copy link
Collaborator

@irvingpop irvingpop commented Feb 24, 2020

Problem: Our current Kubernetes cluster is too expensive to run and also has manageability challenges.

This PR's goal is to fully document the new operationcode-backend EKS cluster, so that it can be rebuilt if there's ever a need. Nearly everything runs "serverless" via Fargate, which should save a considerable amount of money. The only exception is ArgoCD, which requires persistent volumes, must run on an EC2 node.

Add configs for:

  • external-dns controller
  • ALB ingress controller
  • vertical pod autoscaler

Remove:

  • default http backend
  • nginx ingress controller

Rework:

  • python backend ingress
  • ArgoCD
  • resource-api ingress

Fixes: #138
Fixes: #139

Signed-off-by: Irving Popovetsky irving@honeycomb.io

Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
…ternal-dns debug logging

Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
… they recognize the new names

Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
…mall managed node

Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
@irvingpop irvingpop marked this pull request as ready for review February 24, 2020 20:47
@irvingpop irvingpop changed the title v1 rework kube configs to use EKS+fargate for everything Rework Kubernetes deployment to use EKS+fargate for everything Feb 24, 2020
@kylemh
Copy link
Member

kylemh commented Feb 24, 2020

i am out of my depth on this one, @irvingpop - sorry!

@irvingpop
Copy link
Collaborator Author

one thought, regarding the infrastructure node. Because of CNI limitations a t3a.small is limited to 8 pods total and is already at 7 out of 8 just running ArgoCD.

a t3.small is slightly more expensive but can run 12 pods if we ever need to run another persistent service. There doesn't appear to be a way to split a namespace between fargate and EC2 nodes, it's all-or-nothing

Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
@irvingpop
Copy link
Collaborator Author

Update: So that the kops cluster isn't impacted, I've created a kops_stable branch in this repo and pointed all the deployments in the ArgoCD instance on kops at that branch. Pretty sure this should be safe to merge now.

@AllenAnthes
Copy link
Member

one thought, regarding the infrastructure node. Because of CNI limitations a t3a.small is limited to 8 pods total and is already at 7 out of 8 just running ArgoCD.

a t3.small is slightly more expensive but can run 12 pods if we ever need to run another persistent service. There doesn't appear to be a way to split a namespace between fargate and EC2 nodes, it's all-or-nothing

It's a difference of like $2 a month. I say just go for it.

@irvingpop irvingpop merged commit 7396262 into master Mar 1, 2020
@irvingpop irvingpop deleted the eks_fargate branch March 1, 2020 00:00
@irvingpop irvingpop restored the eks_fargate branch March 13, 2020 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants