Skip to content

Commit 7396262

Browse files
irvingpopirvingpop
authored
Rework Kubernetes deployment to use EKS+fargate for everything (#143)
* v1 rework kube configs to use EKS+fargate for everything Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * cleanup: remove grafana ingress bits Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * fix service type for ALB ingress controller to work right, disable external-dns debug logging Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * Cleanup ingresses so they actually allow requests, fix EXTRA_HOSTS so they recognize the new names Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * Add the template for eksctl plus a simple readme Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * worried spot types aren't going to work for us, rather going with 1 small managed node Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * Add ArgoCD install notes and modified install.yaml Signed-off-by: Irving Popovetsky <irving@honeycomb.io> * Update ingresses for resources-api Signed-off-by: Irving Popovetsky <irving@honeycomb.io>
1 parent a664124 commit 7396262

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2705
-845
lines changed

‎SETUP.md

Lines changed: 5 additions & 280 deletions
Original file line numberDiff line numberDiff line change
@@ -2,284 +2,9 @@
22

33
Greetings! Much of Operation Code's web site runs in a [Kubernetes](https://kubernetes.io/) cluster. These instructions will guide you through setting up access to our cluster so you can run rails console, tail logs, and more!
44

5-
## What you need
6-
* An Operation Code Google account in the form of walt@operationcode.org
7-
* Access to 1Password to get the Google Application Client Secret and the Kubernetes cluster Certificate Authority data
8-
9-
# From OS X
10-
11-
## Installing the Kubernetes Command Line
12-
13-
This is what you will use to interact with our Kubernetes cluster - where both the front end and back end of the site runs. If you have not already, install the [Homebrew Package Manager](https://brew.sh/), and run the following:
14-
15-
```bash
16-
brew install kubernetes-cli
17-
```
18-
19-
## Authenticating to the Operation Code Kubernetes Cluster
20-
21-
You will use your email@operationcode.org gmail account to authenticate to our cluster. We use a helper to do this - the k8s-oidc-helper. This helper is written in go - and to use it, we'll need to install the go language and create some configuration.
22-
23-
### Installing Go
24-
25-
First, install the go language onto your workstation:
26-
27-
```bash
28-
$ brew install golang
29-
```
30-
31-
Now, let's add in some configuration for go. Open up your profile file (this is bash_profile if you are using bash as your shell)
32-
33-
```bash
34-
$ vim ~/.bash_profile
35-
```
36-
37-
At the end of the file, add this line:
38-
39-
```bash
40-
export GOPATH=$HOME/gocode
41-
export PATH=$PATH:$GOPATH/bin
42-
```
43-
44-
Now save and close the file, the source it
45-
46-
```bash
47-
$ source ~/.bash_profile
48-
```
49-
50-
Now, check that you can run go commands with this command, you should see it output your version of go
51-
52-
```bash
53-
$ go version
54-
```
55-
56-
## Installing the helper
57-
58-
Alright, now we're ready to install the k8s-oidc-helper. Run this command:
59-
60-
```bash
61-
$ go get github.com/micahhausler/k8s-oidc-helper
62-
```
63-
64-
(Don't fret if you do not see any output, this is normal).
65-
66-
Once it finishes running, check that the helper was installed correctly with:
67-
68-
```bash
69-
$ k8s-oidc-helper --version
70-
```
71-
72-
And it should display the version of the helper.
73-
74-
## Configuring the helper
75-
76-
Now, you'll need to download something from 1Password. If you do not have access to the Operation Code 1Password, reach out to the Project lead, seargent, or any of the maintainers for information. Once you are in 1Password look for a credential called "oauth-oc".
77-
78-
That credential contains a file called client_secret_(...)apps.googleusercontent.com.json. Download this file to your local workstation. I like to save it as "client_secret.json". Now run the helper, passing it this config file.
79-
80-
```bash
81-
$ k8s-oidc-helper -c path/to/client_secret.json
82-
```
83-
84-
If it works correctly, it will tell you to open a url in your browser. Open that url - log in to or select your operation.org account if necessary - and copy the code that is displayed, then paste it next to the prompt "Enter the code Google gave you:"
85-
86-
Copy the output that starts with "#Add the following to your ~/.kube/config".
87-
88-
## Configuring Kubernetes
89-
90-
Now we'll use this to configure access to Operation Code's Kubernetes cluster.
91-
92-
Create a ~/.kube directory
93-
94-
```bash
95-
$ mkdir ~/.kube
96-
```
97-
98-
Now create a file at ~/.kube/config
99-
100-
```bash
101-
$ vim ~/.kube/config
102-
```
103-
104-
And paste in the content you just copied when you ran the k8s-oidc-helper.
105-
106-
Save and close the file.
107-
108-
Alright - we're almost there! First, run a couple of commands to further configure Kubernetes:
109-
110-
```bash
111-
$ kubectl config set-context op-code-prod --cluster k8s.operationcode.org --user nell@operationcode.org
112-
$ kubectl config use-context op-code-prod
113-
```
114-
115-
Now, head back to 1Password and look for a note called "Kubernetes Cluster CA". Copy the content of that note and open your kube config file.
116-
117-
```bash
118-
$ vim ~/.kube/config
119-
```
120-
121-
And replace this line:
122-
123-
```bash
124-
clusters: []
125-
```
126-
127-
With this line:
128-
129-
```bash
130-
clusters:
131-
```
132-
133-
Then, directly after that line, paste the contents of the note you just copied from 1Password.
134-
135-
Save and close the file, then run this command:
136-
137-
```bash
138-
$ kubectl get pods -n operationcode
139-
```
140-
141-
After a few seconds, you should see a list of running Kubernetes pods including operationcode-backend, operationcode-frontend, and more!
142-
143-
# From Linux (Ubuntu)
144-
145-
## Installing the Kubernetes Command Line
146-
147-
This is what you will use to interact with our Kubernetes cluster - where both the front end and back end of the site runs.
148-
149-
* Install the Kubernetes command line
150-
```bash
151-
$ sudo snap install kubectl --classic
152-
```
153-
154-
## Authenticating to the Operation Code Kubernetes Cluster
155-
156-
You will use your email@operationcode.org gmail account to authenticate to our cluster. We use a helper to do this - the k8s-oidc-helper. This helper is written in go - and to use it, we'll need to install the go language and create some configuration.
157-
158-
### Installing Go
159-
160-
First, install the go language on your workstation with these commands (you will want to do it this way, as the one in the ubuntu package manager is quite out of date)
161-
162-
```bash
163-
$ sudo curl -O https://storage.googleapis.com/golang/go1.9.3.linux-amd64.tar.gz
164-
$ sudo tar -xvf go1.9.3.linux-amd64.tar.gz
165-
$ sudo mv go /usr/local
166-
```
167-
168-
Now, let's add in some configuration for go. Open up your profile file
169-
170-
```bash
171-
$ vim ~/.profile
172-
```
173-
174-
At the end of the file, add this line:
175-
176-
```bash
177-
export PATH=$PATH:/usr/local/go/bin
178-
```
179-
180-
Now save and close the file, the source it
181-
182-
```bash
183-
$ source ~/.profile
184-
```
185-
186-
Now, check that you can run go commands with this command, you should see it output your version of go
187-
188-
```bash
189-
$ go version
190-
```
191-
192-
Next, we need to se the $GOPATH environmental variable - I'm going to set mine to /usr/local, but you can set it wherever you would like your go packages to be installed.
193-
194-
```bash
195-
export GOPATH=/usr/local
196-
```
197-
198-
## Installing the helper
199-
200-
Alright, now we're ready to install the k8s-oidc-helper. Run this command:
201-
202-
```bash
203-
$ go get github.com/micahhausler/k8s-oidc-helper
204-
```
205-
206-
(Don't fret if you do not see any output, this is normal).
207-
208-
Once it finishes running, check that the helper was installed correctly with:
209-
210-
```bash
211-
$ k8s-oidc-helper --version
212-
```
213-
214-
And it should display the version of the helper.
215-
216-
## Configuring the helper
217-
218-
Now, you'll need to download something from 1Password. If you do not have access to the Operation Code 1Password, reach out to the Project lead, seargent, or any of the maintainers for information. Once you are in 1Password look for a credential called "oauth-oc".
219-
220-
That credential contains a file called client_secret_(...)apps.googleusercontent.com.json. Download this file to your local workstation. I like to save it as "client_secret.json". Now run the helper, passing it this config file.
221-
222-
```bash
223-
$ k8s-oidc-helper -c path/to/client_secret.json
224-
```
225-
226-
If it works correctly, it will tell you to open a url in your browser. Open that url - log in to or select your operation.org account if necessary - and copy the code that is displayed, then paste it next to the prompt "Enter the code Google gave you:"
227-
228-
Copy the output that starts with "#Add the following to your ~/.kube/config".
229-
230-
## Configuring Kubernetes
231-
232-
Now we'll use this to configure access to Operation Code's Kubernetes cluster.
233-
234-
Create a ~/.kube directory
235-
236-
```bash
237-
$ mkdir ~/.kube
238-
```
239-
240-
Now create a file at ~/.kube/config
241-
242-
```bash
243-
$ vim ~/.kube/config
244-
```
245-
246-
And paste in the content you just copied when you ran the k8s-oidc-helper.
247-
248-
Save and close the file.
249-
250-
Alright - we're almost there! First, run a couple of commands to further configure Kubernetes:
251-
252-
```bash
253-
$ kubectl config set-context op-code-prod --cluster k8s.operationcode.org --user nell@operationcode.org
254-
$ kubectl config use-context op-code-prod
255-
```
256-
257-
Now, head back to 1Password and look for a note called "Kubernetes Cluster CA". Copy the content of that note and open your kube config file.
258-
259-
```bash
260-
$ vim ~/.kube/config
261-
```
262-
263-
And replace this line:
264-
265-
```bash
266-
clusters: []
267-
```
268-
269-
With this line:
270-
271-
```bash
272-
clusters:
273-
```
274-
275-
Then, directly after that line, paste the contents of the note you just copied from 1Password.
276-
277-
Save and close the file, then run this command:
278-
279-
```bash
280-
$ kubectl get pods -n operationcode
281-
```
282-
283-
After a few seconds, you should see a list of running Kubernetes pods including operationcode-backend, operationcode-frontend, and more!
284-
5+
# Getting access to the cluster
2856

7+
1. Ensure you have AWS access, and the aws CLI is operating correctly
8+
2. Install eksctl: https://eksctl.io/introduction/installation/
9+
3. Run: `eksctl utils write-kubeconfig --region us-east-2 --cluster operationcode-backend`
10+
4. Verify everything works: `kubectl get namespaces`

‎kubernetes/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Setup
2+
3+
To re-create a cluster, everything you need is in the eksctl/ folder. Use eksctl with the `operationcode-backend.yaml` config file to create the cluster.
4+
Then install the controllers:
5+
* aws-alb-ingress-controller
6+
* external-dns
7+
* vertical-pod-autoscaler
8+

‎kubernetes/argocd/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Setup
2+
3+
The install.yaml in this folder is slightly modified from the [ArgoCD setup instructions](https://argoproj.github.io/argo-cd/getting_started/), the Redis server has been configured to persist data. Please keep that in mind and don't update from the stock install.yaml file
4+

0 commit comments

Comments
 (0)