Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase prometheus and thanos memory settings #185

Merged
merged 1 commit into from
Mar 19, 2024
Merged

Increase prometheus and thanos memory settings #185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions cluster/terraform_kubernetes/config/production.tfvars.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,11 @@
],
"ingress_nginx_version": "4.8.3",
"enable_lowpriority_app": true,
"prometheus_app_mem": "4Gi",
"prometheus_app_mem": "6Gi",
"prometheus_app_cpu": "0.5",
"thanos_app_mem": "1Gi",
"thanos_querier_mem": "2Gi",
"thanos_app_cpu": "0.5",
"thanos_compactor_app_mem": "2Gi",
"thanos_compactor_mem": "3Gi",
"thanos_store_mem": "2Gi",
"cluster_short": "pd"
}
7 changes: 4 additions & 3 deletions cluster/terraform_kubernetes/config/test.tfvars.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,11 @@
"enable_lowpriority_app": true,
"lowpriority_app_cpu": "0.5",
"lowpriority_app_mem": "1Gi",
"prometheus_app_mem": "4Gi",
"prometheus_app_mem": "6Gi",
"prometheus_app_cpu": "0.5",
"thanos_app_mem": "1Gi",
"thanos_compactor_app_mem": "2Gi",
"thanos_querier_mem": "2Gi",
"thanos_store_mem": "2Gi",
"thanos_compactor_mem": "3Gi",
"thanos_app_cpu": "0.5",
"cluster_short": "ts"
}
6 changes: 3 additions & 3 deletions cluster/terraform_kubernetes/prometheus.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,11 +176,11 @@ resource "kubernetes_deployment" "prometheus" {
resources {
limits = {
cpu = 1
memory = var.prometheus_app_mem
memory = var.thanos_app_mem
}
requests = {
cpu = var.prometheus_app_cpu
memory = var.prometheus_app_mem
cpu = var.thanos_app_cpu
memory = var.thanos_app_mem
}
}

Expand Down
12 changes: 6 additions & 6 deletions cluster/terraform_kubernetes/thanos.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,11 +115,11 @@ resource "kubernetes_deployment" "thanos-querier" {
resources {
limits = {
cpu = 1
memory = var.thanos_app_mem
memory = var.thanos_querier_mem
}
requests = {
cpu = var.thanos_app_cpu
memory = var.thanos_app_mem
memory = var.thanos_querier_mem
}
}
}
Expand Down Expand Up @@ -218,11 +218,11 @@ resource "kubernetes_deployment" "thanos-store-gateway" {
resources {
limits = {
cpu = 1
memory = var.thanos_app_mem
memory = var.thanos_store_mem
}
requests = {
cpu = var.thanos_app_cpu
memory = var.thanos_app_mem
memory = var.thanos_store_mem
}
}

Expand Down Expand Up @@ -307,11 +307,11 @@ resource "kubernetes_deployment" "thanos-compactor" {
resources {
limits = {
cpu = 1
memory = var.thanos_compactor_app_mem
memory = var.thanos_compactor_mem
}
requests = {
cpu = var.thanos_app_cpu
memory = var.thanos_compactor_app_mem
memory = var.thanos_compactor_mem
}
}

Expand Down
14 changes: 12 additions & 2 deletions cluster/terraform_kubernetes/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,15 +114,25 @@ variable "thanos_version" {
}

variable "thanos_app_mem" {
description = "Thanos app memory limit"
description = "Thanos sidecar memory limit"
default = "1Gi"
}

variable "thanos_compactor_app_mem" {
variable "thanos_querier_mem" {
description = "Thanos querier memory limit"
default = "1Gi"
}

variable "thanos_compactor_mem" {
description = "Thanos compactor memory limit"
default = "1Gi"
}

variable "thanos_store_mem" {
description = "Thanos store gateway memory limit"
default = "1Gi"
}

variable "thanos_app_cpu" {
description = "Thanos app cpu request"
default = "100m"
Expand Down
8 changes: 5 additions & 3 deletions documentation/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,11 @@ All are running as single replica deployments.
The default thanos version is hardcoded in the kubernetes variables.tf. It can be overridden for a cluster by adding thanos_version to the env.tfvars.json file.

There are several other variables that can be changed depending on env requirements.
- thanos_app_mem - app memory limit (default 1G)
- thanos_app_cpu - app memory requests (default 100m)
- thanos_compactor_app_mem - app memory limit for the thanos compactor (default 1G)
- thanos_app_mem - sidecar memory limit (default 1G)
- thanos_app_cpu - thanos cpu requests (default 100m)
- thanos_querier_mem - app memory limit for the thanos querier (default 1G)
- thanos_compactor_mem - app memory limit for the thanos compactor (default 1G)
- thanos_store_mem - app memory limit for the thanos store gateway (default 1G)
- thanos_retention_raw - Thanos retention period for raw samples (default 30d)
- thanos_retention_5m - Thanos retention period for 5m samples (default 60d)
- thanos_retention_1h - Thanos retention period for 1h samples (default 90d)
Expand Down