Skip to content

Commit

Permalink
docs: [refinery] Add MaxConnectionAge default to values.yaml (#254)
Browse files Browse the repository at this point in the history
<!--
Thank you for contributing to the project! πŸ’œ
Please see our [OSS process
document](https://github.com/honeycombio/home/blob/main/honeycomb-oss-lifecycle-and-practices.md#)
to get an idea of how we operate.
-->

## Which problem is this PR solving?

The MaxConnectionAge default is 0 which leads to load balancers failing
to re-balance GRPC connections to Refinery. This means that one node in
a multi-node deployment receives all of the traffic and expends more
resources than it should redirecting all the traffic while the other
nodes sit idle.

This PR doesn't change the default but exposes it as a configuration
that can be set so operators can be in a better position to fix the
problem if they run into it.

## Short description of the changes

Adds some new  

## How to verify that this has the expected result

You can use the `helm template` command to see that the config is
present. When it comes to actually verifying the connections remain
open, I haven't cracked that nut yet.
  • Loading branch information
mterhar authored Jun 26, 2023
1 parent eeb0ade commit 3c8e68b
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions charts/refinery/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,19 @@ config:
# By default that setting is 2GB, and this is set to 85% of that limit
# 2 * 1024 * 1024 * 1024 * 0.80 = 1,717,986,918
MaxAlloc: 1717986918

# Reflects: https://pkg.go.dev/google.golang.org/grpc/keepalive#ServerParameters
GRPCServerParameters:

# MaxConnectionAge is a duration for the maximum amount of time a
# connection may exist before it will be closed by sending a GoAway. A
# random jitter of +/-10% will be added to MaxConnectionAge to spread out
# connection storms.
# 0s sets duration to infinity which is the default:
# Recommended to set it to "3m" if there are multiple replicas
# https://github.com/grpc/grpc-go/blob/60a3a7e969c401ca16dbcd0108ad544fb35aa61c/internal/transport/http2_server.go#L220-L222
# Not eligible for live reload.
MaxConnectionAge: "0s"

# Controls the parameters of the stress relief system. There is a metric called
# stress_level that is emitted as part of refinery metrics. It is a measure of
Expand Down

0 comments on commit 3c8e68b

Please sign in to comment.