Use linear buckets in some places

[ndr-ds]

Motivation

We currently use exponential buckets everywhere. As much as they're good in the sense that they generate less buckets, and can be cheaper on Grafana Cloud, etc, they make our buckets super wide, which makes our Prometheus data less accurate.

Proposal

I need to spend some time later looking at the testnet data for these different metrics to adjust the buckets, but for not just changing proxy/server latencies to use linear buckets instead.
I also changed the default starting value to 0.001 as 1 microsecond should be enough for most of what we're measuring, and will take at least one bucket away from metrics using this.

Test Plan

Run a validator locally, see the metrics exported with the new buckets

Release Plan

• Nothing to do / These changes follow the usual release cycle.

[ndr-ds]

• Use linear buckets in some places #3384 👈 (View in Graphite)
• Have one thread per chain on benchmark #3374
• Adjust dashboards to show 1m metrics #3380
• Adjust latency metrics buckets #3379
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[MathieuDutSik]

I do not use the buckets, instead I am using sum and the numbers which gets me more raw data.

With that being said, while a linear scaling might be better for many cases, I think exponential is better for latency. So, I do not like the global change from exponential to linear.

[ndr-ds]

Why is exponential better for latency? It gives us less accurate latency values, as I explained in the PR description

[MathieuDutSik]

Why is exponential better for latency? It gives us less accurate latency values, as I explained in the PR description

I thought the latency was more in an exponential curves, but if you see it differently fine.
But what about the number of buckets? Could we have too much data in the Prometheus.

If it were up to me, I would remove the buckets altogether. I see it mostly as something useful for presentation.

[ndr-ds]

So the issue here is that the wider our buckets, the less accurate our quantiles will be (p50, p90, p99, etc), which is not good, because that's what we'll be looking at in our dashboards to monitor the system. So we need to make our buckets as narrow as possible, without generating too many buckets, which can be expensive when using Grafana Cloud, for example. These were starting at 0.0001 ms and it had a base 3 exponential growth. Now it starts at 1 ms, and grows linearly for both the proxy and server latencies. We're probably not interested in latencies below 1 ms for these, so this is fine. This will generate less buckets than before, and the buckets will be narrower so our data will be more accurate.
So this is a win on every aspect IMHO.

[MathieuDutSik]

I approve because there is less buckets with this system than the preceding.

[ndr-ds]

Merge activity

• Feb 24, 11:56 AM EST: A user started a stack merge that includes this pull request via Graphite.
• Feb 24, 11:56 AM EST: A user merged this pull request with Graphite.