Switch summaries to histograms?

[Aracki]

Is there a plan to start using histograms over summaries for RPC duration metrics?

Even though there are advantages of summaries, I guess it would be big improvement if we can aggregate percentiles over time (especially for checking specific percentiles of specific methods).

[holiman]

What we are using is ResettingSample

 // ResettingSample converts an ordinary sample into one that resets whenever its
// snapshot is retrieved. This will break for multi-monitor systems, but when only
// a single metric is being pushed out, this ensure that low-frequency events don't
// skew th charts indefinitely.

But we are actually using it to back a histogram:

	h := fmt.Sprintf("%s/%s/%s", serveTimeHistName, method, note)
	sampler := func() metrics.Sample {
		return metrics.ResettingSample(
			metrics.NewExpDecaySample(1028, 0.015),
		)
	}
	metrics.GetOrRegisterHistogramLazy(h, nil, sampler).Update(elapsed.Microseconds())
}

So we are aleady using histograms.

Could you be more specific as to what you would like to be different?

[Aracki]

It's still Prometheus Summary and not Histogram (https://prometheus.io/docs/practices/histograms/).

We'd like geth start using Histograms like we have for Lighthouse for example.

All the beacon_block_processing_.*_bucket metrics have le label which enable us to use histogram_quantile Prometheus function to calculate percentiles on Server (Prometheus) side.

[holiman]

Sorry, even after skimming through the prometheus docs, I still don't fully know exactly what the difference is, over the wire. If you have an example of either where we already have it, or how the over-the-wire message would look, then we can figure out what the corresponding meter is.

[Aracki]

Actually in Geth I don't see any Histograms used. All things in code called Histograms are actually Summaries. That should be changed which will cause confusion in future.

If we switch to histograms, for each method we would have something like:

rpc_duration{method="eth_getBalance", le="1"}
rpc_duration{method="eth_getBalance", le="3"}
rpc_duration{method="eth_getBalance", le="10"}
rpc_duration{method="eth_getBalance", le="50"}
rpc_duration{method="eth_getBalance", le="100"}
rpc_duration{method="eth_getBalance", le="1000"}
...

That way, observations are very cheap on Geth side as it only needs to increment counters. For each rpc request which took less than <1ns (le=less equal), it will just increment counter for bucket le=1 and so on.

Then on server side (Prometheus), we could calculate P95 with:

histogram_quantile(0.95, sum(rate(rpc_duration{method="eth_getBalance"}[5m]) >= 0) by (le))

That comes with a lot of advantages and flexibility on server side as explained. Another comparison table taken from https://www.timescale.com/blog/four-types-prometheus-metrics-to-collect/ can be useful to easily understand the differences. Histograms are preferred way over Summaries.

[fjl]

@Aracki is right, the go-ethereum metrics package histograms are a bit weird and do not get exported in the right way to make them usable as histograms within Prometheus. Instead, we do the aggregation on the client side and only export the p99, p999, p9999 properties as plain values. I stumbled on this before but couldn't figure out how to fix it. It's not easy to fix because the structure of the metrics library isn't set up correctly for this.