loadbalancer: Add ConnectTracker and make HealthIndicator extend it

[bryce-anderson]

Motivation:

We can't rely on the request path to pass in connection failures because if we can't get a connection we can never add the observers. That means we need the Host implementation to pass it in directly.

Modifications:

Add a new interface, ConnectTracker, that mirrors RequestTracker in purpose but with the intent of monitoring session establishment related events and have HealthIndicator extend it.

Result:

We should be able to track connection failures now and factor those into the HealthTracker implementations.

Note

See #2817 which is substantially the same but doesn't introduce a new interface.

[tkountis]

I like the separate interface. A few things for your consideration

1. Consider renaming the RequestTracker callbacks to include the request in their name for better readability for use-cases like the health-indicator that implements both.
2. onConnectionSuccess is noop, but I still think this needs to be transactional. If we observed a connection failure we should also track the connection success to negate that effect. In my minds eye, the data path is a separate concern.
3. Consider influencing the score for the connection path too. IMO connection latency is a health indicator too.
4. If you do the above, consider separating the two scoring mechanisms (ie. latency-trackers), in the past we have noted better behavior when the connection path had different decay configuration (possibly even penalties).

[bryce-anderson]

I like the separate interface. A few things for your consideration

1. Consider renaming the RequestTracker callbacks to include the request in their name for better readability for use-cases like the health-indicator that implements both.

👍 done.

2. onConnectionSuccess is noop, but I still think this needs to be transactional. If we observed a connection failure we should also track the connection success to negate that effect. In my minds eye, the data path is a separate concern.
3. Consider influencing the score for the connection path too. IMO connection latency is a health indicator too.
4. If you do the above, consider separating the two scoring mechanisms (ie. latency-trackers), in the past we have noted better behavior when the connection path had different decay configuration (possibly even penalties).

I think these are worth considering but I'd prefer to leave them for later. The only implementation right now is the XDS style health checks and at least the envoy version is pretty similar to what is is here. That isn't to say we shouldn't try to do better but I'd prefer to deliberately examine that in a separate change whereas here I'd prefer to just wire it through.