Skip to content

Fix "metric was collected before" error #433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

colinleroy
Copy link

Hello,

Since one or two weeks, our stackdriver_exporter errors out with logs similar to:

collected metric "stackdriver_network_security_policy_networksecurity_googleapis_com_https_request_count" { label:{name:"backend_target_name"  value:"****-https-lb-spectrum-back"}  label:{name:"blocked"  value:"true"}  label:{name:"location"  value:"global"}  label:{name:"policy_name"  value:"preconfigured-rules-policy"}  label:{name:"project_id"  value:"****-production"}  label:{name:"unit"  value:"1"}  gauge:{value:0}  timestamp_ms:1746012857310} was collected before with the same name and label values

After investigation, it comes from the fact that some of these metrics are returned twice by Stackdriver.

This PR adds a filtering to reportTimeSeriesMetrics() in order to avoid sending duplicate time series to the Prometheus Collector.

I hope this helps and I hope it's good enough, I'm not very well-versed in Go.

Thanks!

Signed-Off-By: Colin Leroy-Mira <colin@colino.net>
The networksecurity.googleapis.com/https/request_count
sometimes returns duplicate series. Filter them out to
avoid errors.

Signed-Off-By: Colin Leroy-Mira <colin@colino.net>
@aarozhkov
Copy link

@colinleroy
Stack on this too.
Try to use binary builded from your PR.

Lost other valid metrics.
Example stackdriver_pubsub_subscription_pubsub_googleapis_com_subscription_ack_latencies_bucket
I have a lot of subscribtion_ids.
Left with only on subscription metric.

@colinleroy
Copy link
Author

Were they metrics with at least one unique key/value label ?

Copy link
Contributor

@kgeckhart kgeckhart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I have some concerns regarding the performance of the implementation as reflect deep equals in a double nested loop like this is going to be very expensive.

We have had to do something similar to this in the AWS CloudWatch exporter (https://github.com/kgeckhart/yet-another-cloudwatch-exporter/blob/master/pkg/promutil/migrate.go#L292-L316) but that implementation is much easier because it can hook in to the intermediate type used which has all the labels in a form that can be directly hashed.

The stackdriver exporter doesn't have a clean injection point or a clean representation. The easiest place is the metrics channel but at that point the data is already packaged as a prometheus.Metric where the labels are a []*LabelPair and must be copied to a new map[string]string to fingerprint. I don't have a quick answer ATM just thinking out loud.

Github issues that's likely related, #166

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants