chore(ACI): Add 1:1 metrics for dual processing #91840
Conversation
github-actions
bot
added
the
Scope: Backend
| if features.has("organizations:workflow-engine-metric-alert-dual-processing-logs"): | ||
| # in order to check if workflow engine is firing 1:1 with the old system, we must only count once rather than each action | ||
| metrics.incr( | ||
| "workflow_engine.process_workflows.fired_actions", | ||
| tags={ | ||
| "detector_type": detector.type, | ||
| "organization_id": detector.project.organization_id, | ||
| }, | ||
| ) |
There was a problem hiding this comment.
thoughts on pulling out the org_id and setting the tag on every metric? something like:
organization_id = None
if features.has("organizations:workflow-engine-metric-alert-dual-processing-logs"):
organization_id = detector.project.organization_id
metrics.incr(
"workflow_engine.process_workflows.triggered_actions",
amount=len(actions),
tags={"detector_type": detector.type, "organization_id": organization_id},
)There was a problem hiding this comment.
I am making this a separate metric because I don't want to have the amount being set to the number of actions as that's not how we track it in the existing system.
There was a problem hiding this comment.
sorry for the confusion, the action one is just meant to illustrate an existing metric decorated with the organization id.
mostly just thinking that we could just add the organization id to all the existing metrics if we pull out the check for the feature and assign the org id to a separate variable. (very similar to the comment that dan had left)
There was a problem hiding this comment.
🤔 all the other metrics in this function pass an amount that's possibly larger than 1, and I only ever want to count a metric issue firing or resolving once otherwise the metrics might not line up
|
Follow up to #91840 The `organization_id` tag is not showing up in Datadog despite manually allowing it for these metrics (I suspect something higher up is blocking it) so this PR removes that tag and adds a metric behind the logging flag for metric alerts firing so that we can see if they are firing at the same rate as in workflow engine.
In order for us to determine whether we are firing metric alerts at the same rate in workflow engine, add org id to the existing metric alert fire/resolve metrics and add a new metric to process workflows that doesn't count the number of actions being fired. We'll have to be careful to not add too many orgs to this flag as that'd result in a high cardinality of tags in Datadog.
Follow up to #91840 The `organization_id` tag is not showing up in Datadog despite manually allowing it for these metrics (I suspect something higher up is blocking it) so this PR removes that tag and adds a metric behind the logging flag for metric alerts firing so that we can see if they are firing at the same rate as in workflow engine.
In order for us to determine whether we are firing metric alerts at the same rate in workflow engine, add org id to the existing metric alert fire/resolve metrics and add a new metric to process workflows that doesn't count the number of actions being fired. We'll have to be careful to not add too many orgs to this flag as that'd result in a high cardinality of tags in Datadog.