feat(aci milestone 3): anomaly detection condition handler #88647

mifu67 · 2025-04-02T22:32:42Z

Create a condition handler for Anomaly Detection detector triggers. This data condition takes in a data packet and passes if the Seer evaluation corresponds to its condition result.

src/sentry/workflow_engine/handlers/condition/anomaly_detection_handler.py

codecov · 2025-04-02T22:59:23Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
25964	1	25963	207

View the top 1 failed test(s) by shortest run time

tests.sentry.incidents.handlers.condition.test_anomaly_detection_handler.TestAnomalyDetectionHandler::test_passes_medium

Stack Traces | 3.95s run time

#x1B[1m#x1B[.../handlers/condition/test_anomaly_detection_handler.py#x1B[0m:103: in test_passes_medium
    assert self.dc.evaluate_value(self.data_packet) == DetectorPriorityLevel.OK
#x1B[1m#x1B[31mE   AssertionError: assert None == <DetectorPriorityLevel.OK: 0>#x1B[0m
#x1B[1m#x1B[31mE    +  where None = <bound method DataCondition.evaluate_value of <DataCondition at 0x7f48d1be56d0: id=12, type=<Condition.ANOMALY_DETECTI...tection'>, condition=None, condition_group=<DataConditionGroup at 0x7f48d1da70b0: id=6, logic_type=<Type.ANY: 'any'>>>>(DataPacket(source_id='24', packet={'subscription_id': '24', 'values': {'value': 1}, 'timestamp': datetime.datetime(2025, 5, 23, 0, 26, 9, 96998, tzinfo=datetime.timezone.utc), 'entity': 'test-entity'}))#x1B[0m
#x1B[1m#x1B[31mE    +    where <bound method DataCondition.evaluate_value of <DataCondition at 0x7f48d1be56d0: id=12, type=<Condition.ANOMALY_DETECTI...tection'>, condition=None, condition_group=<DataConditionGroup at 0x7f48d1da70b0: id=6, logic_type=<Type.ANY: 'any'>>>> = <DataCondition at 0x7f48d1be56d0: id=12, type=<Condition.ANOMALY_DETECTION: 'anomaly_detection'>, condition=None, condition_group=<DataConditionGroup at 0x7f48d1da70b0: id=6, logic_type=<Type.ANY: 'any'>>>.evaluate_value#x1B[0m
#x1B[1m#x1B[31mE    +      where <DataCondition at 0x7f48d1be56d0: id=12, type=<Condition.ANOMALY_DETECTION: 'anomaly_detection'>, condition=None, condition_group=<DataConditionGroup at 0x7f48d1da70b0: id=6, logic_type=<Type.ANY: 'any'>>> = <test_anomaly_detection_handler.TestAnomalyDetectionHandler testMethod=test_passes_medium>.dc#x1B[0m
#x1B[1m#x1B[31mE    +    and   DataPacket(source_id='24', packet={'subscription_id': '24', 'values': {'value': 1}, 'timestamp': datetime.datetime(2025, 5, 23, 0, 26, 9, 96998, tzinfo=datetime.timezone.utc), 'entity': 'test-entity'}) = <test_anomaly_detection_handler.TestAnomalyDetectionHandler testMethod=test_passes_medium>.data_packet#x1B[0m
#x1B[1m#x1B[31mE    +  and   <DetectorPriorityLevel.OK: 0> = DetectorPriorityLevel.OK#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

…dler

mifu67 · 2025-05-20T22:37:19Z

src/sentry/workflow_engine/handlers/condition/anomaly_detection_handler.py

Moved this file to the incidents directory.

mifu67 · 2025-05-20T22:59:20Z

src/sentry/incidents/grouptype.py

@@ -5,6 +5,7 @@
 from typing import Any

 from sentry import features
+from sentry.incidents.handlers.condition import *  # noqa


Imported so that the condition handler gets added to the registry. Maybe there's a better solution?

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

mifu67 · 2025-05-20T23:02:25Z

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

+
+
+# placeholder until we create this in the workflow engine model
+class DetectorError(Exception):


Calling this out as a TODO (need to add it to the workflow engine + build exception handling for it within process_detectors)

Maybe this could live in either sentry/workflow_engine/processors/detector.py or a new utils file in the same dir?

mifu67 · 2025-05-20T23:04:00Z

src/sentry/seer/anomaly_detection/get_anomaly_data.py

+            SEER_ANOMALY_DETECTION_ENDPOINT_URL,
+            json.dumps(detect_anomalies_request).encode("utf-8"),
+        )
+    except (TimeoutError, MaxRetryError):


Everything below this line is error handling (copied from the legacy method)

src/sentry/seer/anomaly_detection/types.py

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

src/sentry/seer/anomaly_detection/types.py

cathteng · 2025-05-20T23:38:00Z

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

+        # covers both None and []
+        if not anomaly_data:
+            # something went wrong during evaluation
+            raise DetectorError("Error during Seer data evaluation process.")


should this return False instead of raising an exception, if you're going to build exception handling for it? what's the outcome of the exception handling?

We want this to raise an error instead of returning False, because returning False indicates that we should set the detector priority level to OK.

I think we actually want to change this condition handler to emit multiple detector priority levels according to the anomaly detection result 🤔

src/sentry/seer/anomaly_detection/get_anomaly_data.py

ceorourke · 2025-05-20T23:28:45Z

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

+        threshold_type = comparison["threshold_type"]
+        confidence = comparison["confidence"]
+
+        subscription: QuerySubscription = QuerySubscription.objects.get(id=update.source_id)


should we have handling around this in case it doesn't exist?

Let's let it fail loudly—I don't think this should happen.

ceorourke · 2025-05-20T23:29:57Z

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

+
+
+# placeholder until we create this in the workflow engine model
+class DetectorError(Exception):


Maybe this could live in either sentry/workflow_engine/processors/detector.py or a new utils file in the same dir?

ceorourke · 2025-05-20T23:36:26Z

src/sentry/incidents/handlers/condition/anomaly_detection_handler.py

+        sensitivity = comparison["sensitivity"]
+        seasonality = comparison["seasonality"]
+        threshold_type = comparison["threshold_type"]
+        confidence = comparison["confidence"]


What else is in comparison? I'm wondering if we could just pass that var to get_anomaly_data_from_seer.

Hmm, that's a good idea—I'd be open to it.

src/sentry/seer/anomaly_detection/get_anomaly_data.py

ceorourke · 2025-05-20T23:59:31Z

src/sentry/seer/anomaly_detection/get_anomaly_data.py

+            },
+        )
+        return None
+    return ts


Can you add tests for this method? Probably can mostly copy/paste from the existing ones, just don't wanna lose the coverage when we delete the old stuff.

…dler

mifu67 · 2025-05-22T21:56:09Z

tests/sentry/incidents/handlers/condition/test_anomaly_detection_handler.py

+                "seasonality": AnomalyDetectionSeasonality.AUTO,
+                "threshold_type": AnomalyDetectionThresholdType.ABOVE_AND_BELOW,
+            },
+            condition_result=DetectorPriorityLevel.HIGH,


@saponifi3d calling out that we want this data condition to emit whatever the result of evaluate_value() is, not a static condition result.

mifu67 · 2025-05-22T22:01:41Z

src/sentry/seer/anomaly_detection/get_anomaly_data.py

+) -> list[TimeSeriesPoint] | None:
+    snuba_query: SnubaQuery = subscription.snuba_query
+    aggregation_value = subscription_update["values"].get("value")
+    source_id = subscription.id


Because we're passing in the query subscription, we don't need to pass source_id and source_type.

mifu67 added 2 commits April 2, 2025 14:24

wip

c1dba54

more progress

14d84dd

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Apr 2, 2025

🛠️ apply pre-commit fixes

30eab83

vercel bot deployed to Preview April 2, 2025 22:39 View deployment

saponifi3d reviewed Apr 2, 2025

View reviewed changes

mifu67 added 3 commits April 7, 2025 15:54

some more work

9054cbb

saving my work

8e20c36

refactor and updates

9d9c7a5

mifu67 requested a review from saponifi3d April 7, 2025 22:55

vercel bot deployed to Preview April 7, 2025 22:59 View deployment

Merge branch 'master' into mifu67/aci/anomaly-detection-condition-han…

a999884

…dler

vercel bot deployed to Preview May 19, 2025 22:33 View deployment

tests + new Seer key

a4eb040

vercel bot deployed to Preview May 20, 2025 22:34 View deployment

mifu67 commented May 20, 2025

View reviewed changes

src/sentry/workflow_engine/handlers/condition/anomaly_detection_handler.py

Copy link

Contributor Author

mifu67 May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this file to the incidents directory.

fixes

4eec4a9

vercel bot deployed to Preview May 20, 2025 22:57 View deployment

remove commented out code

6181e45

vercel bot deployed to Preview May 20, 2025 23:01 View deployment

mifu67 commented May 20, 2025

View reviewed changes

mifu67 requested a review from ceorourke May 20, 2025 23:05

mifu67 marked this pull request as ready for review May 20, 2025 23:05

mifu67 requested review from a team as code owners May 20, 2025 23:05

cathteng reviewed May 20, 2025

View reviewed changes

ceorourke reviewed May 20, 2025

View reviewed changes

mifu67 and others added 4 commits May 21, 2025 16:06

increase email field length on remaining models

7afc678

Merge branch 'master' into mifu67/member-merge/allow-longer-emails-2

aa18e49

Merge branch 'master' into mifu67/aci/anomaly-detection-condition-han…

9b6bcf4

…dler

PR comments + some design changes

a749210

mifu67 requested a review from a team as a code owner May 22, 2025 21:47

Merge branch 'master' into mifu67/aci/anomaly-detection-condition-han…

fec8351

…dler

vercel bot deployed to Preview May 22, 2025 21:53 View deployment

mifu67 commented May 22, 2025

View reviewed changes

mifu67 added 2 commits May 22, 2025 14:59

small fix

7c0cdb5

not sure how that got there

bfbb3a4

getsentry deleted a comment from github-actions bot May 22, 2025

mifu67 removed the request for review from a team May 22, 2025 22:00

vercel bot deployed to Preview May 22, 2025 22:00 View deployment

mifu67 commented May 22, 2025

View reviewed changes

mifu67 requested a review from ceorourke May 22, 2025 22:18

failure tests

1ca68e9

vercel bot deployed to Preview May 23, 2025 00:04 View deployment

seer doesn't support warning alerts yet

f67cbc5

vercel bot deployed to Preview May 23, 2025 00:19 View deployment



		# placeholder until we create this in the workflow engine model
		class DetectorError(Exception):

Uh oh!

feat(aci milestone 3): anomaly detection condition handler #88647

Are you sure you want to change the base?

feat(aci milestone 3): anomaly detection condition handler #88647

Uh oh!

Conversation

mifu67 commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mifu67 commented Apr 2, 2025 •

edited

Loading

codecov bot commented Apr 2, 2025 •

edited

Loading