Skip to content

chore(issues): Setup experiemental N+1 API Call detector #89633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

leeandher
Copy link
Member

@leeandher leeandher commented Apr 15, 2025

For context, I'm auditing the current performance issue detectors and will likely be making some changes to them. Risky changes like touching fingerprinting should probably run silently beforehand to ensure we catch false positives or fingerprint explosion (see guidance).

In this PR, I'm taking the contents of #89362 and creating a silent detector and group type. Since I'm going through a bunch of other detectors as well I added an experiments/ directory to house these duped files.

Going forward, my workflow will probably be the following:

  1. Draft a PR with the detector change & test changes (e.g. fix(perf-issues): Use parameterized URL for similarities on N+1 API Calls #89362) to make it easier to review
  2. Copy the detector and test files into experiments/ in a new PR (e.g. this one)
  3. Add the options via sentry createissueflag (docs) to run in prod
  4. If needed, follow up with a fix/change to the experimental detector. Repeat as needed
  5. When happy with the output, apply changes to the Step 1 PR and delete experiment files.

The reason I'm being so formal about this all is because I'll be parallelizing these experiments so I gotta keep this straight for myself as well 👍

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Apr 15, 2025
@leeandher leeandher marked this pull request as ready for review April 15, 2025 17:53
@leeandher leeandher requested a review from a team as a code owner April 15, 2025 17:53
Copy link

codecov bot commented Apr 15, 2025

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
25229 3 25226 300
View the top 3 failed test(s) by shortest run time
tests.sentry.utils.performance_issues.experiments.test_n_plus_one_api_calls_detector.NPlusOneAPICallsDetectorTest::test_does_detect_problem_with_unparameterized_urls
Stack Traces | 0.153s run time
#x1B[1m#x1B[.../performance_issues/experiments/test_n_plus_one_api_calls_detector.py#x1B[0m:192: in test_does_detect_problem_with_unparameterized_urls
    [problem] = self.find_problems(event)
#x1B[1m#x1B[31mE   ValueError: not enough values to unpack (expected 1, got 0)#x1B[0m
tests.sentry.utils.performance_issues.experiments.test_n_plus_one_api_calls_detector.NPlusOneAPICallsDetectorTest::test_does_detect_problem_with_parameterized_urls
Stack Traces | 0.184s run time
#x1B[1m#x1B[.../performance_issues/experiments/test_n_plus_one_api_calls_detector.py#x1B[0m:199: in test_does_detect_problem_with_parameterized_urls
    assert problem.desc == "/clients/*/info/*/"
#x1B[1m#x1B[31mE   AssertionError: assert 'GET /clients/' == '/clients/*/info/*/'#x1B[0m
#x1B[1m#x1B[31mE     #x1B[0m
#x1B[1m#x1B[31mE     - /clients/*/info/*/#x1B[0m
#x1B[1m#x1B[31mE     + GET /clients/#x1B[0m
tests.sentry.utils.performance_issues.experiments.test_n_plus_one_api_calls_detector.NPlusOneAPICallsDetectorTest::test_detects_problems_with_many_concurrent_calls_to_same_url
Stack Traces | 0.235s run time
#x1B[1m#x1B[.../performance_issues/experiments/test_n_plus_one_api_calls_detector.py#x1B[0m:74: in test_detects_problems_with_many_concurrent_calls_to_same_url
    assert self.find_problems(event) == [
#x1B[1m#x1B[31mE   assert [PerformanceP...rtant=True)])] == [PerformanceP...e_display=[])]#x1B[0m
#x1B[1m#x1B[31mE     #x1B[0m
#x1B[1m#x1B[31mE     At index 0 diff: PerformanceProblem(fingerprint='1-1010-d750ce46bb1b13dd5780aac48098d5e20eea682c', op='http.client', desc='GET .../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A', type=<class 'sentry.issues.grouptype.PerformanceNPlusOneAPICallsGroupType'>, parent_span_ids=['a0c39078d1570b00'], cause_span_ids=[], offender_span_ids=['ba198ace55bdb20f', '8a20c71faa0fb6a7', '9269c825d935b33a', '9ea82f759505e0f3', '8c55019639e94ab3', '9b86746e9cc7fbf0', '806aa31fe1874495', 'bf409b62d9c30197', '896ac7d28addb37f', '9c859aeaf6bfaea9', '950d8f569bbe3d9e', 'b19a2811b457e87a', 'b566d4ce5b46d4f0', 'b33e9da4441a4800', '8b68818410aa45d8', '8ac4e73b53fc2077', '9fe4a1aff019e39e', 'b29cd0c0cd85ae85', 'b3ff0062caa3ea51', 'a3fde2e38a66cc2c', 'b78802cd80762f57', '9e2ea4d33b1c1bc6', 'bb827dc7a11085f4', 'a34089b08b6d0646', '950801c0d7576650'], evidence_data={'op': 'http.client', 'cause_span_ids': [], 'parent_span_ids': ['a0c39078d1570b00'], 'offender_span_ids': ['ba198ace55bdb20f', '8a20c71faa0fb6a7', '9269c825d935b33a', '9ea82f759505e0f3', '8c55019639e94ab3', '9b86746e9cc7fbf0', '806aa31fe1874495', 'bf409b62d9c30197', '896ac7d28addb37f', '9c859aeaf6bfaea9', '950d8f569bbe3d9e', 'b19a2811b457e87a', 'b566d4ce5b46d4f0', 'b33e9da4441a4800', '8b68818410aa45d8', '8ac4e73b53fc2077', '9fe4a1aff019e39e', 'b29cd0c0cd85ae85', 'b3ff0062caa3ea51', 'a3fde2e38a66cc2c', 'b78802cd80762f57', '9e2ea4d33b1c1bc6', 'bb827dc7a11085f4', 'a34089b08b6d0646', '950801c0d7576650'], 'transaction_name': '/organizations/:orgId/issues/', 'num_repeating_spans': '25', 'repeating_spans': '.../organizations/sentry/events/', 'repeating_spans_compact': 'GET .../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A3357551786%20%21replayId%3A%22%22&statsPeriod=14d', 'parameters': ['{field: replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count()}', '{per_page: 50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50}', '{query: issue.id:3357551786 !replayId:"",issue.id:3653952005 !replayId:"",issue.id:3632201767 !replayId:"",issue.id:3615922758 !replayId:"",issue.id:1816441087 !replayId:"",issue.id:2735845732 !replayId:"",issue.id:2021019829 !replayId:"",issue.id:3708674711 !replayId:"",issue.id:3635082165 !replayId:"",issue.id:1888630303 !replayId:"",issue.id:2615251187 !replayId:"",issue.id:3476760148 !replayId:"",issue.id:3254685754 !replayId:"",issue.id:2364324326 !replayId:"",issue.id:3677909695 !replayId:"",issue.id:1816441096 !replayId:"",issue.id:3708657138 !replayId:"",issue.id:3146776098 !replayId:"",issue.id:3542232742 !replayId:"",issue.id:2636653063 !replayId:"",issue.id:2615489334 !replayId:"",issue.id:3696257335 !replayId:"",issue.id:3696257277 !replayId:"",issue.id:3693916318 !replayId:"",issue.id:3696257055 !replayId:""}', '{statsPeriod: 14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d}']}, evidence_display=[IssueEvidence(name='Offending Spans', value='http.client - GET .../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A', important=True)]) != PerformanceProblem(fingerprint='1-1010-d750ce46bb1b13dd5780aac48098d5e20eea682c', op='http.client', desc='GET .../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A', type=<class 'sentry.issues.grouptype.PerformanceNPlusOneAPICallsExperimentalGroupType'>, parent_span_ids=['a0c39078d1570b00'], cause_span_ids=[], offender_span_ids=['ba198ace55bdb20f', '8a20c71faa0fb6a7', '9269c825d935b33a', '9ea82f759505e0f3', '8c55019639e94ab3', '9b86746e9cc7fbf0', '806aa31fe1874495', 'bf409b62d9c30197', '896ac7d28addb37f', '9c859aeaf6bfaea9', '950d8f569bbe3d9e', 'b19a2811b457e87a', 'b566d4ce5b46d4f0', 'b33e9da4441a4800', '8b68818410aa45d8', '8ac4e73b53fc2077', '9fe4a1aff019e39e', 'b29cd0c0cd85ae85', 'b3ff0062caa3ea51', 'a3fde2e38a66cc2c', 'b78802cd80762f57', '9e2ea4d33b1c1bc6', 'bb827dc7a11085f4', 'a34089b08b6d0646', '950801c0d7576650'], evidence_data={'op': 'http.client', 'parent_span_ids': ['a0c39078d1570b00'], 'cause_span_ids': [], 'offender_span_ids': ['ba198ace55bdb20f', '8a20c71faa0fb6a7', '9269c825d935b33a', '9ea82f759505e0f3', '8c55019639e94ab3', '9b86746e9cc7fbf0', '806aa31fe1874495', 'bf409b62d9c30197', '896ac7d28addb37f', '9c859aeaf6bfaea9', '950d8f569bbe3d9e', 'b19a2811b457e87a', 'b566d4ce5b46d4f0', 'b33e9da4441a4800', '8b68818410aa45d8', '8ac4e73b53fc2077', '9fe4a1aff019e39e', 'b29cd0c0cd85ae85', 'b3ff0062caa3ea51', 'a3fde2e38a66cc2c', 'b78802cd80762f57', '9e2ea4d33b1c1bc6', 'bb827dc7a11085f4', 'a34089b08b6d0646', '950801c0d7576650']}, evidence_display=[])#x1B[0m
#x1B[1m#x1B[31mE     #x1B[0m
#x1B[1m#x1B[31mE     Full diff:#x1B[0m
#x1B[1m#x1B[31mE       [#x1B[0m
#x1B[1m#x1B[31mE           PerformanceProblem(#x1B[0m
#x1B[1m#x1B[31mE               fingerprint='1-1010-d750ce46bb1b13dd5780aac48098d5e20eea682c',#x1B[0m
#x1B[1m#x1B[31mE               op='http.client',#x1B[0m
#x1B[1m#x1B[31mE               desc='GET '#x1B[0m
#x1B[1m#x1B[31mE                    '.../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A',#x1B[0m
#x1B[1m#x1B[31mE     -         type=<class 'sentry.issues.grouptype.PerformanceNPlusOneAPICallsExperimentalGroupType'>,#x1B[0m
#x1B[1m#x1B[31mE     ?                                                                         ------------#x1B[0m
#x1B[1m#x1B[31mE     +         type=<class 'sentry.issues.grouptype.PerformanceNPlusOneAPICallsGroupType'>,#x1B[0m
#x1B[1m#x1B[31mE               parent_span_ids=[#x1B[0m
#x1B[1m#x1B[31mE                                   'a0c39078d1570b00',#x1B[0m
#x1B[1m#x1B[31mE                               ],#x1B[0m
#x1B[1m#x1B[31mE               cause_span_ids=[],#x1B[0m
#x1B[1m#x1B[31mE               offender_span_ids=[#x1B[0m
#x1B[1m#x1B[31mE                                     'ba198ace55bdb20f',#x1B[0m
#x1B[1m#x1B[31mE                                     '8a20c71faa0fb6a7',#x1B[0m
#x1B[1m#x1B[31mE                                     '9269c825d935b33a',#x1B[0m
#x1B[1m#x1B[31mE                                     '9ea82f759505e0f3',#x1B[0m
#x1B[1m#x1B[31mE                                     '8c55019639e94ab3',#x1B[0m
#x1B[1m#x1B[31mE                                     '9b86746e9cc7fbf0',#x1B[0m
#x1B[1m#x1B[31mE                                     '806aa31fe1874495',#x1B[0m
#x1B[1m#x1B[31mE                                     'bf409b62d9c30197',#x1B[0m
#x1B[1m#x1B[31mE                                     '896ac7d28addb37f',#x1B[0m
#x1B[1m#x1B[31mE                                     '9c859aeaf6bfaea9',#x1B[0m
#x1B[1m#x1B[31mE                                     '950d8f569bbe3d9e',#x1B[0m
#x1B[1m#x1B[31mE                                     'b19a2811b457e87a',#x1B[0m
#x1B[1m#x1B[31mE                                     'b566d4ce5b46d4f0',#x1B[0m
#x1B[1m#x1B[31mE                                     'b33e9da4441a4800',#x1B[0m
#x1B[1m#x1B[31mE                                     '8b68818410aa45d8',#x1B[0m
#x1B[1m#x1B[31mE                                     '8ac4e73b53fc2077',#x1B[0m
#x1B[1m#x1B[31mE                                     '9fe4a1aff019e39e',#x1B[0m
#x1B[1m#x1B[31mE                                     'b29cd0c0cd85ae85',#x1B[0m
#x1B[1m#x1B[31mE                                     'b3ff0062caa3ea51',#x1B[0m
#x1B[1m#x1B[31mE                                     'a3fde2e38a66cc2c',#x1B[0m
#x1B[1m#x1B[31mE                                     'b78802cd80762f57',#x1B[0m
#x1B[1m#x1B[31mE                                     '9e2ea4d33b1c1bc6',#x1B[0m
#x1B[1m#x1B[31mE                                     'bb827dc7a11085f4',#x1B[0m
#x1B[1m#x1B[31mE                                     'a34089b08b6d0646',#x1B[0m
#x1B[1m#x1B[31mE                                     '950801c0d7576650',#x1B[0m
#x1B[1m#x1B[31mE                                 ],#x1B[0m
#x1B[1m#x1B[31mE               evidence_data={#x1B[0m
#x1B[1m#x1B[31mE                                 'cause_span_ids': [],#x1B[0m
#x1B[1m#x1B[31mE     +                           'num_repeating_spans': '25',#x1B[0m
#x1B[1m#x1B[31mE                                 'offender_span_ids': [#x1B[0m
#x1B[1m#x1B[31mE                                     'ba198ace55bdb20f',#x1B[0m
#x1B[1m#x1B[31mE                                     '8a20c71faa0fb6a7',#x1B[0m
#x1B[1m#x1B[31mE                                     '9269c825d935b33a',#x1B[0m
#x1B[1m#x1B[31mE                                     '9ea82f759505e0f3',#x1B[0m
#x1B[1m#x1B[31mE                                     '8c55019639e94ab3',#x1B[0m
#x1B[1m#x1B[31mE                                     '9b86746e9cc7fbf0',#x1B[0m
#x1B[1m#x1B[31mE                                     '806aa31fe1874495',#x1B[0m
#x1B[1m#x1B[31mE                                     'bf409b62d9c30197',#x1B[0m
#x1B[1m#x1B[31mE                                     '896ac7d28addb37f',#x1B[0m
#x1B[1m#x1B[31mE                                     '9c859aeaf6bfaea9',#x1B[0m
#x1B[1m#x1B[31mE                                     '950d8f569bbe3d9e',#x1B[0m
#x1B[1m#x1B[31mE                                     'b19a2811b457e87a',#x1B[0m
#x1B[1m#x1B[31mE                                     'b566d4ce5b46d4f0',#x1B[0m
#x1B[1m#x1B[31mE                                     'b33e9da4441a4800',#x1B[0m
#x1B[1m#x1B[31mE                                     '8b68818410aa45d8',#x1B[0m
#x1B[1m#x1B[31mE                                     '8ac4e73b53fc2077',#x1B[0m
#x1B[1m#x1B[31mE                                     '9fe4a1aff019e39e',#x1B[0m
#x1B[1m#x1B[31mE                                     'b29cd0c0cd85ae85',#x1B[0m
#x1B[1m#x1B[31mE                                     'b3ff0062caa3ea51',#x1B[0m
#x1B[1m#x1B[31mE                                     'a3fde2e38a66cc2c',#x1B[0m
#x1B[1m#x1B[31mE                                     'b78802cd80762f57',#x1B[0m
#x1B[1m#x1B[31mE                                     '9e2ea4d33b1c1bc6',#x1B[0m
#x1B[1m#x1B[31mE                                     'bb827dc7a11085f4',#x1B[0m
#x1B[1m#x1B[31mE                                     'a34089b08b6d0646',#x1B[0m
#x1B[1m#x1B[31mE                                     '950801c0d7576650',#x1B[0m
#x1B[1m#x1B[31mE                                 ],#x1B[0m
#x1B[1m#x1B[31mE                                 'op': 'http.client',#x1B[0m
#x1B[1m#x1B[31mE     +                           'parameters': [#x1B[0m
#x1B[1m#x1B[31mE     +                               '{field: '#x1B[0m
#x1B[1m#x1B[31mE     +                               'replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count(),replayId,count()}',#x1B[0m
#x1B[1m#x1B[31mE     +                               '{per_page: '#x1B[0m
#x1B[1m#x1B[31mE     +                               '50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50,50}',#x1B[0m
#x1B[1m#x1B[31mE     +                               '{query: issue.id:3357551786 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3653952005 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3632201767 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3615922758 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:1816441087 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:2735845732 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:2021019829 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3708674711 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3635082165 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:1888630303 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:2615251187 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3476760148 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3254685754 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:2364324326 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3677909695 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:1816441096 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3708657138 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3146776098 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3542232742 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:2636653063 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:2615489334 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3696257335 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3696257277 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3693916318 '#x1B[0m
#x1B[1m#x1B[31mE     +                               '!replayId:"",issue.id:3696257055 !replayId:""}',#x1B[0m
#x1B[1m#x1B[31mE     +                               '{statsPeriod: '#x1B[0m
#x1B[1m#x1B[31mE     +                               '14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d,14d}',#x1B[0m
#x1B[1m#x1B[31mE     +                           ],#x1B[0m
#x1B[1m#x1B[31mE                                 'parent_span_ids': [#x1B[0m
#x1B[1m#x1B[31mE                                     'a0c39078d1570b00',#x1B[0m
#x1B[1m#x1B[31mE                                 ],#x1B[0m
#x1B[1m#x1B[31mE     +                           'repeating_spans': '.../organizations/sentry/events/',#x1B[0m
#x1B[1m#x1B[31mE     +                           'repeating_spans_compact': 'GET '#x1B[0m
#x1B[1m#x1B[31mE     +                           '.../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A3357551786%20%21replayId%3A%22%22&statsPeriod=14d',#x1B[0m
#x1B[1m#x1B[31mE     +                           'transaction_name': '/organizations/:orgId/issues/',#x1B[0m
#x1B[1m#x1B[31mE                             },#x1B[0m
#x1B[1m#x1B[31mE     -         evidence_display=[],#x1B[0m
#x1B[1m#x1B[31mE     ?                           --#x1B[0m
#x1B[1m#x1B[31mE     +         evidence_display=[#x1B[0m
#x1B[1m#x1B[31mE     +                              IssueEvidence(#x1B[0m
#x1B[1m#x1B[31mE     +                                  name='Offending Spans',#x1B[0m
#x1B[1m#x1B[31mE     +                                  value='http.client - GET '#x1B[0m
#x1B[1m#x1B[31mE     +                                        '.../organizations/sentry/events/?field=replayId&field=count%28%29&per_page=50&query=issue.id%3A',#x1B[0m
#x1B[1m#x1B[31mE     +                                  important=True,#x1B[0m
#x1B[1m#x1B[31mE     +                              ),#x1B[0m
#x1B[1m#x1B[31mE     +                          ],#x1B[0m
#x1B[1m#x1B[31mE           ),#x1B[0m
#x1B[1m#x1B[31mE       ]#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@leeandher leeandher marked this pull request as draft April 16, 2025 19:47
@leeandher leeandher changed the title chore(issues): Silently run detector change to monitor risks chore(issues): Setup experiemental N+1 API Call detector Apr 17, 2025
@leeandher leeandher closed this Apr 22, 2025
@leeandher leeandher deleted the leander/shadow-n-plus-one branch April 23, 2025 15:01
@github-actions github-actions bot locked and limited conversation to collaborators May 9, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant