[8.x] [Security Solution] Reduce the _review rule upgrade endpoint response size (#211045) #212921
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… size (elastic#211045) **Resolves: elastic#208361 **Resolves: elastic#210544 ## Summary This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances. Memory consumption testing results provided in elastic#211045 (comment). ## Details This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging. The changes are described by the following bullets - The most significant change is the addition of pagination to the `upgrade/_review` endpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint. - New backend methods, such as `ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates. - The `stats` data structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests asserting `Update All` disabled when rules can't be updated. - All changes to the endpoints are backward-compatible. All previously required returned structures still present in response. All newly added structures are optional. - Upgradeable rule tags are now returned from the prebuilt rule status endpoint. - The frontend logic has been updated to move sorting and filtering of prebuilt rules from the client side to the server side. - The `upgrade/_perform` endpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage. - A dry run option has been added to the upgrade perform endpoint. This is needed for the "Update all" rules scenario to determine if any rules contain conflicts and display a confirmation modal to the user. - An option to skip conflicting rules has been added to the upgrade endpoint when called with the `ALL_RULES` mode. - The `install/_review` endpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation. --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co> (cherry picked from commit c4a016e)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Async chunks
Page load bundle
cc @xcrzx |
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this pull request
Mar 4, 2025
…sponse size (elastic#211045) (elastic#212921) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)](elastic#211045) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dmitrii Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-03-03T14:03:07Z","message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com/elastic/pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","performance","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security Solution] Reduce the _review rule upgrade endpoint response size","number":211045,"url":"https://github.com/elastic/kibana/pull/211045","mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com/elastic/pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/211045","number":211045,"mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com/elastic/pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dmitrii Shevchenko <dmitrii.shevchenko@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.x:Questions ?
Please refer to the Backport tool documentation