Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Cache upgrade/_review endpoint calculations #208357

Closed
Tracked by #201502
xcrzx opened this issue Jan 27, 2025 · 3 comments · Fixed by #209551
Closed
Tracked by #201502

[Security Solution] Cache upgrade/_review endpoint calculations #208357

xcrzx opened this issue Jan 27, 2025 · 3 comments · Fixed by #209551
Assignees
@xcrzx
Labels
8.18 candidate bug Fixes for quality problems that affect the customer experience Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. performance Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.18.0

Comments

@xcrzx
Copy link
Contributor

xcrzx commented Jan 27, 2025 •
edited by banderror
Loading

Epic: #174168

Summary

This is related to OOM errors investigation (see #incident-1039-multiple-projects-oomkilled-when-installing-updating-fleet-pa in Slack for more details).

Here's a short summary of what happened.
  • This failure is unrelated to Fleet’s package installation. And external memory consumption wasn’t the issue in that case, but heap memory was.
  • There are a couple of /detection_engine/prebuilt_rules/_bootstrap calls in the logs, but since the endpoint responded in less than 1 second, that means the package installation didn’t occur. The endpoint skips installation if the package is already installed, which was the case here.
  • There were two concurrent detection_engine/prebuilt_rules/installation/_perform requests in the logs. These requests are used to install all prebuilt Elastic rules. Normally, browsers don’t send parallel requests like this, and it’s not something that can be easily triggered through the UI. Given there are many other duplicate requests in the proxy logs, I strongly suspect this was a testing environment where two rule installation requests were deliberately sent at the same time from separate browser tabs to test the workflow.
  • Both concurrent _perform requests were handled successfully, even though each took almost one minute to complete.
  • Then, after rules package installation, Kibana frontend sends a detection_engine/prebuilt_rules/installation/_review request to fetch updated information about prebuilt Elastic rules. This request is memory-intensive, because all rules (more than 1,000) are loaded into memory on the server side to calculate their installation status.
  • With two browser tabs open, two concurrent _review requests were sent, both logged with a 499 status (client closed connection). Immediately after, the browser sent two additional _review requests, resulting in Kibana handling four concurrent _review requests at one point.
  • The heavy memory load from handling multiple simultaneous _review requests appears to have caused the OOM. This conclusion can also be confirmed by proxy logs that show many heavy Elasticsearch responses (around 80Mb summed up).

To avoid OOM errors in the future, we need to introduce some form of caching to reduce memory usage for repeated requests to the upgrade/_review endpoint. We can introduce a short-lived in-memory cache to reduce redundant calculations. For example, an object that holds the current calculation status. If two concurrent requests are received, the first one would set the status to "in progress," while the second one would wait for the first to complete and reuse its result without making expensive DB calls. The cache could automatically clean itself up after a few seconds not being used.

This is a short-term solution. A long-term solution is to add pagination to the _review endpoint: #208361
@xcrzx xcrzx added 8.18 candidate bug Fixes for quality problems that affect the customer experience Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. performance Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team triage_needed labels Jan 27, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

This was referenced Jan 27, 2025
Closed
Open
@xcrzx xcrzx mentioned this issue Feb 4, 2025
Merged
xcrzx added a commit that referenced this issue Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
c5557f3 
…o prebuilt rule routes (#209551)

**Resolves: #208357
**Resolves: #208355

## Summary  

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
14d96cb 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
096b0fe 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Feb 11, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
21e42b1 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)
xcrzx added a commit to xcrzx/kibana that referenced this issue Feb 12, 2025
@xcrzx
[Security Solution] Added concurrency limits and request throttling t…
a6c4954 
…o prebuilt rule routes (elastic#209551)

**Resolves: elastic#208357
**Resolves: elastic#208355

## Summary

To prevent possible OOM errors, we need to limit concurrent requests to
prebuilt rule routes (see attached tickets for more details).

- `installation/_perform` and `upgrade/_perform` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

- `prebuilt_rules/_bootstrap` endpoint
- Install prebuilt rules and endpoint packages sequentially instead of
in parallel to prevent from having them both downloaded into memory
simultaneously.
- Added a 30-minute socket timeout to prevent the proxy from closing the
connection while rule installation is in progress.
- Introduced a `throttleRequests` wrapper, ensuring the endpoint handler
is called only once when multiple concurrent requests are received.
- The first request triggers the handler, while subsequent requests wait
for the first one to complete and reuse its result.
- This prevents costly prebuilt rule package installation from running
in parallel.
- Reusing the response ensures the frontend correctly invalidates cached
prebuilt rule queries. Since concurrent frontend requests should receive
the same installed package information, responding with 421 and using
the retry logic as in cases above is not an option here because the
second request would receive a package installation skipped response
leading to no cache invalidation.

- `installation/_review` and `upgrade/_review` endpoints
- Concurrency is limited to one parallel call. If another call is made
simultaneously, the server responds with 429 Too Many Requests.
- On the front end, all rule install and upgrade operations are retried
in case of a 429 response. This ensures proper handling when a user
clicks multiple times an update or install rule buttons

(cherry picked from commit c5557f3)

# Conflicts:
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/capped_exponential_backoff.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/retry_on_rate_limited_error.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/use_bootstrap_prebuilt_rules.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/prebuilt_rules/use_perform_specific_rules_upgrade_mutation.ts
#	x-pack/plugins/security_solution/public/detection_engine/rule_management/api/hooks/use_bootstrap_prebuilt_rules.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/bootstrap_prebuilt_rules/bootstrap_prebuilt_rules_handler.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/perform_rule_upgrade_route.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_handler.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_installation/review_rule_installation_route.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_handler.ts
#	x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/review_rule_upgrade/review_rule_upgrade_route.ts
#	x-pack/plugins/security_solution/server/utils/throttle_requests.test.ts
#	x-pack/plugins/security_solution/server/utils/throttle_requests.ts
#	x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management/api/hooks/use_bootstrap_prebuilt_rules.ts
#	x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/translations.tsx
#	x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/upgrade_prebuilt_rules_table_context.tsx
@xcrzx xcrzx mentioned this issue Feb 12, 2025
Merged
xcrzx added a commit that referenced this issue Feb 12, 2025
@xcrzx
[8.17] [Security Solution] Added concurrency limits and request throt…
6259401 
…tling to prebuilt rule routes (#209551) (#210773)

# Backport

This will backport the following commits from `main` to `8.17`:
- [[Security Solution] Added concurrency limits and request throttling
to prebuilt rule routes
(#209551)](#209551)

<!--- Backport version: 9.6.4 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Dmitrii
Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-02-11T17:12:03Z","message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","performance","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule
routes","number":209551,"url":"https://github.com/elastic/kibana/pull/209551","mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210642","number":210642,"state":"MERGED","mergeCommit":{"sha":"df87081d8a574c6b0bb2d9fc026776502622dc11","message":"[9.0]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210642)\n\n# Backport\n\nThis will
backport the following commits from `main` to `9.0`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"dmitrii.shevchenko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<dmitrii.shevchenko@elastic.co>"}},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210640","number":210640,"state":"MERGED","mergeCommit":{"sha":"2bd85b19aa84575ec0745ba2ad24d4c7718d824f","message":"[8.18]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210640)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.18`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"dmitrii.shevchenko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<dmitrii.shevchenko@elastic.co>"}},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209551","number":209551,"mergeCommit":{"message":"[Security
Solution] Added concurrency limits and request throttling to prebuilt
rule routes (#209551)\n\n**Resolves:
https://github.com/elastic/kibana/issues/208357**\n**Resolves:
https://github.com/elastic/kibana/issues/208355**\n\n## Summary \n\nTo
prevent possible OOM errors, we need to limit concurrent requests
to\nprebuilt rule routes (see attached tickets for more details).\n\n-
`installation/_perform` and `upgrade/_perform` endpoints\n- Concurrency
is limited to one parallel call. If another call is
made\nsimultaneously, the server responds with 429 Too Many Requests.\n-
On the front end, all rule install and upgrade operations are
retried\nin case of a 429 response. This ensures proper handling when a
user\nclicks multiple times an update or install rule buttons\n\n-
`prebuilt_rules/_bootstrap` endpoint\n- Install prebuilt rules and
endpoint packages sequentially instead of\nin parallel to prevent from
having them both downloaded into memory\nsimultaneously.\n- Added a
30-minute socket timeout to prevent the proxy from closing
the\nconnection while rule installation is in progress.\n- Introduced a
`throttleRequests` wrapper, ensuring the endpoint handler\nis called
only once when multiple concurrent requests are received.\n- The first
request triggers the handler, while subsequent requests wait\nfor the
first one to complete and reuse its result.\n- This prevents costly
prebuilt rule package installation from running\nin parallel.\n- Reusing
the response ensures the frontend correctly invalidates cached\nprebuilt
rule queries. Since concurrent frontend requests should receive\nthe
same installed package information, responding with 421 and using\nthe
retry logic as in cases above is not an option here because the\nsecond
request would receive a package installation skipped response\nleading
to no cache invalidation.\n\n- `installation/_review` and
`upgrade/_review` endpoints\n- Concurrency is limited to one parallel
call. If another call is made\nsimultaneously, the server responds with
429 Too Many Requests.\n- On the front end, all rule install and upgrade
operations are retried\nin case of a 429 response. This ensures proper
handling when a user\nclicks multiple times an update or install rule
buttons","sha":"c5557f33213f699acd9bb656af9166b1449d18f9"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210641","number":210641,"state":"MERGED","mergeCommit":{"sha":"8018c82f7d92b1d4798dea3c0a677bae740d4f8a","message":"[8.x]
[Security Solution] Added concurrency limits and request throttling to
prebuilt rule routes (#209551) (#210641)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.x`:\n- [[Security
Solution] Added concurrency limits and request throttling\nto prebuilt
rule
routes\n(#209551)](https://github.com/elastic/kibana/pull/209551)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Dmitrii\nShevchenko\",\"email\":\"dmitrii.shevchenko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:12:03Z\",\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:fix\",\"performance\",\"v9.0.0\",\"Team:Detections\nand
Resp\",\"Team: SecuritySolution\",\"Team:Detection
Rule\nManagement\",\"Feature:Prebuilt
Detection\nRules\",\"backport:version\",\"v8.18.0\",\"v9.1.0\",\"v8.19.0\",\"v8.17.3\"],\"title\":\"[Security\nSolution]
Added concurrency limits and request throttling to
prebuilt\nrule\nroutes\",\"number\":209551,\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.18\",\"8.x\",\"8.17\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.18\",\"label\":\"v8.18.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209551\",\"number\":209551,\"mergeCommit\":{\"message\":\"[Security\nSolution]
Added concurrency limits and request throttling to prebuilt\nrule routes
(#209551)\\n\\n**Resolves:\nhttps://github.com//issues/208357**\\n**Resolves:\nhttps://github.com//issues/208355**\\n\\n##
Summary \\n\\nTo\nprevent possible OOM errors, we need to limit
concurrent requests\nto\\nprebuilt rule routes (see attached tickets for
more details).\\n\\n-\n`installation/_perform` and `upgrade/_perform`
endpoints\\n- Concurrency\nis limited to one parallel call. If another
call is\nmade\\nsimultaneously, the server responds with 429 Too Many
Requests.\\n-\nOn the front end, all rule install and upgrade operations
are\nretried\\nin case of a 429 response. This ensures proper handling
when a\nuser\\nclicks multiple times an update or install rule
buttons\\n\\n-\n`prebuilt_rules/_bootstrap` endpoint\\n- Install
prebuilt rules and\nendpoint packages sequentially instead of\\nin
parallel to prevent from\nhaving them both downloaded into
memory\\nsimultaneously.\\n- Added a\n30-minute socket timeout to
prevent the proxy from closing\nthe\\nconnection while rule installation
is in progress.\\n- Introduced a\n`throttleRequests` wrapper, ensuring
the endpoint handler\\nis called\nonly once when multiple concurrent
requests are received.\\n- The first\nrequest triggers the handler,
while subsequent requests wait\\nfor the\nfirst one to complete and
reuse its result.\\n- This prevents costly\nprebuilt rule package
installation from running\\nin parallel.\\n- Reusing\nthe response
ensures the frontend correctly invalidates cached\\nprebuilt\nrule
queries. Since concurrent frontend requests should receive\\nthe\nsame
installed package information, responding with 421 and
using\\nthe\nretry logic as in cases above is not an option here because
the\\nsecond\nrequest would receive a package installation skipped
response\\nleading\nto no cache invalidation.\\n\\n-
`installation/_review` and\n`upgrade/_review` endpoints\\n- Concurrency
is limited to one parallel\ncall. If another call is
made\\nsimultaneously, the server responds with\n429 Too Many
Requests.\\n- On the front end, all rule install and upgrade\noperations
are retried\\nin case of a 429 response. This ensures proper\nhandling
when a user\\nclicks multiple times an update or install
rule\nbuttons\",\"sha\":\"c5557f33213f699acd9bb656af9166b1449d18f9\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"8.17\",\"label\":\"v8.17.3\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by:
Dmitrii Shevchenko
<dmitrii.shevchenko@elastic.co>"}},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xcrzx xcrzx

Labels
8.18 candidate bug Fixes for quality problems that affect the customer experience Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. performance Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.18.0
Projects
None yet
Milestone
No milestone
3 participants
@xcrzx @banderror @elasticmachine