[Performance][Security Solution][1/4] - Field Browser Performance #212469

michaelolo24 · 2025-02-26T05:36:36Z

Summary

Testing

For setup see testing section here: #212173 (comment)

Areas to test:

Alert Table (Alerts, Rule Detail, Rule Preview pages)
Security solution field browser component
Flyout table tab.

Background

When investigating the performance of the security solution application, one of the issues that was observed was locking of the page and field browser component when the total number of fields returned were significantly high.

This led to cell values not rendering in the alert table, and the field browser in all the security solution pages causing the page to crash. The relevant images can be seen at the bottom of this description

In short: The push(...fields) is non-performant at scale, and at a significant enough scale (Testing was done with 500k mapped fields), fails to run due to excessive arguments provided to the push method. In this PR improvements are made in the browserFields transformations that are done for the field browser component, expandable flyout table tab, and alert/rule tables via CellValue component.

This work was done to get immediate improvements in the security solution UI, but a longer term consideration will be whether or not the browserFields is even necessary anymore as a concept based on what is available via the fields api. We will revisit once our Sourcerer refactoring work is done.

After the fix

(Done on this branch that has the other changes as well)

Screen.Recording.2025-02-26.at.10.41.17.AM.mov

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Unit or functional tests were updated or added to match the most common scenarios

michaelolo24 · 2025-02-26T06:56:02Z

.../solutions/security/plugins/security_solution/public/common/containers/source/index.test.tsx


 describe('source/index.tsx', () => {
-  describe('useDataView hook', () => {


moved this to it's own file

michaelolo24 · 2025-02-26T06:56:37Z

...ns/security/plugins/security_solution/public/common/containers/source/use_data_view.test.tsx

+ * 2.0.
+ */
+
+import type { IndexFieldSearch } from './use_data_view';


Just a file move, no actual code changes

michaelolo24 · 2025-02-26T15:44:35Z

...rm/packages/shared/response-ops/alerts-fields-browser/components/field_items/field_items.tsx

-    categoryIds.reduce<BrowserFieldItem[]>((fieldItemsAcc, categoryId) => {
+  const getFieldItems = () => {
+    let fieldItemsAcc: BrowserFieldItem[] = [];
+    for (let i = 0; i < categoryIds.length; i += 1) {


for loop for better performance at scale

elasticmachine · 2025-02-26T15:47:46Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine · 2025-02-26T15:47:47Z

Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations)

michaelolo24 · 2025-02-26T18:41:23Z

/ci

michaelolo24 · 2025-02-26T19:02:27Z

/ci

michaelolo24 · 2025-02-26T20:52:49Z

The test failures seem to be related to: #170365

logeekal · 2025-02-27T08:12:13Z

...rm/packages/shared/response-ops/alerts-fields-browser/components/field_items/field_items.tsx

+        const categoryFieldItems = categoryBrowserFields.map(({ name = '', ...field }) => {
+          return {
+            name,
+            type: field.type,
+            description: getDescription(name, EcsFlat as Record<string, EcsMetadata>),
+            example: field.example?.toString(),
+            category: getCategory(name),
+            selected: selectedFieldIds.has(name),
+            isRuntime: !!field.runtimeField,
+          };
+        });
+        fieldItemsAcc = fieldItemsAcc.concat(categoryFieldItems);


Suggested change

const categoryFieldItems = categoryBrowserFields.map(({ name = '', ...field }) => {

return {

name,

type: field.type,

description: getDescription(name, EcsFlat as Record<string, EcsMetadata>),

example: field.example?.toString(),

category: getCategory(name),

selected: selectedFieldIds.has(name),

isRuntime: !!field.runtimeField,

};

});

fieldItemsAcc = fieldItemsAcc.concat(categoryFieldItems);

let fieldSeen = {} // this can go on line 71

for (let j = 0; j < categoryBrowserFields.length; j += 1) {

const field = categoryBrowserFields[j];

const name = field.name || '';

if(name in fieldSeen) continue;

fieldSeen[name] = true

const categoryFieldItem = {

name,

type: field.type,

description: getDescription(name, EcsFlat as Record<string, EcsMetadata>),

example: field.example?.toString(),

category: getCategory(name),

selected: selectedFieldIds.has(name),

isRuntime: !!field.runtimeField,

};

fieldItemsAcc.push(categoryFieldItem);

}

What do you think about above suggestion. With this we can get rid of uniqBy and concat which needs another iteration. It includes mainly 2 changes

Replace uniqBy with simple key existence check in an object.

change map to for loop.

I like it. 👏🏾 Will make the change. Thanks!

Done, with some very minor tweaks!

logeekal

Thank you 🚀

kibanamachine · 2025-03-04T01:22:44Z

Starting backport for target branches: 8.17, 8.18, 9.0

https://github.com/elastic/kibana/actions/runs/13644121804

kibanamachine · 2025-03-04T01:43:08Z

💔 All backports failed

Status	Branch	Result
❌	8.17	Backport failed because of merge conflicts
❌	8.18	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 212469

Questions ?

Please refer to the Backport tool documentation

…astic#212469) ## Summary Part 1 of elastic#212173 ### Testing For setup see testing section here: elastic#212173 (comment) **Areas to test:** - Alert Table (Alerts, Rule Detail, Rule Preview pages) - Security solution field browser component - Flyout table tab. ### Background When investigating the performance of the security solution application, one of the issues that was observed was locking of the page and field browser component when the total number of fields returned were significantly high. This led to cell values not rendering in the alert table, and the field browser in all the security solution pages causing the page to crash. The relevant images can be seen at the bottom of this description In short: The `push(...fields)` is non-performant at scale, and at a significant enough scale (Testing was done with 500k mapped fields), fails to run due to excessive arguments provided to the `push` method. In this PR improvements are made in the `browserFields` transformations that are done for the field browser component, expandable flyout table tab, and alert/rule tables via `CellValue` component. This work was done to get immediate improvements in the security solution UI, but a longer term consideration will be whether or not the `browserFields` is even necessary anymore as a concept based on what is available via the `fields` api. We will revisit once our Sourcerer refactoring work is done. <img width="1728" alt="Screenshot 2025-02-26 at 10 15 29 AM" src="https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020" /> <img width="1445" alt="Screenshot 2025-02-26 at 10 18 36 AM" src="https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6" /> <img width="1469" alt="Screenshot 2025-02-26 at 10 19 48 AM" src="https://github.com/user-attachments/assets/1767aa9b-66ab-46be-bc1a-5311630c2765" /> ![image](https://github.com/user-attachments/assets/5d746b21-fa9b-425b-826a-cc7abd444f21) ![image](https://github.com/user-attachments/assets/4dff2378-d61b-4770-b46b-41cb37d6ead4) ### After the fix (Done on [this branch](elastic#212173) that has the other changes as well) https://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 750e156)

…astic#212469) ## Summary Part 1 of elastic#212173 ### Testing For setup see testing section here: elastic#212173 (comment) **Areas to test:** - Alert Table (Alerts, Rule Detail, Rule Preview pages) - Security solution field browser component - Flyout table tab. ### Background When investigating the performance of the security solution application, one of the issues that was observed was locking of the page and field browser component when the total number of fields returned were significantly high. This led to cell values not rendering in the alert table, and the field browser in all the security solution pages causing the page to crash. The relevant images can be seen at the bottom of this description In short: The `push(...fields)` is non-performant at scale, and at a significant enough scale (Testing was done with 500k mapped fields), fails to run due to excessive arguments provided to the `push` method. In this PR improvements are made in the `browserFields` transformations that are done for the field browser component, expandable flyout table tab, and alert/rule tables via `CellValue` component. This work was done to get immediate improvements in the security solution UI, but a longer term consideration will be whether or not the `browserFields` is even necessary anymore as a concept based on what is available via the `fields` api. We will revisit once our Sourcerer refactoring work is done. <img width="1728" alt="Screenshot 2025-02-26 at 10 15 29 AM" src="https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020" /> <img width="1445" alt="Screenshot 2025-02-26 at 10 18 36 AM" src="https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6" /> <img width="1469" alt="Screenshot 2025-02-26 at 10 19 48 AM" src="https://github.com/user-attachments/assets/1767aa9b-66ab-46be-bc1a-5311630c2765" /> ![image](https://github.com/user-attachments/assets/5d746b21-fa9b-425b-826a-cc7abd444f21) ![image](https://github.com/user-attachments/assets/4dff2378-d61b-4770-b46b-41cb37d6ead4) ### After the fix (Done on [this branch](elastic#212173) that has the other changes as well) https://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 750e156) # Conflicts: # x-pack/platform/plugins/shared/triggers_actions_ui/public/application/sections/field_browser/components/field_items/field_items.tsx

…astic#212469) ## Summary Part 1 of elastic#212173 ### Testing For setup see testing section here: elastic#212173 (comment) **Areas to test:** - Alert Table (Alerts, Rule Detail, Rule Preview pages) - Security solution field browser component - Flyout table tab. ### Background When investigating the performance of the security solution application, one of the issues that was observed was locking of the page and field browser component when the total number of fields returned were significantly high. This led to cell values not rendering in the alert table, and the field browser in all the security solution pages causing the page to crash. The relevant images can be seen at the bottom of this description In short: The `push(...fields)` is non-performant at scale, and at a significant enough scale (Testing was done with 500k mapped fields), fails to run due to excessive arguments provided to the `push` method. In this PR improvements are made in the `browserFields` transformations that are done for the field browser component, expandable flyout table tab, and alert/rule tables via `CellValue` component. This work was done to get immediate improvements in the security solution UI, but a longer term consideration will be whether or not the `browserFields` is even necessary anymore as a concept based on what is available via the `fields` api. We will revisit once our Sourcerer refactoring work is done. <img width="1728" alt="Screenshot 2025-02-26 at 10 15 29 AM" src="https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020" /> <img width="1445" alt="Screenshot 2025-02-26 at 10 18 36 AM" src="https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6" /> <img width="1469" alt="Screenshot 2025-02-26 at 10 19 48 AM" src="https://github.com/user-attachments/assets/1767aa9b-66ab-46be-bc1a-5311630c2765" /> ![image](https://github.com/user-attachments/assets/5d746b21-fa9b-425b-826a-cc7abd444f21) ![image](https://github.com/user-attachments/assets/4dff2378-d61b-4770-b46b-41cb37d6ead4) ### After the fix (Done on [this branch](elastic#212173) that has the other changes as well) https://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 750e156) # Conflicts: # x-pack/plugins/security_solution/public/common/containers/source/index.test.tsx # x-pack/plugins/security_solution/public/common/containers/source/use_data_view.test.tsx # x-pack/plugins/triggers_actions_ui/public/application/sections/field_browser/components/field_items/field_items.tsx

michaelolo24 · 2025-03-04T01:56:25Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.x
✅	8.18
✅	8.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

michaelolo24 · 2025-03-04T13:50:08Z

💚 All backports created successfully

Status	Branch	Result
✅	8.16

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…astic#212469) ## Summary Part 1 of elastic#212173 ### Testing For setup see testing section here: elastic#212173 (comment) **Areas to test:** - Alert Table (Alerts, Rule Detail, Rule Preview pages) - Security solution field browser component - Flyout table tab. ### Background When investigating the performance of the security solution application, one of the issues that was observed was locking of the page and field browser component when the total number of fields returned were significantly high. This led to cell values not rendering in the alert table, and the field browser in all the security solution pages causing the page to crash. The relevant images can be seen at the bottom of this description In short: The `push(...fields)` is non-performant at scale, and at a significant enough scale (Testing was done with 500k mapped fields), fails to run due to excessive arguments provided to the `push` method. In this PR improvements are made in the `browserFields` transformations that are done for the field browser component, expandable flyout table tab, and alert/rule tables via `CellValue` component. This work was done to get immediate improvements in the security solution UI, but a longer term consideration will be whether or not the `browserFields` is even necessary anymore as a concept based on what is available via the `fields` api. We will revisit once our Sourcerer refactoring work is done. <img width="1728" alt="Screenshot 2025-02-26 at 10 15 29 AM" src="https://github.com/user-attachments/assets/a25f577f-f758-415e-9c93-5452eadb8020" /> <img width="1445" alt="Screenshot 2025-02-26 at 10 18 36 AM" src="https://github.com/user-attachments/assets/d70970d3-991a-47ba-b617-5862d18101b6" /> <img width="1469" alt="Screenshot 2025-02-26 at 10 19 48 AM" src="https://github.com/user-attachments/assets/1767aa9b-66ab-46be-bc1a-5311630c2765" /> ![image](https://github.com/user-attachments/assets/5d746b21-fa9b-425b-826a-cc7abd444f21) ![image](https://github.com/user-attachments/assets/4dff2378-d61b-4770-b46b-41cb37d6ead4) ### After the fix (Done on [this branch](elastic#212173) that has the other changes as well) https://github.com/user-attachments/assets/da992296-4eb8-49d4-96ca-b0a19a00f1f0 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 750e156) # Conflicts: # x-pack/plugins/security_solution/public/common/containers/source/index.test.tsx # x-pack/plugins/security_solution/public/common/containers/source/use_data_view.test.tsx # x-pack/plugins/triggers_actions_ui/public/application/sections/field_browser/components/field_items/field_items.tsx

michaelolo24 · 2025-03-04T15:52:43Z

💚 All backports created successfully

Status	Branch	Result
✅	8.16

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ce (#212469) (#213021) # Backport This will backport the following commits from `main` to `9.0`: - [[Performance][Security Solution][1/4] - Field Browser Performance (#212469)](#212469)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…nce (#212469) (#213130) # Backport This will backport the following commits from `main` to `8.16`: - [[Performance][Security Solution][1/4] - Field Browser Performance (#212469)](#212469)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…nce (#212469) (#213025) # Backport This will backport the following commits from `main` to `8.17`: - [[Performance][Security Solution][1/4] - Field Browser Performance (#212469)](#212469)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…ce (#212469) (#213022) # Backport This will backport the following commits from `main` to `8.x`: - [[Performance][Security Solution][1/4] - Field Browser Performance (#212469)](#212469)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…nce (#212469) (#213023) # Backport This will backport the following commits from `main` to `8.18`: - [[Performance][Security Solution][1/4] - Field Browser Performance (#212469)](#212469)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

mistic · 2025-03-04T20:30:57Z

This PR didn't make it into the latest BC of both v8.16.5 and v8.17.3. Updating the labels accordingly.

michaelolo24 force-pushed the security-performance-field-browser branch from a34a4e7 to 8b03ec7 Compare February 26, 2025 06:53

michaelolo24 commented Feb 26, 2025

View reviewed changes

michaelolo24 force-pushed the security-performance-field-browser branch from 8b03ec7 to 529193e Compare February 26, 2025 06:59

michaelolo24 added backport:all-open Backport to all branches that could still receive a release and removed backport:prev-major Backport to (8.x, 8.18, 8.17, 8.16) the previous major branch and other branches in development labels Feb 26, 2025

michaelolo24 mentioned this pull request Feb 26, 2025

[DO NOT MERGE] - Combined performance changes #212173

Draft

michaelolo24 commented Feb 26, 2025

View reviewed changes

michaelolo24 marked this pull request as ready for review February 26, 2025 15:47

michaelolo24 requested review from a team as code owners February 26, 2025 15:47

michaelolo24 force-pushed the security-performance-field-browser branch from 529193e to 0a6dbd3 Compare February 26, 2025 17:09

michaelolo24 force-pushed the security-performance-field-browser branch from 4bf3153 to 7a3e5ad Compare February 26, 2025 20:52

logeekal reviewed Feb 27, 2025

View reviewed changes

michaelolo24 force-pushed the security-performance-field-browser branch from b82e504 to 2573942 Compare February 27, 2025 16:58

michaelolo24 added the v8.18.0 label Feb 27, 2025

logeekal approved these changes Feb 27, 2025

View reviewed changes

michaelolo24 force-pushed the security-performance-field-browser branch from d8b68ad to 81f2b4c Compare February 27, 2025 20:00

michaelolo24 added the v8.19.0 label Mar 4, 2025

michaelolo24 mentioned this pull request Mar 4, 2025

[9.0] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) #213021

Merged

michaelolo24 mentioned this pull request Mar 4, 2025

[8.x] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) #213022

Merged

michaelolo24 mentioned this pull request Mar 4, 2025

[8.18] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) #213023

Merged

michaelolo24 mentioned this pull request Mar 4, 2025

[8.17] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) #213025

Merged

michaelolo24 mentioned this pull request Mar 4, 2025

[8.16] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) #213098

Closed

kibanamachine mentioned this pull request Mar 4, 2025

[Automatic Import] Fix unstructured syslog flow #213042

Merged

2 tasks

michaelolo24 mentioned this pull request Mar 4, 2025

[8.16] [Performance][Security Solution][1/4] - Field Browser Performance (#212469) #213130

Merged

kibanamachine added the v8.16.5 label Mar 4, 2025

mistic added v8.16.6 v8.17.4 and removed v8.17.3 v8.16.5 labels Mar 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Performance][Security Solution][1/4] - Field Browser Performance #212469

[Performance][Security Solution][1/4] - Field Browser Performance #212469

michaelolo24 commented Feb 26, 2025 •

edited by kibanamachine

Loading

michaelolo24 Feb 26, 2025

michaelolo24 Feb 26, 2025

michaelolo24 Feb 26, 2025

elasticmachine commented Feb 26, 2025

elasticmachine commented Feb 26, 2025

michaelolo24 commented Feb 26, 2025

michaelolo24 commented Feb 26, 2025

michaelolo24 commented Feb 26, 2025

logeekal Feb 27, 2025 •

edited

Loading

michaelolo24 Feb 27, 2025 •

edited

Loading

michaelolo24 Feb 27, 2025

logeekal left a comment •

edited

Loading

kibanamachine commented Mar 4, 2025

kibanamachine commented Mar 4, 2025

michaelolo24 commented Mar 4, 2025

michaelolo24 commented Mar 4, 2025

michaelolo24 commented Mar 4, 2025

mistic commented Mar 4, 2025


		describe('source/index.tsx', () => {
		describe('useDataView hook', () => {

[Performance][Security Solution][1/4] - Field Browser Performance #212469

[Performance][Security Solution][1/4] - Field Browser Performance #212469

Conversation

michaelolo24 commented Feb 26, 2025 • edited by kibanamachine Loading

Summary

Testing

Background

After the fix

Checklist

michaelolo24 Feb 26, 2025

Choose a reason for hiding this comment

michaelolo24 Feb 26, 2025

Choose a reason for hiding this comment

michaelolo24 Feb 26, 2025

Choose a reason for hiding this comment

elasticmachine commented Feb 26, 2025

elasticmachine commented Feb 26, 2025

michaelolo24 commented Feb 26, 2025

michaelolo24 commented Feb 26, 2025

michaelolo24 commented Feb 26, 2025

logeekal Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

michaelolo24 Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

michaelolo24 Feb 27, 2025

Choose a reason for hiding this comment

logeekal left a comment • edited Loading

Choose a reason for hiding this comment

kibanamachine commented Mar 4, 2025

kibanamachine commented Mar 4, 2025

💔 All backports failed

Manual backport

Questions ?

michaelolo24 commented Mar 4, 2025

💚 All backports created successfully

Questions ?

michaelolo24 commented Mar 4, 2025

💚 All backports created successfully

Questions ?

michaelolo24 commented Mar 4, 2025

💚 All backports created successfully

Questions ?

mistic commented Mar 4, 2025

michaelolo24 commented Feb 26, 2025 •

edited by kibanamachine

Loading

logeekal Feb 27, 2025 •

edited

Loading

michaelolo24 Feb 27, 2025 •

edited

Loading

logeekal left a comment •

edited

Loading