From 786df79f37642b8ddf214e086a3375204ec63a3b Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Tue, 18 Feb 2025 21:34:15 +0100 Subject: [PATCH] [Security Solution] Refactor prebuilt rule customization test plans, part 2 (#211472) **Epic:** https://github.com/elastic/kibana/issues/174168 **Partially addresses:** https://github.com/elastic/kibana/issues/202068, https://github.com/elastic/kibana/issues/202078, https://github.com/elastic/kibana/issues/202079 **Follow-up to:** https://github.com/elastic/kibana/pull/211300 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates an "entrypoint" file that should help navigate all the test plans for prebuilt rules. - Creates a file for keeping common information about prebuilt rules that can be shared between the test plans. - Extracts duplicated terminology to the file with common information. - Extracts duplicated assumptions to the file with common information. - Extracts duplicated non-functional requirements to the file with common information. - Adds user stories to each test plan. - Updates links to tickets in every test plan. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. This refactoring prepares the test plans for being "functionally" changed and improved in follow-up PRs. For example, we're going to cover the logic described in https://github.com/elastic/kibana/issues/210358 and address any other gaps in coverage. ## Review tip It might be easier to review this PR commit-by-commit as each of them contains logically cohesive changes. --- .../prebuilt_rule_customization.md | 45 +++--- .../prebuilt_rules/prebuilt_rule_export.md | 35 ++++- .../prebuilt_rules/prebuilt_rule_import.md | 33 ++++- .../prebuilt_rule_installation.md | 47 +++---- .../prebuilt_rule_upgrade_diff_algorithms.md | 27 ++-- .../prebuilt_rule_upgrade_with_preview.md | 30 ++-- .../prebuilt_rule_upgrade_without_preview.md | 47 ++----- .../prebuilt_rules/prebuilt_rules.md | 35 +++++ .../prebuilt_rules_common_info.md | 133 ++++++++++++++++++ .../prebuilt_rules/prebuilt_rules_package.md | 44 ++---- 10 files changed, 331 insertions(+), 145 deletions(-) create mode 100644 x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules.md create mode 100644 x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_common_info.md diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md index e98a45d00c32d..bbd7e9de91075 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md @@ -24,8 +24,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Useful information](#useful-information) - [Tickets](#tickets) + - [User stories](#user-stories) - [Terminology](#terminology) - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) - [Scenarios](#scenarios) - [Editing prebuilt rules](#editing-prebuilt-rules) - [**Scenario: User can edit a non-customized prebuilt rule from the rule edit page**](#scenario-user-can-edit-a-non-customized-prebuilt-rule-from-the-rule-edit-page) @@ -54,33 +56,42 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Test plan issue](https://github.com/elastic/kibana/issues/202068) -- [Prebuilt rule customization](https://github.com/elastic/kibana/issues/174168) epic +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) +- [Tests for prebuilt rule customization workflow](https://github.com/elastic/kibana/issues/202068) -### Terminology - -- **Base version**: Prebuilt rule asset we ship in the rule package corresponding to the currently installed prebuilt rules. It represents "original" version of the rule. During prebuilt rules installation prebuilt rule assets data is copied over and becomes an installed prebuilt rule. - -- **Customized prebuilt rule**: An installed prebuilt rule that has been changed by the user in the way rule fields semantically differ from the base version. Also referred to as "Modified" in the UI. - -- **Non-customized prebuilt rule**: An installed prebuilt rule that has rule fields values matching the base version. - -- **Custom rule**: A rule created by the user themselves +### User stories -- **rule source, or ruleSource**: A field on the rule that defines the rule's categorization. Can be `internal` or `external`. +**Prebuilt rule customization workflow:** -- **`is_customized`**: A field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version +- User can edit a single prebuilt rule from the Rule Details page. +- User can edit single prebuilt rules one-by-one from the Rule Management page. +- User can edit multiple prebuilt rules in bulk via bulk actions on the Rule Management page. For example: + - User can bulk add index patterns to prebuilt rules. + - User can bulk update rule schedule in prebuilt rules. +- User can customize most of the fields of prebuilt rules: + - User can edit and customize almost any field of a prebuilt rule, just like it's possible to do with custom rules, via editing it directly or via bulk editing via bulk actions. + - User can't edit the Author and License fields. +- User can see if the rule is customized on the Rule Details page. +- User can see which rules are customized on the Rule Management page in the Upgrade table. +- User can un-customize a prebuilt rule by editing it and reverting its parameters back to their original values. -- **customizable rule field**: A rule field that is able to be customized on a prebuilt rule. A comprehenseive list can be found in `./shared_assets/customizable_rule_fields.md`. - -- **non-customizable rule field**: A rule field that is unable to be customized on a prebuilt rule. A comprehenseive list can be found in `./shared_assets/non_customizable_rule_fields.md`. +### Terminology -- **non-semantic change**: A change to a rule field that is functionally different. We normalize certain fields so for a time-related field such as `from`, `1m` vs `60s` are treated as the same value. We also trim leading and trailing whitespace for query fields. +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). +- **Rule source**, or **`ruleSource`**: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source. +- **`is_customized`**: a field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version. +- **non-semantic change**: a change to a rule field that is functionally different. We normalize certain fields so for a time-related field such as `from`, `1m` vs `60s` are treated as the same value. We also trim leading and trailing whitespace for query fields. ### Assumptions +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). - Rule package used will have all previous rule versions present (no missing base versions) +### Non-functional requirements + +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). + ## Scenarios ### Editing prebuilt rules diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_export.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_export.md index c42e649cddc07..8819a7db15466 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_export.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_export.md @@ -25,7 +25,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Useful information](#useful-information) - [Tickets](#tickets) + - [User stories](#user-stories) - [Terminology](#terminology) + - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) - [Scenarios](#scenarios) - [Core Functionality](#core-functionality) - [Scenario: Exporting prebuilt rule individually from rule details page](#scenario-exporting-prebuilt-rule-individually-from-rule-details-page) @@ -42,15 +45,35 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) -- [Rule Exporting Feature](https://github.com/elastic/kibana/issues/180167#issue-2227974379) -- [Rule Export API PR](https://github.com/elastic/kibana/pull/194498) +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) +- [Allow exporting prebuilt rules at the API level](https://github.com/elastic/kibana/issues/180167) +- [Support exporting prebuilt rules from the Rule Management page](https://github.com/elastic/kibana/issues/180173) +- [Support exporting prebuilt rules from the Rule Details page](https://github.com/elastic/kibana/issues/180176) +- [Tests for prebuilt rule import/export workflow](https://github.com/elastic/kibana/issues/202079) + +### User stories + +**Prebuilt rule export workflow:** + +- User can export a single prebuilt rule from the Rule Details page. +- User can export multiple prebuilt rules one-by-one from the Rule Management page. +- User can export multiple prebuilt rules in bulk from the Rule Management page via bulk actions. +- User can export prebuilt non-customized rules. +- User can export prebuilt customized rules. +- User can export any combination of prebuilt non-customized, prebuilt customized, and custom rules. ### Terminology -- **prebuilt rule**: A rule contained in our `Prebuilt Security Detection Rules` integration in Fleet. -- **custom rule**: A rule defined by the user, which has no relation to the prebuilt rules -- **rule source, or ruleSource**: A field on the rule that defines the rule's categorization +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). + +### Assumptions + +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). + +### Non-functional requirements + +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). ## Scenarios diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_import.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_import.md index 2e61b731efbf0..68063421ce992 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_import.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_import.md @@ -21,7 +21,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Useful information](#useful-information) - [Tickets](#tickets) + - [User stories](#user-stories) - [Terminology](#terminology) + - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) - [Scenarios](#scenarios) - [Core Functionality](#core-functionality) - [Scenario: Importing an unmodified prebuilt rule with a matching rule\_id and version](#scenario-importing-an-unmodified-prebuilt-rule-with-a-matching-rule_id-and-version) @@ -43,15 +46,33 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) -- [Rule Importing Feature](https://github.com/elastic/kibana/issues/180168) -- [Rule Import API PR](https://github.com/elastic/kibana/pull/190198) +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) +- [Allow importing prebuilt rules at the API level](https://github.com/elastic/kibana/issues/180168) +- [Benchmark performance of importing a large number of prebuilt rules](https://github.com/elastic/kibana/issues/195632) +- [Tests for prebuilt rule import/export workflow](https://github.com/elastic/kibana/issues/202079) + +### User stories + +**Prebuilt rule import workflow:** + +- User can import a single prebuilt rule on the Rule Management page. +- User can import multiple prebuilt rules on the Rule Management page. +- User can import prebuilt non-customized rules. +- User can import prebuilt customized rules. +- User can import any combination of prebuilt non-customized, prebuilt customized, and custom rules. ### Terminology -- **prebuilt rule**: A rule contained in our `Prebuilt Security Detection Rules` integration in Fleet. -- **custom rule**: A rule defined by the user, which has no relation to the prebuilt rules -- **rule source, or ruleSource**: A field on the rule that defines the rule's categorization +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). + +### Assumptions + +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). + +### Non-functional requirements + +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). ## Scenarios diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_installation.md index 128e90042f395..56de7b5d64d7c 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_installation.md @@ -20,6 +20,7 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Useful information](#useful-information) - [Tickets](#tickets) + - [User stories](#user-stories) - [Terminology](#terminology) - [Assumptions](#assumptions) - [Non-functional requirements](#non-functional-requirements) @@ -58,50 +59,36 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 2](https://github.com/elastic/kibana/issues/174167) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) -**Milestone 3 - Prebuilt Rules Customization:** -- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) -- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) +### User stories -**Milestone 2:** -- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) -- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) +**Prebuilt rule installation workflow:** -### Terminology - -- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. - -- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). - -- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. +- User can install single prebuilt rules one-by-one from the Rule Installation page. +- User can install multiple prebuilt rules in bulk from the Rule Installation page. +- User can install all available prebuilt rules in bulk from the Rule Installation page. +- User can preview properties of a prebuilt rule before installing it. -- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. - -- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. +### Terminology -- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. - - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. - - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). +- **CTA to install prebuilt rules**: a link button with a counter on the Rule Management page. +- **CTA to upgrade prebuilt rules**: a tab with a counter on the Rule Management page. ### Assumptions -- Below scenarios only apply to prebuilt detection rules. -- Users should be able to install prebuilt rules on the `Basic` license and higher. -- EPR is available for fetching the package unless explicitly indicated otherwise. -- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). ### Non-functional requirements -- Notifications, rule installation workflows should work: - - regardless of the package type: with historical rule versions or without; - - regardless of the package registry availability: i.e., they should also work in air-gapped environments. -- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. -- Kibana should not crash with Out Of Memory exception during package installation. -- For test purposes, it should be possible to use detection rules package versions lower than the latest. +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). ### Functional requirements +- Users should be able to install prebuilt rules on the `Basic` license and higher. - User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties). - If user chooses to preview a prebuilt rule to be installed/upgraded, we currently show this preview in a flyout. - In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed. diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_diff_algorithms.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_diff_algorithms.md index 0db233c3fb357..381e207c3c47a 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_diff_algorithms.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_diff_algorithms.md @@ -19,6 +19,7 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Tickets](#tickets) - [Terminology](#terminology) - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) - [Scenarios](#scenarios) - [Rule field doesn't have an update and has no custom value - `AAA`](#rule-field-doesnt-have-an-update-and-has-no-custom-value---aaa) - [**Scenario: `AAA` - Rule field is any type**](#scenario-aaa---rule-field-is-any-type) @@ -53,21 +54,22 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Users can customize prebuilt detection rules](https://github.com/elastic/kibana/issues/174168) epic -- [Implement single-line string diff algorithm](https://github.com/elastic/kibana/issues/180158) +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) - [Implement number diff algorithm](https://github.com/elastic/kibana/issues/180160) +- [Implement single-line string diff algorithm](https://github.com/elastic/kibana/issues/180158) +- [Implement multi-line string diff algorithm](https://github.com/elastic/kibana/issues/180159) - [Implement array of scalar values diff algorithm](https://github.com/elastic/kibana/issues/180162) +- [Implement data source fields diff algorithm](https://github.com/elastic/kibana/issues/187659) +- [Implement query fields diff algorithms](https://github.com/elastic/kibana/issues/187658) +- [Implement `concurrent_searches` and `items_per_search` fields diff algorithms](https://github.com/elastic/kibana/issues/188061) +- [Implement rule type diff algorithm](https://github.com/elastic/kibana/issues/190482) +- [Tests for prebuilt rule customization workflow](https://github.com/elastic/kibana/issues/202068) +- [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078) ### Terminology -- **Base version**: Also labeled as `base_version`. This is the version of a rule authored by Elastic as it is installed from the `security_detection_engine` package, with no customizations to any fields by the user. - -- **Current version**: Also labeled as `current_version`. This is the version of the rule that the user currently has installed. Consists of the `base_version` of the rules plus all customization applies to its fields by the user. - -- **Target version**: Also labeled as `target_version`. This is the version of the rule that contains the update from Elastic. - -- **Merged version**: Also labeled as `merged_version`. This is the version of the rule that we determine via the various algorithms. It could contain a mix of all the rule versions on a per-field basis to create a singluar version of the rule containing all relevant updates and user changes to display to the user. - +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). - **Grouped fields** - `data_source`: an object that contains a `type` field with a value of `data_view_id` or `index_patterns` and another field that's either `data_view_id` of type string OR `index_patterns` of type string array - `kql_query`: an object that contains a `type` field with a value of `inline_query` or `saved_query` and other fields based on whichever type is defined. If it's `inline_query`, the object contains a `query` string field, a `language` field that's either `kuery` or `lucene`, and a `filters` field which is an array of kibana filters. If the type field is `saved_query`, the object only contains a `saved_query_id` string field. @@ -76,9 +78,14 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Assumptions +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). - All scenarios will contain at least 1 prebuilt rule installed in Kibana. - A new version will be available for rule(s). +### Non-functional requirements + +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). + ## Scenarios ### Rule field doesn't have an update and has no custom value - `AAA` diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md index 31f36d86ed773..759bd352b17da 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md @@ -19,8 +19,10 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Useful information](#useful-information) - [Tickets](#tickets) + - [User stories](#user-stories) - [Terminology](#terminology) - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) - [Functional requirements](#functional-requirements) - [Scenarios](#scenarios) - [Rule upgrade field preview](#rule-upgrade-field-preview) @@ -66,25 +68,33 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) - [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078) -### Terminology - -- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. - - - **CTA to upgrade the prebuilt rule**: the button to upgrade the prebuilt rule currently shown in the Rule Upgrade flyout. +### User stories -- **Non-customized field**: a prebuilt rule's field that has the original value from the originally installed prebuilt rule. +**Prebuilt rule upgrade workflow:** -- **Customized field**: a prebuilt rule's field that has a value that differs from the original field value of the originally installed prebuilt rule. +- User can upgrade a single prebuilt rule to its latest version from the Rule Upgrade table with previewing incoming updates from Elastic and user customizations in the Rule Upgrade flyout. +- Specifically, when upgrading a rule, in the Rule Upgrade flyout: + - User can preview updates from Elastic, for each rule field that has an update from Elastic. + - User can preview their customizations, for each rule field that was customized. + - User can compare their customizations with updates from Elastic and see if there are any conflicts between them, per each rule field. + - User can manually resolve conflicts between their customizations and updates from Elastic, per each rule field. + - User can edit the final field values before submitting the update. +- User can upgrade a rule if its type has been changed by Elastic in the latest version, but can only accept the incoming changes; any user customizations will be lost. -- **Non-customized rule**: a prebuilt rule that doesn't have any customized fields. +### Terminology -- **Customized rule**: a prebuilt rule that has one or more customized fields. +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). +- **CTA to upgrade the prebuilt rule**: the button to upgrade the prebuilt rule currently shown in the Rule Upgrade flyout. ### Assumptions -- Below scenarios only apply to prebuilt detection rules. +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). - A prebuilt rule is shown in the Rule Upgrade table when there's a newer version of this rule in the currently installed package with prebuilt rules. +### Non-functional requirements + +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). + ### Functional requirements - User should be able to upgrade prebuilt rules one-by-one with the ability to preview: diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md index 6cf8ba78270df..db1e1f2b8b66f 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md @@ -20,6 +20,7 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one - [Useful information](#useful-information) - [Tickets](#tickets) + - [User stories](#user-stories) - [Terminology](#terminology) - [Assumptions](#assumptions) - [Non-functional requirements](#non-functional-requirements) @@ -81,52 +82,30 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) epic +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) +- [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078) -**Milestone 3 - Prebuilt Rules Customization:** +### User stories -- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) -- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) +**Prebuilt rule upgrade workflow:** -**Milestone 2:** - -- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) -- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) +- User can upgrade a single prebuilt rule to its latest version without previewing the incoming updates. +- User can bulk upgrade multiple prebuilt rules to their latest versions without previewing the incoming updates. ### Terminology -- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. - -- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). - -- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. - -- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. - -- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. - -- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. - - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. - - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology). +- **CTA to install prebuilt rules**: a link button with a counter on the Rule Management page. +- **CTA to upgrade prebuilt rules**: a tab with a counter on the Rule Management page. ### Assumptions -- Below scenarios only apply to prebuilt detection rules. -- EPR is available for fetching the package with prebuilt rules unless explicitly indicated otherwise. -- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. -- Users have the required [privileges for managing detection rules](https://www.elastic.co/guide/en/security/current/detections-permissions-section.html). -- Users are: - - on the `Basic` license and higher in self-hosted and ECH environments; - - on the `Essentials` tier and higher in Serverless environments. +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). ### Non-functional requirements -- Notifications, rule installation and rule upgrade workflows should work: - - regardless of the package type: with historical rule versions or without; - - regardless of the package registry availability: i.e., they should also work in air-gapped environments. -- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. -- Kibana should not crash with Out Of Memory exception during package installation. -- For test purposes, it should be possible to use detection rules package versions lower than the latest. +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). ### Functional requirements diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules.md new file mode 100644 index 0000000000000..b66e991dd5ef5 --- /dev/null +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules.md @@ -0,0 +1,35 @@ +# Test plans for workflows related to prebuilt rules + +## Common information + +See [common information](./prebuilt_rules_common_info.md) about prebuilt rules, which includes: + +- Overview of our features built around prebuilt rules. +- Common terminology used in the test plans below. +- Common assumptions about testing scenarios. +- And other useful information. + +## Test plans + +Understand how we distribute prebuilt rules via a Fleet package and how prebuilt rules become available for installation and upgrade in the system. + +- [Test plan: prebuilt rules package](./prebuilt_rules_package.md) + +Learn how users can install prebuilt rules: + +- [Test plan: installing prebuilt rules](./prebuilt_rule_installation.md) + +Learn how users can edit and customize prebuilt rules: + +- [Test plan: customizing prebuilt rules](./prebuilt_rule_customization.md) + +Learn how users can upgrade prebuilt rules: + +- [Test plan: upgrading prebuilt rules one-by-one or in bulk without preview](./prebuilt_rule_upgrade_without_preview.md) +- [Test plan: upgrading prebuilt rules one-by-one with preview](./prebuilt_rule_upgrade_with_preview.md) +- [Test plan: diff algorithms for upgrading prebuilt rules](./prebuilt_rule_upgrade_diff_algorithms.md) + +Learn how users can export and import prebuilt rules: + +- [Test plan: exporting prebuilt rules](./prebuilt_rule_export.md) +- [Test plan: importing prebuilt rules](./prebuilt_rule_import.md) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_common_info.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_common_info.md new file mode 100644 index 0000000000000..cca246064fc47 --- /dev/null +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_common_info.md @@ -0,0 +1,133 @@ +# Common information about prebuilt rules + +## Table of contents + + + +- [Tickets](#tickets) +- [Features](#features) +- [Common terminology](#common-terminology) +- [Common assumptions](#common-assumptions) +- [Common non-functional requirements](#common-non-functional-requirements) +- [Common functional requirements](#common-functional-requirements) + +## Tickets + +Epics: + +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 2](https://github.com/elastic/kibana/issues/174167) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) + +Milestone 3: + +- Automated testing: + - [Tests for prebuilt rule customization workflow](https://github.com/elastic/kibana/issues/202068) + - [Tests for prebuilt rule upgrade workflow](https://github.com/elastic/kibana/issues/202078) + - [Tests for prebuilt rule import/export workflow](https://github.com/elastic/kibana/issues/202079) +- Manual testing: + - [Exploratory testing](https://github.com/elastic/kibana/issues/180398) + - [Acceptance testing](https://github.com/elastic/security-team/issues/11572) (internal) +- Documentation: + - [Main ticket](https://github.com/elastic/security-docs/issues/5061) + - [Copy review ticket](https://github.com/elastic/security-docs/issues/6238) + +## Features + +Historically, users were only able to install all the available prebuilt rules at once, and later upgrade them all at once to their latest versions from Elastic. Also, users were only able to add or edit notification actions for prebuilt rules, but it was impossible to edit and customize any other rule parameters. + +With [Milestone 2](https://github.com/elastic/kibana/issues/174167), we introduced the ability to: + +- Install prebuilt rules selectively one-by-one or in bulk. +- Before installing a single rule: + - Preview its properties. +- Upgrade prebuilt rules selectively one-by-one or in bulk to their latest versions from Elastic. +- Before upgrading a single rule: + - Preview properties of its latest version. + - Preview a diff between the currently installed version of the rule and its latest version. + +With [Milestone 3](https://github.com/elastic/kibana/issues/174168), we're introducing the ability to: + +- Edit and customize prebuilt rules (modify almost all rule parameters, besides rule notification actions). +- Export and import prebuilt rules, including customized ones. +- Upgrade prebuilt rules while keeping the user customizations whenever possible. + +Please find more information about Milestone 3 features and user stories in the corresponding test plans. + +## Common terminology + +Terminology related to the package with prebuilt rules: + +- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. +- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. +- **Package**: the `security_detection_engine` Fleet package that we use for distributing prebuilt detection rules in the form of `security-rule` assets (saved objects). +- **Real package**: the actual latest stable package distributed and pulled from EPR via Fleet. +- **Rule asset**: `security-rule` asset saved objects distributed via the package. There can be one or many assets per each prebuilt rule in the package. Each asset can represent either the latest version of a prebuilt rule, or one of its prior historical versions. +- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly from a test during the test setup phase. This allows us to avoid installing the real package in many tests, because this is a heavy, slow and unreliable operation. + +Terminology related to the rule's origin: + +- **Custom rule**: a rule created by the user themselves. +- **Prebuilt rule**: a rule created by Elastic and shipped via the package. + +Terminology related to the various rule versions that can exist in the system: + +- **Base version**, also labeled as `base_version`: the "original" version of a prebuilt rule. This is the version of a rule authored by Elastic as it is installed from the package, without customizations to any fields by the user. It is equal to the prebuilt rule asset from the package that corresponds to the `current_version` of this rule. During the installation of a prebuilt rule its asset data is copied over and becomes an installed prebuilt rule. +- **Current version**, also labeled as `current_version`. This is the version of a rule that the user currently has installed. Can be non-customized (in which case it's equal to the `base_version`) or customized by the user (in which case it's different from the `base_version`). You can think of it as a combination of the `base_version` plus all the user customizations applied to its fields on top of that. +- **Target version**, also labeled as `target_version`. This is a newer version of a rule that contains updates from Elastic and that the user is upgrading the rule to. Currently, we allow users to upgrade prebuilt rules only to their lates versions. +- **Merged version**, also labeled as `merged_version`. This is the version of a prebuilt rule that the rule upgrade workflow proposes to the user by default on upgrade. Can incorporate both user customizations and updates from Elastic, where conflicts between them have been auto-resolved by diff algorithms on a per-field basis. +- We can apply the notion of "versions" to rules as a whole or to each rule field separately. +- Base version's `rule.version` always == current version's `rule.version`. +- Current version's `rule.version` always < target version's `rule.version`. + +Terminology related to prebuilt rule customization: + +- **Customized prebuilt rule**: an installed prebuilt rule that has been changed by the user in the way rule fields semantically differ from the base version. Also referred to as "Modified" in the UI. + - A customized prebuilt rule has one or more customized fields. + - For a customized prebuilt rule, `current_version` != `base_version`. +- **Non-customized prebuilt rule**: an installed prebuilt rule that has rule fields values matching the base version. + - A non-customized prebuilt rule doesn't have any customized fields. + - For a non-customized prebuilt rule, `current_version` == `base_version`. +- **Customized field**: a prebuilt rule's field which value differs from the value from the originally installed prebuilt rule. + - For a customized field, `current_version.field` != `base_version.field`. +- **Non-customized field**: a prebuilt rule's field that has the original value from the originally installed prebuilt rule. + - For a non-customized field, `current_version.field` == `base_version.field`. +- **Customizable rule field**: a rule field that is able to be customized on a prebuilt rule. A comprehenseive list can be found in `./shared_assets/customizable_rule_fields.md`. +- **Non-customizable rule field**: a rule field that is unable to be customized on a prebuilt rule. A comprehenseive list can be found in `./shared_assets/non_customizable_rule_fields.md`. + +Terminology related to the "rule source" object: + +- **Rule source**, also known as `ruleSource` and `rule_source`: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source. +- **`is_customized`**: a field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version. + +Terminology related to UI and UX: + +- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. + +## Common assumptions + +Unless explicitly indicated otherwise: + +- Scenarios in the test plans only apply to prebuilt detection rules. Some scenarios may apply to both prebuilt and custom detection rules, in which case it should be clearly stated. +- EPR is available for fetching the package with prebuilt rules. +- Only the latest **stable** package with prebuilt rules is checked for installation/upgrade. Pre-release packages are ignored. +- User is on the following licenses/tiers: + - on the `Basic` license in a self-hosted or ECH environment; + - on the `Essentials` tier in a Serverless Security environment. +- User has the required [privileges for managing detection rules](https://www.elastic.co/guide/en/security/current/detections-permissions-section.html). + +## Common non-functional requirements + +- Package installation, rule installation and rule upgrade workflows should work: + - regardless of the package type: with historical rule versions or without; + - regardless of the package registry availability: i.e., they should also work in air-gapped environments. +- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. +- Kibana should not crash with Out Of Memory exception during package installation. +- For test purposes, it should be possible to use detection rules package versions lower than the latest. + +## Common functional requirements + +TBD diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_package.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_package.md index 5d0bfee30d8b0..739d15b164db3 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_package.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rules_package.md @@ -35,50 +35,30 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one ### Tickets -- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) - -**Milestone 3 - Prebuilt Rules Customization:** -- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) -- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) - -**Milestone 2:** -- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) -- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) +- [Users can Customize Prebuilt Detection Rules](https://github.com/elastic/security-team/issues/1974) (internal) +- [Users can Customize Prebuilt Detection Rules: Milestone 2](https://github.com/elastic/kibana/issues/174167) + - [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) + - [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) +- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168) ### Terminology -- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. - -- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). - -- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. - -- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. - -- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. - -- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. - - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. - - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. +- [Common terminology](./prebuilt_rules_common_info.md#common-terminology), see the terminology related to the package with prebuilt rules. ### Assumptions -- Below scenarios only apply to prebuilt detection rules. -- Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. -- EPR is available for fetching the package unless explicitly indicated otherwise. -- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. +- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions). ### Non-functional requirements -- Package installation, rule installation and rule upgrade workflows should work: - - regardless of the package type: with historical rule versions or without; - - regardless of the package registry availability: i.e., they should also work in air-gapped environments. -- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. -- Kibana should not crash with Out Of Memory exception during package installation. -- For test purposes, it should be possible to use detection rules package versions lower than the latest. +- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements). ### Functional requirements +- Package should be installable and updatable: + - on any license in self-hosted and ECH environments; + - on any tier in Serverless Security environments; + - regardless of user privileges, as long as the user has access to Security Solution. - User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties). - User should be able to upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions).