Skip to content

Commit

Permalink
[Security Solution] Refactor prebuilt rule customization test plans, …
Browse files Browse the repository at this point in the history
…part 3 (elastic#212420)

**Epic:** elastic#174168
**Partially addresses:**
elastic#202068,
elastic#202078,
elastic#202079
**Follow-up to:** elastic#211472

## Summary

We're cleaning up and refactoring our existing test plans for prebuilt
rule customization, upgrade, and export/import workflows.

Specifically, this PR:

- Updates the common structure of test plans:
- Adds a tip to each test plan's header to help with navigating all our
test plans for prebuilt rules.
- Introduces a new `Requirements` section to each one and consolidates
assumptions, technical requirements and product requirements under it.
- Each `Product requirements` section now contains relevant user stories
and other product details. In the document flow this is kept close to
the `Scenarios` section.
  - Updates the test plan template according to the updated structure.
- Moves the documentation about customizable and non-customizable rule
fields to the common info document.
- Restores the lost changes from commit
elastic@bde4612.

No "functional" changes have been made to any test plans, such as
adding, removing, or updating any scenarios. The only functional-like
changes were made to the user stories when creating/updating the product
requirements sections.

This PR wraps up the refactoring efforts aimed to prepare the test plans
for being "functionally" changed and improved in follow-up PRs. In the
following PRs, we're going to cover the logic described in
elastic#210358, address any gaps in the
coverage, and improve the scenarios structure and wording.

## Review tip

It might be easier to review this PR commit-by-commit as each of them
contains logically cohesive changes.
  • Loading branch information
banderror authored and JoseLuisGJ committed Feb 27, 2025
1 parent ac8b73f commit 03f4cd7
Show file tree
Hide file tree
Showing 14 changed files with 462 additions and 303 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ Each subdomain folder should be owned by a single GitHub team in the `.github/CO
Some examples for reference:

- [Test plan template](./test_plan_template.md).
- [Installation and Upgrade of Prebuilt Rules](./detection_response/prebuilt_rules/installation_and_upgrade.md).
- [Test plans for prebuilt rules](./detection_response/prebuilt_rules/prebuilt_rules.md).

Feel free to tune the structure whenever it makes sense and improves readability or maintainability of your plan: add more sections to `Useful info`, add more top-level sections in addition to `Useful info` and `Scenarios`, etc.
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).

> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->

This is a test plan for the workflows of customizing prebuilt rules via:
Expand All @@ -24,10 +27,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one

- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Editing prebuilt rules](#editing-prebuilt-rules)
- [**Scenario: User can edit a non-customized prebuilt rule from the rule edit page**](#scenario-user-can-edit-a-non-customized-prebuilt-rule-from-the-rule-edit-page)
Expand Down Expand Up @@ -60,9 +64,35 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Users can Customize Prebuilt Detection Rules: Milestone 3](https://github.com/elastic/kibana/issues/174168)
- [Tests for prebuilt rule customization workflow](https://github.com/elastic/kibana/issues/202068)

### User stories
### Terminology

- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **Rule source**, or **`ruleSource`**: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source.
- **`is_customized`**: a field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version.
- **non-semantic change**: a change to a rule field that is functionally different. We normalize certain fields so for a time-related field such as `from`, `1m` vs `60s` are treated as the same value. We also trim leading and trailing whitespace for query fields.

## Requirements

### Assumptions

Assumptions about test environments and scenarios outlined in this test plan.

- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- Rule package used will have all previous rule versions present (no missing base versions).

### Technical requirements

Non-functional requirements for the functionality outlined in this test plan.

- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).

**Prebuilt rule customization workflow:**
### Product requirements

Functional requirements for the functionality outlined in this test plan.

- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).

User stories:

- User can edit a single prebuilt rule from the Rule Details page.
- User can edit single prebuilt rules one-by-one from the Rule Management page.
Expand All @@ -76,22 +106,6 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- User can see which rules are customized on the Rule Management page in the Upgrade table.
- User can un-customize a prebuilt rule by editing it and reverting its parameters back to their original values.

### Terminology

- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).
- **Rule source**, or **`ruleSource`**: a rule field that defines the rule's origin. Can be `internal` or `external`. Currently, custom rules have `internal` rule source and prebuilt rules have `external` rule source.
- **`is_customized`**: a field within `ruleSource` that exists when rule source is set to `external`. It is a boolean value based on if the rule has been changed from its base version.
- **non-semantic change**: a change to a rule field that is functionally different. We normalize certain fields so for a time-related field such as `from`, `1m` vs `60s` are treated as the same value. We also trim leading and trailing whitespace for query fields.

### Assumptions

- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).
- Rule package used will have all previous rule versions present (no missing base versions)

### Non-functional requirements

- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).

## Scenarios

### Editing prebuilt rules
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).

> [!TIP]
> If you're new to prebuilt rules, get started [here](./prebuilt_rules.md) and check an overview of the features of prebuilt rules in [this section](./prebuilt_rules_common_info.md#features).
## Summary <!-- omit from toc -->

This is a test plan for the workflows of:
Expand All @@ -25,10 +28,11 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one

- [Useful information](#useful-information)
- [Tickets](#tickets)
- [User stories](#user-stories)
- [Terminology](#terminology)
- [Requirements](#requirements)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Technical requirements](#technical-requirements)
- [Product requirements](#product-requirements)
- [Scenarios](#scenarios)
- [Core Functionality](#core-functionality)
- [Scenario: Exporting prebuilt rule individually from rule details page](#scenario-exporting-prebuilt-rule-individually-from-rule-details-page)
Expand All @@ -52,28 +56,38 @@ https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
- [Support exporting prebuilt rules from the Rule Details page](https://github.com/elastic/kibana/issues/180176)
- [Tests for prebuilt rule import/export workflow](https://github.com/elastic/kibana/issues/202079)

### User stories

**Prebuilt rule export workflow:**

- User can export a single prebuilt rule from the Rule Details page.
- User can export multiple prebuilt rules one-by-one from the Rule Management page.
- User can export multiple prebuilt rules in bulk from the Rule Management page via bulk actions.
- User can export prebuilt non-customized rules.
- User can export prebuilt customized rules.
- User can export any combination of prebuilt non-customized, prebuilt customized, and custom rules.

### Terminology

- [Common terminology](./prebuilt_rules_common_info.md#common-terminology).

## Requirements

### Assumptions

Assumptions about test environments and scenarios outlined in this test plan.

- [Common assumptions](./prebuilt_rules_common_info.md#common-assumptions).

### Non-functional requirements
### Technical requirements

- [Common non-functional requirements](./prebuilt_rules_common_info.md#common-non-functional-requirements).
Non-functional requirements for the functionality outlined in this test plan.

- [Common technical requirements](./prebuilt_rules_common_info.md#common-technical-requirements).

### Product requirements

Functional requirements for the functionality outlined in this test plan.

- [Common product requirements](./prebuilt_rules_common_info.md#common-product-requirements).

User stories:

- User can export a single prebuilt rule from the Rule Details page.
- User can export multiple prebuilt rules one-by-one from the Rule Management page.
- User can export multiple prebuilt rules in bulk from the Rule Management page via bulk actions.
- User can export prebuilt non-customized rules.
- User can export prebuilt customized rules.
- User can export any combination of prebuilt non-customized, prebuilt customized, and custom rules.

## Scenarios

Expand Down
Loading

0 comments on commit 03f4cd7

Please sign in to comment.