[Security Solution] Exploratory testing of prebuilt rule customization workflows

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

Do comprehensive exploratory testing of the app with the prebuiltRulesCustomizationEnabled feature flag turned on.

Workflows to test that were directly affected by the prebuilt rule customization functionality:

• Prebuilt rule customization: installing, editing, bulk editing, duplicating prebuilt rules; using the Rule Management and Details pages with customized and non-customized prebuilt rules, as well as custom rules.
• Prebuilt rule upgrade: upgrading customized and non-customized prebuilt rules.
• Rule export and import: exporting and importing customized and non-customized prebuilt rules, as well as custom rules.

Workflows to test for regressions:

• Rule management: creating, editing, bulk editing, duplicating custom rules; using the Rule Management and Details pages with custom rules.
• Rule installation workflow: installing prebuilt rules; using the Rule Management and Details pages.

Advanced testing:

• Disable the prebuiltRulesCustomizationEnabled feature flag after using the app with the flag enabled. Test that the app works without issues and errors, even there are some customized prebuilt rules that were created when the flag was turned on.

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

@MadameSheema please reassign to someone from QA Source when you know who could help us with this.

[pborgonovi]

Prebuilt Rule Customization Test Plan

1. Introduction
   This feature allows users to customize prebuilt rules, import/export them, and upgrade while preserving customizations. Testing will focus on rule customization, the Three-Way-Diff component, conflict resolution, bulk upgrades, and import/export functionality to ensure a reliable and seamless user experience.

2. Scope of Testing
   2.1 In-Scope
   Full Integration System Testing.

Feature	Priority	Risk Level	Testing Conditions	Comments	Issues	Status
Prebuilt Rules Customization	High	Low	Test that prebuilt rules can be customized and upgraded correctly while preserving user modifications.	Validate that updates can be done in bulk and that rule integrity is maintained.	206527, 208251	Done
Rule Management Table Updates	Moderate	Low	Test UI functionality for filtering, selecting, and sorting rules; verify distinction between rule types.	Ensure the UI behaves correctly with bulk actions and different rule types.	206132	Done
Bulk Rule Upgrades	High	Low	Test that bulk upgrades process only conflict-free rules, while conflicts are managed with clear notifications.	Validate this workflow thoroughly, as it’s highly impacted by the new changes.		Done
Upgrade Flyout and Conflict Review	High	Low	Test the Three-Way-Diff component’s handling of conflicts during upgrades.	Ensure accurate display of differences and resolution for both solvable and non-solvable conflicts.	206666, 206695	Done
Auto-Merge of Customizations	High	Low	Test auto-merge functionality to ensure changes from different versions are combined correctly during upgrades.	Focus on preserving customizations while ensuring merged results reflect user preferences.		Done
Rule Type Changes	High	Low	Test rule type changes during upgrades, confirming user warnings and allowing cloning or retaining customizations.	Ensure UI handles type changes clearly and informs users of potential data loss or changes.	206649	Done
Import/Export of Customized Rules	Moderate	Low	Test import/export functionality for preserving customizations when moving rules between instances.	Validate behavior between ESS and Serverless instances, retaining customizations during re-import.		Done
Rule Execution	Moderate	Low	Ensure customized and upgraded rules execute correctly, triggering the intended detections.	This can be tested as part of regression testing to verify minimal impact on execution.		Done
Regression Testing	Moderate	Low	Conduct exploratory regression testing to ensure new features do not disrupt existing functionalities.	Check rule installation, duplication, editing of custom rules, and bulk actions.		Done
License	High	Moderate	Ensure Enterprise and Complete tier users can access customization, while others (Basic/Platinum/Essential) cannot.	Validate behavior during tier/license upgrades or downgrades and ensure compliance with restrictions.		Not Started

2.2 Out-of-Scope

• API testing (since testing will be done exclusively through the UI).
• Performance and load testing

3. Objectives
   • Our main objective is to release this functionality free of significant issues.

4. Test Strategy
   4.1 Testing Approach

   • Exploratory Testing (Risk-Based Priority):
     The primary focus will be on exploratory testing, with testing efforts prioritized according to the risk levels identified.
     High-Risk Areas will receive the most attention to ensure critical functionalities are working correctly.

   Prioritized Areas for Exploratory Testing:

   • High-Risk Areas (Focus First):
     • Prebuilt Rule Customization
     • Bulk Rule Upgrades
     • Rule Type Changes
     • Conflict Resolution Logic (Three-Way-Diff)
     • Auto-Merging of Customizations
     • License

   • Moderate-Risk Areas (Secondary Focus):
     • Rule Execution and Alerts
     • Import/Export of Customized Rules
     • Rule Management Table
     • Regression Testing

   • Regression Testing:

     • Conduct exploratory regression testing across key areas to ensure that new features do not break existing functionalities like rule installation, duplication, editing custom rules from both Rule Management and Rule Details page.
     • Turn flag OFF and validate existing functionalities continue to work as expected.

[banderror]

@pborgonovi Great work, and thank you for this initiative. The plan for exploratory testing looks good so far 👍

A few suggestions:

• I would suggest to try to improve the plan's readability. Currently, we repeat pretty much the same thoughts / focus areas multiple times in the "Testing includes", "Objectives", "Prioritized Areas for Exploratory Testing", and "Risk Matrix" sections. Let's try to consolidate this in a single section "Scope", where we'd have a "In-scope" section with a table listing all testing efforts to do, their priority, risk, comments, etc; and a "Out-of-scope" section that could contain a simple list.
• Our objective is to release the feature in time without any significant issues ☝ 🙂
• I would disagree on some of the priorities:
  • "Rule Execution" and "Alerts Based on Customized Rules", while being core features, is a lower risk because we know how it works under the hood, and the new changes shouldn't affect this. Still, worth testing, of course, but could be a part of regression testing with a bit lower priority.
  • "Bulk Rule Upgrades" should be a high risk and high priority. This workflow is highly affected by the new changes.

[pborgonovi]

Thank you so much for the feedback @banderror I'll be working on it.

[pborgonovi]

Exploratory testing findings

206132 | Impact Medium | Rule Updates Table | Incorrect filtering behavior when combining ‘Modifications’ and ‘Tags’ filters in Rule Updates table
206527 | Impact High | Rule Customization | Prebuilt Rule with query filters incorrectly marked as customized after saving without changes
206649 | Impact Medium | Rule Type Change | Update flyout does not display customized fields for rules with type changes
206655 | Impact Low | Update Table | Tags filter in updates table does not reflect newly added tags
206666 | Impact Medium | Update Flyout | Incorrect “my changes” statement displayed in update flyout for fields with no final column changes
206695 | Impact Low | Update Flyout | Misaligned line below section header in update flyout
208251 | Impact High | Rule Customization | Rule incorrectly retains customized status after reverting MITRE ATT&CK changes

[banderror]

Hey @pborgonovi, would you mind to please add a "Status" column to the table in the plan? I'd like to understand the status of each testing area: todo / in progress / done, what has been done so far, what's left to do, etc. If that makes sense.

Please also consider re-assessing and updating the risk level for each area. We have been progressing towards the release and have fixed a lot of bugs, so probably some of these areas are less risky now than they were a couple months ago, because:

• We fixed dozens of bugs in these areas
• You've done a lot of testing and feel that certain areas are stable

We've been continuously reassessing risk levels for each Milestone 3 area in the current checklist.

[pborgonovi]

Hey @banderror

I've made the updates accordingly. Exploratory testing progress as of now Jan 27th, 2024:

• All testing outlined in the risk matrix has been completed successfully except for License ones. Development task ongoing.
• The Risk Level has been adjusted to Low for most features to reflect their stable implementation and successful testing, while License remains "Moderate" due to the pending work.

[pborgonovi]

1st BC is scheduled for tomorrow Jan, 31st and I’ll begin testing the License functionality.

Here is a high-level view of the test coverage based on the defined requirements.
This table represents only the happy path scenarios and does not include edge cases or negative test cases. Additional testing will be conducted to cover potential unexpected behaviors and edge scenarios.

Scenario	Test Conditions	ECH Status	Serverless Status
Editing Prebuilt Rules	Basic/Platinum/Essentials Tier: Only actions can be modified. Non-editable fields show hover message.	Done	Done
	Enterprise License/Complete Tier: Full rule customization enabled. All fields, including previously locked ones, are editable.	Done	Done
Bulk Actions	Basic/Platinum/Essentials Tier: No modifications allowed. Excluded rules display an explanation.	Done	Done
	Enterprise License/Complete Tier: Users can perform full bulk actions (e.g., adding/removing tags, index patterns) on prebuilt rules.	Done	Done
Import/Export	Basic/Platinum/Essentials Tier: Importing and exporting prebuilt rules is allowed, but importing customized prebuilt rules is not allowed.	Not started	Not started
	Enterprise License/Complete Tier: No restrictions on importing/exporting prebuilt or customized prebuilt rules.	Not started	Not started
Rule Updates	Basic/Platinum/Essentials Tier: Read-only mode. Users cannot edit fields, only update to Elastic’s version.	Done	Done
	Enterprise License/Complete Tier: The Upgrade Flyout allows resolving conflicts using three-way diff. Users can accept, edit, or reject changes to customized prebuilt rules.	Done	Done
Bulk Upgrade Behavior	Basic/Platinum/Essentials Tier: Bulk upgrade applies to all rules, and any rule with customization is reverted to its original state.	Done	Done
	Enterprise License/Complete Tier: Bulk upgrades apply only to rules without conflicts. Rules with upgrade conflicts remain unchanged in the Rule Updates table until conflicts are reviewed and resolved in the Upgrade Flyout.	Done	Done
Downgrading License/Tier	Customized rules remain but become read-only. Prebuilt rules can no longer be customized. Bulk actions allow only adding actions/exceptions. Elastic updates revert customized rules to the prebuilt version. Upgrade conflicts must be resolved by accepting Elastic's version. UI displays messages restricting customization.	Done	Done
Upgrading License/Tier	All prebuilt rules become fully customizable. Previously customized rules remain modified and editable. Bulk actions unlock full modification capabilities. Upgrade Flyout enables conflict resolution. Bulk upgrades apply only to rules without conflicts. UI restrictions (disabled fields, hover messages) are removed.	Done	Done

[banderror]

@xcrzx @approksiu Does the above make sense to you?

[approksiu]

@pborgonovi @banderror the plan looks good to me. Thank you!

[xcrzx]

@pborgonovi The plan looks good, just a couple of comments to make sure we are on the same page.

Editing Prebuilt Rules Basic/Platinum: Actions only Non-editable fields show hover message

For the insufficient license case, on the rule editing page, users can modify actions only. The Actions tab remains accessible, while other tabs are locked. We do not restrict the rule editing page field-by-field.

Bulk Actions Basic/Platinum: No modifications Excluded rules display an explanation

Enable, disable, delete, add rule actions, duplicate, and manual run bulk actions should be allowed. All other bulk actions are not allowed.

Customization After Downgrade Rules retain modifications but cannot be changed UI notification appears

We don’t have a separate downgrade scenario in the code. Rule actions can still be edited after a downgrade, and all bulk actions allowed with a lower license should continue to work. The only difference is that a previously customized rule should remain marked as customized even after applying any action. The customization status should only reset upon upgrading the rule to Elastic's version.

Also, not sure what "UI notification appears" means in this context. Could you clarify?

Import/Export Basic: No importing of customized rules

Non-customized prebuilt rules also cannot be exported or imported on basic license. Import/export functionality depends on whether a rule is prebuilt or custom, not on its customization status.

[pborgonovi]

Thank you for collaboration @xcrzx

Also, not sure what "UI notification appears" means in this context. Could you clarify?

The UI notification in this case refers to the message displayed when a user lacks the necessary privileges to modify a prebuilt rule after downgrading to a non-eligible tier/license. For example, if the user hovers over a disabled edit button, a tooltip should appear explaining that rule customization requires an Enterprise license (for ECH) or a Security Complete subscription (for Serverless).

[banderror]

Non-customized prebuilt rules also cannot be exported or imported on basic license. Import/export functionality depends on whether a rule is prebuilt or custom, not on its customization status.

@xcrzx This will change very soon with https://github.com/elastic/security-team/issues/11502, right?

[xcrzx]

For example, if the user hovers over a disabled edit button, a tooltip should appear explaining that rule customization requires an Enterprise license (for ECH) or a Security Complete subscription (for Serverless).

I don’t think we have this scenario. The Edit button should always be enabled, even when users lack the required license.

[xcrzx]

Non-customized prebuilt rules also cannot be exported or imported on basic license. Import/export functionality depends on whether a rule is prebuilt or custom, not on its customization status.

@xcrzx This will change very soon with elastic/security-team#11502, right?

That's right, we’re planning to change this behavior soonish (before the release).

[pborgonovi]

Exploratory Test Plan – Final Changes Validation

Focused on validating the final changes introduced before release to ensure stability, consistency, and expected behavior across core workflows. This includes exploring critical functionalities, verifying UI adjustments, and confirming that rule-related workflows (installation, customization, updates, and export/import) remain intact.

Area	Test Conditions	Status	Results
Rule Installation	- Verify that prebuilt rules install correctly and are available in the Rule Management table. - Ensure versioning, metadata, and UI updates remain consistent. - Verify that filtering and pagination work as expected.	Not Started	-
Rule Customization	- Modify various fields in prebuilt rules (e.g., name, query, timeline template). - Ensure UI correctly reflects the "Modified" badge and customization indicators. - Validate reverting the changes and ensuring the rule is set back to non-customized. - Validate that adding actions, exceptions, or attempting to edit and save without changes does not set the rule as customized. The rule is only set as customized if any configuration is changed.	Not Started	-
Rule Updates Table	- Validate that only conflict-free rules are available for direct upgrade from the table. - Ensure that rules with conflicts have the "Review update" button instead of "Update," requiring users to open the Upgrade Flyout. - Confirm that filtering by "Modified" and "Unmodified" rules correctly updates the table view. - Validate that rules with a missing base version behave identically to rules with a base version, ensuring solvable conflicts are correctly flagged, requiring an upgrade via the flyout, and preventing direct upgrade from the table.	Not Started	-
Upgrade Flyout	- Validate that the Upgrade Flyout opens correctly for rules requiring conflict resolution. - Ensure that conflicted fields are correctly highlighted and users can accept, edit, or reject changes. - Verify that the three-way diff visualization is accurate and correctly reflects the base, current, and target versions. - Confirm that saving modifications applies changes as expected. - Ensure that discarding changes does not apply modifications and closes the flyout without persisting changes.	Not Started	-
Rule Upgrade	- Upgrade prebuilt rules with and without customizations. - Ensure conflict detection in Rule Updates table and that rules with conflicts require flyout review. - Validate bulk upgrade behavior, ensuring only conflict-free rules are upgraded.	Not Started	-
Import & Export	- Test importing prebuilt rules in different states (customized/non-customized). - Validate behavior when overwriting exceptions and actions. - Ensure that import flags function correctly. - Verify that exported rules maintain integrity across Kibana spaces. - Validate that on import of a non-customized prebuilt rule, the rule is NOT wrongly marked as customized.	Not Started	-
Licensing	- Validate that license/tier restrictions correctly limit rule editing, customization, upgrade, and import/export. - Ensure that prebuilt rules remain read-only where applicable. - Confirm that users see appropriate messages when attempting restricted actions.	Not Started	-
UI & Upgrade Callouts	- Validate that the Rule Management page displays a callout encouraging rule upgrades. - Ensure a callout appears on the Rule Details page prompting users to upgrade. - Confirm users can open the Rule Upgrade flyout directly from the Rule Details page. - Verify a callout is shown on the Rule Editing page when applicable. - Ensure a callout appears in the Rule Upgrade flyout when a rule has a missing base version.	Not Started	-

[banderror]

@pborgonovi Maybe it was assumed, but let's add these items to the plan explicitly:

• Rule Installation:
  • Verify that filtering and pagination works as expected.
• Rule Customization:
  • Test the changed logic of handling missing base versions (see #210358, "Editing of prebuilt rules with missing base versions").
• Rule Updates Table:
  • Verify that filtering and pagination works as expected.
  • Test the changed logic of handling missing base versions (see #210358, "Upgrading of prebuilt rules with missing base versions").
• Import & Export:
  • Test the changed logic of handling missing base versions (see #210358, "Importing of prebuilt rules with missing base versions").
• Licensing:
  • Validate that license/tier restrictions correctly limit rule editing, customization, upgrade, and import/export.

Also, these items should be updated:

• "Validate that only conflict-free rules are available for direct upgrade from the table.". We will allow to bulk upgrade rules with solvable conflicts as well, see #210358.
• "Ensure that rules with conflicts have the "Update" button disabled, requiring users to open the Upgrade Flyout.". The "Update" button won't be disabled, it will be replaced with a "Review update" button.

Otherwise, great plan and thanks for posting it!

[pborgonovi]

@banderror Indeed, the changes related to ticket #210358 were missing. Thanks for pointing that out!
I've made the proper adjustments.