[FR] Add Env Var DR_CLI_MAX_WIDTH and DaC Docs Updates

CLI.md

@@ -44,6 +44,9 @@ Using the environment variable `DR_BYPASS_TAGS_VALIDATION` will bypass the Detec
 
 Using the environment variable `DR_BYPASS_TIMELINE_TEMPLATE_VALIDATION` will bypass the timeline template id and title validation for rules. 
 
+Using the environment variable `DR_CLI_MAX_WIDTH` will set a custom max width for the click CLI. 
+For instance, some users may want to increase the default value in cases where help messages are cut off. 
+
 ## Importing rules into the repo
 
 You can import rules into the repo using the `create-rule` or `import-rules-to-repo` commands. Both of these commands will
@@ -281,7 +284,7 @@ Options:
   -id, --rule-id TEXT
   -r, --replace-id           Replace rule IDs with new IDs before export
   -h, --help                 Show this message and exit.
-(detection-rules-build) (base) ➜  detection-rules git:(rule-loader) ✗
+(detection-rules-build) (base) ➜  detection-rules git:(main) ✗
 ```
 
 Alternatively, rules can be exported into a consolidated ndjson file which can be imported in the Kibana security app
@@ -621,7 +624,7 @@ Unknown field
 data_stream.dataset:osquery_manager.result and osquery_meta.counter>0 and osquery_meta.type:diff and osquery.last_run_code:0 and osquery_meta.action:removed
                                                                           ^^^^^^^^^^^^^^^^^
 stack: 8.9.0, beats: 8.9.0, ecs: 8.9.0
-- name - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗
+- name - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}(venv312) ➜  detection-rules-fork git:(main) ✗
 ```
 
 
@@ -670,7 +673,7 @@ method specific to the transform type.
 This applies to osquery and insights for the moment but could expand in the future.
 
 ```
-(venv38) ➜  detection-rules-fork git:(2597-validate-osquery-insights) python -m detection_rules dev transforms -h
+(venv312) ➜  detection-rules-fork git:(main) python -m detection_rules dev transforms -h
 
 █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
 █  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
@@ -692,7 +695,7 @@ Commands:
 
 
 ```
-(venv38) ➜  detection-rules-fork git:(2597-validate-osquery-insights) python -m detection_rules dev transforms guide-plugin-convert
+(venv312) ➜  detection-rules-fork git:(main) python -m detection_rules dev transforms guide-plugin-convert
 
 █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
 █  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄

README.md

@@ -16,6 +16,7 @@ This repository was first announced on Elastic's blog post, [Elastic Security op
   - [Overview of this repository](#overview-of-this-repository)
   - [Getting started](#getting-started)
   - [How to contribute](#how-to-contribute)
+  - [Detections as Code (DaC)](#detections-as-code-dac)
   - [RTAs](#rtas)
   - [Licensing](#licensing)
   - [Questions? Problems? Suggestions?](#questions-problems-suggestions)
@@ -133,11 +134,16 @@ For more advanced command line interface (CLI) usage, refer to the [CLI guide](C
 
 We welcome your contributions to Detection Rules! Before contributing, please familiarize yourself with this repository, its [directory structure](#overview-of-this-repository), and our [philosophy](PHILOSOPHY.md) about rule creation. When you're ready to contribute, read the [contribution guide](CONTRIBUTING.md) to learn how we turn detection ideas into production rules and validate with testing.
 
+## Detections as Code (DaC)
+
+The Detection Rules repo includes a number of commands to help one manage rules with an "as code" philosophy. We recommend starting with our [DaC Specific Documentation](https://dac-reference.readthedocs.io/en/latest/) for strategies and recommended setup information. However, if you would prefer to jump right in, please see our [custom rules documentation](docs/custom-rules.md) for information on how to configure this repo for use with custom rules followed by our [CLI documentation](CLI.md) for information on our commands to import and export rules.
+
 ## RTAs
 
 Red Team Automations (RTAs) used to emulate attacker techniques and verify the rules can be found in dedicated
 repository - [Cortado](https://github.com/elastic/cortado).
 
+
 ## Licensing
 
 Everything in this repository — rules, code, etc. — is licensed under the [Elastic License v2](LICENSE.txt). These rules are designed to be used in the context of the Detection Engine within the Elastic Security application. If you’re using our [Elastic Cloud managed service](https://www.elastic.co/cloud/) or the default distribution of the Elastic Stack software that includes the [full set of free features](https://www.elastic.co/subscriptions), you’ll get the latest rules the first time you navigate to the detection engine.

detection_rules/custom_rules.py

@@ -8,11 +8,10 @@
 
 import click
 import yaml
+from semver import Version
 
 from .main import root
-from .utils import get_etc_path, load_etc_dump, ROOT_DIR
-
-from semver import Version
+from .utils import ROOT_DIR, get_etc_path, load_etc_dump
 
 DEFAULT_CONFIG_PATH = Path(get_etc_path('_config.yaml'))
 CUSTOM_RULES_DOC_PATH = Path(ROOT_DIR).joinpath('docs', 'custom-rules.md')

detection_rules/main.py

@@ -40,7 +40,13 @@
 RULES_DIRS = RULES_CONFIG.rule_dirs
 
 
-@click.group('detection-rules', context_settings={'help_option_names': ['-h', '--help']})
+@click.group(
+    'detection-rules',
+    context_settings={
+        'help_option_names': ['-h', '--help'],
+        'max_content_width': int(os.getenv('DR_CLI_MAX_WIDTH', 240)),
+    },
+)
 @click.option('--debug/--no-debug', '-D/-N', is_flag=True, default=None,
               help='Print full exception stacktrace on errors')
 @click.pass_context

docs/detections-as-code.md

@@ -0,0 +1,57 @@
+# Detection as Code (DaC) Components in Detection-Rules Repo
+
+The **detection-rules** repository contains features for **Detections as Code (DaC)**. These components, including CLI options and workflows, provide methods to help apply DaC principles in practice. The specific DaC architecture should be carefully considered before implementation, for more conceptual ideas and details about DaC, refer to the [DaC Documentation](https://dac-reference.readthedocs.io/en/latest/). Reference implementation is shared to facilitate experimentation and community contributions.
+
+> [!NOTE]
+> This guidance outlines the support scope and best practices for using DaC components within the detection-rules repo. Users should take full responsibility for their usage of this repo, thoroughly test these tools in their environments, and verify functionality before using them.
+
+---
+
+## Support and Scope
+
+Supported DaC components that interact with the Elastic Security Solution:
+- kibana export-rules ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#exporting-rules))
+- kibana import-rules ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#using-import-rules))
+- import-rules-to-repo ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#import-rules-to-repo))
+- export-rules-from-repo ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#uploading-rules-to-kibana))
+
+We welcome general questions, feature requests, and bug reports through the following channels:
+- **GitHub Issues**: For raising general questions, bugs, and feature requests related to the detection-rules repo.  
+  [GitHub Issues](https://github.com/elastic/detection-rules/issues)
+- **Community Slack**: For informal discussions or questions (note that message history is limited to 30 days).  
+  [Elastic Security Community Slack](https://elasticstack.slack.com/archives/C06TE19EP09)
+
+Support tickets related to this DaC reference implementation can be opened with Elastic, however since the logic is just a wrapper of the underlying product API's, we will not be able to treat DaC related support requests as a Severity-1 (Immedediate time frame). Urgent DaC issues can be resolved via direct interaction with the underlying Kibana API or Elastic Security Solution UI. We will ask customers to resort to these methods to resolve downtime or urgent issues.
+
+> [!TIP]
+> Questions about Kibana API usage should be directed to the Kibana repository:  
+> [Kibana Issues](https://github.com/elastic/kibana/issues)
+
+---
+
+## Feature Requests
+
+Feature requests for the DaC components that interact with the Elastic Security Solution (`kibana export-rules`, `kibana import-rules`, `import-rules-to-repo`, and `export-rules-from-repo`) in this repository will be handled similarly to the rest of the detection-rules repo:
+- **Prioritization**: Feature requests will be prioritized along with other development work in the repository.
+- **Schema Updates**: If there are breaking schema changes in Kibana that affect importing/exporting detection rules, those changes will be prioritized.
+- **Rule Engine API**: Current CLI tools leverage the rules engine API, and improvements to this will be treated as part of the ongoing development.
+---
+
+## Reference Implementation of DaC Components
+
+DaC is not a single tool. Detection as Code (DaC) is a modern security approach that applies software development best practices to the creation, management, and deployment of security rules. Here is a short summary of several components that extend upon Elastic's rule management capabilities (e.g. query validation, schema validation, unit tests, etc.) provided to help fast track users ability to implement custom DaC implementations in their private environments. If you are new to these concepts, please refer to the [DaC Documentation](https://dac-reference.readthedocs.io/en/latest/), which also provides a quickstart guide and example end-to-end CI/CD workflows.  These components are configurable by using the [custom-rules](docs-dev/custom-rules.md) setup.
+
+- Kibana's Rule Versioning Mechanism ([link](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-2-defer-to-elastic-security))
+- Local rule management (e.g. autoschema generation, actions and exceptions) ([link](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-1-using-the-built-in-configuration))
+
+---
+
+## Best Practices for Using DaC Components
+
+When implementing DaC in your production environment, follow these best practices:
+
+- **Design and Test Rigorously**: Since every DaC implementation will be user-specific, remember to diligently design, and thoroughly test the tools before deploying them in a production environment.
+- **Version Compatibility**: Before upgrading the detection-rules repo version, ensure that you test compatibility with your environment. For more information, see our [Versioning Documentation](https://github.com/elastic/ia-trade-team/issues/471%23issuecomment-2423259800).
+- **Limited Backward Compatibility**: We do not guarantee backward compatibility across versions for rule schemas. While we aim to make new fields optional where feasible, there may be minimum version requirements for Elastic Stack and are subject to Kibana's rule schema definitions.
+- **Schema Parity**: Not all fields in the schema defined in Kibana are fully supported. Some fields in the detection-rules repo are generalized (e.g., `field = dict()`), while others are more strictly defined. This is due to the complexity of the schemas and the prioritization of Elastic's internal use cases.
+

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "detection_rules"
-version = "0.4.17"
+version = "0.4.18"
 description = "Detection Rules is the home for rules used by Elastic Security. This repository is used for the development, maintenance, testing, validation, and release of rules for Elastic Security’s Detection Engine."
 readme = "README.md"
 requires-python = ">=3.12"