[FR] Add Env Var DR_CLI_MAX_WIDTH and DaC Docs Updates (#4518)

* Add Env Var DR_CLI_MAX_WIDTH

* Version Bump

* Update limit from 120 to 240

* Clean references to reference main

* Update Readme with DaC Info

* Add DaC to Table of Contents

* Bump Patch Version

* Updated naming and add dac md

* Organize Imports

* Deprecate upload-rule

* Update docs/detections-as-code.md

Co-authored-by: Mika Ayenson, PhD <[email protected]>

* move docs to docs-dev

* Sort custom rules imports

* Remove duplicate

* Fix typo

* Bump Patch Version

---------

Co-authored-by: Mika Ayenson, PhD <[email protected]>

Author: eric-forte-elastic

CLI.md

@@ -16,6 +16,7 @@ This repository was first announced on Elastic's blog post, [Elastic Security op
   - [Overview of this repository](#overview-of-this-repository)
   - [Getting started](#getting-started)
   - [How to contribute](#how-to-contribute)
+  - [Detections as Code (DaC)](#detections-as-code-dac)
   - [RTAs](#rtas)
   - [Licensing](#licensing)
   - [Questions? Problems? Suggestions?](#questions-problems-suggestions)
@@ -133,11 +134,16 @@ For more advanced command line interface (CLI) usage, refer to the [CLI guide](C
 
 We welcome your contributions to Detection Rules! Before contributing, please familiarize yourself with this repository, its [directory structure](#overview-of-this-repository), and our [philosophy](PHILOSOPHY.md) about rule creation. When you're ready to contribute, read the [contribution guide](CONTRIBUTING.md) to learn how we turn detection ideas into production rules and validate with testing.
 
+## Detections as Code (DaC)
+
+The Detection Rules repo includes a number of commands to help one manage rules with an "as code" philosophy. We recommend starting with our [DaC Specific Documentation](https://dac-reference.readthedocs.io/en/latest/) for strategies and recommended setup information. However, if you would prefer to jump right in, please see our [custom rules documentation](docs-dev/custom-rules-management.md) for information on how to configure this repo for use with custom rules followed by our [CLI documentation](CLI.md) for information on our commands to import and export rules.
+
 ## RTAs
 
 Red Team Automations (RTAs) used to emulate attacker techniques and verify the rules can be found in dedicated
 repository - [Cortado](https://github.com/elastic/cortado).
 
+
 ## Licensing
 
 Everything in this repository — rules, code, etc. — is licensed under the [Elastic License v2](LICENSE.txt). These rules are designed to be used in the context of the Detection Engine within the Elastic Security application. If you’re using our [Elastic Cloud managed service](https://www.elastic.co/cloud/) or the default distribution of the Elastic Stack software that includes the [full set of free features](https://www.elastic.co/subscriptions), you’ll get the latest rules the first time you navigate to the detection engine.

README.md

@@ -8,12 +8,11 @@
 
 import click
 import yaml
+from semver import Version
 
-from .main import root
-from .utils import get_etc_path, load_etc_dump, ROOT_DIR
 from .docs import REPO_DOCS_DIR
-
-from semver import Version
+from .main import root
+from .utils import ROOT_DIR, get_etc_path, load_etc_dump
 
 DEFAULT_CONFIG_PATH = Path(get_etc_path('_config.yaml'))
 CUSTOM_RULES_DOC_PATH = Path(ROOT_DIR).joinpath(REPO_DOCS_DIR, 'custom-rules.md')

detection_rules/custom_rules.py

@@ -51,10 +51,15 @@ def kibana_group(ctx: click.Context, **kibana_kwargs):
 @click.option('--replace-id', '-r', is_flag=True, help='Replace rule IDs with new IDs before export')
 @click.pass_context
 def upload_rule(ctx, rules: RuleCollection, replace_id):
-    """Upload a list of rule .toml files to Kibana."""
+    """[Deprecated] Upload a list of rule .toml files to Kibana."""
     kibana = ctx.obj['kibana']
     api_payloads = []
 
+    click.secho(
+        "WARNING: This command is deprecated as of Elastic Stack version 9.0. Please use `kibana import-rules`.",
+        fg="yellow",
+    )
+
     for rule in rules:
         try:
             payload = downgrade_contents_from_rule(rule, kibana.version, replace_id=replace_id)

detection_rules/kbwrap.py

@@ -40,7 +40,13 @@
 RULES_DIRS = RULES_CONFIG.rule_dirs
 
 
-@click.group('detection-rules', context_settings={'help_option_names': ['-h', '--help']})
+@click.group(
+    'detection-rules',
+    context_settings={
+        'help_option_names': ['-h', '--help'],
+        'max_content_width': int(os.getenv('DR_CLI_MAX_WIDTH', 240)),
+    },
+)
 @click.option('--debug/--no-debug', '-D/-N', is_flag=True, default=None,
               help='Print full exception stacktrace on errors')
 @click.pass_context

detection_rules/main.py

@@ -383,7 +383,7 @@ def setup_bundle(ctx, model_tag, repo, model_dir):
     click.echo(table)
 
     click.echo('Associated rules and jobs can be found under ML-experimental-detections releases in the repo')
-    click.echo('To upload rules, run: kibana upload-rule <ml-rule.toml>')
+    click.echo('To upload rules, run: kibana import-rules -f <ml-rule.toml>')
     click.echo('To upload ML jobs, run: es experimental upload-ml-job <ml-job.json>')
 
 

detection_rules/ml.py

@@ -0,0 +1,57 @@
+# Detection as Code (DaC) Components in Detection-Rules Repo
+
+The **detection-rules** repository contains features for **Detections as Code (DaC)**. These components, including CLI options and workflows, provide methods to help apply DaC principles in practice. The specific DaC architecture should be carefully considered before implementation, for more conceptual ideas and details about DaC, refer to the [DaC Documentation](https://dac-reference.readthedocs.io/en/latest/). Reference implementation is shared to facilitate experimentation and community contributions.
+
+> [!NOTE]
+> This guidance outlines the support scope and best practices for using DaC components within the detection-rules repo. Users should take full responsibility for their usage of this repo, thoroughly test these tools in their environments, and verify functionality before using them.
+
+---
+
+## Support and Scope
+
+Supported DaC components that interact with the Elastic Security Solution:
+- kibana export-rules ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#exporting-rules))
+- kibana import-rules ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#using-import-rules))
+- import-rules-to-repo ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#import-rules-to-repo))
+- export-rules-from-repo ([link](https://github.com/elastic/detection-rules/blob/main/CLI.md#uploading-rules-to-kibana))
+
+We welcome general questions, feature requests, and bug reports through the following channels:
+- **GitHub Issues**: For raising general questions, bugs, and feature requests related to the detection-rules repo.  
+  [GitHub Issues](https://github.com/elastic/detection-rules/issues)
+- **Community Slack**: For informal discussions or questions (note that message history is limited to 30 days).  
+  [Elastic Security Community Slack](https://elasticstack.slack.com/archives/C06TE19EP09)
+
+Support tickets related to this DaC reference implementation can be opened with Elastic, however since the logic is just a wrapper of the underlying product API's, we ask to resolve urgent DaC issues via direct interaction with the underlying [Kibana APIs](https://www.elastic.co/docs/api/doc/kibana/v8/group/endpoint-security-detections-api) or [Elastic Security Solution UI](https://www.elastic.co/guide/en/security/current/rules-ui-management.html), as we will not be able to treat DaC related support requests as a high severity (immediate time frame).
+
+> [!TIP]
+> Questions about Kibana API usage should be directed to the Kibana repository:  
+> [Kibana Issues](https://github.com/elastic/kibana/issues)
+
+---
+
+## Feature Requests
+
+Feature requests for the DaC components that interact with the Elastic Security Solution (`kibana export-rules`, `kibana import-rules`, `import-rules-to-repo`, and `export-rules-from-repo`) in this repository will be handled similarly to the rest of the detection-rules repo:
+- **Prioritization**: Feature requests will be prioritized along with other development work in the repository.
+- **Schema Updates**: If there are breaking schema changes in Kibana that affect importing/exporting detection rules, those changes will be prioritized.
+- **Rule Engine API**: Current CLI tools leverage the rules engine API, and improvements to this will be treated as part of the ongoing development.
+---
+
+## Reference Implementation of DaC Components
+
+DaC is not a single tool. Detection as Code (DaC) is a modern security approach that applies software development best practices to the creation, management, and deployment of security rules. Here is a short summary of several components that extend upon Elastic's rule management capabilities (e.g. query validation, schema validation, unit tests, etc.) provided to help fast track users ability to implement custom DaC implementations in their private environments. If you are new to these concepts, please refer to the [DaC Documentation](https://dac-reference.readthedocs.io/en/latest/), which also provides a quickstart guide and example end-to-end CI/CD workflows.  These components are configurable by using the [custom-rules](docs-dev/custom-rules.md) setup.
+
+- Kibana's Rule Versioning Mechanism ([link](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-2-defer-to-elastic-security))
+- Local rule management (e.g. autoschema generation, actions and exceptions) ([link](https://dac-reference.readthedocs.io/en/latest/internals_of_the_detection_rules_repo.html#option-1-using-the-built-in-configuration))
+
+---
+
+## Best Practices for Using DaC Components
+
+When implementing DaC in your production environment, follow these best practices:
+
+- **Design and Test Rigorously**: Since every DaC implementation will be user-specific, remember to diligently design, and thoroughly test the tools before deploying them in a production environment.
+- **Version Compatibility**: Before upgrading the detection-rules repo version, ensure that you test compatibility with your environment. For more information, see our [Versioning Documentation](https://github.com/elastic/ia-trade-team/issues/471%23issuecomment-2423259800).
+- **Limited Backward Compatibility**: We do not guarantee backward compatibility across versions for rule schemas. While we aim to make new fields optional where feasible, there may be minimum version requirements for Elastic Stack and are subject to Kibana's rule schema definitions.
+- **Schema Parity**: Not all fields in the schema defined in Kibana are fully supported. Some fields in the detection-rules repo are generalized (e.g., `field = dict()`), while others are more strictly defined. This is due to the complexity of the schemas and the prioritization of Elastic's internal use cases.
+

docs-dev/custom-rules.md docs-dev/custom-rules-management.mddocs-dev/custom-rules.md renamed to docs-dev/custom-rules-management.md

@@ -14,7 +14,7 @@ Rules are now stored in ndjson format and can be imported into Kibana via the se
 
 Earlier releases stored the rules in toml format. These can be uploaded using the 
 [7.12 branch](https://github.com/elastic/detection-rules/tree/7.12) CLI using the 
-[kibana upload-rule](../../CLI.md#uploading-rules-to-kibana) command
+[kibana import-rules](../../CLI.md#uploading-rules-to-kibana) command
 
 ### Uploading ML Jobs and Datafeeds
 

docs-dev/detections-as-code.md

@@ -1,6 +1,6 @@
 [project]
 name = "detection_rules"
-version = "0.4.18"
+version = "0.4.19"
 description = "Detection Rules is the home for rules used by Elastic Security. This repository is used for the development, maintenance, testing, validation, and release of rules for Elastic Security’s Detection Engine."
 readme = "README.md"
 requires-python = ">=3.12"