Deprecate upload-rule

eric-forte-elastic · eric-forte-elastic · commit 40fda60bda38 · 2025-03-07T13:22:55.000-05:00
diff --git a/CLI.md b/CLI.md
@@ -130,28 +130,6 @@ Commands which connect to Elasticsearch or Kibana are embedded under the subcomm
 These command groups will leverage their respective clients and will automatically use parsed config options if
 defined, otherwise arguments should be passed to the sub-command as:
 
-`python -m detection-rules kibana -u <username> -p <password> upload-rule <...>`
-
-
-```console
-python -m detection_rules es -h
-
-Usage: detection_rules es [OPTIONS] COMMAND [ARGS]...
-
-  Commands for integrating with Elasticsearch.
-
-Options:
-  -et, --timeout INTEGER        Timeout for elasticsearch client
-  -ep, --es-password TEXT
-  -eu, --es-user TEXT
-  --cloud-id TEXT
-  -e, --elasticsearch-url TEXT
-  -h, --help                    Show this message and exit.
-
-Commands:
-  collect-events  Collect events from Elasticsearch.
-```
-
 Providers are the name that Elastic Cloud uses to configure authentication in Kibana. When we create deployment, Elastic Cloud configures two providers by default: basic/cloud-basic and saml/cloud-saml (for SSO).
 
 ```console
@@ -168,26 +146,21 @@ Usage: detection_rules kibana [OPTIONS] COMMAND [ARGS]...
 Options:
   --ignore-ssl-errors TEXT
   --space TEXT                 Kibana space
-  --provider-name TEXT         For cloud deployments, Elastic Cloud configures
-                               two providers by default: cloud-basic and
-                               cloud-saml (for SSO)
-  --provider-type TEXT         For cloud deployments, Elastic Cloud configures
-                               two providers by default: basic and saml (for
-                               SSO)
+  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-saml (for SSO)
+  --provider-type TEXT         Elastic Cloud providers: basic and saml (for SSO)
   -ku, --kibana-user TEXT
   --kibana-url TEXT
   -kp, --kibana-password TEXT
   -kc, --kibana-cookie TEXT    Cookie from an authed session
-  --cloud-id TEXT              ID of the cloud instance. Defaults the cloud
-                               provider to cloud-basic if this option is
-                               supplied
+  --api-key TEXT
+  --cloud-id TEXT              ID of the cloud instance.
   -h, --help                   Show this message and exit.
 
 Commands:
   export-rules   Export custom rules from Kibana.
   import-rules   Import custom rules into Kibana.
   search-alerts  Search detection engine alerts with KQL.
-  upload-rule    Upload a list of rule .toml files to Kibana.
+  upload-rule    [Deprecated] Upload a list of rule .toml files to Kibana.
 ```
 
 ## Searching Kibana for Alerts
@@ -198,23 +171,22 @@ Alerts stored in Kibana can be quickly be identified by searching with the `sear
 ```console
 python -m detection_rules kibana search-alerts -h
 
+█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
+█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
+█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
+
 Kibana client:
 Options:
   --ignore-ssl-errors TEXT
   --space TEXT                 Kibana space
-  --provider-name TEXT         For cloud deployments, Elastic Cloud configures
-                               two providers by default: cloud-basic and
-                               cloud-saml (for SSO)
-  --provider-type TEXT         For cloud deployments, Elastic Cloud configures
-                               two providers by default: basic and saml (for
-                               SSO)
+  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-saml (for SSO)
+  --provider-type TEXT         Elastic Cloud providers: basic and saml (for SSO)
   -ku, --kibana-user TEXT
   --kibana-url TEXT
   -kp, --kibana-password TEXT
   -kc, --kibana-cookie TEXT    Cookie from an authed session
-  --cloud-id TEXT              ID of the cloud instance. Defaults the cloud
-                               provider to cloud-basic if this option is
-                               supplied
+  --api-key TEXT
+  --cloud-id TEXT              ID of the cloud instance.
 
 Usage: detection_rules kibana search-alerts [OPTIONS] [QUERY]
 
@@ -224,8 +196,8 @@ Options:
   -d, --date-range <TEXT TEXT>...
                                   Date range to scope search
   -c, --columns TEXT              Columns to display in table
-  -e, --extend                    If columns are specified, extend the
-                                  original columns
+  -e, --extend                    If columns are specified, extend the original columns
+  -m, --max-count INTEGER         The max number of alerts to return
   -h, --help                      Show this message and exit.
 ```
 
@@ -257,70 +229,9 @@ Running the following command will print out a table showing any alerts that hav
 ```
 ## Uploading rules to Kibana
 
-Toml formatted rule files can be uploaded as custom rules using the `kibana upload-rule` command. To upload more than one
-file, specify multiple files at a time as individual args. This command is meant to support uploading and testing of
-rules and is not intended for production use in its current state.
+### Using `kibana import-rules`
 
-This command is built on soon to be deprecated APIs and so should be phased off. For a better option, see below...
-
-```console
-python -m detection_rules kibana upload-rule -h
-
-Kibana client:
-Options:
-  --space TEXT                 Kibana space
-  -kp, --kibana-password TEXT
-  -ku, --kibana-user TEXT
-  --cloud-id TEXT
-  -k, --kibana-url TEXT
-
-Usage: detection_rules kibana upload-rule [OPTIONS]
-
-  Upload a list of rule .toml files to Kibana.
-
-Options:
-  -f, --rule-file FILE
-  -d, --directory DIRECTORY  Recursively export rules from a directory
-  -id, --rule-id TEXT
-  -r, --replace-id           Replace rule IDs with new IDs before export
-  -h, --help                 Show this message and exit.
-(detection-rules-build) (base) ➜  detection-rules git:(main) ✗
-```
-
-Alternatively, rules can be exported into a consolidated ndjson file which can be imported in the Kibana security app
-directly.
-
-```console
-Usage: detection_rules export-rules-from-repo [OPTIONS]
-
-  Export rule(s) and exception(s) into an importable ndjson file.
-
-Options:
-  -f, --rule-file FILE
-  -d, --directory DIRECTORY       Recursively load rules from a directory
-  -id, --rule-id TEXT
-  -o, --outfile PATH              Name of file for exported rules
-  -r, --replace-id                Replace rule IDs with new IDs before export
-  --stack-version [7.8|7.9|7.10|7.11|7.12|7.13|7.14|7.15|7.16|8.0|8.1|8.2|8.3|8.4|8.5|8.6|8.7|8.8|8.9|8.10|8.11|8.12|8.13|8.14]
-                                  Downgrade a rule version to be compatible
-                                  with older instances of Kibana
-  -s, --skip-unsupported          If `--stack-version` is passed, skip rule
-                                  types which are unsupported (an error will
-                                  be raised otherwise)
-  --include-metadata              Add metadata to the exported rules
-  -ac, --include-action-connectors
-                                  Include Action Connectors in export
-  -e, --include-exceptions        Include Exceptions Lists in export
-  -h, --help                      Show this message and exit.
-```
-
-_*To load a custom rule, the proper index must be setup first. The simplest way to do this is to click
-the `Load prebuilt detection rules and timeline templates` button on the `detections` page in the Kibana security app._
-
-
-### Using `import-rules`
-
-This is a better option than `upload-rule` as it is built on refreshed APIs
+To directly load Toml formatted rule files into Kibana, one can use the `kibana import-rules` command as shown below.
 
 ```
 python -m detection_rules kibana import-rules -h
@@ -333,10 +244,8 @@ Kibana client:
 Options:
   --ignore-ssl-errors TEXT
   --space TEXT                 Kibana space
-  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-
-                               saml (for SSO)
-  --provider-type TEXT         Elastic Cloud providers: basic and saml (for
-                               SSO)
+  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-saml (for SSO)
+  --provider-type TEXT         Elastic Cloud providers: basic and saml (for SSO)
   -ku, --kibana-user TEXT
   --kibana-url TEXT
   -kp, --kibana-password TEXT
@@ -355,8 +264,7 @@ Options:
   -o, --overwrite                 Overwrite existing rules
   -e, --overwrite-exceptions      Overwrite exceptions in existing rules
   -ac, --overwrite-action-connectors
-                                  Overwrite action connectors in existing
-                                  rules
+                                  Overwrite action connectors in existing rules
   -h, --help                      Show this message and exit.
 ```
 
@@ -499,6 +407,78 @@ python -m detection_rules kibana import-rules -d test-export-rules -o
 
 </details>
 
+### Using `export-rules-from-repo`
+
+Toml formatted rule files can also be imported into Kibana through Kibana security app via a consolidated ndjson file
+which is exported from detection rules.
+
+```console
+Usage: detection_rules export-rules-from-repo [OPTIONS]
+
+  Export rule(s) and exception(s) into an importable ndjson file.
+
+Options:
+  -f, --rule-file FILE
+  -d, --directory DIRECTORY       Recursively load rules from a directory
+  -id, --rule-id TEXT
+  -o, --outfile PATH              Name of file for exported rules
+  -r, --replace-id                Replace rule IDs with new IDs before export
+  --stack-version [7.8|7.9|7.10|7.11|7.12|7.13|7.14|7.15|7.16|8.0|8.1|8.2|8.3|8.4|8.5|8.6|8.7|8.8|8.9|8.10|8.11|8.12|8.13|8.14]
+                                  Downgrade a rule version to be compatible
+                                  with older instances of Kibana
+  -s, --skip-unsupported          If `--stack-version` is passed, skip rule
+                                  types which are unsupported (an error will
+                                  be raised otherwise)
+  --include-metadata              Add metadata to the exported rules
+  -ac, --include-action-connectors
+                                  Include Action Connectors in export
+  -e, --include-exceptions        Include Exceptions Lists in export
+  -h, --help                      Show this message and exit.
+```
+
+_*To load a custom rule, the proper index must be setup first. The simplest way to do this is to click
+the `Load prebuilt detection rules and timeline templates` button on the `detections` page in the Kibana security app._
+
+
+### Deprecated Methods
+
+Toml formatted rule files can also be uploaded as custom rules using the `kibana upload-rule` command. This command is 
+deprecated as of Elastic Stack version 9.0, but is included for compatibility with older stacks. To upload more than one
+file, specify multiple files at a time as individual args. This command is meant to support uploading and testing of
+rules and is not intended for production use in its current state.
+
+```console
+python -m detection_rules kibana upload-rule -h
+
+█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
+█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
+█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
+
+Kibana client:
+Options:
+  --ignore-ssl-errors TEXT
+  --space TEXT                 Kibana space
+  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-saml (for SSO)
+  --provider-type TEXT         Elastic Cloud providers: basic and saml (for SSO)
+  -ku, --kibana-user TEXT
+  --kibana-url TEXT
+  -kp, --kibana-password TEXT
+  -kc, --kibana-cookie TEXT    Cookie from an authed session
+  --api-key TEXT
+  --cloud-id TEXT              ID of the cloud instance.
+
+Usage: detection_rules kibana upload-rule [OPTIONS]
+
+  [Deprecated] Upload a list of rule .toml files to Kibana.
+
+Options:
+  -f, --rule-file FILE
+  -d, --directory DIRECTORY  Recursively load rules from a directory
+  -id, --rule-id TEXT
+  -r, --replace-id           Replace rule IDs with new IDs before export
+  -h, --help                 Show this message and exit.
+```
+
 ### Exporting rules
 
 This command should be run with the `CUSTOM_RULES_DIR` envvar set, that way proper validation is applied to versioning when the rules are downloaded. See the [custom rules docs](docs/custom-rules.md) for more information.
@@ -514,10 +494,8 @@ Kibana client:
 Options:
   --ignore-ssl-errors TEXT
   --space TEXT                 Kibana space
-  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-
-                               saml (for SSO)
-  --provider-type TEXT         Elastic Cloud providers: basic and saml (for
-                               SSO)
+  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-saml (for SSO)
+  --provider-type TEXT         Elastic Cloud providers: basic and saml (for SSO)
   -ku, --kibana-user TEXT
   --kibana-url TEXT
   -kp, --kibana-password TEXT
diff --git a/detection_rules/kbwrap.py b/detection_rules/kbwrap.py
@@ -51,10 +51,15 @@ def kibana_group(ctx: click.Context, **kibana_kwargs):
 @click.option('--replace-id', '-r', is_flag=True, help='Replace rule IDs with new IDs before export')
 @click.pass_context
 def upload_rule(ctx, rules: RuleCollection, replace_id):
-    """Upload a list of rule .toml files to Kibana."""
+    """[Deprecated] Upload a list of rule .toml files to Kibana."""
     kibana = ctx.obj['kibana']
     api_payloads = []
 
+    click.secho(
+        "WARNING: This command is deprecated as of Elastic Stack version 9.0. Please use `kibana import-rules`.",
+        fg="yellow",
+    )
+
     for rule in rules:
         try:
             payload = downgrade_contents_from_rule(rule, kibana.version, replace_id=replace_id)
diff --git a/detection_rules/ml.py b/detection_rules/ml.py
@@ -383,7 +383,7 @@ def setup_bundle(ctx, model_tag, repo, model_dir):
     click.echo(table)
 
     click.echo('Associated rules and jobs can be found under ML-experimental-detections releases in the repo')
-    click.echo('To upload rules, run: kibana upload-rule <ml-rule.toml>')
+    click.echo('To upload rules, run: kibana import-rules -f <ml-rule.toml>')
     click.echo('To upload ML jobs, run: es experimental upload-ml-job <ml-job.json>')
 
 
diff --git a/docs/experimental-machine-learning/experimental-detections.md b/docs/experimental-machine-learning/experimental-detections.md
@@ -14,7 +14,7 @@ Rules are now stored in ndjson format and can be imported into Kibana via the se
 
 Earlier releases stored the rules in toml format. These can be uploaded using the 
 [7.12 branch](https://github.com/elastic/detection-rules/tree/7.12) CLI using the 
-[kibana upload-rule](../../CLI.md#uploading-rules-to-kibana) command
+[kibana import-rules](../../CLI.md#uploading-rules-to-kibana) command
 
 ### Uploading ML Jobs and Datafeeds
 
diff --git a/generate_rules.py b/generate_rules.py
@@ -0,0 +1,33 @@
+import glob
+import os
+import pytoml as toml
+import pandas as pd
+
+# List to store file paths and rule IDs
+data = []
+
+# Iterate over all TOML files in the directory and its subdirectories
+for file in glob.glob('rules/**/*.toml', recursive=True):
+    # Load the TOML file
+    print(f"Processing {file}")
+    try:
+        with open(file, 'r') as f:
+            toml_data = toml.load(f)
+        
+        # Extract the rule ID
+        rule_id = toml_data.get('rule', {}).get('rule_id', None)
+        
+        # Append the file path and rule ID to the list
+        if rule_id:
+            data.append({'file_path': file, 'rule_id': rule_id})
+    except (toml.TomlError, IndexError) as e:
+        print(f"Error processing {file}: {e}")
+
+# Create a pandas DataFrame from the list
+df = pd.DataFrame(data)
+
+# Write the DataFrame to a CSV file
+df.to_csv('rules_with_ids.csv', index=False)
+
+print("CSV file 'rules_with_ids.csv' has been created.")
+
