Skip to content

Commit 27fcbde

Browse files
terrancedejesustradebot-elastic
authored andcommitted
[Rule Tuning] ESQL Query Field Dynamic Field Standardization (#4912)
* adjusted Potential Widespread Malware Infection Across Multiple Hosts * adjusted Microsoft Azure or Mail Sign-in from a Suspicious Source * adjusted AWS EC2 Multi-Region DescribeInstances API Calls * adjusted AWS Discovery API Calls via CLI from a Single Resource * adjusted AWS Service Quotas Multi-Region Requests * adjusted AWS EC2 EBS Snapshot Shared or Made Public * adjusted AWS S3 Bucket Enumeration or Brute Force * adjusted AWS EC2 EBS Snapshot Access Removed * adjusted Potential AWS S3 Bucket Ransomware Note Uploaded * adjusted AWS S3 Object Encryption Using External KMS Key * adjusted AWS S3 Static Site JavaScript File Uploaded * adjusted AWS Access Token Used from Multiple Addresses * adjusted AWS Signin Single Factor Console Login with Federated User * adjusted AWS IAM AdministratorAccess Policy Attached to Group * adjusted AWS IAM AdministratorAccess Policy Attached to Role * adjusted AWS IAM AdministratorAccess Policy Attached to User * adjusted AWS Bedrock Invocations without Guardrails Detected by a Single User Over a Session * adjusted AWS Bedrock Guardrails Detected Multiple Violations by a Single User Over a Session * adjusted AWS Bedrock Guardrails Detected Multiple Policy Violations Within a Single Blocked Request * adjusted Unusual High Confidence Content Filter Blocks Detected * adjusted Potential Abuse of Resources by High Token Count and Large Response Sizes * AWS Bedrock Detected Multiple Attempts to use Denied Models by a Single User * Unusual High Denied Sensitive Information Policy Blocks Detected * adjusted Unusual High Denied Topic Blocks Detected * adjusted AWS Bedrock Detected Multiple Validation Exception Errors by a Single User * adjusted Unusual High Word Policy Blocks Detected * adjusted Microsoft Entra ID Concurrent Sign-Ins with Suspicious Properties * adjusted Azure Entra MFA TOTP Brute Force Attempts * adjusted Microsoft Entra ID Sign-In Brute Force Activity * adjusted Microsoft Entra ID Exccessive Account Lockouts Detected * adjusted Microsoft 365 Brute Force via Entra ID Sign-Ins * deprecated Azure Entra Sign-in Brute Force Microsoft 365 Accounts by Repeat Source * adjusted Microsoft Entra ID Session Reuse with Suspicious Graph Access * adjusted Suspicious Microsoft OAuth Flow via Auth Broker to DRS * adjusted Potential Denial of Azure OpenAI ML Service * adjusted Azure OpenAI Insecure Output Handling * adjusted Potential Azure OpenAI Model Theft * adjusted M365 OneDrive Excessive File Downloads with OAuth Token * adjusted Multiple Microsoft 365 User Account Lockouts in Short Time Window * adjusted Potential Microsoft 365 User Account Brute Force * adjusted Suspicious Microsoft 365 UserLoggedIn via OAuth Code * adjusted Multiple Device Token Hashes for Single Okta Session * adjusted Multiple Okta User Authentication Events with Client Address * adjusted Multiple Okta User Authentication Events with Same Device Token Hash * adjusted High Number of Okta Device Token Cookies Generated for Authentication * adjusted Okta User Sessions Started from Different Geolocations * adjusted High Number of Egress Network Connections from Unusual Executable * adjusted Unusual Base64 Encoding/Decoding Activity * adjusted Potential Port Scanning Activity from Compromised Host * adjusted Potential Subnet Scanning Activity from Compromised Host * adjusted Unusual File Transfer Utility Launched * adjusted Potential Malware-Driven SSH Brute Force Attempt * adjusted Unusual Process Spawned from Web Server Parent * adjusted Unusual Command Execution from Web Server Parent * adjusted Rare Connection to WebDAV Target * adjusted Potential PowerShell Obfuscation via Invalid Escape Sequences * adjusted Potential PowerShell Obfuscation via Backtick-Escaped Variable Expansion * adjusted Unusual File Creation by Web Server * adjusted Potential PowerShell Obfuscation via High Special Character Proportion * adjusted Potential Malicious PowerShell Based on Alert Correlation * adjusted Potential PowerShell Obfuscation via Character Array Reconstruction * adjusted Potential PowerShell Obfuscation via String Reordering * adjusted Potential PowerShell Obfuscation via String Concatenation * adjusted Potential PowerShell Obfuscation via Reverse Keywords * adjusted PowerShell Obfuscation via Negative Index String Reversal * adjusted Dynamic IEX Reconstruction via Method String Access * adjusted Potential Dynamic IEX Reconstruction via Environment Variables * adjusted Potential PowerShell Obfuscation via High Numeric Character Proportion * adjusted Potential PowerShell Obfuscation via Concatenated Dynamic Command Invocation * adjusted Rare Connection to WebDAV Target * adjusted Potential PowerShell Obfuscation via Invalid Escape Sequences * adjusted Potential PowerShell Obfuscation via Backtick-Escaped Variable Expansion * adjusted Potential PowerShell Obfuscation via Character Array Reconstruction * adjusted Potential PowerShell Obfuscation via High Special Character Proportion * adjusted Potential PowerShell Obfuscation via Special Character Overuse * adjusted Potential PowerShell Obfuscation via String Reordering * adjusted Suspicious Microsoft 365 UserLoggedIn via OAuth Code * adjusted fields that were inconsistent * adjusted additional fields * adjusted esql to Esql * adjusted several rules for common field names * updating rules * updated dates * updated dates * updated ESQL fields * lowercase all functions and logical operators * adjusted dates for unit tests * Update Esql_priv to Esql_temp as these don't hold PII * PowerShell adjustments * Make query comments consistent * update comment * reverted 2856446a-34e6-435b-9fb5-f8f040bfa7ed * Update rules/windows/discovery_command_system_account.toml * removed dot notation --------- Co-authored-by: Jonhnathan <[email protected]> (cherry picked from commit b28338c)
1 parent e51d87a commit 27fcbde

File tree

74 files changed

+2540
-1508
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

74 files changed

+2540
-1508
lines changed

rules/integrations/azure/credential_access_entra_signin_brute_force_microsoft_365_repeat_source.toml renamed to rules/_deprecated/credential_access_entra_signin_brute_force_microsoft_365_repeat_source.toml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
[metadata]
22
creation_date = "2024/09/06"
3+
deprecation_date = "2025/07/16"
34
integration = ["azure"]
4-
maturity = "production"
5-
updated_date = "2025/06/06"
5+
maturity = "deprecated"
6+
updated_date = "2025/07/16"
67

78
[rule]
89
author = ["Elastic"]

rules/cross-platform/execution_potential_widespread_malware_infection.toml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
[metadata]
22
creation_date = "2024/05/08"
33
maturity = "production"
4-
updated_date = "2025/03/20"
4+
updated_date = "2025/07/16"
55

66
[rule]
77
author = ["Elastic"]
@@ -67,8 +67,8 @@ query = '''
6767
from logs-endpoint.alerts-*
6868
| where event.code in ("malicious_file", "memory_signature", "shellcode_thread") and rule.name is not null
6969
| keep host.id, rule.name, event.code
70-
| stats hosts = count_distinct(host.id) by rule.name, event.code
71-
| where hosts >= 3
70+
| stats Esql.host_id_count_distinct = count_distinct(host.id) by rule.name, event.code
71+
| where Esql.host_id_count_distinct >= 3
7272
'''
7373

7474

rules/integrations/aws/discovery_ec2_multi_region_describe_instances.toml

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
creation_date = "2024/08/26"
33
integration = ["aws"]
44
maturity = "production"
5-
updated_date = "2025/01/10"
5+
updated_date = "2025/07/16"
66

77
[rule]
88
author = ["Elastic"]
@@ -89,30 +89,35 @@ query = '''
8989
from logs-aws.cloudtrail-*
9090
9191
// filter for DescribeInstances API calls
92-
| where event.dataset == "aws.cloudtrail" and event.provider == "ec2.amazonaws.com" and event.action == "DescribeInstances"
92+
| where event.dataset == "aws.cloudtrail"
93+
and event.provider == "ec2.amazonaws.com"
94+
and event.action == "DescribeInstances"
9395
9496
// truncate the timestamp to a 30-second window
95-
| eval target_time_window = DATE_TRUNC(30 seconds, @timestamp)
97+
| eval Esql.time_window_date_trunc = date_trunc(30 seconds, @timestamp)
9698
97-
// keep only the relevant fields
98-
| keep target_time_window, aws.cloudtrail.user_identity.arn, cloud.region
99+
// keep only the relevant raw fields
100+
| keep Esql.time_window_date_trunc, aws.cloudtrail.user_identity.arn, cloud.region
99101
100102
// count the number of unique regions and total API calls within the 30-second window
101-
| stats region_count = count_distinct(cloud.region), window_count = count(*) by target_time_window, aws.cloudtrail.user_identity.arn
103+
| stats
104+
Esql.cloud_region_count_distinct = count_distinct(cloud.region),
105+
Esql.event_count = count(*)
106+
by Esql.time_window_date_trunc, aws.cloudtrail.user_identity.arn
102107
103108
// filter for resources making DescribeInstances API calls in more than 10 regions within the 30-second window
104-
| where region_count >= 10 and window_count >= 10
109+
| where Esql.cloud_region_count_distinct >= 10 and Esql.event_count >= 10
105110
106-
// sort the results by time windows in descending order
107-
| sort target_time_window desc
111+
// sort the results by time window in descending order
112+
| sort Esql.time_window_date_trunc desc
108113
'''
109114

110115
[rule.investigation_fields]
111116
field_names = [
112-
"aws.cloudtrail.user_identity.arn",
113-
"target_time_window",
114-
"region_count",
115-
"window_count"
117+
"aws.cloudtrail.user_identity.arn",
118+
"target_time_window",
119+
"region_count",
120+
"window_count"
116121
]
117122

118123
[[rule.threat]]

rules/integrations/aws/discovery_ec2_multiple_discovery_api_calls_via_cli.toml

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
creation_date = "2024/11/04"
33
integration = ["aws"]
44
maturity = "production"
5-
updated_date = "2025/03/20"
5+
updated_date = "2025/07/16"
66

77
[rule]
88
author = ["Elastic"]
@@ -83,11 +83,11 @@ query = '''
8383
from logs-aws.cloudtrail*
8484
8585
// create time window buckets of 10 seconds
86-
| eval time_window = date_trunc(10 seconds, @timestamp)
86+
| eval Esql.time_window_date_trunc = date_trunc(10 seconds, @timestamp)
8787
| where
8888
event.dataset == "aws.cloudtrail"
8989
90-
// filter on CloudTrail audit logs for IAM, EC2, and S3 events only
90+
// filter on CloudTrail audit logs for IAM, EC2, S3, etc.
9191
and event.provider in (
9292
"iam.amazonaws.com",
9393
"ec2.amazonaws.com",
@@ -97,8 +97,7 @@ from logs-aws.cloudtrail*
9797
"dynamodb.amazonaws.com",
9898
"kms.amazonaws.com",
9999
"cloudfront.amazonaws.com",
100-
"elasticloadbalancing.amazonaws.com",
101-
"cloudfront.amazonaws.com"
100+
"elasticloadbalancing.amazonaws.com"
102101
)
103102
104103
// ignore AWS service actions
@@ -117,19 +116,24 @@ from logs-aws.cloudtrail*
117116
starts_with(event.action, "List"),
118117
starts_with(event.action, "Generate")
119118
)
119+
120120
// extract owner, identity type, and actor from the ARN
121-
| dissect aws.cloudtrail.user_identity.arn "%{}::%{owner}:%{identity_type}/%{actor}"
122-
| where starts_with(actor, "AWSServiceRoleForConfig") != true
123-
| keep @timestamp, time_window, event.action, aws.cloudtrail.user_identity.arn
121+
| dissect aws.cloudtrail.user_identity.arn "%{}::%{Esql_priv.aws_cloudtrail_user_identity_arn_owner}:%{Esql.aws_cloudtrail_user_identity_arn_type}/%{Esql.aws_cloudtrail_user_identity_arn_roles}"
122+
| where starts_with(Esql.aws_cloudtrail_user_identity_arn_roles, "AWSServiceRoleForConfig") != true
123+
124+
// keep relevant fields (preserving ECS fields and computed time window)
125+
| keep @timestamp, Esql.time_window_date_trunc, event.action, aws.cloudtrail.user_identity.arn
126+
127+
// count the number of unique API calls per time window and actor
124128
| stats
125-
// count the number of unique API calls per time window and actor
126-
unique_api_count = count_distinct(event.action) by time_window, aws.cloudtrail.user_identity.arn
129+
Esql.event_action_count_distinct = count_distinct(event.action)
130+
by Esql.time_window_date_trunc, aws.cloudtrail.user_identity.arn
127131
128-
// filter for more than 5 unique API calls per time window
129-
| where unique_api_count > 5
132+
// filter for more than 5 unique API calls per 10s window
133+
| where Esql.event_action_count_distinct > 5
130134
131135
// sort the results by the number of unique API calls in descending order
132-
| sort unique_api_count desc
136+
| sort Esql.event_action_count_distinct desc
133137
'''
134138

135139

rules/integrations/aws/discovery_servicequotas_multi_region_service_quota_requests.toml

Lines changed: 61 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
[metadata]
22
creation_date = "2024/08/26"
33
maturity = "production"
4-
updated_date = "2025/01/15"
4+
updated_date = "2025/07/16"
55

66
[rule]
77
author = ["Elastic"]
@@ -15,52 +15,6 @@ from = "now-9m"
1515
language = "esql"
1616
license = "Elastic License v2"
1717
name = "AWS Service Quotas Multi-Region `GetServiceQuota` Requests"
18-
references = [
19-
"https://www.sentinelone.com/labs/exploring-fbot-python-based-malware-targeting-cloud-and-payment-services/",
20-
"https://docs.aws.amazon.com/servicequotas/2019-06-24/apireference/API_GetServiceQuota.html",
21-
]
22-
risk_score = 21
23-
rule_id = "19be0164-63d2-11ef-8e38-f661ea17fbce"
24-
severity = "low"
25-
tags = [
26-
"Domain: Cloud",
27-
"Data Source: AWS",
28-
"Data Source: Amazon Web Services",
29-
"Data Source: AWS Service Quotas",
30-
"Use Case: Threat Detection",
31-
"Tactic: Discovery",
32-
"Resources: Investigation Guide",
33-
]
34-
timestamp_override = "event.ingested"
35-
type = "esql"
36-
37-
query = '''
38-
from logs-aws.cloudtrail-*
39-
40-
// filter for GetServiceQuota API calls
41-
| where event.dataset == "aws.cloudtrail" and event.provider == "servicequotas.amazonaws.com" and event.action == "GetServiceQuota"
42-
43-
// truncate the timestamp to a 30-second window
44-
| eval target_time_window = DATE_TRUNC(30 seconds, @timestamp)
45-
46-
// pre-process the request parameters to extract the service code and quota code
47-
| dissect aws.cloudtrail.request_parameters "{%{?service_code_key}=%{service_code}, %{?quota_code_key}=%{quota_code}}"
48-
49-
// filter for EC2 service quota L-1216C47A (vCPU on-demand instances)
50-
| where service_code == "ec2" and quota_code == "L-1216C47A"
51-
52-
// keep only the relevant fields
53-
| keep target_time_window, aws.cloudtrail.user_identity.arn, cloud.region, service_code, quota_code
54-
55-
// count the number of unique regions and total API calls within the 30-second window
56-
| stats region_count = count_distinct(cloud.region), window_count = count(*) by target_time_window, aws.cloudtrail.user_identity.arn
57-
58-
// filter for resources making DescribeInstances API calls in more than 10 regions within the 30-second window
59-
| where region_count >= 10 and window_count >= 10
60-
61-
// sort the results by time windows in descending order
62-
| sort target_time_window desc
63-
'''
6418
note = """## Triage and analysis
6519
6620
> **Disclaimer**:
@@ -95,6 +49,66 @@ AWS Service Quotas manage resource limits across AWS services, crucial for maint
9549
- Notify the security operations team and relevant stakeholders about the potential compromise and the steps being taken to remediate the issue.
9650
- If evidence of compromise is confirmed, consider engaging AWS Support or a third-party incident response team for further investigation and assistance.
9751
- Review and update IAM policies and permissions to ensure the principle of least privilege is enforced, reducing the risk of future unauthorized access attempts."""
52+
references = [
53+
"https://www.sentinelone.com/labs/exploring-fbot-python-based-malware-targeting-cloud-and-payment-services/",
54+
"https://docs.aws.amazon.com/servicequotas/2019-06-24/apireference/API_GetServiceQuota.html",
55+
]
56+
risk_score = 21
57+
rule_id = "19be0164-63d2-11ef-8e38-f661ea17fbce"
58+
severity = "low"
59+
tags = [
60+
"Domain: Cloud",
61+
"Data Source: AWS",
62+
"Data Source: Amazon Web Services",
63+
"Data Source: AWS Service Quotas",
64+
"Use Case: Threat Detection",
65+
"Tactic: Discovery",
66+
"Resources: Investigation Guide",
67+
]
68+
timestamp_override = "event.ingested"
69+
type = "esql"
70+
71+
query = '''
72+
from logs-aws.cloudtrail-*
73+
74+
// filter for GetServiceQuota API calls
75+
| where
76+
event.dataset == "aws.cloudtrail"
77+
and event.provider == "servicequotas.amazonaws.com"
78+
and event.action == "GetServiceQuota"
79+
80+
// truncate the timestamp to a 30-second window
81+
| eval Esql.time_window_date_trunc = date_trunc(30 seconds, @timestamp)
82+
83+
// dissect request parameters to extract service and quota code
84+
| dissect aws.cloudtrail.request_parameters "{%{?Esql.aws_cloudtrail_request_parameters_service_code_key}=%{Esql.aws_cloudtrail_request_parameters_service_code}, %{?quota_code_key}=%{Esql.aws_cloudtrail_request_parameters_quota_code}}"
85+
86+
// filter for EC2 service quota L-1216C47A (vCPU on-demand instances)
87+
| where Esql.aws_cloudtrail_request_parameters_service_code == "ec2" and Esql.aws_cloudtrail_request_parameters_quota_code == "L-1216C47A"
88+
89+
// keep only the relevant fields
90+
| keep
91+
Esql.time_window_date_trunc,
92+
aws.cloudtrail.user_identity.arn,
93+
cloud.region,
94+
Esql.aws_cloudtrail_request_parameters_service_code,
95+
Esql.aws_cloudtrail_request_parameters_quota_code
96+
97+
// count the number of unique regions and total API calls within the time window
98+
| stats
99+
Esql.cloud_region_count_distinct = count_distinct(cloud.region),
100+
Esql.event_count = count(*)
101+
by Esql.time_window_date_trunc, aws.cloudtrail.user_identity.arn
102+
103+
// filter for API calls in more than 10 regions within the 30-second window
104+
| where
105+
Esql.cloud_region_count_distinct >= 10
106+
and Esql.event_count >= 10
107+
108+
// sort by time window descending
109+
| sort Esql.time_window_date_trunc desc
110+
'''
111+
98112

99113
[[rule.threat]]
100114
framework = "MITRE ATT&CK"

rules/integrations/aws/exfiltration_ec2_ebs_snapshot_shared_with_another_account.toml

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
creation_date = "2024/04/16"
33
integration = ["aws"]
44
maturity = "production"
5-
updated_date = "2025/06/02"
5+
updated_date = "2025/07/16"
66

77
[rule]
88
author = ["Elastic"]
@@ -21,8 +21,7 @@ interval = "5m"
2121
language = "esql"
2222
license = "Elastic License v2"
2323
name = "AWS EC2 EBS Snapshot Shared or Made Public"
24-
note = """
25-
## Triage and analysis
24+
note = """## Triage and analysis
2625
2726
### Investigating AWS EC2 EBS Snapshot Shared or Made Public
2827
@@ -81,10 +80,31 @@ type = "esql"
8180

8281
query = '''
8382
from logs-aws.cloudtrail-* metadata _id, _version, _index
84-
| where event.provider == "ec2.amazonaws.com" and event.action == "ModifySnapshotAttribute" and event.outcome == "success"
85-
| dissect aws.cloudtrail.request_parameters "{%{?snapshotId}=%{snapshotId},%{?attributeType}=%{attributeType},%{?createVolumePermission}={%{operationType}={%{?items}=[{%{?userId}=%{userId}}]}}}"
86-
| where operationType == "add" and cloud.account.id != userId
87-
| keep @timestamp, aws.cloudtrail.user_identity.arn, cloud.account.id, event.action, snapshotId, attributeType, operationType, userId, source.address
83+
| where
84+
event.provider == "ec2.amazonaws.com"
85+
and event.action == "ModifySnapshotAttribute"
86+
and event.outcome == "success"
87+
88+
// Extract snapshotId, attribute type, operation type, and userId
89+
| dissect aws.cloudtrail.request_parameters
90+
"{%{?snapshotId}=%{Esql.aws_cloudtrail_request_parameters_snapshot_id},%{?attributeType}=%{Esql.aws_cloudtrail_request_parameters_attribute_type},%{?createVolumePermission}={%{Esql.aws_cloudtrail_request_parameters_operation_type}={%{?items}=[{%{?userId}=%{Esql_priv.aws_cloudtrail_request_parameters_user_id}}]}}}"
91+
92+
// Check for snapshot permission added for another AWS account
93+
| where
94+
Esql.aws_cloudtrail_request_parameters_operation_type == "add"
95+
and cloud.account.id != Esql_priv.aws_cloudtrail_request_parameters_user_id
96+
97+
// keep ECS and derived fields
98+
| keep
99+
@timestamp,
100+
aws.cloudtrail.user_identity.arn,
101+
cloud.account.id,
102+
event.action,
103+
Esql.aws_cloudtrail_request_parameters_snapshot_id,
104+
Esql.aws_cloudtrail_request_parameters_attribute_type,
105+
Esql.aws_cloudtrail_request_parameters_operation_type,
106+
Esql_priv.aws_cloudtrail_request_parameters_user_id,
107+
source.ip
88108
'''
89109

90110

rules/integrations/aws/impact_aws_s3_bucket_enumeration_or_brute_force.toml

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
[metadata]
22
creation_date = "2024/05/01"
33
maturity = "production"
4-
updated_date = "2025/03/20"
4+
updated_date = "2025/07/16"
55

66
[rule]
77
author = ["Elastic"]
@@ -86,13 +86,29 @@ type = "esql"
8686

8787
query = '''
8888
from logs-aws.cloudtrail*
89-
| where event.provider == "s3.amazonaws.com" and aws.cloudtrail.error_code == "AccessDenied"
90-
// keep only relevant fields
91-
| keep tls.client.server_name, source.address, cloud.account.id
92-
| stats failed_requests = count(*) by tls.client.server_name, source.address, cloud.account.id
93-
// can modify the failed request count or tweak time window to fit environment
94-
// can add `not cloud.account.id in (KNOWN)` or specify in exceptions
95-
| where failed_requests > 40
89+
90+
| where
91+
event.provider == "s3.amazonaws.com"
92+
and aws.cloudtrail.error_code == "AccessDenied"
93+
and tls.client.server_name is not null
94+
and cloud.account.id is not null
95+
96+
// keep only relevant ECS fields
97+
| keep
98+
tls.client.server_name,
99+
source.address,
100+
cloud.account.id
101+
102+
// count access denied requests per server_name, source, and account
103+
| stats
104+
Esql.event_count = count(*)
105+
by
106+
tls.client.server_name,
107+
source.address,
108+
cloud.account.id
109+
110+
// Threshold: more than 40 denied requests
111+
| where Esql.event_count > 40
96112
'''
97113

98114

0 commit comments

Comments
 (0)