Add info to hasThreatMatch and threatlookup operators

jpipkin1 · jpipkin1 · commit e506bfa44878 · 2025-02-13T16:42:33.000-06:00
diff --git a/docs/cse/rules/cse-rules-syntax.md b/docs/cse/rules/cse-rules-syntax.md
@@ -633,41 +633,55 @@ The `hasThreatMatch` Cloud SIEM rules function searches incoming records in Clou
 `hasThreatMatch([<fields>], <filters>, <indicators>)`
 
 Parameters:
-* `<fields>` is a list of comma separated entity field names. At least one field name is required.
-* `<filters>` is a logical expression using indicator attributes. (Allowed are parentheses `()`; `OR` and `AND` boolean operators; and comparison operators `=`, `<`, `>`, `=<`, `=>`, `!=`.)
+* `<fields>` is a list of comma-separated [field names](https://github.com/SumoLogic/cloud-siem-content-catalog/blob/master/schema/full_schema.md). At least one field name is required.
+* `<filters>` is a logical expression using [indicator attributes](/docs/security/threat-intelligence/upload-formats/#normalized-json-format). Allowed in the filtering are parentheses `()`; `OR` and `AND` boolean operators; and comparison operators `=`, `<`, `>`, `=<`, `=>`, `!=`. <br/>You can filter on the following indicator attributes:
+   * `actors`
+   * `confidence`
+   * `id`
+   * `indicator`
+   * `killChain`
+   * `source`
+   * `threatType`
+   * `type`
+   * `validFrom`
+   * `validUntil`
 * `<indicators>` is an optional case insensitive option that describes how indicators should be matched with regard to their validity. Accepted values are:
    * `active_indicators`. Match active indicators only (default).
    * `expired_indicators`. Match expired indicators only.
    * `all_indicators`. Match all indicators.
 
 **Examples**
 
-* `hasThreatMatch([srcDevice_ip, dstDevice_ip], confidence > 50)`
-* `hasThreatMatch([srcDevice_ip], confidence > 50 AND source="FreeTAXII")`
-* `hasThreatMatch([srcDevice_ip], source="s1" OR (source="s2" AND confidence > 50))`
-* `hasThreatMatch([srcDevice_ip, dstDevice_ip], confidence > 50, expired_indicators)`
+* `hasThreatMatch([srcDevice_ip])`
+* `hasThreatMatch([srcDevice_ip, dstDevice_ip])`
+* `hasThreatMatch([srcDevice_ip], type="ipv4-addr")`
+* `hasThreatMatch([srcDevice_ip], confidence > 50)`
+* `hasThreatMatch([srcDevice_ip], confidence > 50 AND source="TAXII2Source")`
+* `hasThreatMatch([srcDevice_ip], source="s1" OR (source="s2" confidence > 50))`
+* `hasThreatMatch([srcDevice_ip], expired_indicators)`
+* `hasThreatMatch([srcDevice_ip], confidence > 50, all_indicators)`
 
 #### Best practice
 
 As a best practice, always include filtering to narrow your search to just the types desired (that is, `type=`). This will ensure that your search results are not overly broad.
 
 For example:
-* `hasThreatMatch([dstDevice_ip], confidence > 1 AND (type ='ipv4-addr:value' OR type='ipv6-addr:value'))` 
-* `hasThreatMatch([file_hash_imphash,file_hash_md5,file_hash_pehash,file_hash_ssdeep,file_hash_sha1,file_hash_sha256], confidence > 1 AND type = 'file:hashes')` 
-* `hasThreatMatch([device_hostname, srcDevice_hostname, dstDevice_hostname, http_hostname, http_referrerHostname, bro_ssl_serverName, bro_ntlm_domainame, bro_ssl_serverName_rootDomain, dns_queryDomain, dns_replyDomain, fromUser_authDomain, http_referrerDomain, http_url_rootDomain, http_url_fqdn], confidence > 1 AND (type='domain-name:value' OR type='url'))` 
-* `hasThreatMatch([http_url], confidence > 1 AND type='url')` 
-* `hasThreatMatch([srcDevice_ip], confidence > 1 AND (type ='ipv4-addr:value' OR type='ipv6-addr:value'))`
+* `hasThreatMatch([dstDevice_ip], confidence > 1 AND (type="ipv4-addr" OR type="ipv6-addr"))` 
+* `hasThreatMatch([file_hash_imphash, file_hash_md5, file_hash_pehash, file_hash_ssdeep, file_hash_sha1, file_hash_sha256], confidence > 1 AND type="file:hashes")` 
+* `hasThreatMatch([device_hostname, srcDevice_hostname, dstDevice_hostname, http_hostname, http_referrerHostname, bro_ssl_serverName, bro_ntlm_domainame, bro_ssl_serverName_rootDomain, dns_queryDomain, dns_replyDomain, fromUser_authDomain, http_referrerDomain, http_url_rootDomain, http_url_fqdn], confidence > 1 AND (type="domain-name" OR type="url"))` 
+* `hasThreatMatch([http_url], confidence > 1 AND type="url")` 
+* `hasThreatMatch([srcDevice_ip], confidence > 1 AND (type="ipv4-addr" OR type="ipv6-addr"))`
 
 Following are the standard indicator types you can filter on:
-* `domain-name:value`. Domain name. 
-* `email-addr:value`. Email address. 
+* `domain-name`. Domain name. 
+* `email-addr`. Email address. 
 * `file:hashes`. File hash. 
 * `file:name`. File name. 
-* `ipv4-addr:value`. IPv4 IP address. 
-* `ipv6-addr:value`. IPv6 IP address. 
-* `mac-addr:value`. Mac address name. 
+* `ipv4-addr`. IPv4 IP address. 
+* `ipv6-addr`. IPv6 IP address. 
+* `mac-addr`. Mac address name. 
 * `process:name`. Process name. 
-* `url:value`. URL. 
+* `url`. URL. 
 * `user-account:user-id`. User ID. 
 * `user-account:login`. Login name. 
 
diff --git a/docs/integrations/amazon-aws/waf.md b/docs/integrations/amazon-aws/waf.md
@@ -65,7 +65,7 @@ _sourceCategory=AWS/WAF {{client_ip}}
 _sourceCategory=AWS/WAF {{client_ip}}
 | parse "\"httpMethod\":\"*\"," as httpMethod,"\"httpVersion\":\"*\"," as httpVersion,"\"uri\":\"*\"," as uri, "{\"clientIp\":\"*\",\"country\":\"*\"" as clientIp,country, "\"action\":\"*\"" as action, "\"matchingNonTerminatingRules\":[*]" as matchingNonTerminatingRules, "\"rateBasedRuleList\":[*]" as rateBasedRuleList, "\"ruleGroupList\":[*]" as ruleGroupList, "\"httpSourceId\":\"*\"" as httpSourceId, "\"httpSourceName\":\"*\"" as httpSourceName, "\"terminatingRuleType\":\"*\"" as terminatingRuleType, "\"terminatingRuleId\":\"*\"" as terminatingRuleId, "\"webaclId\":\"*\"" as webaclId nodrop
 | threatlookup singleIndicator clientip
-| where (_threatlookup.type="ipv4-addr:value" or _threatlookup.type="ipv6-addr:value") and !isNull(_threatlookup.confidence)
+| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
 ```
 -->
 
diff --git a/docs/integrations/security-threat-detection/threat-intel-quick-analysis.md b/docs/integrations/security-threat-detection/threat-intel-quick-analysis.md
@@ -111,7 +111,7 @@ Use [Field Extraction Rules (FER)](/docs/manage/field-extractions/create-field-e
    ```
    | threatlookup singleIndicator src_ip
    | parse regex field=%"_threatlookup.fields" "labels.[^.]+.name\":\"(?<label_name>[^\"]+)\""  multi
-   | where (_threatlookup.type="ipv4-addr:value" or _threatlookup.type="ipv6-addr:value") and !isNull(_threatlookup.confidence)
+   | where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
    | if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor
    | count as threat_count by src_ip, malicious_confidence, Actor,  _source, label_name
    | sort by threat_count
@@ -138,7 +138,7 @@ Use scheduled views with the threat lookup operator to find threats. Scheduled v
     _sourceCategory=cylance
     | threatlookup singleIndicator src_ip
     | parse regex field=%"_threatlookup.fields" "labels.[^.]+.name\":\"(?<label_name>[^\"]+)\""  multi
-    | where (_threatlookup.type="ipv4-addr:value" or _threatlookup.type="ipv6-addr:value") and !isNull(_threatlookup.confidence)
+    | where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
     | if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor
     | lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://default on ip = src_ip
     | count as threat_count by src_ip, malicious_confidence, Actor,  _source,  label_name, city, country_name, raw
@@ -183,7 +183,7 @@ Yes, you can customize the query in the app. For example:
 _sourceCategory= */*/FIREWALL or _sourceCategory=*/*/LB or _sourceCategory=*/*/ROUTER or _sourceCategory=*/*/WINDOWS or _sourceCategory=*/*/SERVER
 | where Your_IP != "0.0.0.0" and Your_IP != "127.0.0.1"
 | threatlookup singleIndicator Your_IP
-| where (_threatlookup.type="ipv4-addr:value" or _threatlookup.type="ipv6-addr:value") and !isNull(_threatlookup.confidence)
+| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
 | if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor
 | count by Actor
 ```
@@ -207,7 +207,7 @@ _sourceCategory= */*/FIREWALL or _sourceCategory=*/*/LB or _sourceCategory=*/*/R
 | where ip_address != "0.0.0.0" and ip_address != "127.0.0.1"
 | threatlookup singleIndicator ip_address
 | parse regex field=%"_threatlookup.fields" "labels.[^.]+.name\":\"(?<label_name>[^\"]+)\""  multi
-| where (_threatlookup.type="ipv4-addr:value" or _threatlookup.type="ipv6-addr:value") and !isNull(_threatlookup.confidence)
+| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
 | where !(label_name matches "*TorProxy*")
 | if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor
 | if (_threatlookup.confidence >= 85, "high", if (_threatlookup.confidence >= 50, "medium", if (_threatlookup.confidence >= 15, "low", if (_threatlookup.confidence >= 0, "unverified", "Unknown")))) as malicious_confidence
diff --git a/docs/observability/aws/integrations/aws-dynamodb.md b/docs/observability/aws/integrations/aws-dynamodb.md
@@ -77,7 +77,7 @@ _sourceCategory=Labs/AWS/DynamoDB account=* namespace=* "\"eventSource\":\"dynam
 | where ip_address != "0.0.0.0" and ip_address != "127.0.0.1"
 | count as ip_count by ip_address
 | threatlookup singleIndicator ip_address
-| where (_threatlookup.type="ipv4-addr:value" or _threatlookup.type="ipv6-addr:value") and !isNull(_threatlookup.confidence)
+| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
 | if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor
 | sum (ip_count) as threat_count
 ```
diff --git a/docs/search/search-query-language/search-operators/threatlookup.md b/docs/search/search-query-language/search-operators/threatlookup.md
@@ -4,16 +4,16 @@ title: threatlookup Search Operator
 sidebar_label: threatlookup
 ---
 
-The `threatlookup` search operator allows you to search logs for matches in [threat intelligence indicators](/docs/security/threat-intelligence/threat-intelligence-indicators/), providing security analytics to help you to detect threats in your environment. 
+The `threatlookup` search operator allows you to search logs for matches in [threat intelligence indicators](/docs/security/threat-intelligence/threat-intelligence-indicators/), providing security analytics to help you to detect threats in your environment.
 
 :::note
 You can also use the [`threatip`](/docs/search/search-query-language/search-operators/threatip/) search operator to search CrowdStrike's threat intelligence data based on IP addresses. 
 :::
 
-### Syntax
+## Syntax
 
 ```
-threatlookup [singleIndicator] [source="<source_value>"] [include="<all|active|expired>"] <indicator_value_field> [,<optional_indicator_value_field_2>, …]
+threatlookup [singleIndicator] [source="<source_value>"] [include="<all|active|expired>"] <indicator> [,<optional_indicator>, …]
 ```
 
 Where:
@@ -29,24 +29,37 @@ Where:
 
 * `source` is the source to search for the threat intelligence indicator. If `source` is not specified, all sources are searched.
 * `include` includes either all, only active, or only expired threat intelligence indicators. If `include` is not specified, only active matching indicators are returned.
-* `<indicator_value_field>` is the indicator to look up. 
-* `<optional_indicator_value_field>` is used to add more indicators to look up.
-
-#### Response fields
-* confidence
-* fields
-* imported
-* indicator
-* valid_from
-* valid_until
-* source
-* threat_type
-* type
-* updated
-* num_match (if `singleIndicator` is used)
-
-### Examples
-
+* `<indicator>` is the [indicator](/docs/security/threat-intelligence/upload-formats/#normalized-json-format) to look up for a [field name](https://github.com/SumoLogic/cloud-siem-content-catalog/blob/master/schema/full_schema.md). At least one field name is required. `<optional_indicator>` is used to add more indicators to look up. Allowed in the filtering are parentheses `()`; `OR` and `AND` boolean operators; and comparison operators `=`, `<`, `>`, `=<`, `=>`, `!=`. <br/>You can filter on the following indicator attributes:
+   * `actors`
+   * `confidence`
+   * `id`
+   * `indicator`
+   * `killChain`
+   * `source`
+   * `threatType`
+   * `type`
+   * `validFrom`
+   * `validUntil`
+
+### Response fields
+
+Query responses return the following fields:
+* `confidence`
+* `fields`
+* `imported`
+* `indicator`
+* `valid_from`
+* `valid_until`
+* `source`
+* `threat_type`
+* `type`
+* `updated`
+* `num_match` (if `singleIndicator` is used)
+
+## Examples
+
+### Simple examples
+ 
 ```
 _index=sec_record*
 | threatlookup srcDevice_ip
@@ -63,7 +76,7 @@ _index=sec_record*
 ```
 ```
 _index=sec_record*
-| threatlookup source="s_CrowdStrike" srcDevice_ip
+| threatlookup source="mysource" srcDevice_ip
 | where _threatlookup.confidence > 50
 | timeslice 1h
 | count by _timeslice
@@ -77,19 +90,99 @@ _index=sec_record*
 ```
 ```
 _index=sec_record*
-| threatlookup source="s_CrowdStrike" dstDevice_ip, srcDevice_ip
+| threatlookup source="mysource" dstDevice_ip, srcDevice_ip
 | where _threatlookup.confidence > 50
 | timeslice 1h
 | count by _timeslice
 ```
 ```
 _index=sec_record*
-| threatlookup source="s_CrowdStrike" include="active" dstDevice_ip, srcDevice_ip
+| threatlookup source="mysource" include="active" dstDevice_ip, srcDevice_ip
 | where _threatlookup.confidence > 50
 | timeslice 1h
 | count by _timeslice
 ```
-### Format timestamp results
+
+### Complex examples
+
+```sql title="Client IP threat info"
+_sourceCategory=AWS/WAF {{client_ip}}
+| parse "\"httpMethod\":\"*\"," as httpMethod,"\"httpVersion\":\"*\"," as httpVersion,"\"uri\":\"*\"," as uri, "{\"clientIp\":\"*\",\"country\":\"*\"" as clientIp,country, "\"action\":\"*\"" as action, "\"matchingNonTerminatingRules\":[*]" as matchingNonTerminatingRules, "\"rateBasedRuleList\":[*]" as rateBasedRuleList, "\"ruleGroupList\":[*]" as ruleGroupList, "\"httpSourceId\":\"*\"" as httpSourceId, "\"httpSourceName\":\"*\"" as httpSourceName, "\"terminatingRuleType\":\"*\"" as terminatingRuleType, "\"terminatingRuleId\":\"*\"" as terminatingRuleId, "\"webaclId\":\"*\"" as webaclId nodrop
+| threatlookup singleIndicator clientip
+| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
+```
+
+```sql title="All IP threat count"
+_sourceCategory=Labs/AWS/DynamoDB account=* namespace=* "\"eventSource\":\"dynamodb.amazonaws.com\""
+| json "eventName", "awsRegion", "requestParameters.tableName", "sourceIPAddress", "userIdentity.userName" as event_name, Region, entity, ip_address, user
+| where Region matches "*" and tolowercase(entity) matches "*"
+| where ip_address != "0.0.0.0" and ip_address != "127.0.0.1"
+| count as ip_count by ip_address
+| threatlookup singleIndicator ip_address
+| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence)
+| if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor
+| sum (ip_count) as threat_count
+```
+
+```sql title="Use threatlookup in a subquery"
+_sourceCategory=weblogs
+[subquery:_sourceCategory="Labs/SecDemo/guardduty" "EC2 Instance" "communicating on an unusual server port 22"
+| json field=_raw "service.action.networkConnectionAction.remoteIpDetails" as remoteIpDetails
+| json field=_raw "service.action.networkConnectionAction.connectionDirection" as connectionDirection
+| where connectionDirection = "OUTBOUND"
+| json field=remoteipdetails "ipAddressV4" as src_ip
+| threatlookup singleIndicator threat| if (_threatlookup.confidence >= 85, "high", if (_threatlookup.confidence >= 50, "medium", if (_threatlookup.confidence >= 15, "low", if (_threatlookup.confidence >= 0, "unverified", "Unknown")))) as malicious_confidence
+| where malicious_confidence = "high"
+| compose src_ip]
+```
+
+<!-- Add this after sumo://threat/cs is replaced by threatlookup":
+
+### Threatlookup queries in dashboards
+
+The `threatlookup` search operator is used for queries in some dashboards, including dashboards in the [Threat Intel Quick Analysis app](/docs/integrations/security-threat-detection/threat-intel-quick-analysis/). These queries provide great examples of how to use the operator.
+
+To see `threatlookup` used in a query:
+1. Open the Threat Intel Quick Analysis app.
+1. Navigate to a dashboard, such as **Overview**.
+1. Click the three-dot kebab in the upper-right corner of the dashboard panel.
+1. Select **Open in Log Search**. 
+1. Look for `threatlookup` used in the query. 
+
+For example, here is the query used for the **Threat Count** panel in the **Threat Intel Quick Analysis - IP** dashboard:
+
+```
+_sourceCategory=<source-category-name> 
+| parse regex "(?<ip_address>\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" 
+| where ip_address != "0.0.0.0" and ip_address != "127.0.0.1"
+| count as ip_count by ip_address
+
+| threatlookup singleIndicator ip_address
+
+// normalize confidence level to a string 
+| if (_threatlookup.confidence >= 85, "high", if (_threatlookup.confidence >= 50, "medium", if (_threatlookup.confidence >= 15, "low", if (_threatlookup.confidence >= 0, "unverified", "unknown")))) as threat_confidence
+
+// filter for threat confidence
+| where  threat_confidence matches "*"
+
+//rename to match threat_<foo> convention
+| %"_threatlookup.actors" as threat_actors
+| %"_threatlookup.type" as type
+| %"_threatlookup.threat_type" as threat_type
+
+//convert threat valid from to human readable time
+| toLong(%"_threatlookup.valid_from" * 1000) as %"_threatlookup.valid_from"
+| formatDate(%"_threatlookup.valid_from", "MM-dd-yyyy") as threat_valid_from
+
+| where type matches "ipv4-addr*" and !isNull(threat_confidence)
+
+| if (isEmpty(threat_actors), "Unassigned", threat_actors) as threat_actors
+
+|sum (ip_count) as threat_count
+```
+-->
+
+## Format timestamp results
 
 Timestamps for the following response fields return results as an integer because they use Unix time (also known as *epoch time*):
 * `_threatlookup.imported`
@@ -107,7 +200,7 @@ _index=sec_record*
 
 <!-- For threat intel. Add this back once we have support for the cat search operator:
 
-#### Run threatlookup with the cat search operator
+## Run threatlookup with the cat search operator
 
 You can run the `threatlookup` search operator with the [`cat` search operator](/docs/search/search-query-language/search-operators/cat/) by using the `sumo://threat-intel` path. This lets you search the entire store of threat intelligence indicators, or just a portion. For example:
 ```
@@ -124,6 +217,6 @@ cat sumo://threat-intel | formatDate(toLong(_threatlookup.valid_until), "yyyy-MM
 ```
 
 :::note
-You cannot use the cat search operator with the `s_CrowdStrike` source.
+You cannot use the cat search operator with the `_sumo_global_feed_cs` source.
 :::
 -->
diff --git a/docs/search/subqueries.md b/docs/search/subqueries.md
@@ -376,7 +376,7 @@ _sourceCategory=search "error while retrying to deploy index"
 
 ### Check Malicious Activity with Subquery
 
-The following search allows a security analyst how to track logs related to a malicious IP address that was flagged by Amazon GuardDuty and also by a CrowdStrike Threat feed. The subquery is returning the field `src_ip` with the IP addresses deemed as threats to the parent query, note that the keywords option was not used so the parent query will expect a field src_ip to exist. The results will include logs from the weblogs sourceCategory that have a `src_ip` value that was deemed a threat from the subquery.
+The following search allows a security analyst to track logs related to a malicious IP address that was flagged by Amazon GuardDuty and also by a CrowdStrike Threat feed. The subquery is returning the field `src_ip` with the IP addresses deemed as threats to the parent query, note that the keywords option was not used so the parent query will expect a field src_ip to exist. The results will include logs from the weblogs sourceCategory that have a `src_ip` value that was deemed a threat from the subquery.
 
 ```sql
 _sourceCategory=weblogs
diff --git a/docs/security/threat-intelligence/find-threats.md b/docs/security/threat-intelligence/find-threats.md
diff --git a/docs/security/threat-intelligence/threat-intelligence-mapping.md b/docs/security/threat-intelligence/threat-intelligence-mapping.md
diff --git a/docs/security/threat-intelligence/upload-formats.md b/docs/security/threat-intelligence/upload-formats.md
