|
| 1 | +--- |
| 2 | +id: threatlookup |
| 3 | +title: threatlookup Search Operator |
| 4 | +sidebar_label: threatlookup |
| 5 | +--- |
| 6 | + |
| 7 | +The `threatlookup` search operator allows you to search logs for matches in [threat intelligence](/docs/security/threat-intelligence/about-threat-intelligence/), providing security analytics to help you to detect threats in your environment. |
| 8 | + |
| 9 | +:::note |
| 10 | +You can also use the [`threatip`](/docs/search/search-query-language/search-operators/threatip/) search operator to search threat intelligence data based on IP addresses. |
| 11 | +::: |
| 12 | + |
| 13 | +## Syntax |
| 14 | + |
| 15 | +``` |
| 16 | +threatlookup [singleIndicator] [source="<source_value>"] [include="<all|active|expired>"] <indicator> [,<optional_indicator>, …] |
| 17 | +``` |
| 18 | + |
| 19 | +Where: |
| 20 | +* `singleIndicator` returns the single best matching indicator. (In the response, `num_match` indicates how many actual matches there are.) If `singleIndicator` is not specified, all matching indicators are returned. |
| 21 | + |
| 22 | + Specifying `singleIndicator` sorts the list of matching indicators using the following priority order, then returns the indicator at the top of the list: |
| 23 | + 1. Active indicators over expired indicators (if you use `include="all"`). |
| 24 | + 1. Higher confidence indicators. |
| 25 | + 1. More malicious indicators. |
| 26 | + 1. Most recently updated indicators. |
| 27 | + |
| 28 | + If there's still a tie at this point, the system picks the indicator the back-end database returned first. |
| 29 | + |
| 30 | +* `source` is the source to search for the threat intelligence indicator. If `source` is not specified, all sources are searched. |
| 31 | +* `include` includes either all, only active, or only expired threat intelligence indicators. If `include` is not specified, only active matching indicators are returned. |
| 32 | +* `<indicator>` is the [indicator](/docs/security/threat-intelligence/upload-formats/#normalized-json-format) to look up for a [field name](https://github.com/SumoLogic/cloud-siem-content-catalog/blob/master/schema/full_schema.md). At least one field name is required. `<optional_indicator>` is used to add more indicators to look up. Allowed in the filtering are parentheses `()`; `OR` and `AND` boolean operators; and comparison operators `=`, `<`, `>`, `=<`, `=>`, `!=`. <br/>You can filter on the following indicator attributes: |
| 33 | + * `actors` |
| 34 | + * `confidence` |
| 35 | + * `id` |
| 36 | + * `indicator` |
| 37 | + * `killChain` |
| 38 | + * `source` |
| 39 | + * `threatType` |
| 40 | + * `type` |
| 41 | + * `validFrom` |
| 42 | + * `validUntil` |
| 43 | + |
| 44 | +### Response fields |
| 45 | + |
| 46 | +Query responses return the following fields: |
| 47 | +* `confidence` |
| 48 | +* `fields` |
| 49 | +* `imported` |
| 50 | +* `indicator` |
| 51 | +* `valid_from` |
| 52 | +* `valid_until` |
| 53 | +* `source` |
| 54 | +* `threat_type` |
| 55 | +* `type` |
| 56 | +* `updated` |
| 57 | +* `num_match` (if `singleIndicator` is used) |
| 58 | + |
| 59 | +## Examples |
| 60 | + |
| 61 | +### Simple examples |
| 62 | + |
| 63 | +``` |
| 64 | +_index=sec_record* |
| 65 | +| threatlookup srcDevice_ip |
| 66 | +| where _threatlookup.confidence > 50 |
| 67 | +| timeslice 1h |
| 68 | +| count by _timeslice |
| 69 | +``` |
| 70 | +``` |
| 71 | +_index=sec_record* |
| 72 | +| threatlookup singleIndicator srcDevice_ip |
| 73 | +| where _threatlookup.confidence > 50 |
| 74 | +| timeslice 1h |
| 75 | +| count by _timeslice |
| 76 | +``` |
| 77 | +``` |
| 78 | +_index=sec_record* |
| 79 | +| threatlookup source="mysource" srcDevice_ip |
| 80 | +| where _threatlookup.confidence > 50 |
| 81 | +| timeslice 1h |
| 82 | +| count by _timeslice |
| 83 | +``` |
| 84 | +``` |
| 85 | +_index=sec_record* |
| 86 | +| threatlookup dstDevice_ip, srcDevice_ip |
| 87 | +| where _threatlookup.confidence > 50 |
| 88 | +| timeslice 1h |
| 89 | +| count by _timeslice |
| 90 | +``` |
| 91 | +``` |
| 92 | +_index=sec_record* |
| 93 | +| threatlookup source="mysource" dstDevice_ip, srcDevice_ip |
| 94 | +| where _threatlookup.confidence > 50 |
| 95 | +| timeslice 1h |
| 96 | +| count by _timeslice |
| 97 | +``` |
| 98 | +``` |
| 99 | +_index=sec_record* |
| 100 | +| threatlookup source="mysource" include="active" dstDevice_ip, srcDevice_ip |
| 101 | +| where _threatlookup.confidence > 50 |
| 102 | +| timeslice 1h |
| 103 | +| count by _timeslice |
| 104 | +``` |
| 105 | + |
| 106 | +### Complex examples |
| 107 | + |
| 108 | +```sql title="Client IP threat info" |
| 109 | +_sourceCategory=AWS/WAF {{client_ip}} |
| 110 | +| parse "\"httpMethod\":\"*\"," as httpMethod,"\"httpVersion\":\"*\"," as httpVersion,"\"uri\":\"*\"," as uri, "{\"clientIp\":\"*\",\"country\":\"*\"" as clientIp,country, "\"action\":\"*\"" as action, "\"matchingNonTerminatingRules\":[*]" as matchingNonTerminatingRules, "\"rateBasedRuleList\":[*]" as rateBasedRuleList, "\"ruleGroupList\":[*]" as ruleGroupList, "\"httpSourceId\":\"*\"" as httpSourceId, "\"httpSourceName\":\"*\"" as httpSourceName, "\"terminatingRuleType\":\"*\"" as terminatingRuleType, "\"terminatingRuleId\":\"*\"" as terminatingRuleId, "\"webaclId\":\"*\"" as webaclId nodrop |
| 111 | +| threatlookup singleIndicator clientip |
| 112 | +| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence) |
| 113 | +``` |
| 114 | + |
| 115 | +```sql title="All IP threat count" |
| 116 | +_sourceCategory=Labs/AWS/DynamoDB account=* namespace=* "\"eventSource\":\"dynamodb.amazonaws.com\"" |
| 117 | +| json "eventName", "awsRegion", "requestParameters.tableName", "sourceIPAddress", "userIdentity.userName" as event_name, Region, entity, ip_address, user |
| 118 | +| where Region matches "*" and tolowercase(entity) matches "*" |
| 119 | +| where ip_address != "0.0.0.0" and ip_address != "127.0.0.1" |
| 120 | +| count as ip_count by ip_address |
| 121 | +| threatlookup singleIndicator ip_address |
| 122 | +| where (_threatlookup.type="ipv4-addr" or _threatlookup.type="ipv6-addr") and !isNull(_threatlookup.confidence) |
| 123 | +| if (isEmpty(_threatlookup.actors), "Unassigned", _threatlookup.actors) as Actor |
| 124 | +| sum (ip_count) as threat_count |
| 125 | +``` |
| 126 | + |
| 127 | +```sql title="Use threatlookup in a subquery" |
| 128 | +_sourceCategory=weblogs |
| 129 | +[subquery:_sourceCategory="Labs/SecDemo/guardduty" "EC2 Instance" "communicating on an unusual server port 22" |
| 130 | +| json field=_raw "service.action.networkConnectionAction.remoteIpDetails" as remoteIpDetails |
| 131 | +| json field=_raw "service.action.networkConnectionAction.connectionDirection" as connectionDirection |
| 132 | +| where connectionDirection = "OUTBOUND" |
| 133 | +| json field=remoteipdetails "ipAddressV4" as src_ip |
| 134 | +| threatlookup singleIndicator threat| if (_threatlookup.confidence >= 85, "high", if (_threatlookup.confidence >= 50, "medium", if (_threatlookup.confidence >= 15, "low", if (_threatlookup.confidence >= 0, "unverified", "Unknown")))) as malicious_confidence |
| 135 | +| where malicious_confidence = "high" |
| 136 | +| compose src_ip] |
| 137 | +``` |
| 138 | + |
| 139 | +<!-- Add this after sumo://threat/cs is replaced by threatlookup": |
| 140 | +### Threatlookup queries in dashboards |
| 141 | +The `threatlookup` search operator is used for queries in some dashboards, including dashboards in the [Threat Intel Quick Analysis app](/docs/integrations/security-threat-detection/threat-intel-quick-analysis/). These queries provide great examples of how to use the operator. |
| 142 | +To see `threatlookup` used in a query: |
| 143 | +1. Open the Threat Intel Quick Analysis app. |
| 144 | +1. Navigate to a dashboard, such as **Overview**. |
| 145 | +1. Click the three-dot kebab in the upper-right corner of the dashboard panel. |
| 146 | +1. Select **Open in Log Search**. |
| 147 | +1. Look for `threatlookup` used in the query. |
| 148 | +For example, here is the query used for the **Threat Count** panel in the **Threat Intel Quick Analysis - IP** dashboard: |
| 149 | +``` |
| 150 | +_sourceCategory=<source-category-name> |
| 151 | +| parse regex "(?<ip_address>\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" |
| 152 | +| where ip_address != "0.0.0.0" and ip_address != "127.0.0.1" |
| 153 | +| count as ip_count by ip_address |
| 154 | +| threatlookup singleIndicator ip_address |
| 155 | +// normalize confidence level to a string |
| 156 | +| if (_threatlookup.confidence >= 85, "high", if (_threatlookup.confidence >= 50, "medium", if (_threatlookup.confidence >= 15, "low", if (_threatlookup.confidence >= 0, "unverified", "unknown")))) as threat_confidence |
| 157 | +// filter for threat confidence |
| 158 | +| where threat_confidence matches "*" |
| 159 | +//rename to match threat_<foo> convention |
| 160 | +| %"_threatlookup.actors" as threat_actors |
| 161 | +| %"_threatlookup.type" as type |
| 162 | +| %"_threatlookup.threat_type" as threat_type |
| 163 | +//convert threat valid from to human readable time |
| 164 | +| toLong(%"_threatlookup.valid_from" * 1000) as %"_threatlookup.valid_from" |
| 165 | +| formatDate(%"_threatlookup.valid_from", "MM-dd-yyyy") as threat_valid_from |
| 166 | +| where type matches "ipv4-addr*" and !isNull(threat_confidence) |
| 167 | +| if (isEmpty(threat_actors), "Unassigned", threat_actors) as threat_actors |
| 168 | +|sum (ip_count) as threat_count |
| 169 | +``` |
| 170 | +--> |
| 171 | + |
| 172 | +## Format timestamp results |
| 173 | + |
| 174 | +Timestamps for the following response fields return results as an integer because they use Unix time (also known as *epoch time*): |
| 175 | +* `_threatlookup.imported` |
| 176 | +* `_threatlookup.valid_from` |
| 177 | +* `_threatlookup.valid_until` |
| 178 | +* `_threatlookup.updated` |
| 179 | + |
| 180 | +To convert the timestamp results to a readable output, you must format it in the search itself with [`formatDate`](/docs/search/search-query-language/search-operators/formatdate). For example: |
| 181 | + |
| 182 | +``` |
| 183 | +_index=sec_record* |
| 184 | +| threatlookup source="mysource" device_ip |
| 185 | +| formatDate(_threatlookup.valid_until, "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'") as valid_until |
| 186 | +``` |
| 187 | + |
| 188 | +<!-- For threat intel. Add this back once we have support for the cat search operator: |
| 189 | +## Run threatlookup with the cat search operator |
| 190 | +You can run the `threatlookup` search operator with the [`cat` search operator](/docs/search/search-query-language/search-operators/cat/) by using the `sumo://threat-intel` path. This lets you search the entire store of threat intelligence indicators, or just a portion. For example: |
| 191 | +``` |
| 192 | +cat sumo://threat-intel | where _threatlookup.indicator = "192.0.2.0" |
| 193 | +``` |
| 194 | +``` |
| 195 | +cat sumo://threat-intel | where _threatlookup.source = "TAXII2Source" and _threatlookup.indicator = "192.0.2.0" |
| 196 | +``` |
| 197 | +In the `cat` output, timestamp fields (like `valid_until`) will appear as integers. You can use the `formatDate()` function to convert them back to timestamps. For example: |
| 198 | +``` |
| 199 | +cat sumo://threat-intel | formatDate(toLong(_threatlookup.valid_until), "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "UTC") as valid_until |
| 200 | +``` |
| 201 | +:::note |
| 202 | +You cannot use the cat search operator with the `SumoLogic_ThreatIntel` source. |
| 203 | +::: |
| 204 | +--> |
0 commit comments