[Detection Engine] Indicator Match Rule's handling of multi-valued fields

[rylnd]

Summary

This is somewhat related to #251361, and arguably to #251362 as well, but: I wanted to create a distinct issue for a few other items I've seen during various SDH investigations, which all seem to fall under the general category of "multiple field values."

Description

There are actually three different behaviors I want to document, here:

1. Indicator match rules will match an unmapped event field to an indicator IFF the field contains a single value. (Test 1)

2. Indicator match rules will match a multi-value field IFF the first value retrieved matches a given indicator's value (Test 2)

3. There is a situation in which a mapped, multi-valued field can match an indicator, but fail to be enriched by said indicator. Screenshot below:

   

I still haven't quite figured this one out, but I think it has something to do with the fact that in the alert in question, the matched field is an IP field with various formats/values. What's clear is that the subsequent reverse lookup to build the enrichment is not symmetrical to the matching logic.

For the other two, here is a diff adding some integration tests demonstrating the referenced behavior:

Click to see diff

diff --git a/x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/indicator_match/trial_license_complete_tier/indicator_match.ts b/x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/indicator_match/trial_license_complete_tier/indicator_match.ts
index 6d704fe7ca5f..720022c477ae 100644
--- a/x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/indicator_match/trial_license_complete_tier/indicator_match.ts
+++ b/x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/indicator_match/trial_license_complete_tier/indicator_match.ts
@@ -181,6 +181,12 @@ export default ({ getService }: FtrProviderContext) => {
     log,
   });
 
+  const { indexListOfDocuments: auditbeatIndexDocuments } = dataGeneratorFactory({
+    es,
+    index: 'auditbeat-8.0.0-2019.02.19-000001',
+    log,
+  });
+
   /**
    * Specific api integration tests for threat matching rule type
    */
@@ -1092,6 +1098,122 @@ export default ({ getService }: FtrProviderContext) => {
         const previewAlerts = await getPreviewAlerts({ es, previewId });
         expect(previewAlerts).toHaveLength(2);
       });
+
+      it.only('does not generate alerts when the indicator matches an unmapped multi-value field', async () => {
+        const id = uuidv4();
+
+        await indexListOfDocuments([
+          {
+            id,
+            '@timestamp': new Date().toISOString(),
+            // source.ip is unmapped in the ecs_compliant index
+            // 45.115.45.3 matches an indicator in filebeat/threat_intel index
+            source: {
+              ip: ['45.115.45.3', 'fe80::784d:7aff:fe69:63b1'],
+            },
+          },
+        ]);
+
+        const rule = createThreatMatchRule({
+          ...threatMatchRuleEcsComplaint(id),
+          threat_query: '*:*',
+          threat_index: ['filebeat-*'],
+          threat_mapping: [
+            {
+              entries: [
+                {
+                  value: 'threat.indicator.ip',
+                  field: 'source.ip',
+                  type: 'mapping',
+                },
+              ],
+            },
+          ],
+        });
+
+        const { previewId } = await previewRule({ supertest, rule });
+        const previewAlerts = await getPreviewAlerts({ es, previewId });
+        expect(previewAlerts).toHaveLength(0);
+      });
+
+      it.only('enriches alerts when the indicator matched one value in a multi-value field', async () => {
+        const id = uuidv4();
+
+        await auditbeatIndexDocuments([
+          // alert that contains multiple `source.ip` values, the SECOND of which matches the indicator (45.115.45.3 in filebeat threat intel)
+          {
+            id,
+            '@timestamp': new Date().toISOString(),
+            source: {
+              ip: ['45.115.45.3', '159.89.120.67'],
+            },
+          },
+        ]);
+
+        const rule = createThreatMatchRule({
+          ...threatMatchRuleEcsComplaint(id),
+          index: ['auditbeat-*'],
+          threat_query: '*:*',
+          threat_index: ['filebeat-*'],
+          threat_mapping: [
+            {
+              entries: [
+                {
+                  value: 'threat.indicator.ip',
+                  field: 'source.ip',
+                  type: 'mapping',
+                },
+              ],
+            },
+          ],
+        });
+
+        const { previewId } = await previewRule({ supertest, rule });
+        const previewAlerts = await getPreviewAlerts({ es, previewId });
+        expect(previewAlerts).toHaveLength(1);
+
+        const [threat] = previewAlerts.map((hit) => hit._source?.threat) as Array<{
+          enrichments: unknown[];
+        }>;
+
+        expect(threat.enrichments).toEqual(
+          expect.arrayContaining([
+            {
+              indicator: {
+                description: 'this should match auditbeat/hosts on both port and ip',
+                first_seen: '2021-01-26T11:06:03.000Z',
+                ip: '45.115.45.3',
+                port: 57324,
+                provider: 'geenensp',
+                type: 'url',
+              },
+              matched: {
+                atomic: ['45.115.45.3', '159.89.120.67'],
+                field: 'source.ip',
+                id: '978785',
+                index: 'filebeat-8.0.0-2021.01.26-000001',
+                type: ENRICHMENT_TYPES.IndicatorMatchRule,
+              },
+            },
+            {
+              indicator: {
+                description: 'this should match auditbeat/hosts on ip',
+                first_seen: '2021-01-26T11:06:03.000Z',
+                ip: '45.115.45.3',
+                provider: 'other_provider',
+                type: 'ip',
+              },
+              matched: {
+                atomic: ['45.115.45.3', '159.89.120.67'],
+                field: 'source.ip',
+                id: '978787',
+                index: 'filebeat-8.0.0-2021.01.26-000001',
+                type: ENRICHMENT_TYPES.IndicatorMatchRule,
+              },
+            },
+          ])
+        );
+      });
     });
 
     describe('indicator enrichment: event-first search', () => {

Related Issues

• [Security Solution] Indicator match rule with normalized fields can yield inconsistent results #251361
• [Security Solution] Indicator match rule does not always handle fields that aren't included in _source #251362

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)