Playbook : ElastAlert query works, but rule doesn't seem to filter properly

[Jaap79]

Hi,

I'm trying to set up a specific Playbook for Office 365, which will alert when certain User Agents are NOT being used. The use case is that people in the respective tenant should only use iOS and MS Edge browsers and everything else should generate an alert. Long story as to why, but it is what it is...

The Playbook Sigma rule is set up as follows:

title: Redacted-Name - Microsoft O365 - Logins from unauthorized User Agents
status: experimental
description: Detects any login from unauthorized User Agents
author: Me
date: 2023/01/20
modified: 2023/01/20
logsource:
index: so-beats*
detection:
selection:
event.dataset:
- o365.audit
o365.audit.Operation:
- UserLoggedIn
host.name:
- redacted-name.onmicrosoft.com
filter:
user_agent.name:
- Edge
- Mobile Safari UI/WKWebView
- Mobile Safari
- Electron
user_agent.original:
- Windows-AzureAD-Authentication-Provider/1.0
source.ip:
- <redacted.ip.address>
condition: selection and not filter
falsepositives:
- Legitimate User Agents
level: high

======================

The thing is, it converts to an ElastAlert query which seems to do the trick just fine. However, the Alerts that I'm getting from this particular Playbook still include the user_agent.name: Edge for instance. The tenant name and everything else works as expected; I only see alerts for that particular O365 tenant and User Logged In-events.

What am I missing?

[InfosecGoon]

Good morning!

Can you share one of the log entries that it is erroneously alerting on? I'd like to see what the user_agent.name field is that's not being matched properly by the filter.

[Jaap79]

@timestamp | 2023-01-27T11:45:26.000Z
event.dataset | alert
event.module | playbook
event.severity | 3
event.severity_label | high
event_data.@timestamp | 2023-01-27T11:36:26Z
event_data.@Version | 1
event_data._id |
event_data._index | -sn01_searchnode:so-o365-2023.01.27
event_data.agent.ephemeral_id | redacted
event_data.agent.id | redacted
event_data.agent.name | redacted-sens01
event_data.agent.type | filebeat
event_data.agent.version | 8.5.3
event_data.client.address | ip.add.re.ss
event_data.client.ip | ip.add.re.ss
event_data.ecs.version | 1.12.0
event_data.event.action | UserLoggedIn
event_data.event.category | authentication
event_data.event.code | AzureActiveDirectoryStsLogon
event_data.event.dataset | o365.audit
event_data.event.id | event-id-number-redacted
event_data.event.ingested | 2023-01-27T11:43:11.490002039Z
event_data.event.kind | event
event_data.event.module | o365
event_data.event.outcome | success
event_data.event.provider | AzureActiveDirectory
event_data.event.type | [ "start", "authentication_success" ]
event_data.fileset.name | audit
event_data.host.id | host-id-redacted
event_data.host.name | tenant-redacted.onmicrosoft.com
event_data.input.type | o365audit
event_data.metadata._id | redacted-number
event_data.metadata.beat | filebeat
event_data.metadata.ip_address | 172.16.x.x
event_data.metadata.pipeline | filebeat-8.5.3-o365-audit-pipeline
event_data.metadata.type | _doc
event_data.metadata.version | 8.5.3
event_data.network.type | ipv4
event_data.num_hits | 2
event_data.num_matches | 2
event_data.o365.audit.Actor | [ { "ID": "actor-number-redacted", "Type": 0 }, { "ID": "[email protected]", "Type": 5 } ]
event_data.o365.audit.ActorContextId | 464171d6-redacted
event_data.o365.audit.ActorIpAddress | ip.ad.re.ss
event_data.o365.audit.ApplicationId | application-id-redacted
event_data.o365.audit.AzureActiveDirectoryEventType | 1
event_data.o365.audit.ClientIP | ip.ad.re.ss
event_data.o365.audit.CreationTime | 2023-01-27T11:36:26
event_data.o365.audit.DeviceProperties | [ { "Name": "Id", "Value": "redacted-number" }, { "Name": "DisplayName", "Value": "TENANT-ID-LT0x" }, { "Name": "OS", "Value": "Windows 10" }, { "Name": "BrowserType", "Value": "Electron" }, { "Name": "IsCompliantAndManaged", "Value": "False" }, { "Name": "TrustType", "Value": "2" }, { "Name": "SessionId", "Value": "number-54635number" } ]
event_data.o365.audit.ErrorNumber | 0
event_data.o365.audit.ExtendedProperties.RequestType | SAS:ProcessAuth
event_data.o365.audit.ExtendedProperties.ResultStatusDetail | Success
event_data.o365.audit.ExtendedProperties.UserAgent | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Teams/1.5.00.36367 Chrome/96.0.4664.174 Electron/16.2.8 Safari/537.36
event_data.o365.audit.Id | id-number-redacted
event_data.o365.audit.InterSystemsId | redacted-number
event_data.o365.audit.IntraSystemId | redacted-number
event_data.o365.audit.ObjectId | redacted-number
event_data.o365.audit.Operation | UserLoggedIn
event_data.o365.audit.OrganizationId | redacted-number
event_data.o365.audit.RecordType | 15
event_data.o365.audit.ResultStatus | Success
event_data.o365.audit.SupportTicketId |  
event_data.o365.audit.Target | [ { "ID": "id-redacted", "Type": 0 } ]
event_data.o365.audit.TargetContextId | redacted-number
event_data.o365.audit.UserId | [email protected]
event_data.o365.audit.UserKey | redacted-number
event_data.o365.audit.UserType | 0
event_data.o365.audit.Version | 1
event_data.o365.audit.Workload | AzureActiveDirectory
event_data.organization.id | redacted-id-number
event_data.organization.name | redacted-id-number
event_data.related.ip | ip.ad.re.ss
event_data.related.user | user.name
event_data.service.type | o365
event_data.source.as.number | number
event_data.source.as.organization.name | ISP-of-User
event_data.source.geo.city_name | Paris
event_data.source.geo.continent_name | Europe
event_data.source.geo.country_iso_code | FR
event_data.source.geo.country_name | France
event_data.source.geo.location.lat | number.12345
event_data.source.geo.location.lon | number.67890
event_data.source.geo.region_iso_code | FR-75
event_data.source.geo.region_name | Paris
event_data.source.ip | ip.ad.re.ss
event_data.tags | [ "forwarded", "beats_input_raw_event" ]
event_data.type | redis-input
event_data.user.domain | domainname.tld
event_data.user.email | [email protected]
event_data.user.id | [email protected]
event_data.user.name | user.name
event_data.user_agent.device.name | Other
event_data.user_agent.name | Electron
event_data.user_agent.original | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Teams/1.5.00.36367 Chrome/96.0.4664.174 Electron/16.2.8 Safari/537.36
event_data.user_agent.os.full | Windows 10
event_data.user_agent.os.name | Windows
event_data.user_agent.os.version | 10
event_data.user_agent.version | 16.2.8
kibana_pivot | https://manager-node/kibana/app/kibana#/discover?_g=()&_a=(columns:!(_source),interval:auto,query:(language:lucene,query:'_id:{[_id]}'),sort:!('@timestamp',desc))
play_url | https://manager-node/playbook/issues/5856
rule.case_template | b0949ad29
rule.category |  
rule.name | REDACTED- Microsoft O365 - Logins from unauthorized User Agents
rule.uuid | b0949ad29
sigma_level | high
soc_pivot | https://cfnc-prd-mn01/#/hunt
soc_id | id-number-redacted
soc_score | 2.392963
soc_type |  
soc_timestamp | 2023-01-27T11:45:26.000Z
soc_source | manager-node:so-playbook-alerts-2023.01.27

[InfosecGoon]

The docs say that the keyword "sofilter" is important for tuning the results - perhaps swapping that in for "filter" would help?

https://docs.securityonion.net/en/2.3/playbook.html#tuning-plays