S3 cloudtrail ingestion to detections and alerts - Almost working

[jmce123]

Version

2.4.190

Installation Method

Cloud image (Amazon, Azure, Google)

Description

configuration

Installation Type

Standalone

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128GB

Storage for /

1TB

Storage for /nsm

1TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I am not sure if this is the best way, its just a way I have found. Please if there is a better way, documented I would appreciate it!

I wrote a custom pipeline to get cloudtrail logs from an S3 bucket. I wrote the yml script and figured out how to get it to load via the UI in logstash > defined_pipelines > manager [adv]

The logs now load and I can see them in kibana. No matter what I do though I cannot get an alert to detect and fire on these logs.

I do have alerting working with CURL and it will fire on failed SOC logins as well.

Kibana->stack manager sees the index as
cloudtrail-2025.12.04

I have an ingestion pipeline and it tags each log with "pipeline applied: true" as below, so I know this is working up to this point.

[
{
"json": {
"field": "message",
"target_field": "cloudtrail"
}
},
{
"set": {
"field": "pipeline_applied",
"value": true
}
}
]

If I try to make the alert in the detections UI with sigma, I see the rule fires in the log but never matches.

Here is the rule

title: CloudTrail Activity (Raw Message Search)
id: aaaabbbb-cccc-dddd-eeee-111122223333
status: test
description: Detects any CloudTrail events searching raw message field
author: Security Team
date: 2025/12/04

logsource:
product: aws
service: cloudtrail

index: cloudtrail-*

detection:
selection:
pipeline_applied: true
condition: selection

realert:
minutes: 0

# ElastAlert-specific tuning
buffer_time:
minutes: 480 # look back 60 minutes per run
run_every:
minutes: 1 # run every 3 minutes

falsepositives:

• Legitimate activity

level: high

tags:

• attack.initial_access
• attack.persistence

Logs look like this:

2025-12-04 23:13:19,394 INFO elastalert Ran CloudTrail - Activity (Raw Message Search) -- aaaabbbb-cccc-dddd-eeee-111122223333 from 2025-12-04 15:13 UTC to 2025-12-04 23:13 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
2025-12-04 23:13:19,394 INFO elastalert CloudTrail - Activity (Raw Message Search) -- aaaabbbb-cccc-dddd-eeee-111122223333 range 28800

So I tried another approach and loaded a rule via yml

/opt/so/rules/elastalert/rules/cloudtrail_pipeline_applied.yml

name: AWS CloudTrail - Pipeline Applied True
type: any
index: cloudtrail-*
filter:

• term:
  pipeline_applied: true
  realert:
  minutes: 0 # Alert on every occurrence
  buffer_time:
  minutes: 5 # Look back 1 minute for events
  alert:
• "slack"

And this gets hits, but then puts the so-elastalert into a "missing state" and never sends the alert
025-12-04 21:58:42,434 INFO elastalert Ran AWS CloudTrail - Pipeline Applied True from 2025-12-04 21:57 UTC to 2025-12-04 21:58 UTC: 903 query hits (0 already seen), 903 matches, 903 alerts sent
2025-12-04 21:58:42,434 INFO elastalert AWS CloudTrail - Pipeline Applied True range 60
2025-12-04 21:58:42,435 INFO apscheduler.executors.default Job "Rule: AWS CloudTrail - Pipeline Applied True (trigger: interval[0:03:00], next run at: 2025-12-04 22:01:30 UTC)" executed successfully

Any help would be greatly appreciated!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Can you review /opt/so/log/elastalert/ for errors? Likely in stderr.log, but also stdout.log

[jmce123]

I figured it out. THanks for replying! It was the way I had slack defined in the rule. Here is the correct way.

name: AWS CloudTrail - Pipeline Applied True
type: any
index: cloudtrail-*

filter:

• term:
  _id: "ixxxxxxxxx3eBA"

realert:
minutes: 0

buffer_time:
minutes: 5

max_query_size: 100

alert:

• "slack"

slack_webhook_url: "https://hooks.slack.com/services/changedForSecurity"
slack_channel: "security-onion"

[jmce123]

The answer was my rule had the slack part malformed but it was still passing the so-elastalert elastalert-test-rule. Here is what the final rule that worked was that fired for me.

name: AWS CloudTrail - Pipeline Applied True
type: any
index: cloudtrail-*

filter:

• term:
  _id: "ix4n7xxxxxxxxxeBA"

realert:
minutes: 0

buffer_time:
minutes: 5

max_query_size: 100

alert:

• "slack"

slack_webhook_url: "https://hooks.slack.com/services/changedForSecurity"
slack_channel: "security-onion"

[jmce123]

For any who find this, I also learned this is not the ideal way to go about this. I since learned to have my SO host elastic agent use the AWS cloudtrail integrations for ingesting the logs which got them in the proper index ".ds-logs-aws.
This then made writing a detections sigma alert in the UI work.

I wish I knew this before I went down the first path of making my own custom pipeline to ingest from S3 and then having to make the above custom rule to check that index to fire the alert.

Hopefully this saves someone else that headache.