Elastalert not sending email

[mksquire66]

I've configured the config.yaml file in the /opt/lme/config/elastalert2 directory and I'm not getting emails.... I want an email for any alert that triggers. My config.yaml file is below - redacted some info..... run_every:
minutes: 1

buffer_time:
minutes: 15

writeback_index: elastalert_status

log_file: /var/log/elastalert.log

alert_time_limit:
days: 2

es_host: lme-elasticsearch
es_port: 9200
start_time: "now"
es_username: "elastic"
es_password: "my password is here"
use_ssl: true
verify_certs: false

smtp_host: internalsmtp.serverIsHere.org
smtp_port: 25
from_addr: "elastalert2@mycompany.org"
email: "myemail@mycompany.org"
smtp_auth_file: /opt/lme/config/elastalert2/smtp_auth.yml

#exists in the container
rules_folder: /opt/elastalert/rules

[aarz-snl]

Do your rules have this configuration in them at least?

alert:
  - email

You may be able to use config.yaml to host your stmpconfig /creds etc for all rules -- but your RULES may still need to be configured with the email alert configuration

Full example:

config.yaml

run_every:
  minutes: 1

buffer_time:
  minutes: 15

writeback_index: elastalert_status

log_file: /var/log/elastalert.log

alert_time_limit:
  days: 2

es_host: lme-elasticsearch
es_port: 9200
start_time: "now"
es_username: "elastic"
es_password: "your_actual_password_here"
use_ssl: true
verify_certs: false

rules_folder: /opt/elastalert/rules

smtp_host: internalsmtp.serverIsHere.org
smtp_port: 25
from_addr: "elastalert2@mycompany.org"
email: "destination_email@mycompany.org"
smtp_auth_file: /opt/lme/config/elastalert2/smtp_auth.yml

creds file:

user: "smtp-username"
password: "smtp-password"

example rule showing email configuration:

name: Simple Test Rule
type: any

index: my-index-*

filter:
- query:
    query_string:
      query: "*"

alert:
  - email

how to use test in container (enter the container)

sudo -i podman exec -it lme-elastalert2 bash

run test script:

elastalert-test-rule /opt/elastalert/rules/alert-name.yaml

• i do recommend using yaml instead of yml
  docs state:

Every file that ends in .yaml in the rules_folder will be run by default. The following configuration settings are common to all types of rules.

[mksquire66]

I don't have rules in the rules folder other than the ones that were there by default. I was trying to avoid having to create a file for every rule that I've enabled and want to get alerted on (I just want an email for all alerts). Am I mis-understanding what you're suggesting? I did just modify the example-email-rule.yml file and added alert:

• email

[aarz-snl]

Yes, you may be misunderstanding -- when you say rules that you've enabled...what rules?

[mksquire66]

The rules in the web interface under "Security"

[aarz-snl]

So those are separate from elastalert... those will alert in kibana but they wont do anyting with email/slack/etc unless you use elastalert.

Something i've been experimenting with is this -- tailored to your use case this may be a good test:

add this to the elastalert rules file at /opt/lme/config/elastalert2/rules

# /opt/lme/config/elastalert2/rules/rollup_kibana_alerts.yaml

name: Rollup Kibana Native Security Alerts
type: any
index: .alerts-security.alerts-*

filter:
  - range:
      "@timestamp":
        gte: "now-5m"
  - query_string:
      query: "kibana.alert.rule.name:*"

realert:
  minutes: 0

aggregation:
  minutes: 0

alert:
  - email

alert_text_type: alert_text_only
alert_text: |
  🚨 New Security Alert 🚨
  Severity: {0}
  Host: {1}
  Rule: {2}
  User: {3}
  Action: {4}

alert_text_args:
  - kibana.alert.severity
  - host.name
  - kibana.alert.rule.name
  - user.name
  - event.action

[mksquire66]

sorry - feeling a little dumb here.... Your first line seems contradictory - "So those are separate from elastalert... those will alert in kibana but they wont do anyting with email/slack/etc unless you use elastalert." I thought I was trying to use elastalert.?

Also - the bottom section under alert_text_args: I put:
alert_text_args:

• kibana.alert.severity = "high"
• host.name = "mysmtpserver.mycompany.org"
• kibana.alert.rule.name = "Hidden User created"
• user.name = "elastic@mycompany.org"
• event.action = "login"

Is this correct?

[mksquire66]

I see the alert in Kibana - no email

[HopperShell]

Those will be args that get populated from the log that comes in.

You don’t have to hard set anything. Just exactly as I provided it

elastalert2 is a separate tool designed to monitor elasticsearch indexes and alert when a search is detected. Kibana alerts are built into kibana. They aren’t related at all.

So you have to do 1 of 2 things. Either create your own elastalert rules (using the example template you see provided you create rules for each scenario of your choice)

or you monitor the index of the kibana alerts with elast alert and alert for anything detected. I hope that makes sense. I can give better explanation when I’m back at a computer

[HopperShell]

These right here

alert_text_args:

• kibana.alert.severity
• host.name
• kibana.alert.rule.name
• user.name
• event.action

are fields in the json from each alert

so every alert has the field kibana.alert.severity field and it will be low, medium, high, or critical. We use an argument here to say get the severity and then add it to the alert

so your email will start with

severity: high or low or critical - whatever that single alert is

does the same for each of those args

you may have broken the rule by adding the = part to the end of each

[aarz-snl]

I got this working for me doing the following:

created a rule at the following location with the following configuration:

# /opt/lme/config/elastalert2/rules/rollup_kibana_alerts.yaml
name: Rollup Kibana Native Security Alerts
type: any
index: .alerts-security.alerts-*
filter:
  - range:
      "@timestamp":
        gte: "now-5m"
  - query_string:
      query: "kibana.alert.rule.name:*"
realert:
  minutes: 0
aggregation:
  minutes: 0
alert:
  - email
alert_text_type: alert_text_only
alert_text: |
  🚨 New Security Alert 🚨
  Severity: {0}
  Host: {1}
  Rule: {2}
  User: {3}
  Action: {4}
alert_text_args:
  - kibana.alert.severity
  - host.name
  - kibana.alert.rule.name
  - user.name
  - event.action
email:
  - "myemail@gmail.com"
smtp_host: "smtp.gmail.com"
smtp_port: 465
smtp_ssl: true
from_addr: "elastalert@elastalert.com"
smtp_auth_file: /opt/elastalert/misc/smtp_auth.yml

created authentication file at the following location:

# /opt/lme/config/elastalert2/misc/smtp_auth.yml
user: "email@gmail.com"
password: "passcode"

My config.yaml

run_every:
  minutes: 5

buffer_time:
  minutes: 15

writeback_index: elastalert_status

alert_time_limit:
  days: 2

es_host: lme-elasticsearch
es_port: 9200
use_ssl: true
verify_certs: false
ssl_show_warn: false

#exists in the container
rules_folder: /opt/elastalert/rules

restarted elastalert verify its online properly

sudo systemctl restart lme-elastalert.service

logs:

sudo -i podman logs -f lme-elastalert2

You may see warnings about http -- this is fine its just private docker network communications

if its up and running you can force the rule to run:

enter the container:

sudo -i podman exec -it lme-elastalert2 bash

Run elastalert rule:

elastalert --verbose --config /opt/elastalert/config.yaml --rule /opt/elastalert/rules/rollup_kibana_alerts.yml

verify in kibana you actually have an alert generated in the last 5 minutes... if not trigger one then run the command above. Check email

[aarz-snl]

updated location of smtp_auth.yml

on host machine it must go here:

/opt/lme/config/elastalert2/misc/smtp_auth.yml

In this example

Just to clarify how this works.. we have a mount that takes

/opt/lme/config/elastalert2 on the HOST machine and mounts it to /opt/elastalert in the container.

so when you see references to /opt/elastalert in configuration files thats basically a function thats existing INSIDE the container -- but its actually mounted on the host machine to /opt/lme/config/elastalert2

[mksquire66]

Thanks so much for sharing these configs, etc. that you got working! I copied what you have above and plugged it into my environment and get to the point of trying to enter the container and had issues. The container keeps dropping and restarting every 2-3 seconds so doesn't stay up long enough to get into it. I had this happen when trying to do the configuration based on documentation initially too, so I'm working through which line is causing me issues. I'll post again when I have a better update.

[HopperShell]

this basically means theres a config or rule somewhere thats improperly configured. Id remove anything that ISN"T the config that we have discussed here. So, the config.yaml and the one single rule above

[cisspUser01]

If you configured everything in ElastAlert2 correctly and still have elastalert2 service dropping every 2-3 seconds. make sure you didnt change the default password. that's what happened to me. after I put back the elastic user default password. All alerts started to work.

[HopperShell]

Yes... it's built into the elastalert configuration to get access to elastic search it has to use the username and password.

We still need to upgrade our password management script so it makes it easier to update passwords

[mksquire66]

Awesome! I finally got it working with a few small tweaks...! First with the 2-3 second restart with the elastalert service: When I copy and pasted - it didn't paste quite right. The line with the bomb icons didn't paste at all and the Severity line was indented a little more to the right. After changing those 2 lines - the container stayed up. :)
alert_text: |
🚨 New Security Alert 🚨
Severity: {0}
Host: {1}
Rule: {2}
User: {3}
Action: {4}

I had to work through some issues with our smtp server (but got errors that indicated what was wrong) and now my mailbox is blowing up with alerts! :). Thanks so much for your help!!!

[aarz-snl]

NP... now since we did it this way all you have to do is go into kibana -> security -> rules and turn off the noisy ones that are full of false positives. Or manually adjust the search they do if its just ONE thing causing hundreds of alerts

[cisspUser01]

are there any recommendation on what alerts to enable?
Thank you,

[aarz-snl]

Not really -- just up to each individually to decide what is important to monitor. Ie. if you're monitoring for specific threat actors -- then use the mitre attack tags to turn on alerts that are tied to those APTs.

You could also just turn on critical and high only and see how that goes.

There's also probably something that can be done with elastalert2 where it detects the field and if its critical or high it will be sent to your email. And if its not it doesn't. however, its still logged in kibana -- so if you have to map an incident out that still exists

Something like this:

name: Filtered Kibana Alerts
type: any
index: .alerts-security.alerts-*

filter:
  - range:
      "@timestamp":
        gte: "now-5m"
  - query_string:
      query: "kibana.alert.rule.name:*"
  - terms:
      kibana.alert.severity: ["high", "critical"]  # only these will trigger

realert:
  minutes: 0

aggregation:
  minutes: 0

alert:
  - "slack"

slack_webhook_url: "https://hooks.slack.com/services/..."
slack_username_override: "Alert Bot"
slack_icon_emoji_override: ":rotating_light:"

alert_text_type: alert_text_only
alert_text: |
  🚨 *New Security Alert*
  *Severity*: {0}
  *Rule*: {1}
  *User*: {2}
  *Action*: {3}

alert_text_args:
  - kibana.alert.severity
  - kibana.alert.rule.name
  - user.name
  - kibana.alert.action

Now those alerts will always live in kibana so you dont actually have to turn them off -- but they wont be sent to your slack this scenario

Using filters and queries in your elastalert rule can trim this down as far as you want to any scenario.

[aarz-snl]

AND i do apologize for my copy and pasting... the icons that get pasted in here is typically something like

:rotating_light: github was just actually changing it into the icon lol because i didn't quote it out

Totally unnecessary addition -- just thought its a nice touch

[cisspUser01]

Do I need to turn on the critical and high alerts on kibana before I can use that in elastalert2? Thank you, From: Andrew Arz ***@***.***> Sent: Friday, May 2, 2025 8:27 AM To: cisagov/LME ***@***.***> Cc: Chenh Hong ***@***.***>; Comment ***@***.***> Subject: Re: [cisagov/LME] Elastalert not sending email (Discussion #659) Glantz External Email Warning: Thoroughly review all content of this email before responding, clicking on any links, or opening any attachments. If anything looks strange please delete the email and contact Glantz IT Not really -- just up to each individually to decide what is important to monitor. Ie. if you're monitoring for specific threat actors -- then use the mitre attack tags to turn on alerts that are tied to those APTs. You could also just turn on critical and high only and see how that goes. There's also probably something that can be done with elastalert2 where it detects the field and if its critical or high it will be sent to your email. And if its not it doesn't. however, its still logged in kibana -- so if you have to map an incident out that still exists Something like this: name: Filtered Kibana Alerts type: any index: .alerts-security.alerts-* filter: - range: ***@***.***": gte: "now-5m" - query_string: query: "kibana.alert.rule.name:*" - terms: severity: ["high", "critical"] # only these will trigger realert: minutes: 0 aggregation: minutes: 0 alert: - "slack" slack_webhook_url: "https://hooks.slack.com/services/..." slack_username_override: "Alert Bot" slack_icon_emoji_override: ":rotating_light:" alert_text_type: alert_text_only alert_text: | 🚨 *New Security Alert* *Severity*: {0} *Rule*: {1} *User*: {2} *Action*: {3} alert_text_args: - severity - kibana.alert.rule.name - user.name - kibana.alert.action Now those alerts will always live in kibana so you dont actually have to turn them off -- but they wont be sent to your slack this scenario — Reply to this email directly, view it on GitHub<#659 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPUNJIGYAE3D6PWXL4LGEWD24NQCXAVCNFSM6AAAAAB4BHEMKKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBRGM3TAMA>. You are receiving this because you commented.Message ID: ***@***.******@***.***>> Chenh Hong Network/Security Manager Glantz 2501 Constant Comment Place Louisville, KY 40299 Tel: 502.568.4429 [https://s3.amazonaws.com/glantz/glantz/content/website/042825eSig.png]<https://www.nglantz.com/search?searchString=4600+GLX&trk=CMP-162399> Shop at Glantz<https://www.nglantz.com> [https://s3.amazonaws.com/glantz/glantz/content/website/facebook-colorful-logo.png]<https://www.facebook.com/GlantzSignSupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/instagram-colorful-logo.png] <https://www.instagram.com/glantzsignsupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/linkedin-colorful-logo.png] <https://www.linkedin.com/company/1303642?trk=tyah&trkInfo=tarId%3A1410786353426%2Ctas%3An%20gl%2Cidx%3A2-2-7> [https://s3.amazonaws.com/glantz/glantz/content/website/youtube-colorful-logo.png] ***@***.***> Disclaimer posted by 766HGC3dXXQ167

[aarz-snl]

yes.. its only going to detect what you turned on in kibana

[cisspUser01]

Thank you. Yes, I have configured some alerts with elastalert2. Included sms/mms when there is alerts via Vonage (formerly Nexmo). From: Andrew Arz ***@***.***> Sent: Friday, May 2, 2025 10:36 AM To: cisagov/LME ***@***.***> Cc: Chenh Hong ***@***.***>; Comment ***@***.***> Subject: Re: [cisagov/LME] Elastalert not sending email (Discussion #659) Glantz External Email Warning: Thoroughly review all content of this email before responding, clicking on any links, or opening any attachments. If anything looks strange please delete the email and contact Glantz IT yes.. its only going to detect what you turned on in kibana — Reply to this email directly, view it on GitHub<#659 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPUNJIHWNA6VXI6SA224RGL24N7E5AVCNFSM6AAAAAB4BHEMKKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBRGUYDAMA>. You are receiving this because you commented.Message ID: ***@***.******@***.***>> Chenh Hong Network/Security Manager Glantz 2501 Constant Comment Place Louisville, KY 40299 Tel: 502.568.4429 [https://s3.amazonaws.com/glantz/glantz/content/website/042825eSig.png]<https://www.nglantz.com/search?searchString=4600+GLX&trk=CMP-162399> Shop at Glantz<https://www.nglantz.com> [https://s3.amazonaws.com/glantz/glantz/content/website/facebook-colorful-logo.png]<https://www.facebook.com/GlantzSignSupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/instagram-colorful-logo.png] <https://www.instagram.com/glantzsignsupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/linkedin-colorful-logo.png] <https://www.linkedin.com/company/1303642?trk=tyah&trkInfo=tarId%3A1410786353426%2Ctas%3An%20gl%2Cidx%3A2-2-7> [https://s3.amazonaws.com/glantz/glantz/content/website/youtube-colorful-logo.png] ***@***.***> Disclaimer posted by 766HGC3dXXQ167

[aarz-snl]

If you can sanitize your configs and share that would be great. Any and all examples we can provide folks would be great

[cisspUser01]

I forget the detail steps on how to do this. I uses ChatGPT (the free one), so it didn’t save my project. But here what I can remember. You have to install vonage in a virtual env. and python 3.12 (need this version to work) inside vonage. I added my vonage api key and secret in the lme-environment. You also have to add the path to elastalert container. Create the py and sh files and put them in the misc of elastalert. You can use any index rules to create your rule. I use “.internal.alerts-security.alerts-dedault-*” You can use elastalert module called Twilio. But they make you jump thru hoops to get sign up with their service. I hope this help. From: Andrew Arz ***@***.***> Sent: Tuesday, May 6, 2025 1:30 PM To: cisagov/LME ***@***.***> Cc: Chenh Hong ***@***.***>; Comment ***@***.***> Subject: Re: [cisagov/LME] Elastalert not sending email (Discussion #659) Glantz External Email Warning: Thoroughly review all content of this email before responding, clicking on any links, or opening any attachments. If anything looks strange please delete the email and contact Glantz IT If you can sanitize your configs and share that would be great. Any and all examples we can provide folks would be great — Reply to this email directly, view it on GitHub<#659 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPUNJIGCXIKRP5BTB6IKQ3D25DWQJAVCNFSM6AAAAAB4BHEMKKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBVGMYDOMQ>. You are receiving this because you commented.Message ID: ***@***.******@***.***>> Chenh Hong Network/Security Manager Glantz 2501 Constant Comment Place Louisville, KY 40299 Tel: 502.568.4429 [https://s3.amazonaws.com/glantz/glantz/content/website/050525eSig.png]<https://www.nglantz.com/search?searchString=3630&facets=fg_brand%7Cfg_brand_3M&facets=fg_series%7Cfg_series_3630&from=1&trk=CMP-144656> Shop at Glantz<https://www.nglantz.com> [https://s3.amazonaws.com/glantz/glantz/content/website/facebook-colorful-logo.png]<https://www.facebook.com/GlantzSignSupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/instagram-colorful-logo.png] <https://www.instagram.com/glantzsignsupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/linkedin-colorful-logo.png] <https://www.linkedin.com/company/1303642?trk=tyah&trkInfo=tarId%3A1410786353426%2Ctas%3An%20gl%2Cidx%3A2-2-7> [https://s3.amazonaws.com/glantz/glantz/content/website/youtube-colorful-logo.png] ***@***.***> Disclaimer posted by 766HGC3dXXQ167 name: "Elastic Defend Alert SMS Notification" type: "any" index: ".internal.alerts-security.alerts-default-*" filter: - term: "event.dataset": "endpoint.alerts" - bool: should: - term: "kibana.alert.severity": "high" - term: "kibana.alert.severity": "medium" timeframe: minutes: 1 alert: - command alert_text: "Elastic Defend Alert" alert_text_type: "alert_text_only" command: - "/usr/local/bin/python3" - "/opt/elastalert/misc/alert_sms.py" - "%(host.hostname)s" - "%(user.name)s" - "%(process.command_line)s" - "%(file.path)s" pipe_match_json: false import sys import os import logging # Add Vonage virtualenv path sys.path.insert(0, "/opt/elastalert/vonage-venv/lib/python3.12/site-packages") import vonage # Log startup and args with open("/tmp/sms_debug.log", "a") as f: f.write("SMS script started\n") f.write(f"Args: {sys.argv}\n") # Parse arguments hostname = sys.argv[1] if len(sys.argv) > 1 else "N/A" username = sys.argv[2] if len(sys.argv) > 2 else "N/A" command_line = sys.argv[3] if len(sys.argv) > 3 else "N/A" file_path = sys.argv[4] if len(sys.argv) > 4 else "N/A" # Build compact alert text alert_text = f"{hostname} | {username} | {command_line} | {file_path}" # Truncate if needed if len(alert_text) > 160: alert_text = alert_text[:157] + "..." # Log final SMS text with open("/tmp/sms_debug.log", "a") as f: f.write(f"Final SMS text: {alert_text}\n") # Vonage setup api_key = os.getenv("VONAGE_API_KEY", "your_api_key") api_secret = os.getenv("VONAGE_API_SECRET", "your_api_secret") from_number = os.getenv("VONAGE_FROM", "xxxxxxxxxxxx") to_number = os.getenv("VONAGE_TO", "xxxxxxxxxxxxx") client = vonage.Client(key=api_key, secret=api_secret) sms = vonage.Sms(client) # Send SMS responseData = sms.send_message({ "from": from_number, "to": to_number, "text": alert_text, }) # Log API response with open("/tmp/sms_debug.log", "a") as f: f.write(f"SMS response: {responseData}\n") #!/bin/bash PYTHONPATH=/opt/elastalert/vonage-venv/lib/python3.12/site-packages python3 /opt/elastalert/misc/alert_sms.py "$1"

[aarz-snl]

Thanks -- config is what I was looking for.

[cisspUser01]

Are you talking about config.yaml for elastalert2. If so, below: run_every: seconds: 10 buffer_time: minutes: 5 writeback_index: elastalert_status alert_time_limit: days: 2 es_host: 'lme-elasticsearch' es_port: 9200 use_ssl: true verify_certs: false #exists in the container rules_folder: /opt/elastalert/rules misc_folder: /opt/elastalert/misc From: Andrew Arz ***@***.***> Sent: Wednesday, May 7, 2025 8:07 AM To: cisagov/LME ***@***.***> Cc: Chenh Hong ***@***.***>; Comment ***@***.***> Subject: Re: [cisagov/LME] Elastalert not sending email (Discussion #659) Glantz External Email Warning: Thoroughly review all content of this email before responding, clicking on any links, or opening any attachments. If anything looks strange please delete the email and contact Glantz IT Thanks -- config is what I was looking for. — Reply to this email directly, view it on GitHub<#659 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPUNJIFVXY5NHWZSARQ4VWD25HZOJAVCNFSM6AAAAAB4BHEMKKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBWGI3DKMQ>. You are receiving this because you commented.Message ID: ***@***.******@***.***>> Chenh Hong Network/Security Manager Glantz 2501 Constant Comment Place Louisville, KY 40299 Tel: 502.568.4429 [https://s3.amazonaws.com/glantz/glantz/content/website/050525eSig.png]<https://www.nglantz.com/search?searchString=3630&facets=fg_brand%7Cfg_brand_3M&facets=fg_series%7Cfg_series_3630&from=1&trk=CMP-144656> Shop at Glantz<https://www.nglantz.com> [https://s3.amazonaws.com/glantz/glantz/content/website/facebook-colorful-logo.png]<https://www.facebook.com/GlantzSignSupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/instagram-colorful-logo.png] <https://www.instagram.com/glantzsignsupplies/> [https://s3.amazonaws.com/glantz/glantz/content/website/linkedin-colorful-logo.png] <https://www.linkedin.com/company/1303642?trk=tyah&trkInfo=tarId%3A1410786353426%2Ctas%3An%20gl%2Cidx%3A2-2-7> [https://s3.amazonaws.com/glantz/glantz/content/website/youtube-colorful-logo.png] ***@***.***> Disclaimer posted by 766HGC3dXXQ167

[aarz-snl]

nope. the elastalert rule you made for it. I got it though. thanks