elastalert and jinja not working

[mtoback]

We are trying to send alerts to Hive and it doesn't seem like elastalert is converting it

I tried starting with the yaml here: https://github.com/Security-Onion-Solutions/securityonion/blob/5b16a6542231877906cb5f2ec0032a5ab50fd43f/salt/elastalert/files/rules/so/suricata_thehive.yaml

the top lines don't seem to work, but I was able to work around it by hard coding the four values.

But when I try to send it I am getting hard coded values like an ip of ip: {match[source][ip]}

It's like it doesn't know jinja exists?

also should we be using things like ip: '{match[source][ip]}' or like ip: '{match[source.ip]}' ?

When I run so-elastalert-test it references items like source.ip

This is what my script looks like:

Elastalert rule to forward Suricata alerts from Security Onion to a specified TheHive instance.

  2 #
  3 es_host: 10.7.70.146
  4 es_port: 9200
  5 name: Suricata-Alert
  6 type: any
  7 index: "*:so-ids-*"
  8 buffer_time:
  9     minutes: 5
 10 query_key: ["rule.uuid","source.ip","destination.ip"]
 11 realert:
 12     days: 1
 13 filter:
 14 - query:
 15    query_string:
 16       query: "event.module: suricata AND rule.severity:(1 OR 2)"
 17 
 18 alert: hivealerter
 19 
 20 hive_connection:
 21   hive_host: http://x.x.x.x
 22   hive_port: 9000
 23   hive_apikey: xxxxxxxxxx
 24 
 25 hive_proxies:
 26   http: ''
 27   https: ''
 28 
 29 hive_alert_config:
 30   title: "{{match[rule.name]}}"
 31   type: "NIDS"
 32   source: "SecurityOnion"
 33   description: "SOC Hunt Pivot: \n\n <https://10.7.70.146/#/hunt?q=network.community_id%3A%20%20%22{{match[network.community_id]}}%22%20%7C%20groupby%20source.ip%20destination.ip,event.module,%20event.dataset>  \n\n `Kibana Dashboard Pivot:` \n\n <https://10.7.70.        146/kibana/app/kibana#/dashboard/30d0ac90-729f-11ea-8dd2-9d8795a1200b?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'2289a0c0-6970-11ea-a0cd-ffa0f6a1bc29',key:network.community_id,negate:!f,params:(query:{{match[network.community_id]        }}),type:phrase),query:(match_phrase:(network.community_id:{{match[network.community_id]}})))),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))>  \n\n `IPs: {{match[source.ip]}}:{{match[source.port]}} --> {{match[destination.ip]}}:{{match[destination.        port]}} \n\n Signature:{{match[rule.rule]}}"
 34   severity: 2
 35   tags: ["{{match[rule.uuid]}}","{{match[source.ip]}}","{{match[destination.ip]}}"]
 36   tlp: 3
 37   status: "New"
 38   follow: True
 39 
 40 hive_observable_data_mapping:
 41   - ip: "{{match[source.ip]}}"
 42   - ip: "{{match[destination.ip]}}"
 
 and this is what I am seeing in Hive for the description:
 
 SOC Hunt Pivot:

https://xx.xx.xx.xx/#/hunt?q=network.community_id%3A%20%20%22{{match[network.community_id]}}%22%20%7C%20groupby%20source.ip%20destination.ip,event.module,%20event.dataset

Kibana Dashboard Pivot:

https://xx.xx.xx.xx/kibana/app/kibana#/dashboard/30d0ac90-729f-11ea-8dd2-9d8795a1200b?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'2289a0c0-6970-11ea-a0cd-ffa0f6a1bc29',key:network.community_id,negate:!f,params:(query:{{match[network.community_id]}}),type:phrase),query:(match_phrase:(network.community_id:{{match[network.community_id]}})))),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))

`IPs: {{match[source.ip]}}:{{match[source.port]}} --> {{match[destination.ip]}}:{{match[destination.port]}}

Signature:{{match[rule.rule]}}

[InfosecGoon]

The YAML that you're starting with was used by Salt to generate the config file via Jinja -- Elastalert doesn't actually interpret Jinja on its own, so that's why its not converting the matches properly.

I would suggest removing the Jinja pieces and using the standard Elastalert syntax for hive_alert_config and hive_observable_data_mapping -- documentation is available here: https://elastalert2.readthedocs.io/en/latest/ruletypes.html#thehive

--Matt

[mtoback]

I tried that, and it's not clear how security onion fields are mapped to the yaml. I tried this and all I see are 400 errors (Bad request):
1. # Elastalert rule to forward Suricata alerts from Security Onion to a specified TheHive instance.
2 #
3 es_host: 10.7.70.146
4 es_port: 9200
5 name: Suricata-Alert
6 type: any
7 index: ":so-ids-"
8 buffer_time:
9 minutes: 5
10 query_key: ["rule.uuid","source.ip","destination.ip"]
11 realert:
12 days: 1
13 filter:
14 - query:
15 query_string:
16 query: "event.module: suricata AND rule.severity:(1 OR 2)"
17
18 alert: hivealerter
19
20 hive_connection:
21 hive_host: http://x.x.x.x
22 hive_port: nnnn
23 hive_apikey: xxxxxxxxx
24
25 hive_proxies:
26 http: ''
27 https: ''
28
29 hive_alert_config:
30 title: rule.name
31 type: 'NIDS'
32 source: 'SecurityOnion'
33 description_args: [rule.name, message]
34 description: '{0} : {1}'
35 severity: 2
36 tags: ['source.ip', 'destination.ip']
37 tlp: 3
38 status: 'New'
39 follow: True
40
41 hive_observable_data_mapping:
42 - ip: source.ip
43 tlp: 2
44 ip: destination.ip

[mtoback]

ok progress. It's gettiing to hive and I'm seeing the IPs. Can't set the title so I hard coded it (should be rule.title but that caused an error? and description is set to message but I'm literally seeing "message"

  1 # Elastalert rule to forward Suricata alerts from Security Onion to a specified TheHive instance.
  2 #
  3 es_host: 10.7.70.146
  4 es_port: 9200
  5 name: Suricata-Alert
  6 type: any
  7 index: "*:so-ids-*"
  8 buffer_time:
  9     minutes: 5
 10 query_key: ["rule.uuid","source.ip","destination.ip"]
 11 realert:
 12     days: 1
 13 filter:
 14 - query:
 15    query_string:
 16       query: "event.module: suricata AND rule.severity:(1 OR 2)"
 17 
 18 alert: hivealerter
 19 
 20 hive_connection:
 21   hive_host: http://x.x.x.x
 22   hive_port: nnnn
 23   hive_apikey: xxxxxxxxxxxxxxx
 24 
 25 hive_proxies:
 26   http: ''
 27   https: ''
 28 
 29 hive_alert_config:
 30   title: 'Security Onion Alert'
 31   type: 'NIDS'
 32   source: 'SecurityOnion'
 33   description: message
 34   severity: 2
 35   tags: [source.ip, destination.ip]
 36   tlp: 3
 37   status: 'New'
 38   follow: True
 39 
 40 hive_observable_data_mapping:
 41   - ip: source.ip
 42   - ip: destination.ip

[InfosecGoon]

Try this:

description: '{0}'
description_args: [message]

I haven't tested, but this looks like the proper syntax from the docs.