Fortigate Syslog Parsing

[StarkZarn]

I'm back in the Security Onion game after a hiatus of around a year or so. Previously my setup included logs sent from a pfSense firewall. These were automatically ingested and parsed into searchable fields wonderfully! Totally hands off.

Now, I'm using a Fortigate and shipping its logs to SO. The logs arrive fine, and I can see them indexed as "syslog" category logs, but they're woefully unparsed. I did some research on how to setup my own parsing, but it looks like with this new major version of SO there's a lot less public discussion about how things work. I learned that rather than using logstash, it's using native elasticsearch parsers now...? My google skills may be failing me, but I can't find any documentation on how this works.

Do I need to write grok patterns to pull out every single field? Or do I just need to get the logs tagged as "firewall" and some magical things happen behind the curtain?

Anybody with Fortinet products out there using security onion as their SIEM?

[weslambert]

We have native firewall log parsing for pfsense, but you'll need to have your own parsing configured to be able to parse the incoming syslog, etc. If you can provide some example records, we may be able to assist in getting you pointed in the right direction.

[StarkZarn]

I'll take a look at this and get back with you.

Thank you!

[stevewillia]

I've got a parser working for sonicwall logs and this is how I came up with it

Log on to your so console and launch Kibana. Click on menu and choose dev tools.

I used the below post for checking. When you execute it will show you the results on the right.

I found it easiest to start from the first field on the left and go one by one.

POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "sonicwall.firewall",
"processors": [
{
"dissect": {
"field": "message",
"pattern" : "id=%{device} sn=%{serial} time="%{syslog_timestamp}" fw=%{fw_address} pri=%{pri} c=%{c_id} m=%{msg_id} msg="%{rule.name}" n=%{seq} src=%{source.ip}:%{source.port}:%{fwlan} dst=%{destination.ip}:%{destination.port}:%{fw_eth} proto=%{network.transport}/%{proto} sent=%{bytes} dpi=%{dpi} fw_action="%{rule.action}""
}
}
]
},
"docs": [
{
"_source": {
"message": "id=firewall sn=C0EAE4EAEB50 time="2021-02-08 20:43:08 UTC" fw=216.147.227.82 pri=6 c=262144 m=98 msg="Connection Opened" n=58091987 src=xx.xx.100.19:58615:X0 dst=xx.xx.161.212:443:X1 proto=tcp/https sent=60 dpi=0 fw_action="NA""
}
}
]
}

You should be able to figure out how I did it by looking at the dissect processor. The only tricky part is in your syslog message if it has quotes in it " you will need to escape every one of them. i.e. "

It's only for the testing data. Real data will not have it.

Try it with one log first.

After you get the data like you want I went and cleaned it up for the final processor.
This actually puts it in Elasticsearch

PUT _ingest/pipeline/sonicwall.firewall
{
"description": "sonicwall.firewall",
"processors": [
{
"dissect": {
"field": "message",
"pattern" : "id=%{device} sn=%{serial} time="%{syslog_timestamp}" fw=%{fw_address} pri=%{pri} c=%{c_id} m=%{msg_id} msg="%{rule.name}" n=%{seq} src=%{source.ip}:%{source.port}:%{fwlan} dst=%{destination.ip}:%{destination.port}:%{fw_eth} proto=%{network.transport}/%{proto} sent=%{bytes} dpi=%{dpi} fw_action="%{rule.action}""
}
},
{ "rename":{ "field": "id", "target_field": "event.module", "ignore_failure": true } },
{ "set": { "field": "_index", "value": "so-sonicwall", "override": true } },
{ "set": { "if": "ctx.network?.transport_id == '0'", "field": "network.transport", "value": "icmp", "override": true } },
{"community_id": { "if": "ctx.network?.transport != null", "field":["source.ip","source.port","destination.ip","destination.port","network.transport"],"target_field":"network.community_id"}},
{ "set": { "field": "module", "value": "sonicwall", "override": true } },
{ "set": { "field": "dataset", "value": "firewall", "override": true } },
{ "set": { "if": "ctx.product != null", "field": "dataset", "value": "{{ product }}", "ignore_failure": true } },
{ "set": { "field": "@timestamp", "value": "{{ syslog_timestamp }}" } },
{ "remove": { "field": ["message", "sn", "pri", "fw_address", "c", "msg_id", "n", "fw_eth", "proto", "sent", "dpi"], "ignore_missing": true, "ignore_failure": true } },
{ "pipeline": { "name": "sonicwall.firewall"} }

],
"on_failure" : [
{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_processor_tag }}"
}
}
]
}

Then you can run tests with the actual processor

POST _ingest/pipeline/sonicwall.firewall/_simulate
{
"docs": [
{
"_index": "so-sonicwall",
"_id": "id",
"_source": {
"message": "id=firewall sn=C0EAE4EAEB50 time="2021-02-08 20:43:08 UTC" fw=216.147.227.82 pri=6 c=262144 m=98 msg="Connection Opened" n=58091987 src=99.99.100.19:58615:X0 dst=99.99.161.212:443:X1 proto=tcp/https sent=60 dpi=0 fw_action="NA""
}
}
]
}

Delete the processor

DELETE /_ingest/pipeline/sonicwall.firewall

Show the processor in Elasticsearch

GET /_ingest/pipeline/sonicwall.firewall

Hope this helps
Steve

[stevewillia]

Also you can disregard the <188>. that is the facilty and priority of the syslog message.

[StarkZarn]

Thanks for that tidbit! I did not know about this tool. Now I just have to adjust my mind from logstash parsers to elasticsearch pipelines... Seems similar enough but also rather foreign to me all together. I'm working on some test messages now.

[stevewillia]

One more thing if you use single quotes ' around your test message you do not have to escape the double quotes.

Steve

[stevewillia]

I sure would like to know how you got them to go into an index. I've been trying for weeks with some sonic wall logs. I've run so-allow for the syslog and I know the logs are coming to the device, but nothing in syslog or an index.

Sonicwall logs are similar to yours. They are a key value pair.

Here is a sample log.
<134> id=firewall sn=XXXXXX347C70 time="2021-02-03 07:23:33" fw=xx.xx.127.198 pri=6 c=1024 m=97 app=48 n=1570020 src=xx.xx.2.58:50728:X0 dst=23.208
.132.89:80:X1 srcMac=xx:xx:01:9e:56:3c dstMac=xx:xx:d3:bb:84:d0 proto=tcp/http op=1 sent=492 rcvd=428 dpi=0 dstname=officecdn.microsoft.com arg=/pr/4923
50f6-3a01-4f97-b9c0-c7c6ddf67d60/Office/Data/16.0.13628.20274/stream.x64.x-none.dat.phf code=27 Category="Information Technology/Computers" note="Policy
: cfsZonePolicy0, Info: 6148 " rule="6 (LAN->WAN)" fw_action="NA"

I know the logs need an ingest processor for elastic search. I created one and it is in Elasticsearch. I can see it when I run

curl -X GET "localhost:9200/_ingest/pipeline/sonicwall?pretty"

But nothing ever runs through it. On the query above it has statistics which show how many times each section was executed. They are all zero.

I've posted my sonic wall ingest parser. I know it loads but do not know if it's accurate with the data.

sonicwall ingest.txt

Steve

[TOoSmOotH]

Did you create a logstash ouput policy to send that data to that pipeline?

example:
https://github.com/Security-Onion-Solutions/securityonion/blob/master/salt/logstash/pipelines/config/so/9000_output_zeek.conf.jinja

[stevewillia]

Yes I verified with tcpdump and ngrep.

I'll get back to you on Logstash.

[stevewillia]

so-allow is broken right now

[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "", line 2, in top-level template code
jinja2.exceptions.UndefinedError: 'list object' has no attribute 'values'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 505, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'list object' has no attribute 'values'
[CRITICAL] Rendering SLS 'base:firewall' failed: Jinja variable 'list object' has no attribute 'values'
local:
Data failed to compile:

Rendering SLS 'base:firewall' failed: Jinja variable 'list object' has no attribute 'values'

[StarkZarn]

No such luck here yet.

[stevewillia]

I've verified the parser doing the simulation and it works and loads in Elasticsearch.

I've created this output policy and I put it here
/opt/so/saltstack/local/salt/logstash/pipelines/config/custom/9999_sonicwall

{%- if grains['role'] == 'so-eval' -%}
{%- set ES = salt'pillar.get' -%}
{%- else %}
{%- set ES = salt'pillar.get' -%}
{%- endif %}
output {
if [module] =~ "sonicwall" {
elasticsearch {
pipeline => "%{module}.%{dataset}"
hosts => "1xx.xx.100.252"
index => "so-firewall"
template_name => "so-firewall"
template => "/templates/so-firewall-template.json"
template_overwrite => true
{%- if grains['role'] in ['so-node','so-heavynode'] %}
ssl => true
ssl_certificate_verification => false
{%- endif %}
}
}
}

and I've added it to
global.sls

logstash:
pipelines:
search:
config:
- custom/9900_sonicwall

and I get this error

      ID: so-logstash
Function: docker_container.running
  Result: False
 Comment: One or more requisite failed: logstash.ls_pipeline_search_custom_9900_sonicwall, logstash.ls_pipeline_search
 Started: 23:57:31.505908
Duration: 0.008 ms
 Changes:

---

So I do not know what the issue is.

Steve

[itpropaul]

Did a parser for Fortigate syslog messages ever get created here?
We're looking to potentially do the same ourselves either via Zeek capturing those or possibly sending them directly to SO.

[StarkZarn]

Did a parser for Fortigate syslog messages ever get created here? We're looking to potentially do the same ourselves either via Zeek capturing those or possibly sending them directly to SO.

I started using the Fortinet filebeat to do it for me. Works okay. All the older style dashboards in kibana don't work still, as it's not classifying DHCP, DNS, "Weird", etc traffic. It does populate as network logs though, with a community_id and everything.

Check out: https://docs.securityonion.net/en/2.3/filebeat.html?highlight=filebeat

[itpropaul]

10-4

[amorphys]

Hi, can you tell me if is working as expected on port 9004 or did you use the default port 514 for syslog
Can you show me a which dashboard did you use in kibana ?

[rosswakelin]

Native Elastic already has ingest pipelines and support for Fortigate logs.