Another Unifi/iptables syslog parsing question.

[mtnsec]

Has anyone been successful in parsing Unifi iptables syslog yet and if so can you share your configs?

With the upgrade to the Features version, I was hoping to use Elastic's built-in ingest node for iptables/unifi: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-iptables.html

However, I'm having problems executing the filebeat commands that would set this up:

$ sudo docker run --net="host" so-lab:5000/security-onion-solutions/so-filebeat:2.3.40 modules enable iptables
Enabled iptables
$ sudo docker run --net="host" so-lab:5000/security-onion-solutions/so-filebeat:2.3.40 modules list
Enabled:

Disabled:
<snip>
iptables

Did I do something wrong with my filebeat commands to have this be automatically set up?

I have access to a working Elastic instance (version 7.8) with iptables enabled so I also tried copying the ingest parser from the Elasticsearch Ingest Node Pipelines configs that I found in the web application and I've been trying to integrate those with SO's syslog parser without luck (I used these super helpful guides #3175 and #3547 - many thanks to the author!!). I've fixed all of the errors that came up, but after restarting all of the elastic components I seem to have lost ALL syslog parsing so I've clearly messed up somewhere. :(

I'm hoping that someone else has already succeeded and can help out - or can point me in the right direction to get this working. Thanks!

[weslambert]

Let me test this and get back with you.

Thanks,
Wes

[mtnsec]

Hi, just checking in on this. I'm still not getting anywhere with filebeats or the parser I was able to grab from the elastic instance..

I would rather not reinvent the wheel when this is something that's supported in Elastic, so any help getting this working in SO would be appreciated.

Thanks!

[mtnsec]

Hi, any updates?

[mtnsec]

It's been a couple of weeks now - just wondering if there are any updates. Thanks!

[pboosten]

Hi, I couldn't get this to work either, but I was wondering what you're trying to achieve with this?

In a 'normal' filebeat -> elasticsearch environment enabling that module will have filebeat listening on port 9001 for incoming syslog traffic. Filebeat on SO is already listening on 514 by default. Also, enabling that module in normal operation will create an ingest pipeline in elastic for iptables, which also fails when trying to do the same in SO (since logstash is in between).

The way I did this was as follows:
I copied the default filebeat ingest pipeline yaml file, converted it to json, modified it, and uploaded it to elasticsearch (via salt). Also, I modified the original syslog pipeline in SO, to make it parse the lines and refer to the newly uploaded iptables ingest pipeline.

Took a bit of experimenting, since documentation is very sparse on this subject, but finally got it working.

Peter

[mtnsec]

Also, I modified the original syslog pipeline in SO, to make it parse the lines and refer to the newly uploaded iptables ingest pipeline.

This is what I can't figure out. The filebeat command I'm trying to run should create those pipelines automatically. That's the only part I'm trying to make work and/or I'm looking for the correct code to put into a new or modified pipeline file.

I am successfully receiving the syslog, and I can search for it in kibana, but the fields are not being applied. I have done this in the past with filebeat to set up the ingest pipeline on the elastic server and syslog on the sender, but that was with a standard elastic stack installation. I don't want to reinvent the wheel and write a parser manually when it's built into elastic stack. And I don't want to have to run separate elastic installs when SO has one built in that should support what I'm doing.

[pboosten]

Creating pipelines automatically is only possible (according to the logs in SO) if your filebeat connects directly to elasticsearch. In SO there's a logstash component inbetween.

I tried once to connect directly to elastic in SO from an external filebeat, and then indeed the pipeline is created, however there are two things that don't work 'out of the box':

1. a standard filebeat (so outside of SO) will create indexes called 'filebeat--', while SO writes indexes as 'so-syslog/beats/whatever'. That means modifying of the external filebeat component (not so hard). I never went down this path. Without this modification default dashboards won't work. disclaimer: I'm not even sure this works in a distributed environment, since there's also a redis service inbetween. I assume that the master in that case will be designated as ingest node, but I'm not certain about this.

2. the SO-filebeat won't use that pipeline ever, because it's standard pipeline is 'syslog'. You would have to modify that pipeline to go through yours.

Maybe there are smarter ways in doing this all, I just haven't found them, because of lack of (detailed) documentation -> a reference to 'Pipelines on the elasticsearch site' isn't detailed documentation.

Peter

[danilansible]

Hi @mtnsec and @pboosten ,

Im exactly at the same point as you were in ensuring the the filebeat modules work with proper indexing to leverage default dashboards.

was there any luck further on this topic, id you make this work.
Can you please share. Thanks

[MrLasVegasSTL]

I was able to get the dashboards imported to SO correctly, but I am having trouble getting SO to ingest the logs and create a index. I think this configuration is %95 of the way there, maybe someone can help me polish this off..

Here is what I tried..

I translated the same documentation SO provided from their netflow video (https://www.youtube.com/watch?v=ew5gtVjAs7g) to the iptables module of filebeat.

Here is that generic configuration..

Update docker and the filebeat module

vi /opt/so/saltstack/local/pillar/minions/onion_standalone.sls

filebeat:
  third_party_filebeat:
    modules:
      iptables:
        log:
          enabled: true
          var.syslog_host: 0.0.0.0
          var.syslog_port: 9001

cp /opt/so/saltstack/default/salt/filebeat/init.sls /opt/so/saltstack/local/salt/filebeat/

chown socore:socore /opt/so/saltstack/local/salt/filebeat/init.sls

vi /opt/so/saltstack/local/salt/filebeat/init.sls

- port_bindings:
    - 0.0.0.0:9001:9001/udp

salt-call state.apply filebeat

Update Firewall

so-firewall addhostgroup iptables
so-firewall addportgroup iptables
so-firewall includehost iptables 10.0.0.0/8
so-firewall addport iptables udp 9001

vi /opt/so/saltstack/local/pillar/minions/onion_standalone.sls

firewall:
  assigned_hostgroups:
    chain:
      DOCKER-USER:
        hostgroups:
          iptables:
            portgroups:
              - portgroups.iptables
      INPUT:
        hostgroups:
          iptables:
            portgroups:
              - portgroups.iptables

salt-call state.apply firewall

Generate logstash pipelines to ingest iptables.

This is where I am stuck at, it Errors "Exiting: module iptables is configured but has no enabled filesets"

docker exec -i so-filebeat filebeat setup modules -pipelines -modules iptables -c /usr/share/filebeat/module-setup.yml

so-filebeat-restart

To import the dashboards, I exported the iptables objects from a elasticsearch/kibana setup, and in the json file changes filebeat-* to so-*.

Attached below, save it as .ndjson and import it to SO Kibana saved objects.
iptables.txt

[MrLasVegasSTL]

#8481

This article helped get this working, the trick was to create this file and use it instead. https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/salt/filebeat/thirdpartydefaults.yaml

/opt/so/saltstack/local/salt/filebeat/thirdpartydefaults.yaml