Filebeat panw module, no enabled filesets?

[endpointsec]

Version: 2.3.140
On-Prem, standalone installation
Installed from ISO

Issue:

Following the filebeat Netflow example to permit Palo Alto logs, Building the logstash pipeline produces an error.

docker exec -i so-filebeat filebeat setup modules -pipelines -modules panw -c /usr/share/filebeat/module-setup.yml
Exiting: module panw is configured but has no enabled filesets

Enable 3rd Party Module:

/opt/so/saltstack/local/pillar/minions/*.sls

filebeat:
  third_party_filebeat:
    modules:
      panw:
        panos:
          enabled: true
          var.syslog_host: 0.0.0.0
          var.syslog_port: 9002

Update firewall configuration:

so-firewall addhostgroup palo
so-firewall addportgroup palo
so-firewall includehost palo INTERNALSUBNET/24
so-firewall addport palo udp 9002

/opt/so/saltstack/local/pillar/minions/*.sls

firewall:
  assigned_hostgroups:
    chain:
      DOCKER-USER:
        hostgroups:
          palo:
            portgroups:
              - portgroups.palo

Troubleshooting:

• salt-call state.highstate: No errors
• so-status: All services OK.
• tcpdump shows traffic coming to SO on port 9002
• Debugged filebeat logging and palo logs are evident at /opt/so/log/filebeat
• so-filebeat-restart: Succeeded: 140 (changed=6) Failed: 0
• salt-call state.apply common: Succeeded: 37 (changed=1) Failed: 0
• so-filebeat-module-setup: panw.panos Loaded Ingest pipelines
• so-elasticsearch-pipelines-list | grep panw

  "filebeat-7.17.4-panw-panos-globalprotect",
  "filebeat-7.17.4-panw-panos-hipmatch",
  "filebeat-7.17.4-panw-panos-pipeline",
  "filebeat-7.17.4-panw-panos-threat",
  "filebeat-7.17.4-panw-panos-traffic",
  "filebeat-7.17.4-panw-panos-userid",
  "filebeat-8.3.2-panw-panos-globalprotect",
  "filebeat-8.3.2-panw-panos-hipmatch",
  "filebeat-8.3.2-panw-panos-pipeline",
  "filebeat-8.3.2-panw-panos-threat",
  "filebeat-8.3.2-panw-panos-traffic",
  "filebeat-8.3.2-panw-panos-userid",

• iptables --list -n | grep 9002
  ACCEPT udp -- INTERNALSUBNET/24 0.0.0.0/0 udp dpt:9002
• docker ps | grep 9002
  "/usr/local/bin/dock…" 0.0.0.0:514->514/tcp, 0.0.0.0:5066->5066/tcp, 0.0.0.0:514->514/udp, 0.0.0.0:9002->9002/tcp, 0.0.0.0:9002->9002/udp so-filebeat

Related topics:
#8481
#8392

Any suggestions on what to try next? I can't seem to build the pipeline for the data to show in Elastic.

[weslambert]

It looks like panw is already listed in the pipeline output (for several pipelines). Have you confirmed anything is hitting the intended pipeline (using so-elasticsearch -pipeline-stats $pipeline)?

Thanks,
Wes

[endpointsec]

Hi Wes, thank you for the reply. It doesn't look like it.

pipeline:filebeat-8.3.2-panw-panos-traffic": { "type": "conditional", "stats": { "count": 0, "time_in_millis": 0, "current": 0, "failed": 0

"pipeline:filebeat-7.17.4-panw-panos-traffic": { "type": "conditional", "stats": { "count": 0, "time_in_millis": 0, "current": 0, "failed": 0

Inside the debug filebeat logs, I do see panw module traffic logs from the fw appliance:

"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.3.2\",\n \"pipeline\": \"filebeat-8.3.2-panw-panos-pipeline\"

"filebeat-8.3.2-panw-panos-pipeline": { "count": 0, "time_in_millis": 0, "current": 0, "failed": 0, "processors": [ { "set": { "type": "set", "stats": { "count": 0, "time_in_millis": 0, "current": 0, "failed": 0

[endpointsec]

Still haven't gotten this to work -- anyone have any tips to investigate? Debugged filebeat again and quickly generating a ton of logs but does not appear to go anywhere. I have never had a so-panw* index created.

Today filebeat ndjson log example:
{"log.level":"debug","@timestamp":"2022-08-25T12:16:26.641Z","log.logger":"logstash","log.origin":{"file.name":"logstash/async.go","file.line":172},"message":"145 events out of 145 events sent to logstash host securityonion2:5644. Continue sending","service.name":"filebeat","ecs.version":"1.6.0"}

"message":"Publish event: {\n \"@timestamp\": \"2022-08-25T08:17:40.000Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.3.3\",\n \"truncated\": false,\n \"pipeline\": \"filebeat-8.3.3-panw-panos-pipeline\"\n },\n \"panw\"

[endpointsec]

Yes, unfortunately. No progress has been made.

I ran (zcat *gz | grep -i panw) and also cat grep on the log files for Logstash and Elasticsearch. No hits for either term. I enabled filebeat debug again and instantly get large (11mb) ndjson files created in so/log/filebeat which is littered with a lot of the palo logs showing "version": "8.3.3",\n "pipeline": "filebeat-8.3.3-panw-panos-pipeline" within.

This was a fresh install about a month ago and the only mod I did was a custom 0009 input to enable winlogbeat encryption. Zero issues with winlogbeat coming in.

[root@xxx]# docker exec -i so-filebeat filebeat setup modules -pipelines -modules panw -c /usr/share/filebeat/module-setup.yml
Exiting: module panw is configured but has no enabled filesets

so-filebeat-module-setup
Loaded Ingest pipelines
panw.panos

so-elasticsearch-query _cat/indices | grep pan
No results

so-elasticsearch-pipelines-list | grep panw
"filebeat-7.17.4-panw-panos-globalprotect",
"filebeat-7.17.4-panw-panos-hipmatch",
"filebeat-7.17.4-panw-panos-pipeline",
"filebeat-7.17.4-panw-panos-threat",
"filebeat-7.17.4-panw-panos-traffic",
"filebeat-7.17.4-panw-panos-userid",
"filebeat-8.3.2-panw-panos-globalprotect",
"filebeat-8.3.2-panw-panos-hipmatch",
"filebeat-8.3.2-panw-panos-pipeline",
"filebeat-8.3.2-panw-panos-threat",
"filebeat-8.3.2-panw-panos-traffic",
"filebeat-8.3.2-panw-panos-userid",
"filebeat-8.3.3-panw-panos-globalprotect",
"filebeat-8.3.3-panw-panos-hipmatch",
"filebeat-8.3.3-panw-panos-pipeline",
"filebeat-8.3.3-panw-panos-threat",
"filebeat-8.3.3-panw-panos-traffic",
"filebeat-8.3.3-panw-panos-userid",

Spun up 2.3.160 in a fresh VM. Edited the minions SLS file to enable the panw module.
Ran so-filebeat-module-setup and panw is ingested.
so-elasticsearch-pipeslies-list | grep panw (confirms this).
Go to execute the docker command but am told no enabled filesets.

[weslambert]

The Docker command syntax changed with Elastic 8. However, you shouldn't need to run that if you have everything correctly defined in the config and the pipelines are loaded.

Are you able to share some sample data of what is actually being sent to Security Onion?

[weslambert]

Following up on this -- were you able to get the issue resolved?

[endpointsec]

I have not been able to. Here is one sample from the filebeat debug:

{"log.level":"debug","@timestamp":"2022-09-15T11:58:52.179Z","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":210},"message":"Publish event: {\n \"@timestamp\": \"2022-09-15T07:58:52.000Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.3.3\",\n \"truncated\": false,\n \"pipeline\": \"filebeat-8.3.3-panw-panos-pipeline\"\n },\n \"tags\": [\n \"pan-os\",\n \"forwarded\"\n ],\n \"input\": {\n \"type\": \"syslog\"\n },\n \"destination\": {\n \"packets\": \"4\",\n \"ip\": \"xxxxx\",\n \"address\": \"xxxxx\",\n \"nat\": {\n \"ip\": \"0.0.0.0\",\n \"port\": \"0\"\n },\n \"port\": \"443\",\n \"bytes\": \"392\"\n },\n \"hostname\": \"xxxx.com\",\n \"event\": {\n \"created\": \"2022/09/15 07:58:52\",\n \"start\": \"2022/09/15 07:57:33\",\n \"duration\": \"77\",\n \"severity\": 6,\n \"module\": \"panw\",\n \"dataset\": \"panw.panos\",\n \"timezone\": \"+00:00\"\n },\n \"log\": {\n \"source\": {\n \"address\": \"xxxxx:55560\"\n }\n },\n \"fileset\": {\n \"name\": \"panos\"\n },\n \"service\": {\n \"type\": \"panw\"\n },\n \"observer\": {\n \"product\": \"PAN-OS\",\n \"type\": \"firewall\",\n \"serial_number\": \"xxxxx\",\n \"ingress\": {\n \"zone\": \"xxxxx-zone\",\n \"interface\": {\n \"name\": \"xxx.xx\"\n }\n },\n \"egress\": {\n \"zone\": \"xxxxx-zone\",\n \"interface\": {\n \"name\": \"xxx.xx\"\n }\n },\n \"hostname\": \"xxxxx\",\n \"vendor\": \"Palo Alto Networks\"\n },\n \"_temp_\": {\n \"ietf_header\": \"1\",\n \"generated_time\": \"2022/09/15 07:58:52\",\n \"labels\": \"0x100001c\",\n \"external_zones\": [\n \"untrust\"\n ],\n \"internal_zones\": [\n \"trust\"\n ]\n },\n \"client\": {\n \"ip\": \"xxxxx\",\n \"nat\": {\n \"ip\": \"0.0.0.0\",\n \"port\": \"0\"\n },\n \"user\": {\n \"name\": \"xxxxx\\\\xxxxx\"\n },\n \"port\": \"53136\",\n \"bytes\": \"1010\",\n \"packets\": \"7\"\n },\n \"syslog\": {\n \"severity_label\": \"Informational\",\n \"facility\": 1,\n \"facility_label\": \"user-level\",\n \"priority\": 14\n },\n \"ecs\": {\n \"version\": \"1.12.0\"\n },\n \"agent\": {\n \"type\": \"filebeat\",\n \"version\": \"8.3.3\",\n \"ephemeral_id\": \"xxxxx\",\n \"id\": \"xxxxx\",\n \"name\": \"securityonion2\"\n },\n \"server\": {\n \"bytes\": \"392\",\n \"packets\": \"4\",\n \"ip\": \"xxxxx\",\n \"nat\": {\n \"ip\": \"0.0.0.0\",\n \"port\": \"0\"\n },\n \"port\": \"443\"\n },\n \"panw\": {\n \"panos\": {\n \"type\": \"TRAFFIC\",\n \"virtual_sys\": \"vsys1\",\n \"url\": {\n \"category\": \"any\"\n },\n \"flow_id\": \"3452911\",\n \"action\": \"allow\",\n \"sequence_number\": \"7143197660338123367\",\n \"endreason\": \"tcp-fin\",\n \"sub_type\": \"end\",\n \"ruleset\": \"xxxxx\",\n \"source\": {\n \"zone\": \"xxxxx\",\n \"interface\": \"xxxxx\",\n \"nat\": {\n \"ip\": \"0.0.0.0\",\n \"port\": 0\n }\n },\n \"destination\": {\n \"zone\": \"xxxxx\",\n \"interface\": \"xxxxx\",\n \"nat\": {\n \"ip\": \"0.0.0.0\",\n \"port\": 0\n }\n }\n }\n },\n \"source\": {\n \"user\": {\n \"name\": \"xxxxx\\\\xxxxx\"\n },\n \"port\": \"53136\",\n \"bytes\": \"1010\",\n \"packets\": \"7\",\n \"address\": \"xxxxx\",\n \"ip\": \"xxxxx\",\n \"nat\": {\n \"ip\": \"0.0.0.0\",\n \"port\": \"0\"\n }\n },\n \"network\": {\n \"application\": \"ssl\",\n \"transport\": \"tcp\",\n \"bytes\": \"1402\",\n \"packets\": \"11\"\n },\n \"message\": \"1,2022/09/15 07:58:52,xxxxx,TRAFFIC,end,2305,2022/09/15 07:58:52,xxxxx,xxxxx,0.0.0.0,0.0.0.0,xxxxx,xxxxx\\\\xxxxx,,ssl,vsys1,xxxxx,xxxxx,xxxxx,xxxxx,Syslog,2022/09/15 07:58:52,3452911,1,53136,443,0,0,0x100001c,tcp,allow,1402,1010,392,11,2022/09/15 07:57:33,77,any,0,xxxxx,0x0,10.0.0.0-10.255.255.255,United States,0,7,4,tcp-fin,0,0,0,0,,xxxxx,from-policy,,,0,,0,,N/A,0,0,0,0,xxxxx,0,0,,,,,,,\"\n}","service.name":"filebeat","ecs.version":"1.6.0"}

Additionally

{"log.level":"info","@timestamp":"2022-09-22T11:08:58.752Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":108},"message":"Enabled modules/filesets: panw (panos)","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.755Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://securityonion2:9200","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.779Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.3.3","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.788Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.3.3-panw-panos-pipeline","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.790Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.3.3-panw-panos-traffic","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.791Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.3.3-panw-panos-threat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.792Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.3.3-panw-panos-globalprotect","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.793Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.3.3-panw-panos-userid","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-09-22T11:08:58.794Z","log.logger":"modules","log.origin":{"file.name":"fileset/pipelines.go","file.line":133},"message":"Elasticsearch pipeline loaded.","service.name":"filebeat","pipeline":"filebeat-8.3.3-panw-panos-hipmatch","ecs.version":"1.6.0"}

[weslambert]

Thanks, I'll see if I can l look into this more as I get time.

[kpsmiley23]

I ran into this as well and got around it by:

1. moving the panw stanza from /opt/so/saltstack/local/pillar/minions/*.sls to /opt/so/saltstack/local/salt/filebeat/thirdpartydefaults.yaml
2. removing the port mapping from /opt/so/saltstack/local/salt/filebeat/init.sls
3. restart the so-filebeat container

Hope this helps!

[endpointsec]

Thanks for the reply. I did not edit the init.sls because enabling the module alone opens up the port specified. I did create the thirdpartydefaults.yaml at that location but it has not resolved the issue. Glad you got yours working though!

[justdiver]

Were you ever able to resolve this issue? I seem to be having the same problem, and I've tried all the same suggestions that you have. I'm banging my head against this right now and I can't solve it.

[kpsmiley23]

Sorry @justdiver I’ve since moved on to another tool, but if the above suggestions don’t work for you maybe try adjusting your search timeframe as I also had issues with time zones being in the future or too far in the past.

[capitangiaco]

try to set the Firewall timezone to UTC, and then add the processors: add_locale: yourlocale to your *.sls

my example:

filebeat: third_party_filebeat: modules: panw: panos: enabled: true var.syslog_host: 0.0.0.0 var.syslog_port: 9002 processors: - add_locale: Europe/Rome