Some logs from Pfsense are not parsed to Secuirty Onion

[MaleGift]

Version

2.4.70

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

128

Storage for /

280 GB

Storage for /nsm

7.3 TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I had configured two pfsense firewalls to be sending logs to security onion using the pfsense integration, i could collect 120 million logs in a day , however all of a sudden the log count dropped to 34 millions a day. I checked to confirm if there was a change in the logging on the firewall but this is not the case, all types of logs are sent some many of them dont reflect in the pfsense logs.
I have removed the configuration and re added the firewalls but the same issue is persistent. Even the native parsing using syslog does not work.

Errors when i run sudo salt-call state.highstate

local:
Data failed to compile:

The function "state.highstate" is running as PID 3081386 and was started at 2024, Jul 12 12:55:58.197638 with jid 20240712125558197638

/opt/so/log/logstash
][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on siem-hn:9696 (Redis::Timeo

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Can you povide more of the logstash log? Is that from your manager or search node?

[cm-ops]

If you are using the integration, then check the pfsense pipeline stats. For the general pipeline it is logs-pfsense.log-1.16.0, if you want to see the other ingest pipelines for pfsense, run sudo so-elasticsearch-pipelines-list | grep pfsense.

If they aren't making it to the ingest pipeline, work backwards to Logstash then Elastic Agent.

[MaleGift]

This is the output for sudo so-elasticsearch-pipeline-stats logs-pfsense.log-1.16.0, am seeing failure in one section of grok
"count": 3408102,
"time_in_millis": 1584753,
"current": 0,
"failed": 0,
"processors": [
{
"set": {
"type": "set",
"stats": {
"count": 3408102,
"time_in_millis": 10591,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 3408102,
"time_in_millis": 5072,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 3408102,
"time_in_millis": 3558,
"current": 0,
"failed": 0
}
}
},
{
"rename": {
"type": "rename",
"stats": {
"count": 3408102,
"time_in_millis": 15840,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 3408102,
"time_in_millis": 2902,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"grok": {
"type": "grok",
"stats": {
"count": 3408102,
"time_in_millis": 144777,
"current": 0,
"failed": 0
}
}
},
{
"date": {
"type": "date",
"stats": {
"count": 3408102,
"time_in_millis": 103823,
"current": 0,
"failed": 0
}
}
},
{
"date": {
"type": "date",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"grok": {
"type": "grok",
"stats": {
"count": 3408102,
"time_in_millis": 36258,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-firewall": {
"type": "pipeline",
"stats": {
"count": 3403719,
"time_in_millis": 484772,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-openvpn": {
"type": "pipeline",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-ipsec": {
"type": "pipeline",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-dhcp": {
"type": "pipeline",
"stats": {
"count": 433,
"time_in_millis": 235,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-unbound": {
"type": "pipeline",
"stats": {
"count": 43,
"time_in_millis": 4,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-haproxy": {
"type": "pipeline",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-php-fpm": {
"type": "pipeline",
"stats": {
"count": 2,
"time_in_millis": 1,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-squid": {
"type": "pipeline",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log-1.16.0-suricata": {
"type": "pipeline",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"drop": {
"type": "drop",
"stats": {
"count": 3905,
"time_in_millis": 6,
"current": 0,
"failed": 0
}
}
},
{
"append": {
"type": "append",
"stats": {
"count": 3404152,
"time_in_millis": 11202,
"current": 0,
"failed": 0
}
}
},
{
"convert": {
"type": "convert",
"stats": {
"count": 3404154,
"time_in_millis": 14834,
"current": 0,
"failed": 0
}
}
},
{
"convert": {
"type": "convert",
"stats": {
"count": 3404154,
"time_in_millis": 9673,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 33075,
"time_in_millis": 63,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 3370649,
"time_in_millis": 4987,
"current": 0,
"failed": 0
}
}
},
{
"geoip": {
"type": "geoip",
"stats": {
"count": 3404154,
"time_in_millis": 35168,
"current": 0,
"failed": 0
}
}
},
{
"geoip": {
"type": "geoip",
"stats": {
"count": 3404154,
"time_in_millis": 52785,
"current": 0,
"failed": 0
}
}
},
{
"geoip": {
"type": "geoip",
"stats": {
"count": 3404154,
"time_in_millis": 17967,
"current": 0,
"failed": 0
}
}
},
{
"geoip": {
"type": "geoip",
"stats": {
"count": 3404154,
"time_in_millis": 23473,
"current": 0,
"failed": 0
}
}
},
{
"rename": {
"type": "rename",
"stats": {
"count": 3404154,
"time_in_millis": 5769,
"current": 0,
"failed": 0
}
}
},
{
"rename": {
"type": "rename",
"stats": {
"count": 3404154,
"time_in_millis": 2838,
"current": 0,
"failed": 0
}
}
},
{
"rename": {
"type": "rename",
"stats": {
"count": 3404154,
"time_in_millis": 7785,
"current": 0,
"failed": 0
}
}
},
{
"rename": {
"type": "rename",
"stats": {
"count": 3404154,
"time_in_millis": 6687,
"current": 0,
"failed": 0
}
}
},
{
"community_id": {
"type": "community_id",
"stats": {
"count": 3404154,
"time_in_millis": 54739,
"current": 0,
"failed": 5
}
}
},
{
"grok": {
"type": "grok",
"stats": {
"count": 3404154,
"time_in_millis": 50014,
"current": 0,
"failed": 255654
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 3404154,
"time_in_millis": 12824,
"current": 0,
"failed": 0
}
}
},
{
"append": {
"type": "append",
"stats": {
"count": 3403720,
"time_in_millis": 40754,
"current": 0,
"failed": 0
}
}
},
{
"append": {
"type": "append",
"stats": {
"count": 3403724,
"time_in_millis": 23421,
"current": 0,
"failed": 0
}
}
},
{
"append": {
"type": "append",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"append": {
"type": "append",
"stats": {
"count": 0,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"append": {
"type": "append",
"stats": {
"count": 2,
"time_in_millis": 0,
"current": 0,
"failed": 0
}
}
},
{
"set": {
"type": "set",
"stats": {
"count": 3403719,
"time_in_millis": 26197,
"current": 0,
"failed": 0
}
}
},
{
"remove": {
"type": "remove",
"stats": {
"count": 3404154,
"time_in_millis": 5310,
"current": 0,
"failed": 0
}
}
},
{
"script": {
"type": "script",
"stats": {
"count": 3404154,
"time_in_millis": 131967,
"current": 0,
"failed": 0
}
}
},
{
"remove": {
"type": "remove",
"stats": {
"count": 3404154,
"time_in_millis": 11969,
"current": 0,
"failed": 0
}
}
},
{
"pipeline:logs-pfsense.log@custom": {
"type": "pipeline",
"stats": {
"count": 3404154,
"time_in_millis": 1875,
"current": 0,
"failed": 0
}
}
}
]
}

[cm-ops]

    {
      "grok": {
        "field": "observer.ingress.interface.name",
        "patterns": [
          "%{DATA}.%{NONNEGINT:observer.ingress.vlan.id}"
        ],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },

That is what the grok failure is. Has ignore_missing and ignore_failure so the rest of the processors run. Does the number of events look about right or low?

[MaleGift]

The number of events are very low,
from about 10,000,000 to jus 3 millions.

[cm-ops]

If the ingest seems low, check backwards through the pipeline. Start at Logstash on your search node (both the log and so-logstash-pipeline-stats search) for errors or drops, then work to the manager and Logstash there same two places except the Logstash pipeline on the manager is "manager" not "search".