Not all filebeat logs are showing up in manager-search

[jjvquest]

Hello team/community,

I have some trouble getting Zeek logs transported from our sensor to manager-search node. In Kibana some logs get through, but only on the hour. From my understanding the logs should be shipped with filebeat. All machines are running 2.3.52 from ISO. Zeek is used for metadata and Suricata and ossec events are showing up in SOC Console.
The /nsm/zeek/logs/current/dns.log on the sensor also gets filled correctly.

Sensor logs
cat /opt/so/log/filebeat/filebeat.log

2021-06-23T06:56:05.769Z        WARN    beater/filebeat.go:178  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-23T06:56:06.708Z        WARN    beater/filebeat.go:381  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-23T07:07:44.285Z        ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: write tcp 172.17.0.10:59986->10.100.100.1:5644: write: connection reset by peer
2021-06-23T07:07:45.896Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: write tcp 172.17.0.10:59986->10.100.100.1:5644: write: connection reset by peer
2021-06-23T07:07:48.332Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://mgr1:5644)): dial tcp 10.100.100.1:5644: connect: no route to host
2021-06-23T07:07:55.554Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://mgr1:5644)): dial tcp 10.100.100.1:5644: connect: no route to host
2021-06-23T07:08:05.060Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://mgr1:5644)): dial tcp 10.100.100.1:5644: connect: connection refused
2021-06-23T07:08:36.363Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://mgr1:5644)): dial tcp 10.100.100.1:5644: connect: connection refused

Manager-search logs
cat /opt/so/log/filebeat/filebeat.log

2021-06-23T06:53:58.460Z        WARN    beater/filebeat.go:178  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-23T06:53:58.866Z        WARN    beater/filebeat.go:381  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-06-23T06:57:16.444Z        ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: EOF
2021-06-23T06:57:16.445Z        ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: client is not connected
2021-06-23T06:57:18.008Z        ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: client is not connected

Logstash and Elastic logs don't seem to contain relevant errors.

sudo so-redis-count
(integer) 0

cat /opt/so/log/redis/redis-server.log

1:M 23 Jun 2021 09:16:29.017 * 10000 changes in 60 seconds. Saving...
1:M 23 Jun 2021 09:16:29.018 * Background saving started by pid 161
161:C 23 Jun 2021 09:16:29.019 * DB saved on disk
161:C 23 Jun 2021 09:16:29.020 * RDB: 2 MB of memory used by copy-on-write
1:M 23 Jun 2021 09:16:29.118 * Background saving terminated with success

sudo salt * state.highstate
finishes with no errors. Also rebooted the machines, but to no avail.

I have tried to troubleshoot with tips from the disccusion boards and documentation.

• from sensor tried to use openssl to connect to the tcp/5644 on the manager and noticed a SSL handshake exception in the manager-search logstash.log. Forgot that it had to do with me connecting and tried to regenerate the certificates via
  IDS/NSM events not visible #4298

• tried to add the following to global.sls:

logstash:
  pipelines:
    manager:
      config:
        - so/0009_input_beats.conf
        - so/0010_input_hhbeats.conf
        - so/9999_output_redis.conf.jinja
    search:
      config:
        - so/0900_input_redis.conf.jinja
        - so/9000_output_zeek.conf.jinja
        - so/9002_output_import.conf.jinja
        - so/9034_output_syslog.conf.jinja
        - so/9100_output_osquery.conf.jinja
        - so/9400_output_suricata.conf.jinja
        - so/9500_output_beats.conf.jinja
        - so/9600_output_ossec.conf.jinja
        - so/9700_output_strelka.conf.jinja

• adjusted the logstash settings in managersearch.sls (got 16 cores, adjusted the workers to 20; upped the batch size after reading another discussion on the board; upped the heap to max allowed, box had 64GB RAM):

logstash_settings:
  ls_pipeline_batch_size: 511
  ls_input_threads: 1
  lsheap: 4000m
  ls_pipeline_workers: 20

• adjusted the redis memory in global.sls:

redis_settings:
  redis_maxmemory: 2000

Can someone point me in the right direction here? Could it have to do with resources? Mem/CPU seems fine. Or certificates? Where can I easily disable SSL for the filebeat part for testing purposes?

Thanks in advance!
Jay

[jjvquest]

Still struggling with this.

• Sensor node and Manager-search node don't seem to have issues with CPU/MEM/DISK according to Grafana.
• Tried disabling steno on the sensor server to see if disk usage had anything to do with the issue
• Tried different parameters for the logstash_settings but did not help
• No errors in filebeat, logstash or redis
• Can it have something to do with manager-search role combined on one server and the global.sls logstash config specifically stating a manager and a search part?
• Restarting filebeat on sensor and manager-search does not help. It is still strange that exactly on the hour some logs do make it into Elastic.

Next step could be debugging the Filebeat network (container), I'll have to look into that. Does anyone have tips?

Jay

[jjvquest]

Seems the following did the trick in the sensor sls file:

filebeat:
ls_bulk_max_size: 4096
ls_workers: 12
mem_events: 81920
mem_flush_min_events: 2048

Thanks to #4796

First I was getting ~8k Zeek events per minute, now it is ~150k. I will look further into tuning it even better, but maybe this already helps others.

Jay

[thegreekman]

I seem to be having a similar issue. Filebeat is causing my redis to fill up, not sure why. Is this the only change you made to fix this problem?

[EbolaWare]

I seem to be having a similar issue. Filebeat is causing my redis to fill up, not sure why. Is this the only change you made to fix this problem?

If your redis is filling up, it likely means your elasticsearch cluster cannot handle the load. Additional searchnodes would likely be the fix.

(I'm sure you likely fixed it already, or have moved on by now. This comment is for anyone else with a similar problem.)