logs from filebeat not showing up in logstash

[Sabrine-se]

Hello,
A production deployment
Here's some informations:

/etc/soversion : 2.3.270
On-prem with Airgap=True no access to internet
Installed with Security onion image ISO
2 Nodes (1 Manager node and 1 search node )
No monitoring network traffic
So-status show all services running on the 2 nodes
Soc grid page don't show any failures (status OK)
run salt '' test.ping with et salt '' state.hignstateTrue pour manager et pour search Minion did not return/Minion returned with non-zero exit code
Salt-key -L show that keys are accepted
=>i confugured my filebeat.yml file on my remote log server to send the logs to output logstash
/[root@srvlog log]# filebeat test config
Config OK
/[root@srvlog filebeat]# filebeat test output
logstash: ...:5044...
connection...
parse host... OK
dns lookup... OK
addresses: ...
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK

and on the manager I allowed the traffic with so-allow but I do not receive the logs here are the pipelines

/[root@srv-so-manager so]# cp /opt/so/saltstack/local/salt/logstash/pipelines/config/so/0009_input_beats.conf /opt/so/saltstack/local/salt/logstash/pipelines/config/so/

input {
2 beats {
3 port => "5044"
4 tags => [ "beat-ext" ]
5 }
6 }
7 filter {
8 mutate {
9 rename => {"@metadata" => "metadata"}
10 }
11 }

/[root@srv-so-manager ~]# cp /opt/so/saltstack/default/salt/logstash/pipelines/config/so/9999_output_redis.conf.jinja /opt/so/saltstack/local/salt/logstash/pipelines/config/so/

8 output {
9 if "beat-ext" in [tags] and "import" not in [tags] and "filebeat" not in [metadata][pipeline] {
10 if [metadata][_id] {
11 elasticsearch {
12 pipeline => "beats.common"
13 hosts => "...:9200"
14 {% if salt'pillar.get' is sameas true %}
15 user => " so_elastic "
16 password => " Ga[a76NGEwMaYP(R4ut|-Q6y?x5QXvE(1[KpVtN-ObZXW9K!YASR@l=e&-QKGYPL+TUBirD "
17 {% endif %}
18 index => "so-beats"
19 ssl => true
20 ssl_certificate_verification => false
21 document_id => "%{[metadata][_id]}"
22 }
23 } else {
24 elasticsearch {
25 pipeline => "beats.common"
26 hosts => "...:9200"
27 {% if salt'pillar.get' is sameas true %}
28 user => " so_elastic "
29 password => " Ga[a76NGEwMaYP(R4ut|-Q6y?x5QXvE(1[KpVtN-ObZXW9K!YASR@l=e&-QKGYPL+TUBirD "
30 {% endif %}
31 index => "so-beats"
32 ssl => true
33 ssl_certificate_verification => false
34 }
35 }
36 }
37 }

help me plz to resolve this!

[cm-ops]

Have you ran tcpdump as well on the manager for port 5044? Also, if you check /opt/so/log/logstash/logstash.log for any issues. You can also check the manager pipeline to see if events are making it through the input with so-logstash-pipeline-stats manager

[Sabrine-se]

When i run tcpdump =>
[root@srv-so-manager so]# tcpdump port 5044
65 packets captured
154 packets received by filter
89 packets dropped by kernel
(witn an IP 172.17.0.13)

[root@srv-so-manager so]# cd /opt/so/log/logstash/
[root@srv-so-manager logstash]# cat logstash.log
[2024-03-18T09:15:08,896][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2024-03-18T09:15:17,218][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"manager"}
[2024-03-18T09:15:18,351][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:manager}
[2024-03-18T09:15:18,375][INFO ][logstash.runner ] Logstash shut down.
[2024-03-18T09:16:52,004][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
.
.
[2024-03-18T09:16:53,613][WARN ][logstash.inputs.beats ] You are using a deprecated config setting "ssl" set in beats. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Use 'ssl_enabled' instead.
.
.
[2024-03-18T09:16:55,155][INFO ][logstash.inputs.beats ] Starting input listener {:address=>"0.0.0.0:5644"}
[2024-03-18T09:16:55,199][INFO ][logstash.inputs.beats ] Starting input listener {:address=>"0.0.0.0:5044"}
[2024-03-18T09:16:55,200][INFO ][logstash.inputs.http ] Starting http input listener {:address=>"0.0.0.0:3765", :ssl=>"true"}
[2024-03-18T09:16:55,215][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2024-03-18T09:16:55,215][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"manager"}
[2024-03-18T09:16:55,217][INFO ][org.logstash.beats.Server] Starting server on port: 5644
[2024-03-18T09:16:55,227][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:manager], :non_running_pipelines=>[]}

No response for so-logstash-pipeline-stats manager

[cm-ops]

This command so-logstash-pipeline-stats manager should return information about the manager's logstash plugins (input, filters, output) if it does not return any information there is a problem with Logstash.

I see you copied 0009_input_beats.conf to the local path, but it looks to be default. 9999_output_redis.conf.jinja doesn't send the events to Redis like the default https://github.com/Security-Onion-Solutions/securityonion/blob/master/salt/logstash/pipelines/config/so/9999_output_redis.conf.jinja

What type of logs are you trying to send from your remote server?

[Sabrine-se]

I want to send all the logs from my remote log server via filebeat
My conf was on manager editing 4 file 0009_input_beats.conf ,9999_output_redis.conf.jinja,9500_output_beats.conf.jinja, 0900_input_redis.conf.jinja on this directory /opt/so/saltstack/local/salt/logstash/pippelines/config/so so( i copied files from default ti edit it) .

what configuration I must do to ensure the transfer of my logs from server log via filebeat to logstash on the manager to elasticsearch on the search node plz?

[cm-ops]

Can you share you filebeat.yaml on your remote server?

[Sabrine-se]

======================== Filebeat inputs ===============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input-specific configurations.

filestream is an input for collecting log messages from files.

• type: log

  Unique ID among all inputs, an ID is required.

  id: my-log-id

  Change to true to enable this input configuration.

  enabled: true

  Paths that should be crawled and fetched. Glob based paths.

  paths:

  • /var/log/*.log

================================== Outputs =============================

Configure what output to use when sending the data collected by the beat.

---------------------------- Elasticsearch Output ----------------------------

#output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]

Protocol - either http (default) or https.

#protocol: "https"

Authentication credentials - either API key or username/password.

#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"

------------------------------ Logstash Output -------------------------------

output.logstash:

The Logstash hosts

hosts: "10.24.38.153:5044"

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

[cm-ops]

Thanks, have your tried remove your two local modified files from /opt/so/saltstack/local/salt/logstash/pippelines/config/so and restarting Logstash?