Kibana Not Showing any Data

[och0Sec]

I have done so-status on manager, search and sensor nodes and they all show everything in good working order.
Are there any other commands that I can do to figure out why the Kibana dashboard isn't displaying any data? I have confirmed that my Gigamon tap is forwarding traffic to the monitor interface on the sensor host.

[dougburks]

Is your sensor writing logs to /nsm/zeek/logs/current/?

[och0Sec]

I thought it was since I configured the sensor node to report back to my manager node in the distributed setup. Now when I try adding it manually, I receive the error below:

[user@so2-fwd current]# so-allow
This program allows you to add a firewall rule to allow connections from a new IP address.

Choose the role for the IP or Range you would like to add

[a] - Analyst - ports 80/tcp and 443/tcp
[b] - Logstash Beat - port 5044/tcp
[e] - Elasticsearch REST API - port 9200/tcp
[f] - Strelka frontend - port 57314/tcp
[o] - Osquery endpoint - port 8090/tcp
[s] - Syslog device - 514/tcp/udp
[w] - Wazuh agent - port 1514/tcp/udp
[p] - Wazuh API - port 55000/tcp
[r] - Wazuh registration service - 1515/tcp

Please enter your selection:
e
Enter a single ip address or range to allow (example: 10.10.10.10 or 10.10.0.0/16):
169.254.23.2
Adding 169.254.23.2 to the elasticsearch_rest role. This can take a few seconds
Traceback (most recent call last):
File "/usr/sbin/so-firewall", line 305, in
main()
File "/usr/sbin/so-firewall", line 296, in main
code = cmd(args[1:])
File "/usr/sbin/so-firewall", line 246, in includehost
result = addIp(args[0], args[1], 'insert')
File "/usr/sbin/so-firewall", line 72, in addIp
content = loadYaml(hostgroupsFilename)
File "/usr/sbin/so-firewall", line 52, in loadYaml
file = open(filename, "r")
FileNotFoundError: [Errno 2] No such file or directory: '/opt/so/saltstack/local/salt/firewall/hostgroups.local.yaml'

[dougburks]

When you joined the search node to your manager, it should have automatically configured firewall rules on both the manager and the search node. Do you have any network firewalls or any other devices that may be blocking traffic between the manager and the search node?

[och0Sec]

No there aren't any firewalls in the path. Both nodes reside on the same subnet.

[dougburks]

Have you checked the logs on your search node for additional clues?

[och0Sec]

I see these errors in the search node, it is interesting to see that logstash in the search node is using its LAN IP to connect back into elasticsearch instead of 127.0.0.1.
When I try to manually connect to that IP:Port it times out for me as well
[user@so2-srch log]# tail elasticsearch/so2-srch
so2-srch-2020-10-21.log so2-srch-2020-10-23.log so2-srch-2020-10-25.log so2-srch-2020-10-27.log so2-srch.log
so2-srch-2020-10-22.log so2-srch-2020-10-24.log so2-srch-2020-10-26.log so2-srch-2020-10-28.log
[user@so2-srch log]# tail -n 10 elasticsearch/so2-srch.log
[2020-10-29T02:31:00,439][INFO ][org.elasticsearch.monitor.jvm.JvmGcMonitorService] [gc][5208] overhead, spent [411ms] collecting in the last [1.2s]
[2020-10-29T02:31:05,442][INFO ][org.elasticsearch.monitor.jvm.JvmGcMonitorService] [gc][5213] overhead, spent [404ms] collecting in the last [1s]
[2020-10-29T02:31:05,552][INFO ][org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService] attempting to trigger G1GC due to high heap usage [608429056]
[2020-10-29T02:31:05,555][INFO ][org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService] GC did bring memory usage down, before [608429056], after [607510656], allocations [1], duration [3]
[2020-10-29T02:31:12,650][WARN ][org.elasticsearch.monitor.jvm.JvmGcMonitorService] [gc][5220] overhead, spent [989ms] collecting in the last [1.1s]
[2020-10-29T02:31:13,898][WARN ][org.elasticsearch.monitor.jvm.JvmGcMonitorService] [gc][5221] overhead, spent [1.2s] collecting in the last [1.2s]
[2020-10-29T02:31:14,899][INFO ][org.elasticsearch.monitor.jvm.JvmGcMonitorService] [gc][5222] overhead, spent [439ms] collecting in the last [1s]
[2020-10-29T02:31:16,036][INFO ][org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService] attempting to trigger G1GC due to high heap usage [597733504]
[2020-10-29T02:31:16,040][INFO ][org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService] GC did bring memory usage down, before [597733504], after [575413960], allocations [1], duration [4]
[2020-10-29T02:31:17,025][WARN ][org.elasticsearch.monitor.jvm.JvmGcMonitorService] [gc][5224] overhead, spent [811ms] collecting in the last [1.1s]

[user@so2-srch log]# tail -n 10 logstash/logstash.log
[2020-10-29T15:46:58,228][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}
[2020-10-29T15:46:58,543][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}
[2020-10-29T15:47:00,154][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}
[2020-10-29T15:47:03,559][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}
[2020-10-29T15:47:07,656][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}
[2020-10-29T15:47:08,109][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>64}
[2020-10-29T15:47:09,557][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}
[2020-10-29T15:47:11,162][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}
[2020-10-29T15:47:14,576][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}
[2020-10-29T15:47:20,573][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://169.254.23.164:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://169.254.23.164:9200/][Manticore::ConnectTimeout] Read timed out"}

I can confirm that the search node is listening and has established sessions on port 9200:
[user@so2-srch log]# netstat -ano | grep 169.254.23.164:9200
tcp 0 142 169.254.23.164:55274 169.254.23.164:9200 ESTABLISHED on (1.93/5/0)
tcp 0 0 169.254.23.164:49798 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:50002 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 142 169.254.23.164:54932 169.254.23.164:9200 ESTABLISHED on (1.14/8/0)
tcp 0 142 169.254.23.164:55042 169.254.23.164:9200 ESTABLISHED on (31.16/8/0)
tcp 0 142 169.254.23.164:55152 169.254.23.164:9200 ESTABLISHED on (9.66/7/0)
tcp 0 173 169.254.23.164:55182 169.254.23.164:9200 ESTABLISHED on (15.42/7/0)
tcp 0 0 169.254.23.164:49766 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:49732 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:49834 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:49920 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:50136 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:49964 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:49750 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:50176 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:50052 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:50092 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:49884 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)
tcp 0 0 169.254.23.164:50228 169.254.23.164:9200 ESTABLISHED off (0.00/0/0)

[dougburks]

Did you specifically define a static IP address of 169.254.23.164? That's usually considered an APIPA address.

[och0Sec]

That's just a replacement IP scheme I decided to utilize for this purpose since these posts are publicly available to anyone. They are all on the same 10.x.x.0/24 network. I did noticed that Suricata was not able to download any rules and found out that the manager was not trusting our Gigamon TLS inline certificate. I am waiting 15 minutes to test and see if I fixed this because Suricata will not attempt to download the rules again if so-rule-update was ran less than 15 minutes ago.

[och0Sec]

The issue wasn't fixed, but I can now see that I have 28000+ rules enabled.

[och0Sec]

Looks like the Manager node is unable to connect to the Search node judging by the logs below. It is interesting to node that firewalld is completely disabled (not running).
Logs from Manager:
[2020-11-02T13:09:49,777][WARN ][org.elasticsearch.transport.SniffConnectionStrategy] fetching nodes from external cluster [so2-srch_searchnode] failed
org.elasticsearch.transport.ConnectTransportException: [][X.X.X.164:9300] general node connection failure
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:956) ~[elasticsearch-7.9.3.jar:7.9.3]
[root@so2-mstr log]# ip a | grep X.X.X.
inet X.X.X.94/24 brd X.X.X.255 scope global noprefixroute dynamic ens192

Logs from Search node:
[root@so2-srch log]# so-status

Checking Docker status

Docker ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    

Checking container statuses

so-curator --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    
so-elasticsearch --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    
so-filebeat -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    
so-logstash -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    
so-nginx ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    
so-telegraf -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    
so-wazuh ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ OK ]    

[root@so2-srch log]# netstat -ano | grep 9300
tcp6 0 0 :::9300 :::* LISTEN off (0.00/0/0)
[root@so2-srch log]# service firewalld status
Redirecting to /bin/systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@parc-so2-srch log]# ip a | grep X.X.X.
inet X.X.X.164/24 brd X.X.X.255 scope global noprefixroute dynamic ens192

[och0Sec]

Redeploying the Search node fixed my issue.