Empty Dashboards on Manager

[markmaunu]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

64G

Storage for /

293GB

Storage for /nsm

14TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

The sensor node is running without issues and collecting suricata logs, zeek logs and pcaps locally . The manager node is not displaying any data in the UI . There are no errors on the sensor node running sudo salt-call state.highstate

Two errors on the manager node running sudo salt-call state.highstate

1.    ID: so-elastic-fleet-auto-configure-elasticsearch-urls
   

   Function: cmd.run
   Name: /usr/sbin/so-elastic-fleet-es-url-update
   Result: False
   Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
   Command "/usr/sbin/so-elastic-fleet-es-url-update" run
   Started: 18:30:25.658136
   Duration: 30180.413 ms
   Changes:
   ----------
   pid:
   3647601
   retcode:
   1
   stderr:
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed

                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100    98  100    98    0     0   1960      0 --:--:-- --:--:-- --:--:--  1960
          stdout:
              Failed to query for current Fleet Server Elasticsearch URLs...
   

2.      ID: so-elastic-fleet-auto-configure-logstash-outputs
   

   Function: cmd.run
   Name: /usr/sbin/so-elastic-fleet-outputs-update
   Result: False
   Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
   Command "/usr/sbin/so-elastic-fleet-outputs-update" run
   Started: 18:29:51.232606
   Duration: 30166.332 ms
   Changes:
   ----------
   pid:
   3646249
   retcode:
   1
   stderr:
   % Total % Received % Xferd Average Speed Time Time Time Current
   Dload Upload Total Spent Left Speed

                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100    98  100    98    0     0   3266      0 --:--:-- --:--:-- --:--:--  3266
          stdout:
              Failed to query for current Logstash Outputs...
   

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[stondino00]

I think this is similar issue I'm having as well on another thread.

[markmaunu]

We are not seeing the webui errors you are seeing about the Elasticsearch cluster . Our dashboards are completely empty with no data yet the grid health component shows everything as green . The only errors I see are the 2 posted above running sudo salt-call state.highstate on the manager which you also show. The one sensor node does not show any errors when I run sudo salt-call state.highstate. Reiterating that the sensor node is collecting pcaps, zeek and suricata logs locally - over 13TB filled under /nsm.

This exact hardware was running without issues on the latest 2.3.x version of Security Onion but we installed a fresh copy of 2.4.20 from ISO and have not been able to use the system since . At this stage without a working manager node running 2.4 and no visibility we are looking to go back and install 2.3 .

Does anyone have any suggestions to troubleshoot ?

[bumfighter]

I think I'm having the same error where all the hardware worked on 2.3 but now I can't get any dashboards to populate on 2.4 with the same hardware. Were you able to get a resolution for this?

[cm-ops]

What does salt \* cmd.run "elastic-agent status" return?

[markmaunu]

I defanged IP addresses and API key in paste below but its showing 2 errors

[admin@so-manager-node-manager ~]sudo salt * cmd.run "elastic-agent status"
so-manager-node-manager:
State: HEALTHY
Message: Running
Fleet State: FAILED
Fleet Message: fail to checkin to fleet-server: all hosts failed: 2 errors occurred:
* requester 0/2 to host https://10.X.X.21:8220/ errored: Post "https://10.X.X.21:8220/api/fleet/agents/XXXX/checkin?": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "so-manager-node-manager")
* requester 1/2 to host https://so-manager-node-manager:8220/ errored: Post "https://so-manager-node-manager:8220/api/fleet/agents/XXXX/checkin?": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "so-manager-node-manager")

Components:
  * log         (HEALTHY)
                Healthy: communicating with pid '1674'
  * osquery     (HEALTHY)
                Healthy: communicating with pid '1734'
  * udp         (HEALTHY)
                Healthy: communicating with pid '1740'
  * tcp         (HEALTHY)
                Healthy: communicating with pid '1854'
  * filestream  (HEALTHY)
                Healthy: communicating with pid '1923'

so-forwarder-node-sensor:
State: HEALTHY
Message: Running
Fleet State: FAILED
Fleet Message: fail to checkin to fleet-server: all hosts failed: 2 errors occurred:
* requester 0/2 to host https://so-manager-manager:8220/ errored: Post "https://so-manager-node-manager:8220/api/fleet/agents/XXXX/checkin?": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "so-manager-node-manager")
* requester 1/2 to host https://10.X.X.21:8220/ errored: Post "https://10.X.X.21:8220/api/fleet/agents/XXXX/checkin?": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "so-manager-node-manager")

Components:
  * osquery     (HEALTHY)
                Healthy: communicating with pid '3036191'
  * udp         (HEALTHY)
                Healthy: communicating with pid '3036192'
  * filestream  (HEALTHY)
                Healthy: communicating with pid '3036193'
  * tcp         (HEALTHY)
                Healthy: communicating with pid '3036194'
  * log         (HEALTHY)
                Healthy: communicating with pid '3036200'

[defensivedepth]

@markmaunu @stondino00 Do you run setup with proxy settings?

[markmaunu]

I did not run setup with proxy settings

[defensivedepth]

x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "so-manager-node-manager"

Has the hostname or IP of the Manager changed since you originally ran setup? Did you re-run setup on the Manager?

You can try the following (from the Manager):

salt \* cmd.run "elastic-agent uninstall -f"

salt \* state.highstate

This will uninstall the Elastic Agent on all Grid Nodes & reinstall it.

[markmaunu]

The hostname or IP hasn't changed since I originally ran setup but I did re-run setup on the manager before I posted here as I was trying to see if it fixed the empty dashboards by re-running setup . I never received any errors running setup the first or second time.

I have run the commands above and the first successfully uninstalled the elastic agent but when I run salt * state.highstate I get the original 2 errors I was having . Pasting them below:

---

      ID: so-elastic-fleet-auto-configure-logstash-outputs
Function: cmd.run
    Name: /usr/sbin/so-elastic-fleet-outputs-update
  Result: False
 Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
          Command "/usr/sbin/so-elastic-fleet-outputs-update" run
 Started: 20:32:28.411501
Duration: 30158.871 ms
 Changes:   
          ----------
          pid:
              961320
          retcode:
              1
          stderr:
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
              
                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100    98  100    98    0     0   3920      0 --:--:-- --:--:-- --:--:--  3920
          stdout:
              Failed to query for current Logstash Outputs...

---

      ID: so-elastic-fleet-auto-configure-elasticsearch-urls
Function: cmd.run
    Name: /usr/sbin/so-elastic-fleet-es-url-update
  Result: False
 Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
          Command "/usr/sbin/so-elastic-fleet-es-url-update" run
 Started: 20:33:03.330347
Duration: 30163.074 ms
 Changes:   
          ----------
          pid:
              963024
          retcode:
              1
          stderr:
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
              
                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100    98  100    98    0     0   3161      0 --:--:-- --:--:-- --:--:--  3161
          stdout:
              Failed to query for current Fleet Server Elasticsearch URLs...

---

Succeeded: 625 (changed=39)
Failed: 2

[bigdaddyschaefer]

My Dashboard is also empty and I'm getting the same two errors. I also noticed a "bad_certificate" error in the logstash log, it seems nothing is getting ingested by logstash in my case. I have created #11947
Not sure if this is just one issue or two separate ones.

[dougburks]

@markmaunu Is it possible that your network is already using the 172.17.x.x range? If so, have you tried adjusting the Docker network range as shown at https://docs.securityonion.net/en/2.4/docker.html#networking-and-bridging?