Highstate Failures

[stondino00]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

16

RAM

16G

Storage for /

500GB

Storage for /nsm

16TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Getting these 2 failures when running highstate or the hotfix upgrade from 2.4.20.

---

      ID: so-elastic-fleet-auto-configure-elasticsearch-urls
Function: cmd.run
    Name: /usr/sbin/so-elastic-fleet-es-url-update
  Result: False
 Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
          Command "/usr/sbin/so-elastic-fleet-es-url-update" run
 Started: 18:55:08.571418
Duration: 30168.718 ms
 Changes:
          ----------
          pid:
              1132299
          retcode:
              1
          stderr:
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed

                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100    98  100    98    0     0   3161      0 --:--:-- --:--:-- --:--:--  3161
          stdout:
              Failed to query for current Fleet Server Elasticsearch URLs...

---

      ID: so-elastic-fleet-auto-configure-logstash-outputs
Function: cmd.run
    Name: /usr/sbin/so-elastic-fleet-outputs-update
  Result: False
 Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
          Command "/usr/sbin/so-elastic-fleet-outputs-update" run
 Started: 18:54:34.512331
Duration: 30186.526 ms
 Changes:
          ----------
          pid:
              1130237
          retcode:
              1
          stderr:
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed

                0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
              100    98  100    98    0     0   2969      0 --:--:-- --:--:-- --:--:--  2969
          stdout:
              Failed to query for current Logstash Outputs...

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[markmaunu]

I have the same new deployment as above but with more cpus/memory/diskspace . I am seeing the same errors as above and no dashboards are being populated on the manager node . The sensor node is collecting zeek -suricata - pcaps locally with no issues. Since deployment 2 days ago no dashboards are populated on the manager node .

[defensivedepth]

@markmaunu Please start a new Discussion.

[stondino00]

I even tried a reinstall on the manager and same 2 errors happen. My manager is basically not usable anymore. Nothing shows for dashboard, hunt, alerts. Just getting this message.

[defensivedepth]

@stondino00 What errors are you seeing when you highstate it? sudo salt-call state.highstate

Can you confirm that you are running 16GB RAM on your Manager?

[stondino00]

Yes I'm running 16Gigs of RAM on the manager. When I run that highstate, I get the 2 errors in the original post above.

[stondino00]

Where should I look next?

[defensivedepth]

Please post a (santized as needed) screencap of your Fleet Settings. This can be found by navigating to

Elastic Fleet --> Settings

[defensivedepth]

Also, as I asked in the other thread - did you configure Security Onion to use a proxy during setup?

[stondino00]

No did not configure a proxy during setup. I just went ahead and did a new install of my manager and reran the setup script on my heavy nodes. That seemed to fix the problem with this. I'm not getting any kibana logs but I'll open another discussion on that.