Elasticsearch Status: Fault after Upgrade 2.4.70

[cherryCokeBack]

Version

2.4.70

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

12

RAM

20

Storage for /

2to

Storage for /nsm

20to

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hello ,
after Upgrade to 2.4.70 my manager have Elasticsearch Status: Fault
after restart manager and service it's not ok

Log for so-elasticsearch after commande restart :

Loading Security Onion index templates...
Loading template file so-case-template.json
Executing command with retry support: so-elasticsearch-query _index_template/so-case [email protected] -XPUT
Results: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} (0)
Did not find expectedOutput: '{"acknowledged":true}' in the output below from running the command: 'so-elasticsearch-query _index_template/so-case [email protected] -XPUT'

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400}

Forcing exit code to 1
Command failed with exit code 1; will retry in 5 seconds (1 / 3)...
Executing command with retry support: so-elasticsearch-query _index_template/so-case [email protected] -XPUT
Results: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} (0)
Did not find expectedOutput: '{"acknowledged":true}' in the output below from running the command: 'so-elasticsearch-query _index_template/so-case [email protected] -XPUT'

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400}

Forcing exit code to 1
Command failed with exit code 1; will retry in 5 seconds (2 / 3)...
Executing command with retry support: so-elasticsearch-query _index_template/so-case [email protected] -XPUT
Results: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} (0)
Did not find expectedOutput: '{"acknowledged":true}' in the output below from running the command: 'so-elasticsearch-query _index_template/so-case [email protected] -XPUT'

{"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400}

Forcing exit code to 1
Command failed with exit code 1; will retry in 5 seconds (3 / 3)...
Command continues to fail; giving up.
ERROR: Could not load template file: so-case-template.json
Exiting.

      ID: so-elasticsearch-pipelines
Function: cmd.run
    Name: /usr/sbin/so-elasticsearch-pipelines manager
  Result: True
 Comment: Command "/usr/sbin/so-elasticsearch-pipelines manager" run
 Started: 10:05:01.550430
Duration: 14.354 ms
 Changes:   
          ----------
          pid:
              240337
          retcode:
              0
          stderr:
          stdout:

---

      ID: so-elasticsearch-roles-load
Function: cmd.run
    Name: /usr/sbin/so-elasticsearch-roles-load
  Result: True
 Comment: Command "/usr/sbin/so-elasticsearch-roles-load" run
 Started: 10:05:01.567090
Duration: 30599.705 ms
 Changes:   
          ----------
          pid:
              240341
          retcode:
              0
          stderr:
          stdout:
              Waiting for ElasticSearch...connected!
              Loading roles...
              {"role":{"created":false}}{"role":{"created":false}}{"role":{"created":false}}{"role":{"created":false}}

---

      ID: so-elasticsearch-indices-delete
Function: cron.present
    Name: /usr/sbin/so-elasticsearch-indices-delete > /opt/so/log/elasticsearch/cron-elasticsearch-indices-delete.log 2>&1
  Result: True
 Comment: Cron /usr/sbin/so-elasticsearch-indices-delete > /opt/so/log/elasticsearch/cron-elasticsearch-indices-delete.log 2>&1 already present
 Started: 10:05:32.167142
Duration: 6.261 ms
 Changes:   

Summary for local

Succeeded: 351 (changed=7)
Failed: 1

Total states run: 352
Total run time: 77.640 s

Many thanks for your help

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cherryCokeBack]

After enter this commande :

• so-elasticsearch-query _index_template/so-case [email protected] -XPUT
• sudo so-elasticsearch-templates-load
• sudo so-elasticsearch-index-templates-list

it's ok for Elasticsearch.
But for "detection" the sync are on error :

I have no log in the hunt section

many thanks for your help

[cm-ops]

#13124 (comment)

Are you running into this?

[cherryCokeBack]

Hello,

thank you for your reply.
I currently get the following error after rebooting the manager

I can't do anything about the manager from the web interface

the following command gives me this result :

sudo so-elasticsearch-templates-load
Templates already loaded

I followed steps 1 and 2 in your link before rebooting the manager.
I changed the index lifetime to 90d

but when I run the state.highstate command I get the following message

[cherryCokeBack]

after a few minutes following the reboot the service is now ok

in the detections section I get this

the result for state.highstate :
Summary for local

Succeeded: 704 (changed=33)
Failed: 1

Total states run: 705
Total run time: 101.840 s

the error is located on influxdb at dashboard level
ID: metrics_link_file
Function: cmd.run
Name: so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt
Result: False
Comment: Command "so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt" run
Started: 07:29:55.977545
Duration: 186.458 ms
Changes:
----------
pid:
215750
retcode:
1
stderr:
Wed Jun 5 07:29:56 UTC 2024 | so-influxdb-manage | Dashboard not found
stdout:

Actually :

[cherryCokeBack]

it's ok.
changed the index lifetime to 90d + reboot

then you have to be patient

Many thanks