Update error caused by changing global_overides/phases/delete value

[Bal33p]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

128

Storage for /

500g

Storage for /nsm

30tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

more than 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Updated 2.4.50 the reboot now system will not complete Highstate with the following error

<{% set ES_INDEX_SETTINGS = {} %}
{% do ES_INDEX_SETTINGS_GLOBAL_OVERRIDES.update(salt['defaults.merge'](ES_INDEX_SETTINGS_GLOBAL_OVERRIDES, ES_INDEX_PILLAR, in_place=False)) %}
{% for index, settings in ES_INDEX_SETTINGS_GLOBAL_OVERRIDES.items() %}
{% if settings.index_template is defined %}
{% if not settings.get('index_sorting', False) | to_bool and settings.index_template.template.settings.index.sort is defined %} <======================
{% do settings.index_template.template.settings.index.pop('sort') %}
{% endif %}
{% endif %}
{% do ES_INDEX_SETTINGS.update({index | replace("x", "."): ES_INDEX_SETTINGS_GLOBAL_OVERRIDES[index]}) %}
{% endfor %}

[CRITICAL] Rendering SLS 'base:elasticsearch.enabled' failed: Jinja variable 'dict object' has no attribute 'template'
/var/cache/salt/minion/files/base/elasticsearch/template.map.jinja(23):

[...]

{% set ES_INDEX_SETTINGS = {} %}
{% do ES_INDEX_SETTINGS_GLOBAL_OVERRIDES.update(salt['defaults.merge'](ES_INDEX_SETTINGS_GLOBAL_OVERRIDES, ES_INDEX_PILLAR, in_place=False)) %}
{% for index, settings in ES_INDEX_SETTINGS_GLOBAL_OVERRIDES.items() %}
{% if settings.index_template is defined %}
{% if not settings.get('index_sorting', False) | to_bool and settings.index_template.template.settings.index.sort is defined %} <======================
{% do settings.index_template.template.settings.index.pop('sort') %}
{% endif %}
{% endif %}
{% do ES_INDEX_SETTINGS.update({index | replace("x", "."): ES_INDEX_SETTINGS_GLOBAL_OVERRIDES[index]}) %}
{% endfor %}

local:
Data failed to compile:

Rendering SLS 'base:elasticsearch.enabled' failed: Jinja variable 'dict object' has no attribute 'template'

/var/cache/salt/minion/files/base/elasticsearch/template.map.jinja(23):

[...]

{% set ES_INDEX_SETTINGS = {} %}
{% do ES_INDEX_SETTINGS_GLOBAL_OVERRIDES.update(salt['defaults.merge'](ES_INDEX_SETTINGS_GLOBAL_OVERRIDES, ES_INDEX_PILLAR, in_place=False)) %}
{% for index, settings in ES_INDEX_SETTINGS_GLOBAL_OVERRIDES.items() %}
{% if settings.index_template is defined %}
{% if not settings.get('index_sorting', False) | to_bool and settings.index_template.template.settings.index.sort is defined %} <======================
{% do settings.index_template.template.settings.index.pop('sort') %}
{% endif %}
{% endif %}
{% do ES_INDEX_SETTINGS.update({index | replace("x", "."): ES_INDEX_SETTINGS_GLOBAL_OVERRIDES[index]}) %}
{% endfor %}>

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Bal33p]

Regarding this error
I rolled back from .50 to .40 using my backup files and got the system running again.
Although it is running, I still see a single failure during the salt-call run for Elasticsearch associated with a change I made to the global_overrides-Phases-Delete-min_age setting where I place a 60d value.

The 60d value was appended to the so-case-template.json file, which now looks like this.
{ "composed_of": [ "case-mappings", "case-settings" ], "index_patterns": [ "so-case*" ], "priority": 500, "template": { "mappings": { "date_detection": false, "dynamic_templates": [ { "strings_as_keyword": { "mapping": { "ignore_above": 1024, "type": "keyword" }, "match_mapping_type": "string" } } ] }, "settings": { "index": { "mapping": { "total_fields": { "limit": 1500 } }, "number_of_replicas": 0, "number_of_shards": 1, "refresh_interval": "30s" } } }, "phases": { "delete": { "min_age": "" } } }

The salt error i get when running a salt-call state.apply Elasticsearch looks like this

Loading template file so-case-template.json Executing command with retry support: so-elasticsearch-query _index_template/so-case [email protected] -XPUT Results: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} (0) Did not find expectedOutput: '{"acknowledged":true}' in the output below from running the command: 'so-elasticsearch-query _index_template/so-case [email protected] -XPUT' <Start of output> {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} <End of output> Forcing exit code to 1 Command failed with exit code 1; will retry in 5 seconds (1 / 3)... Executing command with retry support: so-elasticsearch-query _index_template/so-case [email protected] -XPUT Results: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} (0) Did not find expectedOutput: '{"acknowledged":true}' in the output below from running the command: 'so-elasticsearch-query _index_template/so-case [email protected] -XPUT' <Start of output> {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} <End of output> Forcing exit code to 1 Command failed with exit code 1; will retry in 5 seconds (2 / 3)... Executing command with retry support: so-elasticsearch-query _index_template/so-case [email protected] -XPUT Results: {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} (0) Did not find expectedOutput: '{"acknowledged":true}' in the output below from running the command: 'so-elasticsearch-query _index_template/so-case [email protected] -XPUT' <Start of output> {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:547] [index_template] unknown field [phases]"},"status":400} <End of output> Forcing exit code to 1 Command failed with exit code 1; will retry in 5 seconds (3 / 3)... Command continues to fail; giving up. ERROR: Could not load template file: so-case-template.json Exiting.

So it seems the appended "phases" attribute for the so-case-template is not parsable and, therefore, causes a failure.
I have since removed the 60d reference from global_overides, but the "phases" reference merely removed the 60d value but left the phases stanza. as can be seen above. So, of course, this bares the question. How, do I remove the phases stanza from the template.json file so that it is not loaded, thus causing the failure?

[Bal33p]

Removed the following reference from soc_elasticsearch.sls re-ran salt-call state.apply elasticsearch
I watched the reference get removed during the run and confirmed no failures.

[m0duspwnens]

This is an issue that will be fixed in 2.4.60. #12382