Cant access My Manager Server UI #14338

izidood · 2025-03-05T10:05:10Z

izidood
Mar 5, 2025

Version

2.4.120

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

100GB

Storage for /nsm

1.5TB

Network Traffic Collection

other (please provide detail below)
Netflow

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

So from about 45 hours ago, a bunch of my containers on the manager node went down, and i only have 2 running containers. one of my search nodes crashed to a point where it wanted to make a complete reinstallation, so i did.
The problem is, that i cant enter my manager nodes webui when i go for an so-status i simply get the message "⌛ System appears to be starting. No highstate has completed since the system was restarted." but that message is there forever, i tried to make a salt-call state.highstate to see if i got anything, and got the following:

 sudo salt-call state.highstate
[sudo] password for soadmin:
[INFO    ] Loading fresh modules for state activity
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/clouds'
[INFO    ] Syncing clouds for environment 'base'
[INFO    ] Loading cache from salt://_clouds, for base
[INFO    ] Caching directory '_clouds/' for environment 'base'
[INFO    ] Syncing beacons for environment 'base'
[INFO    ] Loading cache from salt://_beacons, for base
[INFO    ] Caching directory '_beacons/' for environment 'base'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_beacons/zeek.py' to '/var/cache/salt/minion/extmods/beacons/zeek.py'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base
[INFO    ] Caching directory '_modules/' for environment 'base'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/healthcheck.py' to '/var/cache/salt/minion/extmods/modules/healthcheck.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/needs_restarting.py' to '/var/cache/salt/minion/extmods/modules/needs_restarting.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/so.py' to '/var/cache/salt/minion/extmods/modules/so.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/telegraf.py' to '/var/cache/salt/minion/extmods/modules/telegraf.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/zeekctl.py' to '/var/cache/salt/minion/extmods/modules/zeekctl.py'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base
[INFO    ] Caching directory '_states/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[INFO    ] Syncing sdb for environment 'base'
[INFO    ] Loading cache from salt://_sdb, for base
[INFO    ] Caching directory '_sdb/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base
[INFO    ] Caching directory '_grains/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base
[INFO    ] Caching directory '_renderers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base
[INFO    ] Caching directory '_returners/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/output'
[INFO    ] Syncing output for environment 'base'
[INFO    ] Loading cache from salt://_output, for base
[INFO    ] Caching directory '_output/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base
[INFO    ] Caching directory '_utils/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[INFO    ] Syncing log_handlers for environment 'base'
[INFO    ] Loading cache from salt://_log_handlers, for base
[INFO    ] Caching directory '_log_handlers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/executors'
[INFO    ] Syncing executors for environment 'base'
[INFO    ] Loading cache from salt://_executors, for base
[INFO    ] Caching directory '_executors/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[INFO    ] Syncing proxy for environment 'base'
[INFO    ] Loading cache from salt://_proxy, for base
[INFO    ] Caching directory '_proxy/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/engines'
[INFO    ] Syncing engines for environment 'base'
[INFO    ] Loading cache from salt://_engines, for base
[INFO    ] Caching directory '_engines/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/thorium'
[INFO    ] Syncing thorium for environment 'base'
[INFO    ] Loading cache from salt://_thorium, for base
[INFO    ] Caching directory '_thorium/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/serializers'
[INFO    ] Syncing serializers for environment 'base'
[INFO    ] Loading cache from salt://_serializers, for base
[INFO    ] Caching directory '_serializers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/matchers'
[INFO    ] Syncing matchers for environment 'base'
[INFO    ] Loading cache from salt://_matchers, for base
[INFO    ] Caching directory '_matchers/' for environment 'base'
[INFO    ] Loading fresh modules for state activity
[INFO    ] Executing command rpm in directory '/root'
[INFO    ] Executing command dnf in directory '/root'
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 45, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 45

---
[...]
    - group: 939
    - mode: 770
    - template: jinja
    - source: salt://telegraf/scripts/{{script}}
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{% endfor %}

telegraf_sbin:
  file.recurse:
    - name: /usr/sbin
[...]
---
[CRITICAL] Rendering SLS 'base:telegraf.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 45

---
[...]
    - group: 939
    - mode: 770
    - template: jinja
    - source: salt://telegraf/scripts/{{script}}
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{% endfor %}

telegraf_sbin:
  file.recurse:
    - name: /usr/sbin
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 44, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 44

---
[...]
    {% endif %}
    - environment:
      {% if (GLOBALS.role in GLOBALS.manager_roles and ELASTICSEARCH_SEED_HOSTS | length == 1) or GLOBALS.role == 'so-heavynode' %}
      - discovery.type=single-node
      {% endif %}
      - ES_JAVA_OPTS=-Xms{{ GLOBALS.elasticsearch.es_heap }} -Xmx{{ GLOBALS.elasticsearch.es_heap }} -Des.transport.cname_in_publish_address=true -Dlog4j2.formatMsgNoLookups=true    <======================
      ulimits:
      - memlock=-1:-1
      - nofile=65536:65536
      - nproc=4096
      {% if DOCKER.containers['so-elasticsearch'].extra_env %}
[...]
---
[CRITICAL] Rendering SLS 'base:elasticsearch.enabled' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 44

---
[...]
    {% endif %}
    - environment:
      {% if (GLOBALS.role in GLOBALS.manager_roles and ELASTICSEARCH_SEED_HOSTS | length == 1) or GLOBALS.role == 'so-heavynode' %}
      - discovery.type=single-node
      {% endif %}
      - ES_JAVA_OPTS=-Xms{{ GLOBALS.elasticsearch.es_heap }} -Xmx{{ GLOBALS.elasticsearch.es_heap }} -Des.transport.cname_in_publish_address=true -Dlog4j2.formatMsgNoLookups=true    <======================
      ulimits:
      - memlock=-1:-1
      - nofile=65536:65536
      - nproc=4096
      {% if DOCKER.containers['so-elasticsearch'].extra_env %}
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 26, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 26

---
[...]
  cmd.script:
    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
    - template: jinja
    - cwd: /opt/so
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{%   endif %}

cacertz:
  file.managed:
    - name: /opt/so/conf/ca/cacerts
[...]
---
[CRITICAL] Rendering SLS 'base:elasticsearch.ca' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 26

---
[...]
  cmd.script:
    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
    - template: jinja
    - cwd: /opt/so
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{%   endif %}

cacertz:
  file.managed:
    - name: /opt/so/conf/ca/cacerts
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 89, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 89

---
[...]
    - source: salt://logstash/pipelines/config/{{CONFIGFILE}}
      {% if 'jinja' in CONFIGFILE.split('.')[-1] %}
    - name: /opt/so/conf/logstash/pipelines/{{assigned_pipeline}}/{{CONFIGFILE.split('/')[1] | replace(".jinja", "")}}
    - template: jinja
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
        ES_USER: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') }}"
        ES_PASS: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') }}"
        THREADS: {{ LOGSTASH_MERGED.config.pipeline_x_workers }}
        BATCH: {{ LOGSTASH_MERGED.config.pipeline_x_batch_x_size }}
      {% else %}
[...]
---
[CRITICAL] Rendering SLS 'base:logstash.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 89

---
[...]
    - source: salt://logstash/pipelines/config/{{CONFIGFILE}}
      {% if 'jinja' in CONFIGFILE.split('.')[-1] %}
    - name: /opt/so/conf/logstash/pipelines/{{assigned_pipeline}}/{{CONFIGFILE.split('/')[1] | replace(".jinja", "")}}
    - template: jinja
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
        ES_USER: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') }}"
        ES_PASS: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') }}"
        THREADS: {{ LOGSTASH_MERGED.config.pipeline_x_workers }}
        BATCH: {{ LOGSTASH_MERGED.config.pipeline_x_batch_x_size }}
      {% else %}
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 52, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 52

---
[...]
    - user: 932
    - group: 939
    - file_mode: 755
    - template: jinja
    - defaults:
      GLOBALS: {{ GLOBALS }}    <======================

kibanaconfig:
  file.managed:
    - name: /opt/so/conf/kibana/etc/kibana.yml
    - source: salt://kibana/etc/kibana.yml.jinja
[...]
---
[CRITICAL] Rendering SLS 'base:kibana.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 52

---
[...]
    - user: 932
    - group: 939
    - file_mode: 755
    - template: jinja
    - defaults:
      GLOBALS: {{ GLOBALS }}    <======================

kibanaconfig:
  file.managed:
    - name: /opt/so/conf/kibana/etc/kibana.yml
    - source: salt://kibana/etc/kibana.yml.jinja
[...]
---
[INFO    ] Runner completed: 20250305094831848019
local:
    Data failed to compile:
----------
    Rendering SLS 'base:telegraf.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 45

---
[...]
    - group: 939
    - mode: 770
    - template: jinja
    - source: salt://telegraf/scripts/{{script}}
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{% endfor %}

telegraf_sbin:
  file.recurse:
    - name: /usr/sbin
[...]
---
----------
    Rendering SLS 'base:elasticsearch.enabled' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 44

---
[...]
    {% endif %}
    - environment:
      {% if (GLOBALS.role in GLOBALS.manager_roles and ELASTICSEARCH_SEED_HOSTS | length == 1) or GLOBALS.role == 'so-heavynode' %}
      - discovery.type=single-node
      {% endif %}
      - ES_JAVA_OPTS=-Xms{{ GLOBALS.elasticsearch.es_heap }} -Xmx{{ GLOBALS.elasticsearch.es_heap }} -Des.transport.cname_in_publish_address=true -Dlog4j2.formatMsgNoLookups=true    <======================
      ulimits:
      - memlock=-1:-1
      - nofile=65536:65536
      - nproc=4096
      {% if DOCKER.containers['so-elasticsearch'].extra_env %}
[...]
---
----------
    Rendering SLS 'base:elasticsearch.ca' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 26

---
[...]
  cmd.script:
    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
    - template: jinja
    - cwd: /opt/so
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{%   endif %}

cacertz:
  file.managed:
    - name: /opt/so/conf/ca/cacerts
[...]
---
----------
    Rendering SLS 'base:logstash.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 89

---
[...]
    - source: salt://logstash/pipelines/config/{{CONFIGFILE}}
      {% if 'jinja' in CONFIGFILE.split('.')[-1] %}
    - name: /opt/so/conf/logstash/pipelines/{{assigned_pipeline}}/{{CONFIGFILE.split('/')[1] | replace(".jinja", "")}}
    - template: jinja
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
        ES_USER: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') }}"
        ES_PASS: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') }}"
        THREADS: {{ LOGSTASH_MERGED.config.pipeline_x_workers }}
        BATCH: {{ LOGSTASH_MERGED.config.pipeline_x_batch_x_size }}
      {% else %}
[...]
---
----------
    Rendering SLS 'base:kibana.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 52

---
[...]
    - user: 932
    - group: 939
    - file_mode: 755
    - template: jinja
    - defaults:
      GLOBALS: {{ GLOBALS }}    <======================

kibanaconfig:
  file.managed:
    - name: /opt/so/conf/kibana/etc/kibana.yml
    - source: salt://kibana/etc/kibana.yml.jinja
[...]
---

i dont know what i should look after in the /opt/so/log ?
of what i see there is something with es_heap ?.

Im just wondering how this just happened by itself over the weekend.

Guidelines

I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

Answered by dougburks

Mar 13, 2025

It looks like your manager and all of your search nodes have somehow lost their esheap setting. For each of those files, please try adding an esheap line to the elasticsearch section under the enabled setting. It should look like this:

elasticsearch:
  enabled: true
  esheap: 5000m

You can replace 5000m with a value that is appropriate for your deployment.

View full answer

rh-ops · 2025-03-06T14:47:03Z

rh-ops
Mar 6, 2025

Try the following command to reinstall the salt packages:
yum reinstall salt

Then run a highstate with so-checkin

2 replies

izidood Mar 7, 2025
Author

sudo yum reinstall salt
[sudo] password for soadmin:
Last metadata expiration check: 3:21:46 ago on Fri 07 Mar 2025 03:00:45 AM UTC.
Dependencies resolved.
========================================================================================================================
 Package                 Architecture              Version                       Repository                        Size
========================================================================================================================
Reinstalling:
 salt                    x86_64                    3006.9-0                      securityonion                     47 M

Transaction Summary
========================================================================================================================

Total size: 47 M
Installed size: 142 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                1/1
  Running scriptlet: salt-3006.9-0.x86_64                                                                           1/2
salt:x:938:
usermod: no changes

  Reinstalling     : salt-3006.9-0.x86_64                                                                           1/2
  Running scriptlet: salt-3006.9-0.x86_64                                                                           1/2
  Running scriptlet: salt-3006.9-0.x86_64                                                                           2/2
  Cleanup          : salt-3006.9-0.x86_64                                                                           2/2
  Running scriptlet: salt-3006.9-0.x86_64                                                                           2/2
  Verifying        : salt-3006.9-0.x86_64                                                                           1/2
  Verifying        : salt-3006.9-0.x86_64                                                                           2/2

Reinstalled:
  salt-3006.9-0.x86_64

Complete!
[soadmin@so-manager-v01 ~]$ so-checkin
This script must be run using sudo!
[soadmin@so-manager-v01 ~]$ sudo so-checkin

so-checkin will run a full salt highstate to apply all salt states. If a highstate is already running, this request will be queued and so it may pause for a few minutes before you see any more output. For more information about so-checkin and salt, please see:
https://docs.securityonion.net/en/2.4/salt.html

[INFO    ] Loading fresh modules for state activity
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/clouds'
[INFO    ] Syncing clouds for environment 'base'
[INFO    ] Loading cache from salt://_clouds, for base
[INFO    ] Caching directory '_clouds/' for environment 'base'
[INFO    ] Syncing beacons for environment 'base'
[INFO    ] Loading cache from salt://_beacons, for base
[INFO    ] Caching directory '_beacons/' for environment 'base'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_beacons/zeek.py' to '/var/cache/salt/minion/extmods/beacons/zeek.py'
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base
[INFO    ] Caching directory '_modules/' for environment 'base'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/healthcheck.py' to '/var/cache/salt/minion/extmods/modules/healthcheck.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/needs_restarting.py' to '/var/cache/salt/minion/extmods/modules/needs_restarting.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/so.py' to '/var/cache/salt/minion/extmods/modules/so.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/telegraf.py' to '/var/cache/salt/minion/extmods/modules/telegraf.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/zeekctl.py' to '/var/cache/salt/minion/extmods/modules/zeekctl.py'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base
[INFO    ] Caching directory '_states/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[INFO    ] Syncing sdb for environment 'base'
[INFO    ] Loading cache from salt://_sdb, for base
[INFO    ] Caching directory '_sdb/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base
[INFO    ] Caching directory '_grains/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base
[INFO    ] Caching directory '_renderers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base
[INFO    ] Caching directory '_returners/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/output'
[INFO    ] Syncing output for environment 'base'
[INFO    ] Loading cache from salt://_output, for base
[INFO    ] Caching directory '_output/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base
[INFO    ] Caching directory '_utils/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[INFO    ] Syncing log_handlers for environment 'base'
[INFO    ] Loading cache from salt://_log_handlers, for base
[INFO    ] Caching directory '_log_handlers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/executors'
[INFO    ] Syncing executors for environment 'base'
[INFO    ] Loading cache from salt://_executors, for base
[INFO    ] Caching directory '_executors/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[INFO    ] Syncing proxy for environment 'base'
[INFO    ] Loading cache from salt://_proxy, for base
[INFO    ] Caching directory '_proxy/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/engines'
[INFO    ] Syncing engines for environment 'base'
[INFO    ] Loading cache from salt://_engines, for base
[INFO    ] Caching directory '_engines/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/thorium'
[INFO    ] Syncing thorium for environment 'base'
[INFO    ] Loading cache from salt://_thorium, for base
[INFO    ] Caching directory '_thorium/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/serializers'
[INFO    ] Syncing serializers for environment 'base'
[INFO    ] Loading cache from salt://_serializers, for base
[INFO    ] Caching directory '_serializers/' for environment 'base'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/matchers'
[INFO    ] Syncing matchers for environment 'base'
[INFO    ] Loading cache from salt://_matchers, for base
[INFO    ] Caching directory '_matchers/' for environment 'base'
[INFO    ] Loading fresh modules for state activity
[INFO    ] Executing command rpm in directory '/root'
[INFO    ] Executing command dnf in directory '/root'
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 45, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 45

---
[...]
    - group: 939
    - mode: 770
    - template: jinja
    - source: salt://telegraf/scripts/{{script}}
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{% endfor %}

telegraf_sbin:
  file.recurse:
    - name: /usr/sbin
[...]
---
[CRITICAL] Rendering SLS 'base:telegraf.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 45

---
[...]
    - group: 939
    - mode: 770
    - template: jinja
    - source: salt://telegraf/scripts/{{script}}
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{% endfor %}

telegraf_sbin:
  file.recurse:
    - name: /usr/sbin
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 44, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 44

---
[...]
    {% endif %}
    - environment:
      {% if (GLOBALS.role in GLOBALS.manager_roles and ELASTICSEARCH_SEED_HOSTS | length == 1) or GLOBALS.role == 'so-heavynode' %}
      - discovery.type=single-node
      {% endif %}
      - ES_JAVA_OPTS=-Xms{{ GLOBALS.elasticsearch.es_heap }} -Xmx{{ GLOBALS.elasticsearch.es_heap }} -Des.transport.cname_in_publish_address=true -Dlog4j2.formatMsgNoLookups=true    <======================
      ulimits:
      - memlock=-1:-1
      - nofile=65536:65536
      - nproc=4096
      {% if DOCKER.containers['so-elasticsearch'].extra_env %}
[...]
---
[CRITICAL] Rendering SLS 'base:elasticsearch.enabled' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 44

---
[...]
    {% endif %}
    - environment:
      {% if (GLOBALS.role in GLOBALS.manager_roles and ELASTICSEARCH_SEED_HOSTS | length == 1) or GLOBALS.role == 'so-heavynode' %}
      - discovery.type=single-node
      {% endif %}
      - ES_JAVA_OPTS=-Xms{{ GLOBALS.elasticsearch.es_heap }} -Xmx{{ GLOBALS.elasticsearch.es_heap }} -Des.transport.cname_in_publish_address=true -Dlog4j2.formatMsgNoLookups=true    <======================
      ulimits:
      - memlock=-1:-1
      - nofile=65536:65536
      - nproc=4096
      {% if DOCKER.containers['so-elasticsearch'].extra_env %}
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 26, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 26

---
[...]
  cmd.script:
    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
    - template: jinja
    - cwd: /opt/so
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{%   endif %}

cacertz:
  file.managed:
    - name: /opt/so/conf/ca/cacerts
[...]
---
[CRITICAL] Rendering SLS 'base:elasticsearch.ca' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 26

---
[...]
  cmd.script:
    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
    - template: jinja
    - cwd: /opt/so
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{%   endif %}

cacertz:
  file.managed:
    - name: /opt/so/conf/ca/cacerts
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 89, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 89

---
[...]
    - source: salt://logstash/pipelines/config/{{CONFIGFILE}}
      {% if 'jinja' in CONFIGFILE.split('.')[-1] %}
    - name: /opt/so/conf/logstash/pipelines/{{assigned_pipeline}}/{{CONFIGFILE.split('/')[1] | replace(".jinja", "")}}
    - template: jinja
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
        ES_USER: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') }}"
        ES_PASS: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') }}"
        THREADS: {{ LOGSTASH_MERGED.config.pipeline_x_workers }}
        BATCH: {{ LOGSTASH_MERGED.config.pipeline_x_batch_x_size }}
      {% else %}
[...]
---
[CRITICAL] Rendering SLS 'base:logstash.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 89

---
[...]
    - source: salt://logstash/pipelines/config/{{CONFIGFILE}}
      {% if 'jinja' in CONFIGFILE.split('.')[-1] %}
    - name: /opt/so/conf/logstash/pipelines/{{assigned_pipeline}}/{{CONFIGFILE.split('/')[1] | replace(".jinja", "")}}
    - template: jinja
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
        ES_USER: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') }}"
        ES_PASS: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') }}"
        THREADS: {{ LOGSTASH_MERGED.config.pipeline_x_workers }}
        BATCH: {{ LOGSTASH_MERGED.config.pipeline_x_batch_x_size }}
      {% else %}
[...]
---
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 468, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 52, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 245, in __str__
    output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'esheap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 211, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 474, in render_jinja_tmpl
    raise SaltRenderError(f"Jinja variable {exc}{out}", line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'esheap'; line 52

---
[...]
    - user: 932
    - group: 939
    - file_mode: 755
    - template: jinja
    - defaults:
      GLOBALS: {{ GLOBALS }}    <======================

kibanaconfig:
  file.managed:
    - name: /opt/so/conf/kibana/etc/kibana.yml
    - source: salt://kibana/etc/kibana.yml.jinja
[...]
---
[CRITICAL] Rendering SLS 'base:kibana.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 52

---
[...]
    - user: 932
    - group: 939
    - file_mode: 755
    - template: jinja
    - defaults:
      GLOBALS: {{ GLOBALS }}    <======================

kibanaconfig:
  file.managed:
    - name: /opt/so/conf/kibana/etc/kibana.yml
    - source: salt://kibana/etc/kibana.yml.jinja
[...]
---
[INFO    ] Runner completed: 20250307062323491735
local:
    Data failed to compile:
----------
    Rendering SLS 'base:telegraf.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 45

---
[...]
    - group: 939
    - mode: 770
    - template: jinja
    - source: salt://telegraf/scripts/{{script}}
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{% endfor %}

telegraf_sbin:
  file.recurse:
    - name: /usr/sbin
[...]
---
----------
    Rendering SLS 'base:elasticsearch.enabled' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 44

---
[...]
    {% endif %}
    - environment:
      {% if (GLOBALS.role in GLOBALS.manager_roles and ELASTICSEARCH_SEED_HOSTS | length == 1) or GLOBALS.role == 'so-heavynode' %}
      - discovery.type=single-node
      {% endif %}
      - ES_JAVA_OPTS=-Xms{{ GLOBALS.elasticsearch.es_heap }} -Xmx{{ GLOBALS.elasticsearch.es_heap }} -Des.transport.cname_in_publish_address=true -Dlog4j2.formatMsgNoLookups=true    <======================
      ulimits:
      - memlock=-1:-1
      - nofile=65536:65536
      - nproc=4096
      {% if DOCKER.containers['so-elasticsearch'].extra_env %}
[...]
---
----------
    Rendering SLS 'base:elasticsearch.ca' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 26

---
[...]
  cmd.script:
    - source: salt://elasticsearch/tools/sbin_jinja/so-catrust
    - template: jinja
    - cwd: /opt/so
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
{%   endif %}

cacertz:
  file.managed:
    - name: /opt/so/conf/ca/cacerts
[...]
---
----------
    Rendering SLS 'base:logstash.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 89

---
[...]
    - source: salt://logstash/pipelines/config/{{CONFIGFILE}}
      {% if 'jinja' in CONFIGFILE.split('.')[-1] %}
    - name: /opt/so/conf/logstash/pipelines/{{assigned_pipeline}}/{{CONFIGFILE.split('/')[1] | replace(".jinja", "")}}
    - template: jinja
    - defaults:
        GLOBALS: {{ GLOBALS }}    <======================
        ES_USER: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') }}"
        ES_PASS: "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') }}"
        THREADS: {{ LOGSTASH_MERGED.config.pipeline_x_workers }}
        BATCH: {{ LOGSTASH_MERGED.config.pipeline_x_batch_x_size }}
      {% else %}
[...]
---
----------
    Rendering SLS 'base:kibana.config' failed: Jinja variable 'dict object' has no attribute 'esheap'; line 52

---
[...]
    - user: 932
    - group: 939
    - file_mode: 755
    - template: jinja
    - defaults:
      GLOBALS: {{ GLOBALS }}    <======================

kibanaconfig:
  file.managed:
    - name: /opt/so/conf/kibana/etc/kibana.yml
    - source: salt://kibana/etc/kibana.yml.jinja
[...]
---

izidood Mar 10, 2025
Author

Any other suggestions ? i cant seem to solve this issue and is quite bad, as i cant get to my manager ui.

dougburks · 2025-03-11T15:10:58Z

dougburks
Mar 11, 2025
Maintainer

of what i see there is something with es_heap ?.

Yes, it looks like there are multiple errors related to es_heap. Have you recently changed your Elasticsearch heap setting or any other configuration?

What is the output of the following?

sudo grep esheap /opt/so/saltstack/local/pillar/minions/*

9 replies

izidood Mar 12, 2025
Author

the only thing i changed for some time ago where this was a problem afterwards, was changing the NTP. but i reversed that setting via the terminal, as i cant get in to the manager webui.

The output from the command you just provided is

sudo ls -alh /opt/so/saltstack/local/pillar/minions/
[sudo] password for soadmin:
total 88K
drwxr-xr-x.  2 socore socore 4.0K Mar  1  2024 .
drwxr-xr-x. 39 socore socore 4.0K Feb 13 08:54 ..
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_dataswch_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  8  2024 adv_ftpsrv-v01_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_git-dev-v01_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_isaproxy-v01_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_legacydb-v01_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_nas2012-f01_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_ntp-v01_idh.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_smtprelay-v01_idh.sls
-rw-r--r--.  1 socore socore    0 Jan 25  2024 adv_so-manager-v01_manager.sls
-rw-r--r--.  1 socore socore    0 Jan 25  2024 adv_so-receiver-v01_receiver.sls
-rw-r--r--.  1 socore socore    0 Jan 25  2024 adv_so-receiver-v02_receiver.sls
-rw-r--r--.  1 socore socore    0 Jan 25  2024 adv_so-search-v01_searchnode.sls
-rw-r--r--.  1 socore socore    0 Jan 25  2024 adv_so-search-v02_searchnode.sls
-rw-r--r--.  1 socore socore    0 Jan 25  2024 adv_so-search-v03_searchnode.sls
-rw-r--r--.  1 socore socore    0 Feb  8  2024 adv_so-search-v04_searchnode.sls
-rw-r--r--.  1 socore socore    0 Feb  8  2024 adv_so-search-v05_searchnode.sls
-rw-r--r--.  1 socore socore    0 Jan 26  2024 adv_so-sensor-f01_sensor.sls
-rw-r--r--.  1 socore socore    0 Feb  5  2024 adv_so-sensor-f02_sensor.sls
-rw-r--r--.  1 socore socore    0 Feb  7  2024 adv_so-sensor-f03_sensor.sls
-rw-r--r--.  1 socore socore    0 Feb  9  2024 adv_sql-v01_idh.sls
-rw-r--r--.  1 socore socore  210 Feb  9  2024 dataswch_idh.sls
-rw-r--r--.  1 socore socore  289 Feb 16  2024 ftpsrv-v01_idh.sls
-rw-r--r--.  1 socore socore  289 Feb 16  2024 git-dev-v01_idh.sls
-rw-r--r--.  1 socore socore  295 Feb 21  2024 isaproxy-v01_idh.sls
-rw-r--r--.  1 socore socore  291 Feb 21  2024 legacydb-v01_idh.sls
-rw-r--r--.  1 socore socore  321 Feb 21  2024 nas2012-f01_idh.sls
-rw-r--r--.  1 socore socore  289 Feb 16  2024 ntp-v01_idh.sls
-rw-r--r--.  1 socore socore  209 Feb  9  2024 smtprelay-v01_idh.sls
-rw-r--r--.  1 socore socore 1.2K Feb 14 13:27 so-manager-v01_manager.sls
-rw-r--r--.  1 socore socore  265 Jan 25  2024 so-receiver-v01_receiver.sls
-rw-r--r--.  1 socore socore  265 Jan 25  2024 so-receiver-v02_receiver.sls
-rw-r--r--.  1 socore socore  288 Feb 14 13:26 so-search-v01_searchnode.sls
-rw-r--r--.  1 socore socore  288 Feb 14 13:27 so-search-v02_searchnode.sls
-rw-r--r--.  1 socore socore  288 Feb 14 13:27 so-search-v03_searchnode.sls
-rw-r--r--.  1 socore socore  288 Feb 14 13:27 so-search-v04_searchnode.sls
-rw-r--r--.  1 socore socore  288 Feb 14 13:26 so-search-v05_searchnode.sls
-rw-r--r--.  1 socore socore  607 May 29  2024 so-sensor-f01_sensor.sls
-rw-r--r--.  1 socore socore  607 May 29  2024 so-sensor-f02_sensor.sls
-rw-r--r--.  1 socore socore  605 May 29  2024 so-sensor-f03_sensor.sls
-rw-r--r--.  1 socore socore  290 Feb 21  2024 sql-v01_idh.sls```

dougburks Mar 12, 2025
Maintainer

the only thing i changed for some time ago where this was a problem afterwards, was changing the NTP. but i reversed that setting via the terminal

How exactly did you reverse that setting via the terminal?

What is the output of the following?

sudo grep -A5 "elasticsearch:" /opt/so/saltstack/local/pillar/minions/*

izidood Mar 13, 2025
Author

i did a sudo salt-call pillar.get ntp and after i did sudo find /opt/so/ -type f -iname "*ntp*" and then found the config in /opt/so/saltstack/local/pillar/ntp/soc_ntp.sls and changed it back to default.

This is what i from your command:

sudo grep -A5 "elasticsearch:" /opt/so/saltstack/local/pillar/minions/*
/opt/so/saltstack/local/pillar/minions/so-manager-v01_manager.sls:elasticsearch:
/opt/so/saltstack/local/pillar/minions/so-manager-v01_manager.sls-    enabled: true
/opt/so/saltstack/local/pillar/minions/so-manager-v01_manager.sls-host:
/opt/so/saltstack/local/pillar/minions/so-manager-v01_manager.sls-    mainint: ens192
/opt/so/saltstack/local/pillar/minions/so-manager-v01_manager.sls-    mainip: 172.xx.x.xxx
/opt/so/saltstack/local/pillar/minions/so-manager-v01_manager.sls-idstools:
--
/opt/so/saltstack/local/pillar/minions/so-search-v01_searchnode.sls:elasticsearch:
/opt/so/saltstack/local/pillar/minions/so-search-v01_searchnode.sls-    enabled: true
/opt/so/saltstack/local/pillar/minions/so-search-v01_searchnode.sls-host:
/opt/so/saltstack/local/pillar/minions/so-search-v01_searchnode.sls-    mainint: ens192
/opt/so/saltstack/local/pillar/minions/so-search-v01_searchnode.sls-    mainip: 172.xx.x.xxx
/opt/so/saltstack/local/pillar/minions/so-search-v01_searchnode.sls-logstash:
--
/opt/so/saltstack/local/pillar/minions/so-search-v02_searchnode.sls:elasticsearch:
/opt/so/saltstack/local/pillar/minions/so-search-v02_searchnode.sls-    enabled: true
/opt/so/saltstack/local/pillar/minions/so-search-v02_searchnode.sls-host:
/opt/so/saltstack/local/pillar/minions/so-search-v02_searchnode.sls-    mainint: ens192
/opt/so/saltstack/local/pillar/minions/so-search-v02_searchnode.sls-    mainip: 172.xx.x.xxx
/opt/so/saltstack/local/pillar/minions/so-search-v02_searchnode.sls-logstash:
--
/opt/so/saltstack/local/pillar/minions/so-search-v03_searchnode.sls:elasticsearch:
/opt/so/saltstack/local/pillar/minions/so-search-v03_searchnode.sls-    enabled: true
/opt/so/saltstack/local/pillar/minions/so-search-v03_searchnode.sls-host:
/opt/so/saltstack/local/pillar/minions/so-search-v03_searchnode.sls-    mainint: ens192
/opt/so/saltstack/local/pillar/minions/so-search-v03_searchnode.sls-    mainip: 172.xx.x.xxx
/opt/so/saltstack/local/pillar/minions/so-search-v03_searchnode.sls-logstash:
--
/opt/so/saltstack/local/pillar/minions/so-search-v04_searchnode.sls:elasticsearch:
/opt/so/saltstack/local/pillar/minions/so-search-v04_searchnode.sls-    enabled: true
/opt/so/saltstack/local/pillar/minions/so-search-v04_searchnode.sls-host:
/opt/so/saltstack/local/pillar/minions/so-search-v04_searchnode.sls-    mainint: ens192
/opt/so/saltstack/local/pillar/minions/so-search-v04_searchnode.sls-    mainip: 172.xx.x.xxx
/opt/so/saltstack/local/pillar/minions/so-search-v04_searchnode.sls-logstash:
--
/opt/so/saltstack/local/pillar/minions/so-search-v05_searchnode.sls:elasticsearch:
/opt/so/saltstack/local/pillar/minions/so-search-v05_searchnode.sls-    enabled: true
/opt/so/saltstack/local/pillar/minions/so-search-v05_searchnode.sls-host:
/opt/so/saltstack/local/pillar/minions/so-search-v05_searchnode.sls-    mainint: ens192
/opt/so/saltstack/local/pillar/minions/so-search-v05_searchnode.sls-    mainip: 172.xx.x.xxx
/opt/so/saltstack/local/pillar/minions/so-search-v05_searchnode.sls-logstash:

dougburks Mar 13, 2025
Maintainer

It looks like your manager and all of your search nodes have somehow lost their esheap setting. For each of those files, please try adding an esheap line to the elasticsearch section under the enabled setting. It should look like this:

elasticsearch:
  enabled: true
  esheap: 5000m

You can replace 5000m with a value that is appropriate for your deployment.

Answer selected by izidood

izidood Mar 13, 2025
Author

Hey Doug, that fixed my problem with getting into the manager UI thanks!!. Now i just got a bunch of failed stuff, fx. i cant enter kibana, elastic fleet, osquery manager, i get an 404. should i create a new ticket ?

dougburks Mar 13, 2025
Maintainer

Yes, you can create a new discussion. However, please note that this is already a strange issue in that I don't think we've ever had a report of machines losing their esheap setting like this. That would seem to indicate that perhaps a person made the change. I'm not blaming you, but perhaps there are others who have access to your deployment. If that's the case, it may prove difficult to try to fix all of the issues to get the deployment functioning again at least via this Discussions forum. It may be a better use of your time to consider purchasing Professional Services to help you via screenshare. Alternatively, you could perform a fresh installation of Security Onion in a virtual machine and then compare that working system to your broken deployment side-by-side looking for any differences.

izidood Mar 21, 2025
Author

im the only person configurating SO here, and all this happaned after i tried to change the ntp. which is odd, as i dont see what the relation would be. but yea, atleast i got to enter the UI, so i'll close this ticket. Thanks again Doug.

Cant access My Manager Server UI #14338

Uh oh!

Uh oh!

izidood Mar 5, 2025

Version

Installation Method

Description

Installation Type

Location

Hardware Specs

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

Network Traffic Speeds

Status

Salt Status

Logs

Detail

Guidelines

Replies: 2 comments · 11 replies

Uh oh!

rh-ops Mar 6, 2025

Uh oh!

izidood Mar 7, 2025 Author

Uh oh!

izidood Mar 10, 2025 Author

Uh oh!

dougburks Mar 11, 2025 Maintainer

Uh oh!

Uh oh!

izidood Mar 12, 2025 Author

Uh oh!

dougburks Mar 12, 2025 Maintainer

Uh oh!

Uh oh!

izidood Mar 13, 2025 Author

Uh oh!

dougburks Mar 13, 2025 Maintainer

Uh oh!

izidood Mar 13, 2025 Author

Uh oh!

Uh oh!

dougburks Mar 13, 2025 Maintainer

Uh oh!

izidood Mar 21, 2025 Author

izidood
Mar 5, 2025

Replies: 2 comments 11 replies

rh-ops
Mar 6, 2025

izidood Mar 7, 2025
Author

izidood Mar 10, 2025
Author

dougburks
Mar 11, 2025
Maintainer

izidood Mar 12, 2025
Author

dougburks Mar 12, 2025
Maintainer

izidood Mar 13, 2025
Author

dougburks Mar 13, 2025
Maintainer

izidood Mar 13, 2025
Author

dougburks Mar 13, 2025
Maintainer

izidood Mar 21, 2025
Author