Template Error - Log Ingestion Not Working

[mckenziemmack]

Version

2.4.140

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

64

RAM

251G

Storage for /

558G

Storage for /nsm

38T

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hello,

I am currently experiencing the following errors when running sudo salt-call state.highstate:

As a result of this error, logs are not being ingested, and I suspect this template problem is the root cause.

so-status show all services are running, SOC Grid page doesn't show any failures, and all indices are in the green.

Could anyone offer advice on how to troubleshoot or resolve this template error? Any insights would be greatly appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Have you made any configuration changes to existing templates or created custom index templates?

Are you able to post the full log so we can see which template(s) is having an issue? I only see templates successfully loading in the screenshot

[rfuller318]

I am having the same issue. The error comes from an OSQuery Manager json template load. This is the information I receive from running salt-run state.event pretty=True:

"comment": "Command \"/usr/sbin/so-elasticsearch-templates-load\" run",
"duration": 16819.586,
"fun": "state.highstate",
"id": "xxxxxxxxxxxx",
"jid": "20250403150203641038",
"name": "/usr/sbin/so-elasticsearch-templates-load",
"result": false,
"retcode": 2,
"return": "Error: cmd.run",
"start_time": "14:54:56.686988",
"success": false

}

Forcing exit code to 1 Command failed with exit code 1; will retry in 1 seconds (3 / 3)... Command continues to fail; giving up. ERROR: Could not load template file: so-logs-osquery-manager.action.responses-template.json

[reyesj2]

Thank you, can you run this and share the output

ls -1 /opt/so/conf/elasticsearch/templates/index/so-logs-osquery* 

[rfuller318]

I get this output when I run the command as sudo:

ls: cannot access '/opt/so/conf/elasticsearch/templates/index/so-logs-osquery*': No such file or directory

But when I log in as root and run the command:

/opt/so/conf/elasticsearch/templates/index/so-logs-osquery-manager-action.responses-template.json
/opt/so/conf/elasticsearch/templates/index/so-logs-osquery-manager.action.responses-template.json
/opt/so/conf/elasticsearch/templates/index/so-logs-osquery-manager-actions-template.json
/opt/so/conf/elasticsearch/templates/index/so-logs-osquery-manager.result-template.json

[reyesj2]

Have you made any configuration changes to existing templates or created custom index templates?

Or copied a defaults.yaml to another location?

[rfuller318]

No, I have not made any template changes, custom templates, or copied defaults.yaml.

[mckenziemmack]

My issue was with the logs-cisco_duo.auth-* template and was due to some custom configurations in the SO Console under elasticsearch > advanced [adv]. Initially, I was searching for a solution in Elastic's Index Management > Templates, but the overriding settings in the advanced configurations were the root cause.