After upgrade from 2.4.120 to 2.4.130 the Custom Logs integration is broken (unable to manually add or change Custom Logs integration)

[zenon8000]

Version

2.4.130

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

50GB

Storage for /

1500GB

Storage for /nsm

1300GB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

After upgrading from 2.4.120 to 2.4.130 the Custom Logs integration is broken. Is is also not possible to manually add or change Custom Logs integration.

Custom logs are predefined under so-grid-nodes_general policy.

If I try to open the integration, there will not be any result.

SO-status is OK.

Here is the error from so-checkin command:

The command: **so-elastic-fleet-integration-upgrade; echo $?** results:
Current log package version (2.3.0) is not the same as the latest available package (2.3.3)...
Upgrading import-evtx-logs...
Starting dry run...
jq: error (at <stdin>:1): Cannot index number with string "hasErrors"
Errors detected during dry run. Stopping upgrade...
1


Here are the ERROR logs from: so-elastic-fleet-restart command
=========================================================================
Restarting elastic-fleet...

This could take a while if another Salt job is running.
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
so-elastic-fleet
so-elastic-fleet
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [elastic-fleet] at time 12:40:57.569589
[INFO    ] Executing state group.present for [elastic-fleet]
[INFO    ] Group elastic-fleet is present and up to date
[INFO    ] Completed state [elastic-fleet] at time 12:40:57.573748 (duration_in_ms=4.16)
[INFO    ] Running state [elastic-fleet] at time 12:40:57.574811
[INFO    ] Executing state user.present for [elastic-fleet]
[INFO    ] User elastic-fleet is present and up to date
[INFO    ] Completed state [elastic-fleet] at time 12:40:57.602262 (duration_in_ms=27.451)
[INFO    ] Running state [/usr/sbin] at time 12:40:57.603615
[INFO    ] Executing state file.recurse for [/usr/sbin]
[INFO    ] The directory /usr/sbin is in the correct state
[INFO    ] Completed state [/usr/sbin] at time 12:40:57.749422 (duration_in_ms=145.808)
[INFO    ] Running state [/usr/sbin] at time 12:40:57.749612
[INFO    ] Executing state file.recurse for [/usr/sbin]
[INFO    ] The directory /usr/sbin is in the correct state
[INFO    ] Completed state [/usr/sbin] at time 12:40:58.943210 (duration_in_ms=1193.599)
[INFO    ] Running state [/opt/so/conf/elastic-fleet] at time 12:40:58.943391
[INFO    ] Executing state file.directory for [/opt/so/conf/elastic-fleet]
[INFO    ] The directory /opt/so/conf/elastic-fleet is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet] at time 12:40:58.944420 (duration_in_ms=1.029)
[INFO    ] Running state [/opt/so/log/elasticfleet] at time 12:40:58.944639
[INFO    ] Executing state file.directory for [/opt/so/log/elasticfleet]
[INFO    ] The directory /opt/so/log/elasticfleet is in the correct state
[INFO    ] Completed state [/opt/so/log/elasticfleet] at time 12:40:58.945606 (duration_in_ms=0.967)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/state] at time 12:40:58.945743
[INFO    ] Executing state file.directory for [/opt/so/conf/elastic-fleet/state]
[INFO    ] The directory /opt/so/conf/elastic-fleet/state is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/state] at time 12:40:58.946596 (duration_in_ms=0.854)
[INFO    ] Running state [/nsm/custom-mappings] at time 12:40:58.946726
[INFO    ] Executing state file.directory for [/nsm/custom-mappings]
[INFO    ] The directory /nsm/custom-mappings is in the correct state
[INFO    ] Completed state [/nsm/custom-mappings] at time 12:40:58.948231 (duration_in_ms=1.505)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-package-upgrade] at time 12:40:58.948364
[INFO    ] Executing state file.managed for [/usr/sbin/so-elastic-fleet-package-upgrade]
[INFO    ] File /usr/sbin/so-elastic-fleet-package-upgrade is in the correct state
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-package-upgrade] at time 12:40:58.966273 (duration_in_ms=17.908)
[INFO    ] Executing command git in directory '/root'
[INFO    ] Running state [https://github.com/Security-Onion-Solutions/securityonion-resources.git] at time 12:40:58.971377
[INFO    ] Executing state git.latest for [https://github.com/Security-Onion-Solutions/securityonion-resources.git]
[INFO    ] Executing command git in directory '/root'
[INFO    ] Checking remote revision for https://github.com/Security-Onion-Solutions/securityonion-resources.git
[INFO    ] Executing command git in directory '/root'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Checking local revision for /nsm/securityonion-resources
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Checking local branch for /nsm/securityonion-resources
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Executing command git in directory '/nsm/securityonion-resources'
[INFO    ] Repository /nsm/securityonion-resources is up-to-date
[INFO    ] Completed state [https://github.com/Security-Onion-Solutions/securityonion-resources.git] at time 12:40:59.816923 (duration_in_ms=845.546)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/defend-exclusions/rulesets] at time 12:40:59.817112
[INFO    ] Executing state file.directory for [/opt/so/conf/elastic-fleet/defend-exclusions/rulesets]
[INFO    ] The directory /opt/so/conf/elastic-fleet/defend-exclusions/rulesets is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/defend-exclusions/rulesets] at time 12:40:59.818285 (duration_in_ms=1.173)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml] at time 12:40:59.818465
[INFO    ] Executing state file.managed for [/opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml]
[INFO    ] File /opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/defend-exclusions/disabled-filters.yaml] at time 12:40:59.828132 (duration_in_ms=9.666)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters-raw] at time 12:40:59.828374
[INFO    ] Executing state file.managed for [/opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters-raw]
[INFO    ] File /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters-raw is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters-raw] at time 12:40:59.837129 (duration_in_ms=8.756)
[INFO    ] Running state [python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/elastic-fleet/defend                                                                                                -exclusions/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters/                                                                                                 &>> /opt/so/log/elasticfleet/elastic-defend-manage-filters.log] at time 12:40:59.839020
[INFO    ] Executing state cron.absent for [python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/el                                                                                                astic-fleet/defend-exclusions/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/ruleset                                                                                                s/custom-filters/ &>> /opt/so/log/elasticfleet/elastic-defend-manage-filters.log]
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] Cron python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/elastic-fleet/defend-exclusion                                                                                                s/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters/ &>> /opt/s                                                                                                o/log/elasticfleet/elastic-defend-manage-filters.log already absent
[INFO    ] Completed state [python3 /sbin/so-elastic-defend-manage-filters.py -c /opt/so/conf/elasticsearch/curl.config -d /opt/so/conf/elastic-fleet/defe                                                                                                nd-exclusions/disabled-filters.yaml -i /nsm/securityonion-resources/event_filters/ -i /opt/so/conf/elastic-fleet/defend-exclusions/rulesets/custom-filters                                                                                                / &>> /opt/so/log/elasticfleet/elastic-defend-manage-filters.log] at time 12:40:59.843534 (duration_in_ms=4.513)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations] at time 12:40:59.843699
[INFO    ] Executing state file.directory for [/opt/so/conf/elastic-fleet/integrations]
[INFO    ] The directory /opt/so/conf/elastic-fleet/integrations is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations] at time 12:40:59.844587 (duration_in_ms=0.887)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations] at time 12:40:59.844724
[INFO    ] Executing state file.recurse for [/opt/so/conf/elastic-fleet/integrations]
[INFO    ] The directory /opt/so/conf/elastic-fleet/integrations is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations] at time 12:40:59.926122 (duration_in_ms=81.398)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations] at time 12:40:59.926325
[INFO    ] Executing state file.recurse for [/opt/so/conf/elastic-fleet/integrations]
[INFO    ] The directory /opt/so/conf/elastic-fleet/integrations is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations] at time 12:41:00.223893 (duration_in_ms=297.569)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations-optional] at time 12:41:00.224207
[INFO    ] Executing state file.directory for [/opt/so/conf/elastic-fleet/integrations-optional]
[INFO    ] The directory /opt/so/conf/elastic-fleet/integrations-optional is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations-optional] at time 12:41:00.225443 (duration_in_ms=1.236)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc] at time 12:41:00.225669
[INFO    ] Executing state file.directory for [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc]
[INFO    ] The directory /opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc] at time 12:41:00.227049 (duration_in_ms=1.381)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/sublime_platform.json] at time 12:41:00.227272
[INFO    ] Executing state file.absent for [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/sublime_platform.json]
[INFO    ] File /opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/sublime_platform.json is not present
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/sublime_platform.json] at time 12:41:00.227976 (duration_i                                                                                                n_ms=0.704)
[INFO    ] Running state [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/kismet.json] at time 12:41:00.228167
[INFO    ] Executing state file.absent for [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/kismet.json]
[INFO    ] File /opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/kismet.json is not present
[INFO    ] Completed state [/opt/so/conf/elastic-fleet/integrations-optional/FleetServer_so-poc/kismet.json] at time 12:41:00.228870 (duration_in_ms=0.703                                                                                                )
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 12:41:00.229997
[INFO    ] Executing state file.append for [/opt/so/conf/so-status/so-status.conf]
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and                                                                                                 instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon)                                                                                                .
  salt.utils.versions.warn_until(
[INFO    ] Executing command 'grep' in directory '/root'
[INFO    ] ['unless condition is true']
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 12:41:01.095745 (duration_in_ms=865.747)
[INFO    ] Running state [/etc/pki/issued_certs] at time 12:41:01.095980
[INFO    ] Executing state file.directory for [/etc/pki/issued_certs]
[INFO    ] The directory /etc/pki/issued_certs is in the correct state
[INFO    ] Completed state [/etc/pki/issued_certs] at time 12:41:01.097322 (duration_in_ms=1.341)
[INFO    ] Running state [/etc/salt/minion.d/signing_policies.conf] at time 12:41:01.097512
[INFO    ] Executing state file.managed for [/etc/salt/minion.d/signing_policies.conf]
[INFO    ] File /etc/salt/minion.d/signing_policies.conf is in the correct state
[INFO    ] Completed state [/etc/salt/minion.d/signing_policies.conf] at time 12:41:01.107127 (duration_in_ms=9.614)
[INFO    ] Running state [/etc/pki/ca.crt] at time 12:41:01.108015
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/ca.crt] at time 12:41:01.152666
[INFO    ] Executing state file.managed for [/etc/pki/ca.crt]
[INFO    ] File /etc/pki/ca.crt not updated
[INFO    ] Completed state [/etc/pki/ca.crt] at time 12:41:01.155761 (duration_in_ms=3.096)
[INFO    ] Running state [/etc/pki/ca.crt] at time 12:41:01.428244
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/ca.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/ca.crt] at time 12:41:01.468281
[INFO    ] Executing state file.managed for [/etc/pki/ca.crt]
[INFO    ] File /etc/pki/ca.crt not updated
[INFO    ] Completed state [/etc/pki/ca.crt] at time 12:41:01.471535 (duration_in_ms=3.255)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/ca.crt] at time 12:41:01.735968 (duration_in_ms=307.724)
[INFO    ] Running state [/etc/pki/ca.key] at time 12:41:01.736682
[INFO    ] Executing state file.managed for [/etc/pki/ca.key]
[INFO    ] File /etc/pki/ca.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/ca.key] at time 12:41:01.737906 (duration_in_ms=1.223)
[INFO    ] Running state [/etc/pki/tls/certs] at time 12:41:01.738049
[INFO    ] Executing state file.directory for [/etc/pki/tls/certs]
[INFO    ] The directory /etc/pki/tls/certs is in the correct state
[INFO    ] Completed state [/etc/pki/tls/certs] at time 12:41:01.738863 (duration_in_ms=0.814)
[INFO    ] Running state [/etc/pki/tls/certs/intca.crt] at time 12:41:01.739004
[INFO    ] Executing state x509.pem_managed for [/etc/pki/tls/certs/intca.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/tls/certs/intca.crt] at time 12:41:01.782043
[INFO    ] Executing state file.managed for [/etc/pki/tls/certs/intca.crt]
[INFO    ] File /etc/pki/tls/certs/intca.crt is in the correct state
[INFO    ] Completed state [/etc/pki/tls/certs/intca.crt] at time 12:41:01.786198 (duration_in_ms=4.156)
[INFO    ] File /etc/pki/tls/certs/intca.crt is in the correct state
[INFO    ] Completed state [/etc/pki/tls/certs/intca.crt] at time 12:41:01.786527 (duration_in_ms=47.522)
[INFO    ] Running state [m2cryptopkgs] at time 12:41:01.792265
[INFO    ] Executing state pkg.installed for [m2cryptopkgs]
[INFO    ] Executing command rpm in directory '/root'
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [m2cryptopkgs] at time 12:41:01.940404 (duration_in_ms=148.138)
[INFO    ] Running state [/etc/pki/influxdb.crt] at time 12:41:01.941130
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/influxdb.crt] at time 12:41:01.984097
[INFO    ] Executing state file.managed for [/etc/pki/influxdb.crt]
[INFO    ] File /etc/pki/influxdb.crt not updated
[INFO    ] Completed state [/etc/pki/influxdb.crt] at time 12:41:01.986826 (duration_in_ms=2.728)
[INFO    ] Publishing 'x509.sign_remote_certificate' to tcp://127.0.0.1:4506
[INFO    ] Running state [/etc/pki/influxdb.crt] at time 12:41:02.574574
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/influxdb.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/influxdb.crt] at time 12:41:02.675710
[INFO    ] Executing state file.managed for [/etc/pki/influxdb.crt]
[INFO    ] File /etc/pki/influxdb.crt not updated
[INFO    ] Completed state [/etc/pki/influxdb.crt] at time 12:41:02.678821 (duration_in_ms=3.112)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/influxdb.crt] at time 12:41:02.940302 (duration_in_ms=365.729)
[INFO    ] Running state [/etc/pki/influxdb.key] at time 12:41:02.940537
[INFO    ] Executing state file.managed for [/etc/pki/influxdb.key]
[INFO    ] File /etc/pki/influxdb.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/influxdb.key] at time 12:41:02.941750 (duration_in_ms=1.213)
[INFO    ] Running state [/etc/pki/redis.crt] at time 12:41:02.942459
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/redis.crt] at time 12:41:02.981544
[INFO    ] Executing state file.managed for [/etc/pki/redis.crt]
[INFO    ] File /etc/pki/redis.crt not updated
[INFO    ] Completed state [/etc/pki/redis.crt] at time 12:41:02.984030 (duration_in_ms=2.486)
[INFO    ] Publishing 'x509.sign_remote_certificate' to tcp://127.0.0.1:4506
[INFO    ] Running state [/etc/pki/redis.crt] at time 12:41:03.588627
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/redis.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/redis.crt] at time 12:41:03.634229
[INFO    ] Executing state file.managed for [/etc/pki/redis.crt]
[INFO    ] File /etc/pki/redis.crt not updated
[INFO    ] Completed state [/etc/pki/redis.crt] at time 12:41:03.637221 (duration_in_ms=2.992)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/redis.crt] at time 12:41:03.899360 (duration_in_ms=310.733)
[INFO    ] Running state [/etc/pki/redis.key] at time 12:41:03.899617
[INFO    ] Executing state file.managed for [/etc/pki/redis.key]
[INFO    ] File /etc/pki/redis.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/redis.key] at time 12:41:03.901041 (duration_in_ms=1.424)
[INFO    ] Running state [/etc/pki/elasticfleet-server.crt] at time 12:41:03.901768
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-server.crt] at time 12:41:03.943614
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-server.crt]
[INFO    ] File /etc/pki/elasticfleet-server.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-server.crt] at time 12:41:03.946593 (duration_in_ms=2.979)
[INFO    ] Publishing 'x509.sign_remote_certificate' to tcp://127.0.0.1:4506
[INFO    ] Running state [/etc/pki/elasticfleet-server.crt] at time 12:41:04.528575
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/elasticfleet-server.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-server.crt] at time 12:41:04.579358
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-server.crt]
[INFO    ] File /etc/pki/elasticfleet-server.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-server.crt] at time 12:41:04.582133 (duration_in_ms=2.774)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/elasticfleet-server.crt] at time 12:41:04.842568 (duration_in_ms=313.993)
[INFO    ] Running state [/etc/pki/elasticfleet-server.key] at time 12:41:04.842834
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-server.key]
[INFO    ] File /etc/pki/elasticfleet-server.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-server.key] at time 12:41:04.844182 (duration_in_ms=1.347)
[INFO    ] Running state [/etc/pki/elasticfleet-server.crt] at time 12:41:04.844371
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-server.crt]
[INFO    ] File /etc/pki/elasticfleet-server.crt exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-server.crt] at time 12:41:04.845512 (duration_in_ms=1.14)
[INFO    ] Running state [/etc/pki/elasticfleet-server.key] at time 12:41:04.845651
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-server.key]
[INFO    ] File /etc/pki/elasticfleet-server.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-server.key] at time 12:41:04.846697 (duration_in_ms=1.046)
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:04.847433
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:04.889642
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-logstash.crt]
[INFO    ] File /etc/pki/elasticfleet-logstash.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:04.892463 (duration_in_ms=2.821)
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:05.156262
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/elasticfleet-logstash.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:05.200157
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-logstash.crt]
[INFO    ] File /etc/pki/elasticfleet-logstash.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:05.204680 (duration_in_ms=4.523)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:05.471538 (duration_in_ms=315.276)
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.key] at time 12:41:05.472178
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-logstash.key]
[INFO    ] File /etc/pki/elasticfleet-logstash.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-logstash.key] at time 12:41:05.473663 (duration_in_ms=1.485)
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:05.473822
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-logstash.crt]
[INFO    ] File /etc/pki/elasticfleet-logstash.crt exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-logstash.crt] at time 12:41:05.474888 (duration_in_ms=1.066)
[INFO    ] Running state [/etc/pki/elasticfleet-logstash.key] at time 12:41:05.475043
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-logstash.key]
[INFO    ] File /etc/pki/elasticfleet-logstash.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-logstash.key] at time 12:41:05.476106 (duration_in_ms=1.064)
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:05.476852
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:05.519821
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-lumberjack.crt]
[INFO    ] File /etc/pki/elasticfleet-lumberjack.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:05.523923 (duration_in_ms=4.101)
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:05.820811
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/elasticfleet-lumberjack.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:05.864143
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-lumberjack.crt]
[INFO    ] File /etc/pki/elasticfleet-lumberjack.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:05.866930 (duration_in_ms=2.788)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:06.125514 (duration_in_ms=304.703)
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.key] at time 12:41:06.126057
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-lumberjack.key]
[INFO    ] File /etc/pki/elasticfleet-lumberjack.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.key] at time 12:41:06.127291 (duration_in_ms=1.234)
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.p8] at time 12:41:06.127434
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-lumberjack.p8]
[INFO    ] File /etc/pki/elasticfleet-lumberjack.p8 exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.p8] at time 12:41:06.128466 (duration_in_ms=1.033)
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:06.128599
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-lumberjack.crt]
[INFO    ] File /etc/pki/elasticfleet-lumberjack.crt exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.crt] at time 12:41:06.129571 (duration_in_ms=0.972)
[INFO    ] Running state [/etc/pki/elasticfleet-lumberjack.key] at time 12:41:06.129701
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-lumberjack.key]
[INFO    ] File /etc/pki/elasticfleet-lumberjack.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-lumberjack.key] at time 12:41:06.130676 (duration_in_ms=0.974)
[INFO    ] Running state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.131370
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.172281
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-agent.crt]
[INFO    ] File /etc/pki/elasticfleet-agent.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.174909 (duration_in_ms=2.628)
[INFO    ] Running state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.437195
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/elasticfleet-agent.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.482729
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-agent.crt]
[INFO    ] File /etc/pki/elasticfleet-agent.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.485969 (duration_in_ms=3.241)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.769573 (duration_in_ms=332.378)
[INFO    ] Running state [/etc/pki/elasticfleet-agent.key] at time 12:41:06.770265
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-agent.key]
[INFO    ] File /etc/pki/elasticfleet-agent.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-agent.key] at time 12:41:06.771845 (duration_in_ms=1.58)
[INFO    ] Running state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.772039
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-agent.crt]
[INFO    ] File /etc/pki/elasticfleet-agent.crt exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-agent.crt] at time 12:41:06.773207 (duration_in_ms=1.168)
[INFO    ] Running state [/etc/pki/elasticfleet-agent.key] at time 12:41:06.773369
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-agent.key]
[INFO    ] File /etc/pki/elasticfleet-agent.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-agent.key] at time 12:41:06.774430 (duration_in_ms=1.061)
[INFO    ] Running state [/etc/pki/filebeat.crt] at time 12:41:06.775250
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/filebeat.crt] at time 12:41:06.818130
[INFO    ] Executing state file.managed for [/etc/pki/filebeat.crt]
[INFO    ] File /etc/pki/filebeat.crt not updated
[INFO    ] Completed state [/etc/pki/filebeat.crt] at time 12:41:06.821251 (duration_in_ms=3.121)
[INFO    ] Publishing 'x509.sign_remote_certificate' to tcp://127.0.0.1:4506
[INFO    ] Running state [/etc/pki/filebeat.crt] at time 12:41:07.412670
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/filebeat.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/filebeat.crt] at time 12:41:07.456597
[INFO    ] Executing state file.managed for [/etc/pki/filebeat.crt]
[INFO    ] File /etc/pki/filebeat.crt not updated
[INFO    ] Completed state [/etc/pki/filebeat.crt] at time 12:41:07.459280 (duration_in_ms=2.682)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/filebeat.crt] at time 12:41:07.716455 (duration_in_ms=303.785)
[INFO    ] Running state [/etc/pki/filebeat.key] at time 12:41:07.717048
[INFO    ] Executing state file.managed for [/etc/pki/filebeat.key]
[INFO    ] File /etc/pki/filebeat.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/filebeat.key] at time 12:41:07.718295 (duration_in_ms=1.247)
[INFO    ] Running state [/etc/pki/filebeat.p8] at time 12:41:07.718435
[INFO    ] Executing state file.managed for [/etc/pki/filebeat.p8]
[INFO    ] File /etc/pki/filebeat.p8 exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/filebeat.p8] at time 12:41:07.719435 (duration_in_ms=1.0)
[INFO    ] Running state [/opt/so/saltstack/local/salt/filebeat/files] at time 12:41:07.719567
[INFO    ] Executing state file.directory for [/opt/so/saltstack/local/salt/filebeat/files]
[INFO    ] The directory /opt/so/saltstack/local/salt/filebeat/files is in the correct state
[INFO    ] Completed state [/opt/so/saltstack/local/salt/filebeat/files] at time 12:41:07.720372 (duration_in_ms=0.804)
[INFO    ] Running state [/opt/so/saltstack/local/salt/filebeat/files/filebeat.p8] at time 12:41:07.720501
[INFO    ] Executing state file.symlink for [/opt/so/saltstack/local/salt/filebeat/files/filebeat.p8]
[INFO    ] Symlink /opt/so/saltstack/local/salt/filebeat/files/filebeat.p8 is present and owned by socore:socore
[INFO    ] Completed state [/opt/so/saltstack/local/salt/filebeat/files/filebeat.p8] at time 12:41:07.721912 (duration_in_ms=1.411)
[INFO    ] Running state [/opt/so/saltstack/local/salt/filebeat/files/filebeat.crt] at time 12:41:07.722044
[INFO    ] Executing state file.symlink for [/opt/so/saltstack/local/salt/filebeat/files/filebeat.crt]
[INFO    ] Symlink /opt/so/saltstack/local/salt/filebeat/files/filebeat.crt is present and owned by socore:socore
[INFO    ] Completed state [/opt/so/saltstack/local/salt/filebeat/files/filebeat.crt] at time 12:41:07.723268 (duration_in_ms=1.224)
[INFO    ] Running state [/etc/pki/registry.crt] at time 12:41:07.723973
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/registry.crt] at time 12:41:07.764007
[INFO    ] Executing state file.managed for [/etc/pki/registry.crt]
[INFO    ] File /etc/pki/registry.crt not updated
[INFO    ] Completed state [/etc/pki/registry.crt] at time 12:41:07.766962 (duration_in_ms=2.955)
[INFO    ] Running state [/etc/pki/registry.crt] at time 12:41:08.036548
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/registry.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/registry.crt] at time 12:41:08.082181
[INFO    ] Executing state file.managed for [/etc/pki/registry.crt]
[INFO    ] File /etc/pki/registry.crt not updated
[INFO    ] Completed state [/etc/pki/registry.crt] at time 12:41:08.085407 (duration_in_ms=3.226)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/registry.crt] at time 12:41:08.350533 (duration_in_ms=313.985)
[INFO    ] Running state [/etc/pki/registry.key] at time 12:41:08.350790
[INFO    ] Executing state file.managed for [/etc/pki/registry.key]
[INFO    ] File /etc/pki/registry.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/registry.key] at time 12:41:08.352048 (duration_in_ms=1.259)
[INFO    ] Running state [/etc/pki/elasticsearch.crt] at time 12:41:08.352862
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticsearch.crt] at time 12:41:08.395780
[INFO    ] Executing state file.managed for [/etc/pki/elasticsearch.crt]
[INFO    ] File /etc/pki/elasticsearch.crt not updated
[INFO    ] Completed state [/etc/pki/elasticsearch.crt] at time 12:41:08.398600 (duration_in_ms=2.82)
[INFO    ] Running state [/etc/pki/elasticsearch.crt] at time 12:41:08.676827
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/elasticsearch.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticsearch.crt] at time 12:41:08.719647
[INFO    ] Executing state file.managed for [/etc/pki/elasticsearch.crt]
[INFO    ] File /etc/pki/elasticsearch.crt not updated
[INFO    ] Completed state [/etc/pki/elasticsearch.crt] at time 12:41:08.722708 (duration_in_ms=3.062)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/elasticsearch.crt] at time 12:41:08.984694 (duration_in_ms=307.868)
[INFO    ] Running state [/etc/pki/elasticsearch.key] at time 12:41:08.985314
[INFO    ] Executing state file.managed for [/etc/pki/elasticsearch.key]
[INFO    ] File /etc/pki/elasticsearch.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticsearch.key] at time 12:41:08.986785 (duration_in_ms=1.471)
[INFO    ] Running state [/etc/pki/elasticsearch.p12] at time 12:41:08.986951
[INFO    ] Executing state file.managed for [/etc/pki/elasticsearch.p12]
[INFO    ] File /etc/pki/elasticsearch.p12 exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticsearch.p12] at time 12:41:08.988011 (duration_in_ms=1.06)
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki] at time 12:41:08.988156
[INFO    ] Executing state file.directory for [/opt/so/conf/filebeat/etc/pki]
[INFO    ] The directory /opt/so/conf/filebeat/etc/pki is in the correct state
[INFO    ] Completed state [/opt/so/conf/filebeat/etc/pki] at time 12:41:08.988996 (duration_in_ms=0.839)
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:08.989731
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:09.106805
[INFO    ] Executing state file.managed for [/opt/so/conf/filebeat/etc/pki/filebeat.crt]
[INFO    ] File /opt/so/conf/filebeat/etc/pki/filebeat.crt not updated
[INFO    ] Completed state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:09.109483 (duration_in_ms=2.679)
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:09.380766
[INFO    ] Executing state x509.certificate_managed for [/opt/so/conf/filebeat/etc/pki/filebeat.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:09.424046
[INFO    ] Executing state file.managed for [/opt/so/conf/filebeat/etc/pki/filebeat.crt]
[INFO    ] File /opt/so/conf/filebeat/etc/pki/filebeat.crt not updated
[INFO    ] Completed state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:09.426797 (duration_in_ms=2.75)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/opt/so/conf/filebeat/etc/pki/filebeat.crt] at time 12:41:09.701291 (duration_in_ms=320.526)
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki/filebeat.key] at time 12:41:09.701954
[INFO    ] Executing state file.managed for [/opt/so/conf/filebeat/etc/pki/filebeat.key]
[INFO    ] File /opt/so/conf/filebeat/etc/pki/filebeat.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/opt/so/conf/filebeat/etc/pki/filebeat.key] at time 12:41:09.703583 (duration_in_ms=1.629)
[INFO    ] Running state [/opt/so/conf/filebeat/etc/pki/filebeat.p8] at time 12:41:09.703774
[INFO    ] Executing state file.managed for [/opt/so/conf/filebeat/etc/pki/filebeat.p8]
[INFO    ] File /opt/so/conf/filebeat/etc/pki/filebeat.p8 exists with proper permissions. No changes made.
[INFO    ] Completed state [/opt/so/conf/filebeat/etc/pki/filebeat.p8] at time 12:41:09.705092 (duration_in_ms=1.318)
[INFO    ] Running state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:09.705939
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:09.745891
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-kafka.crt]
[INFO    ] File /etc/pki/elasticfleet-kafka.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:09.748849 (duration_in_ms=2.959)
[INFO    ] Publishing 'x509.sign_remote_certificate' to tcp://127.0.0.1:4506
[INFO    ] Running state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:10.341985
[INFO    ] Executing state x509.certificate_managed for [/etc/pki/elasticfleet-kafka.crt]
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:10.392195
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-kafka.crt]
[INFO    ] File /etc/pki/elasticfleet-kafka.crt not updated
[INFO    ] Completed state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:10.395835 (duration_in_ms=3.641)
[INFO    ] The certificate is in the correct state
[INFO    ] Completed state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:10.662741 (duration_in_ms=320.756)
[INFO    ] Running state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:10.663011
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-kafka.crt]
[INFO    ] File /etc/pki/elasticfleet-kafka.crt exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-kafka.crt] at time 12:41:10.664319 (duration_in_ms=1.308)
[INFO    ] Running state [/etc/pki/elasticfleet-kafka.key] at time 12:41:10.664479
[INFO    ] Executing state file.managed for [/etc/pki/elasticfleet-kafka.key]
[INFO    ] File /etc/pki/elasticfleet-kafka.key exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/pki/elasticfleet-kafka.key] at time 12:41:10.665642 (duration_in_ms=1.162)
[INFO    ] Running state [so-elasticsearch-wait] at time 12:41:10.665794
[INFO    ] Executing state cmd.run for [so-elasticsearch-wait]
[INFO    ] Executing command 'so-elasticsearch-wait' in directory '/root'
[INFO    ] {'pid': 1305210, 'retcode': 0, 'stdout': "Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)\nReceived ex                                                                                                pected response; proceeding.", 'stderr': ''}
[INFO    ] Completed state [so-elasticsearch-wait] at time 12:41:10.707045 (duration_in_ms=41.25)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-outputs-update] at time 12:41:10.707366
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-outputs-update]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-outputs-update' in directory '/root'
[INFO    ] {'pid': 1305218, 'retcode': 0, 'stdout': '\nHashes match - no update needed.\nCurrent List: ["so-poc:5055"]\nNew List: ["so-poc:5055"]\nCheckin                                                                                                g if the correct output policy is set as default\nDefault output policy is set - no update needed.', 'stderr': '  % Total    % Received % Xferd  Average S                                                                                                peed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0                                                                                                      0      0 --:--:-- --:--:-- --:--:--     0\r100  7713  100  7713    0     0   313k      0 --:--:-- --:--:-- --:--:--  313k'}
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-outputs-update] at time 12:41:14.795507 (duration_in_ms=4088.14)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-urls-update] at time 12:41:14.795927
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-urls-update]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-urls-update' in directory '/root'
[INFO    ] {'pid': 1306269, 'retcode': 0, 'stdout': '\nHashes match - no update needed.\nCurrent List: ["https://so-poc:8220"]\nNew List: ["https://so-poc                                                                                                :8220"]', 'stderr': '  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Tot                                                                                                al   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   131  100   131    0     0  10076      0                                                                                                 --:--:-- --:--:-- --:--:-- 10076'}
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-urls-update] at time 12:41:18.803788 (duration_in_ms=4007.861)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-es-url-update] at time 12:41:18.804038
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-es-url-update]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-es-url-update' in directory '/root'
[INFO    ] {'pid': 1307292, 'retcode': 0, 'stdout': '\nHashes match - no update needed.\nCurrent List: ["https://so-poc:9200"]\nNew List: ["https://so-poc                                                                                                :9200"]', 'stderr': '  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Tot                                                                                                al   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100  2250  100  2250    0     0   129k      0                                                                                                 --:--:-- --:--:-- --:--:--  129k'}
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-es-url-update] at time 12:41:18.851829 (duration_in_ms=47.792)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-artifacts-url-update] at time 12:41:18.852085
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-artifacts-url-update]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-artifacts-url-update' in directory '/root'
[INFO    ] {'pid': 1307311, 'retcode': 0, 'stdout': 'All expected artifact URLs are present. No updates needed.', 'stderr': '  % Total    % Received % Xfe                                                                                                rd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0                                                                                                    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   311  100   311    0     0  23923      0 --:--:-- --:--:-- --:--:-- 23923'}
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-artifacts-url-update] at time 12:41:18.883668 (duration_in_ms=31.584)
[INFO    ] Running state [so-elastic-fleet] at time 12:41:18.884589
[INFO    ] Executing state docker_container.running for [so-elastic-fleet]
[INFO    ] {'container': {'Networks': {'bridge': {'IPConfiguration': {'old': 'automatic', 'new': 'not connected'}}, 'sobridge': {'IPAMConfig': {'old': Non                                                                                                e, 'new': {'IPv4Address': '172.17.1.21'}}}}}, 'container_id': {'added': '85d5177709fe0d821da4f6477e86ad0239ad9ae53b505f4354dc60c919260bda'}, 'state': {'ol                                                                                                d': None, 'new': 'running'}}
[INFO    ] Completed state [so-elastic-fleet] at time 12:41:20.378061 (duration_in_ms=1493.472)
[INFO    ] Running state [/opt/so/state/elastic_fleet_packages.txt] at time 12:41:20.378352
[INFO    ] Executing state file.managed for [/opt/so/state/elastic_fleet_packages.txt]
[INFO    ] File /opt/so/state/elastic_fleet_packages.txt is in the correct state
[INFO    ] Completed state [/opt/so/state/elastic_fleet_packages.txt] at time 12:41:20.380649 (duration_in_ms=2.298)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-integration-policy-load] at time 12:41:20.381364
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-integration-policy-load]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-integration-policy-load' in directory '/root'
[INFO    ] {'pid': 1307481, 'retcode': 0, 'stdout': '', 'stderr': ''}
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-integration-policy-load] at time 12:41:20.389329 (duration_in_ms=7.966)
[INFO    ] Running state [/usr/sbin/so-elastic-agent-grid-upgrade] at time 12:41:20.389471
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-agent-grid-upgrade]
[INFO    ] Executing command '/usr/sbin/so-elastic-agent-grid-upgrade' in directory '/root'
[INFO    ] {'pid': 1307486, 'retcode': 0, 'stdout': 'No Agents need updates... Exiting', 'stderr': '  % Total    % Received % Xferd  Average Speed   Time                                                                                                    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0                                                                                                 --:--:-- --:--:-- --:--:--     0\r100   189  100   189    0     0   7560      0 --:--:-- --:--:-- --:--:--  7560'}
[INFO    ] Completed state [/usr/sbin/so-elastic-agent-grid-upgrade] at time 12:41:20.442300 (duration_in_ms=52.829)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-integration-upgrade] at time 12:41:20.442554
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-integration-upgrade]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-integration-upgrade' in directory '/root'
[**ERROR   ] Command '/usr/sbin/so-elastic-fleet-integration-upgrade' failed with return code: 1
[ERROR   ] stdout: Current log package version (2.3.0) is not the same as the latest available package (2.3.3)...
Upgrading import-evtx-logs...
Starting dry run...
Errors detected during dry run. Stopping upgrade...
[ERROR   ] stderr: jq: error (at <stdin>:1): Cannot index number with string "hasErrors"
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 1307503, 'retcode': 1, 'stdout': 'Current log package version (2.3.0) is not the same as the latest available package (2.3.3)...\nUpgra                                                                                                ding import-evtx-logs...\nStarting dry run...\nErrors detected during dry run. Stopping upgrade...', 'stderr': 'jq: error (at <stdin>:1): Cannot index num                                                                                                ber with string "hasErrors"'}**
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-integration-upgrade] at time 12:41:21.113566 (duration_in_ms=671.012)
[INFO    ] Running state [/usr/sbin/so-elastic-fleet-optional-integrations-load] at time 12:41:21.113795
[INFO    ] Executing state cmd.run for [/usr/sbin/so-elastic-fleet-optional-integrations-load]
[INFO    ] Executing command '/usr/sbin/so-elastic-fleet-optional-integrations-load' in directory '/root'
[INFO    ] {'pid': 1307578, 'retcode': 0, 'stdout': "Executing command with retry support: curl -s -K /opt/so/conf/elasticsearch/curl.config --output /dev                                                                                                /null --silent --head --fail localhost:5601/api/fleet/epm/packages\nResults:  (0)\nbeaconing integration requires an Elastic license of platinum or greate                                                                                                r... skipping\ncloud_defend integration requires an Elastic license of enterprise or greater... skipping\nded integration requires an Elastic license of p                                                                                                latinum or greater... skipping\nlmd integration requires an Elastic license of platinum or greater... skipping\nproblemchild integration requires an Elast                                                                                                ic license of platinum or greater... skipping\nElastic integrations don't appear to need installation/updating...", 'stderr': ''}
[INFO    ] Completed state [/usr/sbin/so-elastic-fleet-optional-integrations-load] at time 12:41:32.041266 (duration_in_ms=10927.47)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 12:41:32.041486
[INFO    ] Executing state file.uncomment for [/opt/so/conf/so-status/so-status.conf]
[INFO    ] Pattern already uncommented
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 12:41:32.043273 (duration_in_ms=1.787)
local:

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Can you try upgrading the integration via the UI? You mentioned it wasn't possible to edit the custom logs integration, when you navigate to custom logs in the URL it likely says 2.3.0 can you try changing that to 2.3.3 and see if you have an option under 'settings' to upgrade the integration manually?

If possible can you post your /root/soup.log sanitized as needed? So it can be reviewed

[zenon8000]

so-elastic-fleet-integration-upgrade

Current log package version (2.3.0) is not the same as the latest available package (2.3.3)...
Upgrading import-evtx-logs...
Starting dry run...
jq: error (at :1): Cannot index number with string "hasErrors"
Errors detected during dry run. Stopping upgrade...

There have never been such and index logs-suricata-s4a in our system. I double checked this. Currently there are only these suricata related indexes:

[reyesj2]

Under index templates you don't have a logs-suricata-s4a? Try enabling all options

If you don't have that template I would be confused since previously you shared

If you try these instructions again does the custom logs integration get upgraded ?
#14398 (comment)

[zenon8000]

I enabled all options and now I finally found it: "logs-suricata-s4a" :)

I think that this is somehow related to Elastic Agent, because I had Elastic Agent named s4a in my environment. At the present moment the Elastic Agent named to s4a is removed, but I still see unenrolled agents in Fleet:

In our environment we used Elastic Agent to get third-party Suricata alerts (not logs) into the SecurityOnion. We used predefined "suricata-logs" integration under "so-grid-nodes_general" policy where we defined additional "Log file path" for eve.json and applied the policy to the Elastic-Agent to get the alerts into the SecurityOnion Alerts page.

[reyesj2]

Using third-party suricata alerts is probably fine so long as the formatting is the same as when output via Security Onion bundled Suricata. Otherwise you would need to create your own ingest pipelines to properly ingest the logs. What isn't going to be fine is having two index templates with conflicting index patterns & at the same priority (logs-suricata & logs-suricata-s4a).
My guess is whatever you're doing for creating the third party alerts is also creating an index template in ES as part of its setup process.

Try just increasing or lowering the set priority for logs-suricata-s4a. Then follow previous instructions to update logs integration

[zenon8000]

At first I lowered the priority of the logs-suricata-s4a index template, then I deleted the template and rebooted the SO, but I still receiving the same error:

Executing: so-elastic-fleet-integration-upgrade
Current log package version (2.3.0) is not the same as the latest available package (2.3.3)...
Upgrading import-evtx-logs...
Starting dry run...
jq: error (at :1): Cannot index number with string "hasErrors"
Errors detected during dry run. Stopping upgrade...

ALSO dry-run output:

cat /tmp/dryrun-output
{"statusCode":404,"error":"Not Found","message":"[email protected] not found"}

I can't figure out where logs-suricata-s4a is taken if the index template itself is deleted and Elastic agent is unenrolled

[zfcarsonb]

I am having this same issue, except the broken package is "AbuseCH". I am still searching for a solution; if I find one, I will reply back to this thread.

[zfcarsonb]

@zfcarsonb if you are having the same jq: error (at <stdin>:1): Cannot index number with string "hasErrors" error can you try the above aswell and post output? You would just need to sub out 'import-evtx-logs' for the name you gave the 'AbuseCH' integration in your fleet policy. Example if you kept the default abusech integration name when adding it to your policy

INTEGRATION_ID=$(elastic_fleet_integration_id "so-grid-nodes_general" "ti_abusech-1")

*If you created a custom agent policy for your abusech integration you won't use 'so-grid-nodes_general' policy name. You need the policy id. The easiest way to get this is via the UI. Screenshot shows an example where I add abusech to a custom policy and sub out the 'so-grid-nodes_general' for the policy id.

INTEGRATION_ID=$(elastic_fleet_integration_id "2283b935-3c52-4fa6-a489-c7aceaed97bf" "ti_abusech-1")

I tried logging in to the manager to apply this advice, however I ran into a roadblock: It appears that during the upgrade, the integration was deleted from the agent policy. That is:

• Prior to the upgrade, the agent policy logcollector had a ti_abusech-logcollector integration
• After running soup, the logcollector agent policy no longer has the integration. All other configured integrations are still present.

I also tried the original advice of going to the integration's URL and bumping the version string, however, this causes Kibana to spin indefinitely:

I also have the following error messages in the browser console:

[reyesj2]

What did you bump the version to? It should be 2.6.0

Try to upgrade it via the UI or reinstall if its an option

Can you share what the output of the following

so-status

docker logs so-kibana

[zfcarsonb]

Replacing "2.3.0" with "2.6.0" in the Kibana integration URL got me to the integration page where I successfully upgraded it. This resolved my issue, thanks for the help! I was erroneously trying version "2.3.3"

I have successfully re-added the integration to the logcollector agent policy, and can now search for AbuseCH threat intel from SOC.

My guess on what happened:

From the Kibana logs, it appears as if soup tried to upgrade the integration to v2.6.0, however - likely due to a one-off fluke - the
upgrade failed, so Kibana tried to roll back to v2.3.0. However, the previous version v2.3.0 was possibly either already removed,
or otherwise in an inconsistent state, causing the rollback to fail as well.