Sensor dont send events after reboot #12475

tuamortemo · 2024-03-01T11:38:33Z

tuamortemo
Mar 1, 2024

Version

2.4.50

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

other (please provide detail below)

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

16

Storage for /

200

Storage for /nsm

800

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hello,
I have installed the new version SO 2.4.50. It is installed from 3 servers - manager, search and sensor. After a fresh installation, a message appears in the grid menu telling me to restart nodes. Before restarting, the sensor sends zeek and suricata data. After reboot, all statuses go to the ok state, but no data is received from the modules, only system.syslog.
I tried to reinstall the sensor, but after the reboot the situation does not change

Guidelines

I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

tuamortemo · 2024-03-01T12:23:43Z

tuamortemo
Mar 1, 2024
Author

2 hours after the reboot, the sensor started sending events. What could be the reason for such a delay, I did not observe any problems according to the logs from /opt/so/log

6 replies

tuamortemo Mar 6, 2024
Author

[2024-03-04T10:00:11,808][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-03-04T10:00:11,809][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@manager:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}

cm-ops Mar 7, 2024
Maintainer

Also check it on your search node, looks like Redis is getting backed up.

tuamortemo Mar 12, 2024
Author

Hello,
Sorry for the long absence of feedback

On search

[2024-03-12T10:37:54,213][INFO ][logstash.javapipeline ] Pipeline search is configured with pipeline.ecs_compatibility: disabled setting. All plugins in this pipeline will default to ecs_compatibility => disabled unless explicitly configured otherwise.
[2024-03-12T10:37:54,224][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//search-01"]}
[2024-03-12T10:37:54,228][WARN ][logstash.outputs.elasticsearch] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set ssl_verification_mode => full
[2024-03-12T10:37:54,373][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://so_elastic:xxxxxx@search-01:9200/]}}
[2024-03-12T10:37:54,434][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:37:54,436][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}

after several such messages

[2024-03-12T10:37:54,473][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://so_elastic:xxxxxx@manager:9200/]}}
[2024-03-12T10:37:57,567][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"No route to host", :exception=>Manticore::SocketException, :cause=>#<Java::JavaNet::NoRouteToHostException: No route to host>}
[2024-03-12T10:37:57,569][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@manager:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://manager:9200/][Manticore::SocketException] No route to host"}
[2024-03-12T10:37:57,569][INFO ][logstash.outputs.elasticsearch] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While ecs_compatibility can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an ecs_compatibility mode, and the pipeline.ecs_compatibility setting can be used to opt-in for all plugins in a pipeline.
[2024-03-12T10:37:57,570][INFO ][logstash.outputs.elasticsearch] Data streams auto configuration (data_stream => auto or unset) resolved to false
[2024-03-12T10:37:57,585][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"search", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipelines/search/0900_input_redis.conf", "/usr/share/logstash/pipelines/search/9805_output_elastic_agent.conf", "/usr/share/logstash/pipelines/search/9900_output_endgame.conf"], :thread=>"#<Thread:0x3b803c63 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

[2024-03-12T10:37:58,887][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@manager:9696/0 list:logstash:unparsed"}
[2024-03-12T10:37:58,887][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@manager:9696/0 list:logstash:unparsed"}
[2024-03-12T10:37:58,887][INFO ][logstash.inputs.redis ] Registering Redis {:identity=>"redis://@manager:9696/0 list:logstash:unparsed"}
[2024-03-12T10:37:58,899][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"search"}
[2024-03-12T10:37:58,925][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:search], :non_running_pipelines=>[]}
[2024-03-12T10:37:59,447][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:37:59,449][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}
[2024-03-12T10:37:59,458][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:37:59,470][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}
[2024-03-12T10:37:59,482][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:37:59,484][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}
[2024-03-12T10:38:03,040][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Errno::ECONNREFUSED)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:38:03,040][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Errno::ECONNREFUSED)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:38:03,040][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Errno::ECONNREFUSED)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:38:03,040][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Errno::ECONNREFUSED)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:38:03,618][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to manager:9200 [manager/10.99.1.10] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to manager:9200 [manager/10.99.1.10] failed: Connection refused>}
[2024-03-12T10:38:03,621][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@manager:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://manager:9200/][Manticore::SocketException] Connect to manager:9200 [manager/10.99.1.10] failed: Connection refused"}
[2024-03-12T10:38:03,916][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Errno::ECONNREFUSED)", :exception=>Redis::CannotConnectError}

later only timeout or con refused on search or manager like this

[2024-03-12T10:41:06,285][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:41:06,287][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:41:09,743][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:41:09,743][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:41:09,744][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}
[2024-03-12T10:41:09,744][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}
[2024-03-12T10:41:09,754][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused>}
[2024-03-12T10:41:09,754][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@search-01:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://search-01:9200/][Manticore::SocketException] Connect to search-01:9200 [search-01/10.99.1.15] failed: Connection refused"}
[2024-03-12T10:41:11,578][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on manager:9696 (Errno::EHOSTUNREACH)", :exception=>Redis::CannotConnectError}
[2024-03-12T10:41:13,690][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"No route to host", :exception=>Manticore::SocketException, :cause=>#<Java::JavaNet::NoRouteToHostException: No route to host>}

cm-ops Mar 13, 2024
Maintainer

Looks like there are some issues connecting to Redis on the manager and connecting to Elasticsearch on search-01. Can you check your search node Elasticsearch log in /opt/so/log/elasticsearch for any errors as well?

tuamortemo Mar 13, 2024
Author

[2024-03-12T10:52:11,518][INFO ][org.apache.lucene.util.VectorUtilPanamaProvider] Java vector incubator API enabled; uses preferredBitSize=256
[2024-03-12T10:52:12,294][INFO ][org.elasticsearch.node.Node] version[8.10.4], pid[117], build[docker/b4a62ac808e886ff032700c391f45f1408b2538c/2023-10-11T22:04:35.506990650Z], OS[Linux/5.15.0-203.146.5.1.el9uek.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21/21+35-2513]
[2024-03-12T10:52:12,294][INFO ][org.elasticsearch.node.Node] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2024-03-12T10:52:12,294][INFO ][org.elasticsearch.node.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-16229454662554675157, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms5310m, -Xmx5310m, -Des.transport.cname_in_publish_address=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=2785017856, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=org.elasticsearch.preallocate, -Djdk.module.main=org.elasticsearch.server]
[2024-03-12T10:52:13,626][WARN ][com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader] Unable to load config file null
java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/elasticsearch/.aws/config" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]

[2024-03-12T10:52:29,696][INFO ][org.elasticsearch.node.Node] initialized
[2024-03-12T10:52:29,697][INFO ][org.elasticsearch.node.Node] starting ...
[2024-03-12T10:52:29,727][INFO ][org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache] persistent cache index loaded
[2024-03-12T10:52:29,728][INFO ][org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent] deprecation component started
[2024-03-12T10:52:29,827][INFO ][org.elasticsearch.transport.TransportService] publish_address {search-01/10.99.1.15:9300}, bound_addresses {0.0.0.0:9300}
[2024-03-12T10:52:32,211][INFO ][org.elasticsearch.bootstrap.BootstrapChecks] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-03-12T10:52:32,236][INFO ][org.elasticsearch.cluster.coordination.ClusterBootstrapService] this node is locked into cluster UUID [VX37XBqtRfmSLZbPGS33ig] and will not attempt further cluster bootstrapping
[2024-03-12T10:52:33,618][INFO ][org.elasticsearch.cluster.service.ClusterApplierService] master node changed {previous [], current [{manager}{KUtICoU9T-6TkNu_c3l3Hg}{EaArXnLsS82-iNMwO9OBCw}{manager}{manager}{10.99.1.10:9300}{dmrt}{8.10.4}{7000099-8100499}]}, added {{manager}{KUtICoU9T-6TkNu_c3l3Hg}{EaArXnLsS82-iNMwO9OBCw}{manager}{manager}{10.99.1.10:9300}{dmrt}{8.10.4}{7000099-8100499}}, term: 10, version: 3876, reason: ApplyCommitRequest{term=10, version=3876, sourceNode={manager}{KUtICoU9T-6TkNu_c3l3Hg}{EaArXnLsS82-iNMwO9OBCw}{manager}{manager}{10.99.1.10:9300}{dmrt}{8.10.4}{7000099-8100499}{xpack.installed=true, transform.config_version=10.0.0}}
[2024-03-12T10:52:33,629][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.seconi-manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,629][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,632][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.seconi-manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,632][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,633][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.seconi-manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,633][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,635][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,635][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.seconi-manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,636][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.seconi-manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:33,636][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.manager.seeds] from [[]] to [["127.0.0.1:9300"]]
[2024-03-12T10:52:36,880][WARN ][org.elasticsearch.ingest.common.GrokProcessor] nested repeat operator '+' and '' was replaced with '' in regular expression /^((?:<(?NONNEGINT:log.syslog.priority:long\b(?:[0-9]+)\b)>(\d )?))?(?:(?:(?:(?SYSLOGTIMESTAMP:_tmp.timestamp(?:\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|Mm?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y|i)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|Oo?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b) +(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?!<[0-9])(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9])))((?:\s*)(?:(?:\b(?NAME:process.name[[[:alnum:]]-]+)|((?NAME:process.name[[[:alnum:]]-]+))))|(?:\s*)(?:(?:(?IP:observer.ip(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9]))))|(?HOSTNAME:observer.name\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))))(?:\s)(?:(?:\b(?NAME:process.name[[[:alnum:]]-]+)|((?NAME:process.name[[[:alnum:]]-]+)))))([(?POSINT:process.pid:long\b(?:[1-9][0-9])\b)])?:)|(?:(?<TIMESTAMP_ISO8601:_tmp.timestamp8601>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?<ISO8601_TIMEZONE:event.timezone>(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?)(?:\s)(?:(?:(?IP:observer.ip(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])...)(?![0-9]))))|(?HOSTNAME:observer.name\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b))))(?:\s)(?:(((?DATA:process.name.?))|(?:(?:(/([\w_%!$@:.,+~-]+|\.))+)/)?(?BASEPATH:process.name[[[:alnum:]]_%!$@:.,+~-]+)))(?:\s)((?POSINT:process.pid:long\b(?:[1-9][0-9])\b)|-) - (-|(?:[[^\]]]))))) (?GREEDYDATA:message.*)/
[2024-03-12T10:52:36,982][WARN ][org.elasticsearch.ingest.common.GrokProcessor] character class has '-' without escape
[2024-03-12T10:52:36,982][WARN ][org.elasticsearch.ingest.common.GrokProcessor] character class has '-' without escape
[2024-03-12T10:52:36,982][WARN ][org.elasticsearch.ingest.common.GrokProcessor] character class has '-' without escape

but this did not happen for more than 2 minutes.
Now I went into Elastic Fleet and saw a lot of errors of next kind. And degraded agent status of sensor

[elastic_agent][error] Unit state changed log-default-logfile-logs-9cacb7c8-c04e-4ca4-be26-a9784360b29f (STARTING->FAILED): Failed: pid '695069' missed 3 check-ins and will be killed
20:29:55.623
elastic_agent
[elastic_agent][error] Unit state changed log-default-logfile-logs-f552ffcc-1400-455c-add7-b952fc5a01ed (STARTING->FAILED): Failed: pid '695069' missed 3 check-ins and will be killed
20:29:55.623
elastic_agent
[elastic_agent][error] Unit state changed log-default-logfile-logs-92071e16-a4a7-4a21-a55f-6b775191b5a1 (STARTING->FAILED): Failed: pid '695069' missed 3 check-ins and will be killed
20:29:55.623
elastic_agent
[elastic_agent][error] Unit state changed log-default-logfile-logs-22dd36c4-a0a7-48e7-a5ed-62f214bc3702 (STARTING->FAILED): Failed: pid '695069' missed 3 check-ins and will be killed
20:29:55.623
elastic_agent
[elastic_agent][error] Unit state changed log-default-logfile-elasticsearch-009644b9-0db9-4657-90ee-a07d9534c68d (STARTING->FAILED): Failed: pid '695069' missed 3 check-ins and will be killed
20:29:55.623
elastic_agent
[elastic_agent][error] Unit state changed log-default-logfile-logs-zeek-logs (STARTING->FAILED): Failed: pid '695069' missed 3 check-ins and will be killed

cm-ops · 2024-03-14T17:45:39Z

cm-ops
Mar 14, 2024
Maintainer

Would you let me know what the following returns? Run this on your manager and search node - sudo so-elasticsearch-query /

7 replies

tuamortemo Mar 20, 2024
Author

[adminso@manager ~]$ sudo salt * elastic-agent status
sensor-##_sensor:
'elastic-agent' is not available.
search-01_searchnode:
'elastic-agent' is not available.
sensor-##_sensor:
'elastic-agent' is not available.
search-02_searchnode:
'elastic-agent' is not available.
manager_manager:
'elastic-agent' is not available.
ERROR: Minions returned with non-zero exit code

cm-ops Mar 20, 2024
Maintainer

Okay, forgot to add one thing in my command, sudo salt \* cmd.run "elastic-agent status"

tuamortemo Mar 21, 2024
Author

30 mins after reboot
Wanna try to reinstall on 2.4.60, I hope there will be changes. The update did not solve the problem

[root@manager adminso]# sudo salt * cmd.run "elastic-agent status"
sensor-##_sensor:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a degraded state
├─ filestream-monitoring
│ ├─ status: (DEGRADED) Degraded: pid '26809' missed 2 check-ins
│ ├─ filestream-monitoring
│ │ └─ status: (STARTING) Starting: spawned pid '26809'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting: spawned pid '26809'
├─ log-default
│ ├─ status: (DEGRADED) Degraded: pid '26701' missed 2 check-ins
│ ├─ log-default
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-elasticsearch-009644b9-0db9-4657-90ee-a07d9534c68d
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-22dd36c4-a0a7-48e7-a5ed-62f214bc3702
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-4d093fd7-686a-4d81-b211-ebe1e6517294
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-92071e16-a4a7-4a21-a55f-6b775191b5a1
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-9771fe8d-e213-46c8-b199-7898fd9b53f9
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-9cacb7c8-c04e-4ca4-be26-a9784360b29f
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-9f5d2ae2-15f2-459a-bb33-6c126bc00965
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-b5c46489-9816-4604-9a79-c49ac66be210
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-d18a3cb6-39ac-4de0-b4f7-85b545fa19a5
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-d613a48e-d3cf-485d-8fa1-d66d9cfd5e08
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-ebec0140-20ae-4981-8dac-6ce5d27912e7
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-f552ffcc-1400-455c-add7-b952fc5a01ed
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-faa60692-95de-4caf-8bcf-bcd6fa498a12
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-logs-zeek-logs
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ ├─ log-default-logfile-redis-8fb22280-c729-4e1f-908f-ff980356665c
│ │ └─ status: (STARTING) Starting: spawned pid '26701'
│ └─ log-default-logfile-system-502035c9-92f7-4e23-8836-8eab6f9324a9
│ └─ status: (STARTING) Starting: spawned pid '26701'
├─ osquery-default
│ ├─ status: (DEGRADED) Degraded: pid '26772' missed 2 check-ins
│ ├─ osquery-default
│ │ └─ status: (STARTING) Starting: spawned pid '26772'
│ └─ osquery-default-97fb8d09-24dd-4b5f-a087-e31763b8ea1c
│ └─ status: (STARTING) Starting: spawned pid '26772'
├─ tcp-default
│ ├─ status: (DEGRADED) Degraded: pid '26729' missed 2 check-ins
│ ├─ tcp-default
│ │ └─ status: (STARTING) Starting: spawned pid '26729'
│ └─ tcp-default-tcp-tcp-51b0ab68-b9ac-4387-bc2d-91c7c676dd9e
│ └─ status: (STARTING) Starting: spawned pid '26729'
└─ udp-default
├─ status: (DEGRADED) Degraded: pid '26754' missed 2 check-ins
├─ udp-default
│ └─ status: (STARTING) Starting: spawned pid '26754'
└─ udp-default-udp-udp-74b20225-3425-412e-9f05-16b0c42ca00f
└─ status: (STARTING) Starting: spawned pid '26754'
sensor-##_sensor:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a degraded state
├─ filestream-monitoring
│ ├─ status: (DEGRADED) Degraded: pid '106393' missed 1 check-in
│ ├─ filestream-monitoring
│ │ └─ status: (STARTING) Starting: spawned pid '106393'
│ └─ filestream-monitoring-filestream-monitoring-agent
│ └─ status: (STARTING) Starting: spawned pid '106393'
├─ log-default
│ ├─ status: (DEGRADED) Degraded: pid '106359' missed 1 check-in
│ ├─ log-default
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-elasticsearch-009644b9-0db9-4657-90ee-a07d9534c68d
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-22dd36c4-a0a7-48e7-a5ed-62f214bc3702
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-4d093fd7-686a-4d81-b211-ebe1e6517294
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-92071e16-a4a7-4a21-a55f-6b775191b5a1
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-9771fe8d-e213-46c8-b199-7898fd9b53f9
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-9cacb7c8-c04e-4ca4-be26-a9784360b29f
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-9f5d2ae2-15f2-459a-bb33-6c126bc00965
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-b5c46489-9816-4604-9a79-c49ac66be210
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-d18a3cb6-39ac-4de0-b4f7-85b545fa19a5
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-d613a48e-d3cf-485d-8fa1-d66d9cfd5e08
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-ebec0140-20ae-4981-8dac-6ce5d27912e7
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-f552ffcc-1400-455c-add7-b952fc5a01ed
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-faa60692-95de-4caf-8bcf-bcd6fa498a12
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-logs-zeek-logs
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ ├─ log-default-logfile-redis-8fb22280-c729-4e1f-908f-ff980356665c
│ │ └─ status: (STARTING) Starting: spawned pid '106359'
│ └─ log-default-logfile-system-502035c9-92f7-4e23-8836-8eab6f9324a9
│ └─ status: (STARTING) Starting: spawned pid '106359'
├─ osquery-default
│ ├─ status: (DEGRADED) Degraded: pid '106316' missed 1 check-in
│ ├─ osquery-default
│ │ └─ status: (STARTING) Starting: spawned pid '106316'
│ └─ osquery-default-97fb8d09-24dd-4b5f-a087-e31763b8ea1c
│ └─ status: (STARTING) Starting: spawned pid '106316'
├─ tcp-default
│ ├─ status: (DEGRADED) Degraded: pid '106278' missed 1 check-in
│ ├─ tcp-default
│ │ └─ status: (STARTING) Starting: spawned pid '106278'
│ └─ tcp-default-tcp-tcp-51b0ab68-b9ac-4387-bc2d-91c7c676dd9e
│ └─ status: (STARTING) Starting: spawned pid '106278'
└─ udp-default
├─ status: (DEGRADED) Degraded: pid '106277' missed 1 check-in
├─ udp-default
│ └─ status: (STARTING) Starting: spawned pid '106277'
└─ udp-default-udp-udp-74b20225-3425-412e-9f05-16b0c42ca00f
└─ status: (STARTING) Starting: spawned pid '106277'
search-01_searchnode:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
└─ status: (HEALTHY) Running
manager_manager:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
└─ status: (HEALTHY) Running
search-02_searchnode:
┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
└─ status: (HEALTHY) Running

cm-ops Mar 22, 2024
Maintainer

Have you tried to stop/start/restart the Elastic Agent on the sensors? If so, do they still show degraded?

tuamortemo Mar 26, 2024
Author

I tried to deploy the same configuration of 3 servers (manager, search and sensor) on 2.4.60, the problem remained at first glance. Next, I tried restarting the elastic agent and it looks like it helped, thank you!

Sensor dont send events after reboot #12475

Uh oh!

tuamortemo Mar 1, 2024

Version

Installation Method

Description

Installation Type

Location

Hardware Specs

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

Network Traffic Speeds

Status

Salt Status

Logs

Detail

Guidelines

Replies: 2 comments · 13 replies

Uh oh!

tuamortemo Mar 1, 2024 Author

Uh oh!

tuamortemo Mar 6, 2024 Author

Uh oh!

cm-ops Mar 7, 2024 Maintainer

Uh oh!

Uh oh!

tuamortemo Mar 12, 2024 Author

Uh oh!

cm-ops Mar 13, 2024 Maintainer

Uh oh!

Uh oh!

tuamortemo Mar 13, 2024 Author

Uh oh!

cm-ops Mar 14, 2024 Maintainer

Uh oh!

tuamortemo Mar 20, 2024 Author

Uh oh!

cm-ops Mar 20, 2024 Maintainer

Uh oh!

tuamortemo Mar 21, 2024 Author

Uh oh!

cm-ops Mar 22, 2024 Maintainer

Uh oh!

tuamortemo Mar 26, 2024 Author

tuamortemo
Mar 1, 2024

Replies: 2 comments 13 replies

tuamortemo
Mar 1, 2024
Author

tuamortemo Mar 6, 2024
Author

cm-ops Mar 7, 2024
Maintainer

tuamortemo Mar 12, 2024
Author

cm-ops Mar 13, 2024
Maintainer

tuamortemo Mar 13, 2024
Author

cm-ops
Mar 14, 2024
Maintainer

tuamortemo Mar 20, 2024
Author

cm-ops Mar 20, 2024
Maintainer

tuamortemo Mar 21, 2024
Author

cm-ops Mar 22, 2024
Maintainer

tuamortemo Mar 26, 2024
Author