Upgrading to 2.4.210 causing i_o_exception errors #15550

HaydenB0101 · 2026-03-03T20:47:34Z

HaydenB0101
Mar 3, 2026

Version

2.4.210

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

12

RAM

32

Storage for /

200

Storage for /nsm

400

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

After upgrading to 2.4.210 this morning, my dashboard is displaying a warning saying "i_o_exception: Invalid string; unexpected character: 255 hex: ff". It also displays a full error saying:

Error: An internal server error occurred
    at https://siem/kibana/a2a937354781/bundles/plugin/dataViews/1.0.0/dataViews.plugin.js:1:51306
    at async DataViewsServicePublic.refreshFieldSpecMap (https://siem/kibana/a2a937354781/bundles/plugin/dataViews/1.0.0/dataViews.plugin.js:1:31168)
    at async DataViewsServicePublic.initFromSavedObjectLoadFields (https://siem/kibana/a2a937354781/bundles/plugin/dataViews/1.0.0/dataViews.plugin.js:1:32865)
    at async DataViewsServicePublic.initFromSavedObject (https://siem/kibana/a2a937354781/bundles/plugin/dataViews/1.0.0/dataViews.plugin.js:1:33382)

This is resulting in no data being displayed, but data is being ingested in the SOC.

Running GET _cluster/health?pretty in the Dev console on my manager displays:

{
  "cluster_name": "securityonion",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 4,
  "number_of_data_nodes": 4,
  "active_primary_shards": 728,
  "active_shards": 928,
  "relocating_shards": 2,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "unassigned_primary_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

While running it on my webserver displays:

{
  "cluster_name": "elasticsearch",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 92,
  "active_shards": 92,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 8,
  "unassigned_primary_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 92
}

I don't know if this is a result of unassigned shards or not, and am not sure where to start looking.

Guidelines

I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

Answered by HaydenB0101

Mar 6, 2026

A solution I found was updating my webserver to ES/Kibana 9.1.10, as there was some release order issues referenced with 8.19.x to 9.0.x in this elastic discussion post.
I also had to update the keyring and update repository with these commands

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list

View full answer

reyesj2 · 2026-03-04T18:39:20Z

reyesj2
Mar 4, 2026
Maintainer

Try running sudo so-elasticsearch-troubleshoot and sharing the output here.

Did you have any custom Data Views in Kibana?

What does docker images | grep -E "^$HOSTNAME.*(2\.4\.210|9\.0\.8)" output?

Might also be worth running so-docker-refresh and restarting Kibana if this is a network installation, should check to make sure that the correct docker container was downloaded and is being used for services.

4 replies

HaydenB0101 Mar 4, 2026
Author

Try running sudo so-elasticsearch-troubleshoot and sharing the output here.

================ Elasticsearch Status ================
 Ilm: An index has stayed on the same action longer than expected.
  Cause: Some indices have been stagnated on the action [rollover] longer than the expected time.

  Action: Check the current status of the Index Lifecycle Management for every affected index using the [GET /<affected_index_name>/_ilm/explain] API. Please replace the <affected_index_name> in the API with the actual index name.
  Affected indices:
    - logs-soc-so-clone-001

================ Disk Usage Check ================
LOW:75% HIGH:80% FLOOD:95%

 node1 disk usage: 20.06%
 manager disk usage: 32.45%
 node2 disk usage: 22.51%
 node3 disk usage: 65.79%

================ Unassigned Shards Check ================
 All shards are assigned

Did you have any custom Data Views in Kibana?
the only views I have are siem:* siem:logs-* and siem:logs-panw*

What does docker images | grep -E "^$HOSTNAME.*(2\.4\.210|9\.0\.8)" output?

ghcr.io/security-onion-solutions/so-elastalert:2.4.210                                04a021ea01f0        528MB             0B   U
ghcr.io/security-onion-solutions/so-elastic-agent-builder:2.4.210                     6f1ad0955ee5        235MB             0B
ghcr.io/security-onion-solutions/so-elastic-agent:2.4.210                             18218d79a33b       3.35GB             0B   U
ghcr.io/security-onion-solutions/so-elastic-fleet-package-registry:2.4.210            f0c5a8cee99e       1.95GB             0B   U
ghcr.io/security-onion-solutions/so-elasticsearch:9.0.8                               02664a9e3809       1.55GB             0B   U
ghcr.io/security-onion-solutions/so-hydra:2.4.210                                     5f8eb4b3e3fc        234MB             0B
ghcr.io/security-onion-solutions/so-idh:2.4.210                                       bd67b0f0c21c        265MB             0B
ghcr.io/security-onion-solutions/so-influxdb:2.4.210                                  912c5317991b        476MB             0B   U
ghcr.io/security-onion-solutions/so-kafka:2.4.210                                     ff0f55dd7762        499MB             0B
ghcr.io/security-onion-solutions/so-kibana:2.4.210                                    3bed2d9f7b0d       2.39GB             0B
ghcr.io/security-onion-solutions/so-kratos:2.4.210                                    ca5c50234653        251MB             0B   U
ghcr.io/security-onion-solutions/so-logstash:2.4.210                                  c1479acb3dac       1.62GB             0B   U
ghcr.io/security-onion-solutions/so-nginx:2.4.210                                     3fb51ac25ef1        183MB             0B   U
ghcr.io/security-onion-solutions/so-pcaptools:2.4.210                                 24a2dba8a050        244MB             0B
ghcr.io/security-onion-solutions/so-redis:2.4.210                                     d67abda8b1f0       40.9MB             0B   U
ghcr.io/security-onion-solutions/so-soc:2.4.210                                       5820ff399e2f        726MB             0B   U
ghcr.io/security-onion-solutions/so-steno:2.4.210                                     c2141c4ed1d5        826MB             0B
ghcr.io/security-onion-solutions/so-strelka-backend:2.4.210                           8e5ff904e042       2.19GB             0B
ghcr.io/security-onion-solutions/so-strelka-manager:2.4.210                           bbea20dfc7f2       81.3MB             0B
ghcr.io/security-onion-solutions/so-suricata:2.4.210                                  aa9f02893a46        524MB             0B
ghcr.io/security-onion-solutions/so-telegraf:2.4.210                                  607bc46b61c5        328MB             0B   U
ghcr.io/security-onion-solutions/so-zeek:2.4.210                                      537f56df66b2       1.03GB             0B
siemmanager:5000/security-onion-solutions/so-elastalert:2.4.210                       04a021ea01f0        528MB             0B   U
siemmanager:5000/security-onion-solutions/so-elastic-agent-builder:2.4.210            6f1ad0955ee5        235MB             0B
siemmanager:5000/security-onion-solutions/so-elastic-agent:2.4.210                    18218d79a33b       3.35GB             0B   U
siemmanager:5000/security-onion-solutions/so-elastic-fleet-package-registry:2.4.210   f0c5a8cee99e       1.95GB             0B   U
siemmanager:5000/security-onion-solutions/so-elasticsearch:9.0.8                      02664a9e3809       1.55GB             0B   U
siemmanager:5000/security-onion-solutions/so-hydra:2.4.210                            5f8eb4b3e3fc        234MB             0B
siemmanager:5000/security-onion-solutions/so-idh:2.4.210                              bd67b0f0c21c        265MB             0B
siemmanager:5000/security-onion-solutions/so-influxdb:2.4.210                         912c5317991b        476MB             0B   U
siemmanager:5000/security-onion-solutions/so-kafka:2.4.210                            ff0f55dd7762        499MB             0B
siemmanager:5000/security-onion-solutions/so-kibana:2.4.210                           3bed2d9f7b0d       2.39GB             0B
siemmanager:5000/security-onion-solutions/so-kratos:2.4.210                           ca5c50234653        251MB             0B   U
siemmanager:5000/security-onion-solutions/so-logstash:2.4.210                         c1479acb3dac       1.62GB             0B   U
siemmanager:5000/security-onion-solutions/so-nginx:2.4.210                            3fb51ac25ef1        183MB             0B   U
siemmanager:5000/security-onion-solutions/so-pcaptools:2.4.210                        24a2dba8a050        244MB             0B
siemmanager:5000/security-onion-solutions/so-redis:2.4.210                            d67abda8b1f0       40.9MB             0B   U
siemmanager:5000/security-onion-solutions/so-soc:2.4.210                              5820ff399e2f        726MB             0B   U
siemmanager:5000/security-onion-solutions/so-steno:2.4.210                            c2141c4ed1d5        826MB             0B
siemmanager:5000/security-onion-solutions/so-strelka-backend:2.4.210                  8e5ff904e042       2.19GB             0B
siemmanager:5000/security-onion-solutions/so-strelka-manager:2.4.210                  bbea20dfc7f2       81.3MB             0B
siemmanager:5000/security-onion-solutions/so-suricata:2.4.210                         aa9f02893a46        524MB             0B
siemmanager:5000/security-onion-solutions/so-telegraf:2.4.210                         607bc46b61c5        328MB             0B   U
siemmanager:5000/security-onion-solutions/so-zeek:2.4.210                             537f56df66b2       1.03GB             0B

HaydenB0101 Mar 5, 2026
Author

One thing I have seen is my webserver didn't update elasticsearch to 9.x, only staying on 8.19.12. running updates from 4.9.2-0ubuntu0~22.04.1 to 4.10.2-0ubuntu0~22.04.1 broke access to the server from the web but allowed me to download elasticsearch 9.x.
Before the update I updated the repository file to https://artifacts.elastic.co/packages/9.x/apt stable main which made the update choose to update KIbana to 9.3.1, which when online says is incompatible with the webserver as an elastic node.
Forcing elastic to update to 9.0.8 returns a fatal exception while booting Elasticsearch

[2026-03-05T16:35:49,721][ERROR][o.e.b.Elasticsearch ] [siem] fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchException: failed to load metadata at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:118)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.node.Node.start(Node.java:315
 ~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:649)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:446)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:102)
~[elasticsearch-9.0.8.jar:?] Caused by: org.elasticsearch.gateway.CorruptStateException: org.elasticsearch.xcontent.XContentParseException: [-1:15374] [data_stream] failed to parse field [data_stream] at org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:679)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$6(PersistedClusterStateService.java:580)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:722)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:579)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:503)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:414)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.GatewayMetaState.createOnDiskPersistedState(GatewayMetaState.java:179)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.GatewayMetaState.createPersistedState(GatewayMetaState.java:147)
~[elasticsearch-9.0.8.jar:?] at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:105)
~[elasticsearch-9.0.8.jar:?] ... 4 more Caused by: org.elasticsearch.xcontent.XContentParseException: [-1:15374] [data_stream] failed to parse field [data_stream]

reyesj2 Mar 6, 2026
Maintainer

I am a bit confused as you're referencing Elastic versions that Security Onion hasn't used, like 8.19.12 and 9.3.1 ?

What repository file are you manually modifying and what was the reasoning?

Forcing elastic to update to 9.0.8

Security Onion 2.4.210 uses 9.0.8, just running sudo soup would have upgraded to this version, where/how did you 'force' an update to 9.0.8?

HaydenB0101 Mar 6, 2026
Author

This work from the One thing I have seen reply has been on my webserver that I use to display Data. the server is not within the security onion cluster and is running ubuntu. It's using Kibana to display the data, and to my knowledge needs Elasticsearch to retrieve the data from the SO cluster. All of my SO nodes are running 9.0.8 and I only ran soup, no forcing updates.

HaydenB0101 · 2026-03-06T20:08:55Z

HaydenB0101
Mar 6, 2026
Author

A solution I found was updating my webserver to ES/Kibana 9.1.10, as there was some release order issues referenced with 8.19.x to 9.0.x in this elastic discussion post.
I also had to update the keyring and update repository with these commands

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading to 2.4.210 causing i_o_exception errors #15550

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Upgrading to 2.4.210 causing i_o_exception errors #15550

Uh oh!

Uh oh!

HaydenB0101 Mar 3, 2026

Version

Installation Method

Description

Installation Type

Location

Hardware Specs

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

Network Traffic Speeds

Status

Salt Status

Logs

Detail

Guidelines

Replies: 2 comments · 4 replies

Uh oh!

reyesj2 Mar 4, 2026 Maintainer

Uh oh!

HaydenB0101 Mar 4, 2026 Author

Uh oh!

Uh oh!

HaydenB0101 Mar 5, 2026 Author

Uh oh!

reyesj2 Mar 6, 2026 Maintainer

Uh oh!

HaydenB0101 Mar 6, 2026 Author

Uh oh!

HaydenB0101 Mar 6, 2026 Author

HaydenB0101
Mar 3, 2026

Replies: 2 comments 4 replies

reyesj2
Mar 4, 2026
Maintainer

HaydenB0101 Mar 4, 2026
Author

HaydenB0101 Mar 5, 2026
Author

reyesj2 Mar 6, 2026
Maintainer

HaydenB0101 Mar 6, 2026
Author

HaydenB0101
Mar 6, 2026
Author