Kibana Error After Upgrade

[craigsmooth]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128 GB

Storage for /

5 TB

Storage for /nsm

10 TB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

After upgrading to version 2.4.100-20240903 kibana will not start. If I navigate to the kibana url, I get "Kibana server is not ready yet." Any ideas on what might be causing this and how to resolve? Thank you. Here is what I see in kibana.log:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-09-06T20:15:30.959+00:00","message":"[.kibana_analytics] Action failed with '[index_not_yellow_timeout] Timeout waiting for the status of the [.kibana_analytics_8.10.4_001] index to become 'yellow' Refer to https://www.elastic.co/guide/en/kibana/8.14/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 11 in 64 seconds.","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":7,"uptime":3708.421826568},"trace":{"id":"49f5ec245b57523dd373b32cde266843"},"transaction":{"id":"c33c11dd6d6e14cd"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-09-06T20:15:31.000+00:00","message":"[.kibana_analytics] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 364454ms.","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":7,"uptime":3708.422309281},"trace":{"id":"49f5ec245b57523dd373b32cde266843"},"transaction":{"id":"c33c11dd6d6e14cd"}}

I am seeing a few errors in salt related to kibana not being available.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Can you show the output of the following commands?

sudo so-status
sudo so-elasticsearch-query _cluster/health?pretty=true
sudo so-checkin

[craigsmooth]

Thanks @reyesj2 . Here are the results from my two unassigned shards. I'm guessing the kibana one is the reason my kibana instance is not running "404 page not found".

.ds-logs-zeek-so-2024.09.06-000003                                 0     p      UNASSIGNED
.kibana_analytics_8.10.4_001                                       0     p      UNASSIGNED

.ds-logs-zeek-so-2024.09.06-000003 :

{
  "index" : ".ds-logs-zeek-so-2024.09.06-000003",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2024-09-06T21:47:05.346Z",
    "failed_allocation_attempts" : 1,
    "details" : "failed shard on node [9-5ExZMARvWbUSXDwZyIyw]: shard failure, reason [merge failed], failure org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge\n\tat [email protected]/org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2867)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge\n\tat [email protected]/org.apache.lucene.index.IndexWriter.hasPendingMerges(IndexWriter.java:2433)\n\tat [email protected]/org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler.afterMerge(InternalEngine.java:2825)\n\tat [email protected]/org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:123)\n\tat [email protected]/org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:700)\nCaused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ff022664 actual=89a051cb (resource=BufferedChecksumIndexInput(MemorySegmentIndexInput(path=\"/usr/share/elasticsearch/data/indices/hQgubL61RpCtH92MbMcdcQ/0/index/_pe.cfs\") [slice=_pe_ES812Postings_0.tim]))\n\tat [email protected]/org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:441)\n\tat [email protected]/org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:620)\n\tat [email protected]/org.apache.lucene.codecs.lucene90.blocktree.Lucene90BlockTreeTermsReader.checkIntegrity(Lucene90BlockTreeTermsReader.java:335)\n\tat [email protected]/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:370)\n\tat [email protected]/org.apache.lucene.codecs.perfield.PerFieldMergeState$FilterFieldsProducer.checkIntegrity(PerFieldMergeState.java:296)\n\tat [email protected]/org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:83)\n\tat [email protected]/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:205)\n\tat [email protected]/org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:209)\n\tat [email protected]/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:298)\n\tat [email protected]/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)\n\tat [email protected]/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5252)\n\tat [email protected]/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4740)\n\tat [email protected]/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6541)\n\tat [email protected]/org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:639)\n\tat [email protected]/org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:118)\n\t... 1 more\n",
    "last_allocation_status" : "no_valid_s
      "node_name" : "so2mgr",
      "transport_address" : "172.30.1.102:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0"
      },
      "roles" : [
        "data",
        "data_hot",
        "ingest",
        "master",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "gm7cScxFQdm2mrdXmS08Lw",
        "store_exception" : {
          "type" : "corrupt_index_exception",
          "reason" : "failed engine (reason: [merge failed]) (resource=preexisting_corruption)",
          "caused_by" : {
            "type" : "i_o_exception",
            "reason" : "failed engine (reason: [merge failed])",
            "caused_by" : {
              "type" : "corrupt_index_exception",
              "reason" : "checksum failed (hardware problem?) : expected=ff022664 actual=89a051cb (resource=BufferedChecksumIndexInput(MemorySegmentIndexInput(path=\"/usr/share/elasticsearch/data/indices/hQgubL61RpCtH92MbMcdcQ/0/index/_pe.cfs\") [slice=_pe_ES812Postings_0.tim]))"
            }
          }
        }
      }
    }
  ]
}`

.kibana_analytics_8.10.4_001 : 


    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "Elasticsearch can't allocate this shard because all the copies of its data in the cluster are stale or corrupt. Elasticsearch will allocate this shard when a node containing a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.",
  "node_allocation_decisions" : [
    {
      "node_id" : "9-5ExZMARvWbUSXDwZyIyw",
      "node_name" : "so2mgr",
      "transport_address" : "172.30.1.102:9300",
      "node_attributes" : {
        "ml.config_version" : "12.0.0",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0"
      },
      "roles" : [
        "data",
        "data_hot",
        "ingest",
        "master",
        "remote_cluster_client",
        "transform"
      ],
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "PFQd_zTmQ7uYF0krM0bTuQ",
        "store_exception" : {
          "type" : "corrupt_index_exception",
          "reason" : "failed engine (reason: [merge failed]) (resource=preexisting_corruption)",
          "caused_by" : {
            "type" : "i_o_exception",
            "reason" : "failed engine (reason: [merge failed])",
            "caused_by" : {
              "type" : "corrupt_index_exception",
              "reason" : "checksum failed (hardware problem?) : expected=23283a85 actual=7bc04cd6 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path=\"/usr/share/elasticsearch/data/indices/PEfGQOHrSd-OdDuuPSrgOA/0/index/_1dr.fdt\")))"
            }
          }
        }
      }
    }
  ]
}

[reyesj2]

Was the server unexpectedly shutdown? Or having any hardware issues you know of?
Looks like there is some data corruption going on in those shards. Your best bet is to delete the corrupt shards

so-elasticsearch-query .kibana_analytics_8.10.4_001 -XDELETE
so-elasticsearch-query .ds-logs-zeek-so-2024.09.06-000003 -XDELETE

To help avoid this in the future you can add a searchnode to your distributed grid and then set the Elasticsearch replication factor to 1. That will tell Elasticsearch to keep an additional copy of your data, so it can attempt to recover from corruption

[craigsmooth]

Thanks. No hardware or unexpected shutdowns during the upgrade. I'm just starting to wonder if this cluster is just unrecoverable. Here are the results when I try to delete those problem indices.
sudo so-elasticsearch-query .ds-logs-zeek-so-2024.09.06-000003 -XDELETE

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"index [.ds-logs-zeek-so-2024.09.06-000003] is the write index for data stream [logs-zeek-so] and cannot be deleted"}],"type":"illegal_argument_exception","reason":"index [.ds-logs-zeek-so-2024.09.06-000003] is the write index for data stream [logs-zeek-so] and cannot be deleted"},"status":400}

sudo so-elasticsearch-query .kibana_analytics_8.10.4_001 -XDELETE8.10.4_001 -XDELETE

{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana_analytics_8.10.4_001], this action is granted by the index privileges [delete_index,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/delete] is unauthorized for user [so_elastic] with effective roles [superuser] on restricted indices [.kibana_analytics_8.10.4_001], this action is granted by the index privileges [delete_index,manage,all]"},"status":403}

[reyesj2]

Looks like the zeek index is the current write try rolling it over, closing and then deleting it

so-elasticsearch-query logs-zeek-so/_rollover -XPOST
so-elasticsearch-query .ds-logs-zeek-so-2024.09.06-000003/_close -XPOST
so-elasticsearch-query .ds-logs-zeek-so-2024.09.06-000003 -XDELETE

For the kibana one try

kpass=$(salt-call pillar.get elasticsearch:auth:users:so_kibana_user:pass --out=newline_values_only)
curl --user "so_kibana:$kpass" -s -k -L -H "Content-Type:application/json" "https://localhost:9200/.kibana_analytics_8.10.4_001" -XDELETE

Then check the cluster health again

so-elasticsearch-query _cluster/health?pretty=true

[craigsmooth]

That worked, @reyesj2 . Thank you!

[craigsmooth]

Actually, just FYI @reyesj2 , it looks like this action recovered kibana access however at the cost of:
-all dashboards are gone
-all integrations are gone
-looks like most of the kibana configuration is gone
Looks like this is going to be a rebuild. Not a huge deal since this install was in a lab environment. Just a lesson learned I guess. Have good backups and a replica.