Kibana 404 page not found after update to 2.4.100

[cklfe]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32 GB

Storage for /

130 GB

Storage for /nsm

270 GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

After upgrade to 2.4.100 link to open Kibana is showing "404 page not found" error despite showing healthy status in Grid. Possibly related to the other discussion about Kibana but I don't have any unassigned Elastic shards so not certain.

"sudo so-status" shows all healthy.

"sudo salt-call state.highstate" produces this error:

local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 903158 and was started at 2024, Sep 12 16:07:03.378788 with jid 20240912160703378788

"sudo so-checkin" shows 4 failures:

----------
          ID: so-elastic-fleet-auto-configure-logstash-outputs
    Function: cmd.run
        Name: /usr/sbin/so-elastic-fleet-outputs-update
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
              Attempt 3: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-outputs-update" run"
              Command "/usr/sbin/so-elastic-fleet-outputs-update" run
     Started: 14:13:44.352171
    Duration: 90101.50200000001 ms
     Changes:   
              ----------
              pid:
                  379990
              retcode:
                  1
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Failed to query for current Logstash Outputs...
----------
          ID: so-elastic-fleet-auto-configure-server-urls
    Function: cmd.run
        Name: /usr/sbin/so-elastic-fleet-urls-update
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-urls-update" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-urls-update" run"
              Attempt 3: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-urls-update" run"
              Command "/usr/sbin/so-elastic-fleet-urls-update" run
     Started: 14:15:14.525277
    Duration: 90106.533 ms
     Changes:   
              ----------
              pid:
                  382204
              retcode:
                  1
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (52) Empty reply from server
              stdout:
                  Failed to query for current Fleet Server URLs...
----------
          ID: so-elastic-fleet-auto-configure-elasticsearch-urls
    Function: cmd.run
        Name: /usr/sbin/so-elastic-fleet-es-url-update
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
              Attempt 3: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-fleet-es-url-update" run"
              Command "/usr/sbin/so-elastic-fleet-es-url-update" run
     Started: 14:16:44.684622
    Duration: 90112.26000000001 ms
     Changes:   
              ----------
              pid:
                  385826
              retcode:
                  1
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Failed to query for current Fleet Server Elasticsearch URLs...
				  
				  
----------
          ID: so-elastic-agent-grid-upgrade
    Function: cmd.run
        Name: /usr/sbin/so-elastic-agent-grid-upgrade
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "/usr/sbin/so-elastic-agent-grid-upgrade" run"
              Command "/usr/sbin/so-elastic-agent-grid-upgrade" run
     Started: 14:18:16.248475
    Duration: 30087.73 ms
     Changes:   
              ----------
              pid:
                  386251
              retcode:
                  56
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  curl: (52) Empty reply from server
                  /usr/sbin/so-elastic-agent-grid-upgrade: line 20: [: : integer expression expected
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  100     4    0     0  100     4      0   2000 --:--:-- --:--:-- --:--:--  2000
                  curl: (56) Recv failure: Connection reset by peer
              stdout:
                  Initiating upgrades for  Agents to Elastic 8.14.3...

Kibana log contains these errors:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-09-12T14:54:33.097+00:00","message":"Error trying to transform document: Cannot read properties of undefined (reading 'value')","log":{"level":"ERROR","logger":"savedobjects-service"},"process":{"pid":7,"uptime":16.260642551},"trace":{"id":"49ebb0134d38ca875d7422d72d8ac62d"},"transaction":{"id":"5b623c8d93c4a664"}}

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.11.0"},"@timestamp":"2024-09-12T14:54:33.200+00:00","message":"Reason: Unable to complete saved object migrations for the [.kibana_ingest] index: Migrations failed. Reason: 1 transformation errors were encountered:\n- ingest-package-policies:db891ae0-6e87-11ee-92c9-b1d7c736422b: Error: Migration function for version 10.6.0 threw an error\n    at Object.convertedTransform [as transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/model_version.js:80:13)\n    at DocumentUpgradePipeline.run (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.js:174:21)\n    at DocumentMigrator.transformUp (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:196:18)\n    at DocumentMigrator.transform (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:173:19)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:131:14)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:62:88)\n    at Immediate._onImmediate (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/core/migrate_raw_docs.js:138:17)\n    at processImmediate (node:internal/timers:478:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)\nCaused by:\nTypeError: Cannot read properties of undefined (reading 'value')\n    at Object.migratePackagePolicyToV8140 [as backfillFn] (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/saved_objects/migrations/security_solution/to_v8_14_0.js:25:40)\n    at /usr/share/kibana/node_modules/@kbn/core-saved-objects-base-server-internal/src/model_version/build_transform_fn.js:50:27\n    at Object.convertedTransform [as transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/model_version.js:73:22)\n    at DocumentUpgradePipeline.run (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.js:174:21)\n    at DocumentMigrator.transformUp (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:196:18)\n    at DocumentMigrator.transform (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:173:19)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:131:14)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:62:88)\n    at Immediate._onImmediate (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/core/migrate_raw_docs.js:138:17)\n    at processImmediate (node:internal/timers:478:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)\n\nTo allow migrations to proceed, please delete or fix these documents.\nNote that you can configure Kibana to automatically discard corrupt documents and transform errors for this migration.\nPlease refer to https://www.elastic.co/guide/en/kibana/8.14/resolve-migrations-failures.html for more information.\nError: Unable to complete saved object migrations for the [.kibana_ingest] index: Migrations failed. Reason: 1 transformation errors were encountered:\n- ingest-package-policies:db891ae0-6e87-11ee-92c9-b1d7c736422b: Error: Migration function for version 10.6.0 threw an error\n    at Object.convertedTransform [as transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/model_version.js:80:13)\n    at DocumentUpgradePipeline.run (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.js:174:21)\n    at DocumentMigrator.transformUp (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:196:18)\n    at DocumentMigrator.transform (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:173:19)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:131:14)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:62:88)\n    at Immediate._onImmediate (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/core/migrate_raw_docs.js:138:17)\n    at processImmediate (node:internal/timers:478:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)\nCaused by:\nTypeError: Cannot read properties of undefined (reading 'value')\n    at Object.migratePackagePolicyToV8140 [as backfillFn] (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/saved_objects/migrations/security_solution/to_v8_14_0.js:25:40)\n    at /usr/share/kibana/node_modules/@kbn/core-saved-objects-base-server-internal/src/model_version/build_transform_fn.js:50:27\n    at Object.convertedTransform [as transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/model_version.js:73:22)\n    at DocumentUpgradePipeline.run (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.js:174:21)\n    at DocumentMigrator.transformUp (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:196:18)\n    at DocumentMigrator.transform (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:173:19)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:131:14)\n    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:62:88)\n    at Immediate._onImmediate (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/core/migrate_raw_docs.js:138:17)\n    at processImmediate (node:internal/timers:478:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)\n\nTo allow migrations to proceed, please delete or fix these documents.\nNote that you can configure Kibana to automatically discard corrupt documents and transform errors for this migration.\nPlease refer to https://www.elastic.co/guide/en/kibana/8.14/resolve-migrations-failures.html for more information.\n    at migrationStateActionMachine (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/migrations_state_action_machine.js:98:29)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 5)\n    at SavedObjectsService.start (/usr/share/kibana/node_modules/@kbn/core-saved-objects-server-internal/src/saved_objects_service.js:197:7)\n    at Server.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/server.js:374:31)\n    at Root.start (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/root/index.js:65:14)\n    at bootstrap (/usr/share/kibana/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:118:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:242:5)","log":{"level":"FATAL","logger":"root"},"process":{"pid":7,"uptime":16.363890917},"trace":{"id":"49ebb0134d38ca875d7422d72d8ac62d"},"transaction":{"id":"5b623c8d93c4a664"}}

I'm not sure where I should start troubleshooting so any help would be appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Do you see anything in the docker logs? Run sudo docker logs so-kibana

Also, have you tried to stop the Kibana contaner, remove it, and apply the Kibana state?

sudo docker stop so-kibana
sudo docker rm so-kibana
sudo salt-call state.apply kibana

Recheck the log to see if there are still errors.

[cklfe]

Thanks for the advice, you got me on the right track and I got it fixed. Notes on the issue and fix below for anyone else who may run into this.

Stopping, removing and applying the Kibana state didn't help. The Kibana logs were indicating the problem was the migration of the custom Kibana saved objects. I'm assuming this error was being caused by some of the custom Kibana dashboards I created. I tried running "so-kibana-savedobjects-defaults" but that didn't help either.

This is what the Kibana log was showing:

[2024-09-13T19:16:23.569+00:00][INFO ][root] Kibana is shutting down
[2024-09-13T19:16:23.570+00:00][FATAL][root] Reason: Unable to complete saved object migrations for the [.kibana_ingest] index: Migrations failed. Reason: 1 transformation errors were encountered:
- ingest-package-policies:db891ae0-6e87-11ee-92c9-b1d7c736422b: Error: Migration function for version 10.6.0 threw an error
    at Object.convertedTransform [as transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/model_version.js:80:13)
    at DocumentUpgradePipeline.run (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.js:174:21)
    at DocumentMigrator.transformUp (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:196:18)
    at DocumentMigrator.transform (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:173:19)
    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:131:14)
    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:62:88)
    at Immediate._onImmediate (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/core/migrate_raw_docs.js:138:17)
    at processImmediate (node:internal/timers:478:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
Caused by:
TypeError: Cannot read properties of undefined (reading 'value')
    at Object.migratePackagePolicyToV8140 [as backfillFn] (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/saved_objects/migrations/security_solution/to_v8_14_0.js:25:40)
    at /usr/share/kibana/node_modules/@kbn/core-saved-objects-base-server-internal/src/model_version/build_transform_fn.js:50:27
    at Object.convertedTransform [as transform] (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/model_version.js:73:22)
    at DocumentUpgradePipeline.run (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/pipelines/upgrade_pipeline.js:174:21)
    at DocumentMigrator.transformUp (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:196:18)
    at DocumentMigrator.transform (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:173:19)
    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:131:14)
    at DocumentMigrator.migrateAndConvert (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/document_migrator/document_migrator.js:62:88)
    at Immediate._onImmediate (/usr/share/kibana/node_modules/@kbn/core-saved-objects-migration-server-internal/src/core/migrate_raw_docs.js:138:17)
    at processImmediate (node:internal/timers:478:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)


To allow migrations to proceed, please delete or fix these documents.
Note that you can configure Kibana to automatically discard corrupt documents and transform errors for this migration.
Please refer to https://www.elastic.co/guide/en/kibana/8.14/resolve-migrations-failures.html for more information.

Following that link I found the Kibana flags you can set to automatically discard the objects causing the issue. To do that, I modified the file at "/opt/so/saltstack/local/pillar/kibana/soc_kibana.sls" to the following:

kibana:
    config:
        migrations:
            discardCorruptObjects: 8.14.3
            discardUnknownObjects: 8.14.3

Then I did "sudo so-kibana-restart" and now things look back to normal.