Kibana 404 after a week of uptime

[geistchevalier]

SO Version: 2.3.240
Deployment: Standalone, physical machine
RAM: 16 GB
CPU: E3-1240 V2
Storage for /nsm: 8TB
Log sources: 4 windows server vms, 1 firewall (the firewall outputs the most logs)

First of all I want to mention that I've remedied the kibana 404 issue by applying the solution from the following the discussion #9850

What I want to ask is why does this happen quite frequently for me, is it because of the relatively small amount of ram that is available for the system? or is there a configuration or flag that I need to set so that the index doesn't hang up every week?

[InfosecGoon]

Can you post your kibana.log file from the last time it was failing? It would be helpful to see what exactly the error message you're seeing is.

The file should be in /opt/so/log/kibana.

[geistchevalier]

This is what I found, after the last line it just repeats the same ECONNREFUSED and license information is not available message, it will usually be resolved after a kibana service restart, but sometimes I still need to reboot the machine to resolve it.

Strange thing is that when I run so-status everything is marked as [ OK ]

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T06:43:20.285+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":7},"trace":{"id":"2f4fe3301ff8eca83b78d3e300688ae9"},"transaction":{"id":"aae3f9698dd237d5"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T06:58:23.214+00:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":7},"trace":{"id":"2f4fe3301ff8eca83b78d3e300688ae9"},"transaction":{"id":"3551b7fd952d1708"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.372+00:00","message":"[Task Poller Monitor]: Observable Monitor: Hung Observable restarted after 33000ms of inactivity","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":7},"trace":{"id":"c5b2b43152973bd17084a54318f5aff6"},"transaction":{"id":"0df6f3176ff9583a"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.940+00:00","message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 192.#.#.#:9200 error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":7},"trace":{"id":"4a990e6db163d3d83064972e24db65b9"},"transaction":{"id":"7a670334e4aea144"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.956+00:00","message":"X-Pack Monitoring Cluster Alerts will not be available: connect ECONNREFUSED 192.#.#.#:9200","log":{"level":"WARN","logger":"plugins.monitoring.monitoring"},"process":{"pid":7},"trace":{"id":"4a990e6db163d3d83064972e24db65b9"},"transaction":{"id":"7a670334e4aea144"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.957+00:00","message":"License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 192.#.#.#:9200 error","log":{"level":"WARN","logger":"plugins.licensing"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.962+00:00","message":"You cannot use searchprofiler because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.searchprofiler"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.963+00:00","message":"You cannot use painlessLab because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.painlessLab"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.963+00:00","message":"You cannot use snapshot_restore because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.snapshotRestore"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.967+00:00","message":"You cannot use transform because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.transform"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.967+00:00","message":"You cannot use Remote Clusters because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.remoteClusters"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.968+00:00","message":"You cannot use index_lifecycle_management because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.indexLifecycleManagement"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:34.968+00:00","message":"You cannot use rollup because license information is not available at this time.","log":{"level":"INFO","logger":"plugins.rollup"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.088+00:00","message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped","log":{"level":"INFO","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.103+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 192.#.#.#:9200","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.113+00:00","message":"ConnectionError: connect ECONNREFUSED 192.#.#.#:9200\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n    at Monitoring.bulk (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/monitoring.js:53:16)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:19:10)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:161:12)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:150:9)","log":{"level":"WARN","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.113+00:00","message":"Unable to bulk upload the stats payload to the local cluster","log":{"level":"WARN","logger":"plugins.monitoring.monitoring.kibana-monitoring"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.114+00:00","message":"[WorkloadAggregator]: ConnectionError: connect ECONNREFUSED 192.#.#.#:9200","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.137+00:00","message":"Unable to verify endpoint policies in line with license change: failed to fetch package policies: connect ECONNREFUSED 192.#.#.#:9200","log":{"level":"WARN","logger":"plugins.securitySolution"},"process":{"pid":7},"trace":{"id":"3e211dbc84e1a3a0259e4716b160918e"},"transaction":{"id":"3e8d8e0f8af8c8d7"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-05-05T07:07:35.198+00:00","message":"Kibana is now critical (was available)","log":{"level":"INFO","logger":"status"},"process":{"pid":7},"trace":{"id":"c5b2b43152973bd17084a54318f5aff6"},"transaction":{"id":"0df6f3176ff9583a"}}```

[geistchevalier]

I ended up reducing the keep warm time in global.sls to 3-5 days for most things and the kibana in SO seems to be stable, going through my system logs it seems that 16GB ram isn't enough when SO consumes a firewall log that has a really heavy traffic, it kept causing the system to run out of memory and crashes the kibana instance and a bunch of other services.