kibana server is not ready yet

[corrcorr]

Hi Team,

I got the following error when i try to lunch Kibana "kibana server is not ready yet" I followed this steps https://docs.securityonion.net/en/2.3/kibana.html#diagnostic-logging but i get this error logs instead.
I've also tried to restart Kibana but I keep having the same error "(287/300)\nServer is not ready\nWaiting for value 'Elastic' at 'http://localhost:5601/app/kibana' " so it gives up after 300 retries or so.

Thanks in advance

{"type":"log","@timestamp":"2021-06-14T11:15:10+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":294,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-06-14T11:15:11+00:00","tags":["info","savedobjects-service"],"pid":294,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-06-14T11:15:12+00:00","tags":["info","savedobjects-service"],"pid":294,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-06-14T11:15:12+00:00","tags":["warning","savedobjects-service"],"pid":294,"message":"Unable to connect to Elasticsearch. Error: search_phase_execution_exception"}
{"type":"log","@timestamp":"2021-06-14T11:15:12+00:00","tags":["warning","savedobjects-service"],"pid":294,"message":"Unable to connect to Elasticsearch. Error: search_phase_execution_exception"}

[TheN00bBuilder]

Sounds like Elasticsearch isn't coming up (or something like that). What version of SO are you on? And do you have the logs from /opt/so/log/elasticsearch/<sensorname>.log?

[TheN00bBuilder]

That's not good. Does the cluster API still work? curl -XGET localhost:9200/_cluster/stats?pretty

[corrcorr]

this is what i get

curl: (52) Empty reply from server

[TheN00bBuilder]

Still?! Hmm, this is quite odd and indicative of something being seriously broken with Elasticsearch. Has the API ever worked? Can you try restarting Elasticsearch and try both commands again?

[corrcorr]

the output is still the same after restarting elastic search. please noter that i can hunt and do all other stuffs except for kibana.

[TheN00bBuilder]

Ah, okay. Probably my fault for not asking. This is getting a little out of my known domain (I've messed with SO only for 2 weeks now, this is not my job haha) so hopefully someone else has an idea. The last thing I'd check is firewall rules - have you created any or deleted any recently?

[weslambert]

I would first consider updating your version of Security Onion to the latest.

What is the output of the following from the manager and the search node?

curl -sk https://localhost:9200/_cluster/health?pretty

curl -sk https://localhost:9200/_indices | sort -n

[TheN00bBuilder]

I agree with the upgrade, or even a full reinstall. They were getting a lot of "empty response from server" errors from CURL when I asked them to run a few other API calls, so these may net the same results.