No elastalert and kibana is not responding (ES License issue) - Solved

[kawaiipantsu]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

other (please provide detail below)

Hardware Specs

Exceeds minimum requirements

CPU

20

RAM

32GB

Storage for /

1TB

Storage for /nsm

5TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

THIS IS SOLVED

This is for others to be able to find this solution when searching as the problem/identifier does not really lead one in the right direction...

Identifier
The behaviour is that you start up/rebooted your Security Onion 2.4.xx installation and then suddenly you notice the following things:

• so-status show that so-elastalert is not running!
• Trying to access Kibana leaves you with a blank/404 page
• elastalert's logs does not really give you any insight

Solution
All of this turned out for me to be related to my "trail" license had expired. Seems like as long as you don't "reboot" then an expired ElasticSearch (Enterprise) Lisense will not break anything already running but once you reboot the license is then "hard" expired and ElasticSearch will not let you run the server as the "security" features is enabled but license expired.

You can check this by doing the following:
sudo so-elasticsearch-query _license

You need to see that it says: "status" : "active" amongst the json output.
If you have an license that is Expired then make sure to change it back to either the default "Basic License" that is free or to buy a new License/Update your current.

To update/revert your Elastic Search License back to "Basic" free license:
sudo so-elasticsearch-query _license/start_basic?acknowledge=true "-X POST"

Please then rerun the check to see if it's active. Then i recommend to reboot your SO instance.
After reboot - It all comes right back up as usual and so-status once again show that everything is working.

Other discussions posts related to the same issue and/or with alternative solutions

• Kibana 404 #12712
• Upgrading 2.4.50 -> 2.4.60 - grid doesn't come up again #12648

Cheers

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[kawaiipantsu]

• Solved

[Nicalicious]

I have a post over at #13338 which seems to be in the same vein. Mine is on a fresh install of the iso. The license status is active but elastalert will not start no matter what. There are no logs which may even imply that it isn't installed at all.

[kawaiipantsu]

I have a post over at #13338 which seems to be in the same vein. Mine is on a fresh install of the iso. The license status is active but elastalert will not start no matter what. There are no logs which may even imply that it isn't installed at all.

You should always start by doing a salt-call state.highstate and see what errors it throws or check if you got any errors in either elastalert.logor soup.log.

If it's a salt problem, you can try to reset the high state on that thing that breaks by: sudo salt-call <xxxxx>.update then run sudo so-checkin

Sometimes you can also just to a manual sudo so-checkin to rebuild and refresh everything