so-elastalert missing

[ejgh-oe]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

10

RAM

32GB

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hi,
After rebooting my 2.4.80 cluster because OS-Updates had been installed requiring a reboot (status for all nodes was "Reboot") I ended up with the manager node going to "Fault".

Already tried rebooting the manager node several times - didn't help.

so-elastalert-restart didn't solve the problem either: ran for several minutes and ended up with

          ID: wait_for_elasticsearch
    Function: cmd.run
        Name: so-elasticsearch-wait
      Result: False
     Comment: Command "so-elasticsearch-wait" run
     Started: 11:03:37.248883
    Duration: 314398.239 ms
     Changes:
              ----------
              pid:
                  647405
              retcode:
                  1
              stderr:
              stdout:
...
          ID: so-elastalert
    Function: docker_container.running
      Result: False
     Comment: One or more requisite failed: elastalert.enabled.wait_for_elasticsearch
     Started: 11:08:51.648008
    Duration: 0.003 ms
     Changes:
----------
          ID: delete_so-elastalert_so-status.disabled
    Function: file.uncomment
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: Pattern already uncommented
     Started: 11:08:51.648069
    Duration: 1.609 ms
     Changes:

Summary for local
-------------
Succeeded: 15 (changed=2)
Failed:     2
-------------
Total states run:     17
Total run time:  315.606 s

Running sudo so-elasticsearch-query _cluster/health?pretty gives

{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 281,
  "active_shards" : 281,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 47,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 85.67073170731707
}

Any ideas as to what's wrong here and how I can get my system back running again?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[raphaeleid]

yo mate hope all is good, yesterday i faced the same issue while adding rules in elastalert, under /opt/so/rules/elastalert/rules i created a folder called failed-loginz and added two rules, two different names but elastic thought it was duplicate and it got removed, when we deleted the rule, we restarted elastalert using so-elastalert-restart and it appeared

good luck

[mohasmr]

i faced this issue when create a miss configured rule and restart the so-elastalert , please check if you have any suspesious rule

[ejgh-oe]

Thanks for the hints. It had to do with misconfigured rules, but not elastalert rules (I've got no custom elastalert rules for now), but there seems to be a problem with my custom suricata rules. Commented out all of them and the elastalert error was gone :-).
Now my manager node is in the "Pending" state but that seems to be a different story...