ElastAlert missing

[oliverhavrila]

Hello,

I did upgrade from 2.4.20-20231012 to the latest version 2.4.30-20231113

After that so-status showed me that so-elastalert is missing.

[root@company /]# so-status

                       Security Onion Status
                         Container │ Status  │              Details
───────────────────────────────────┼─────────┼──────────────────────
                        so-curator │ running │           Up 2 hours
                 so-dockerregistry │ running │           Up 2 hours
                     so-elastalert │ missing │

docker ps -a

[root@company log]# docker ps -a | grep elastalert
643844650a61   company:5000/security-onion-solutions/so-elastalert:2.4.30                       "/opt/elastalert/run…"   9 minutes ago            Exited (1) 9 minutes ago 

docker logs so-elastalert
[root@company /]# docker logs so-elastalert Error response from daemon: No such container: so-elastalert

docker images | grep elastalert

[root@company /]# docker images | grep elastalert
ghcr.io/security-onion-solutions/so-elastalert                          2.4.30    43a80ed3ab58   3 weeks ago    414MB
company:5000/security-onion-solutions/so-elastalert                       2.4.30    43a80ed3ab58   3 weeks ago    414MB
company:5000/security-onion-solutions/so-elastalert                       2.4.20    3e63b1e4686e   6 weeks ago    414MB

so-elastalert-restart

[root@company /]# so-elastalert-restart
=========================================================================
Restarting elastalert...

This could take a while if another Salt job is running.
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
Error response from daemon: No such container: so-elastalert
Error response from daemon: No such container: so-elastalert
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [elastalert] at time 16:22:49.496245
[INFO    ] Executing state group.present for [elastalert]
[INFO    ] Group elastalert is present and up to date
[INFO    ] Completed state [elastalert] at time 16:22:49.505448 (duration_in_ms=9.204)
[INFO    ] Running state [elastalert] at time 16:22:49.507218
[INFO    ] Executing state user.present for [elastalert]
[INFO    ] User elastalert is present and up to date
[INFO    ] Completed state [elastalert] at time 16:22:49.587993 (duration_in_ms=80.775)
[INFO    ] Running state [/opt/so/log/elastalert] at time 16:22:49.592124
[INFO    ] Executing state file.directory for [/opt/so/log/elastalert]
[INFO    ] The directory /opt/so/log/elastalert is in the correct state
[INFO    ] Completed state [/opt/so/log/elastalert] at time 16:22:49.595212 (duration_in_ms=3.087)
[INFO    ] Running state [/usr/sbin] at time 16:22:49.595652
[INFO    ] Executing state file.recurse for [/usr/sbin]
[INFO    ] {'/usr/sbin': {'/usr/sbin': {'user': 933}, 'user': 933}}
[INFO    ] Loading fresh modules for state activity
[INFO    ] Completed state [/usr/sbin] at time 16:22:50.239035 (duration_in_ms=643.382)
[INFO    ] Running state [/opt/so/rules/elastalert] at time 16:22:50.244221
[INFO    ] Executing state file.directory for [/opt/so/rules/elastalert]
[INFO    ] The directory /opt/so/rules/elastalert is in the correct state
[INFO    ] Completed state [/opt/so/rules/elastalert] at time 16:22:50.251520 (duration_in_ms=7.299)
[INFO    ] Running state [/opt/so/conf/elastalert] at time 16:22:50.251966
[INFO    ] Executing state file.directory for [/opt/so/conf/elastalert]
[INFO    ] The directory /opt/so/conf/elastalert is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastalert] at time 16:22:50.254850 (duration_in_ms=2.884)
[INFO    ] Running state [/opt/so/conf/elastalert/modules/so] at time 16:22:50.255317
[INFO    ] Executing state file.directory for [/opt/so/conf/elastalert/modules/so]
[INFO    ] The directory /opt/so/conf/elastalert/modules/so is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastalert/modules/so] at time 16:22:50.258136 (duration_in_ms=2.818)
[INFO    ] Running state [/opt/so/conf/elastalert/modules/custom] at time 16:22:50.258577
[INFO    ] Executing state file.directory for [/opt/so/conf/elastalert/modules/custom]
[INFO    ] The directory /opt/so/conf/elastalert/modules/custom is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastalert/modules/custom] at time 16:22:50.261443 (duration_in_ms=2.866)
[INFO    ] Running state [/opt/so/conf/elastalert/modules/so] at time 16:22:50.261906
[INFO    ] Executing state file.recurse for [/opt/so/conf/elastalert/modules/so]
[INFO    ] The directory /opt/so/conf/elastalert/modules/so is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastalert/modules/so] at time 16:22:50.481603 (duration_in_ms=219.696)
[INFO    ] Running state [/opt/so/conf/elastalert/elastalert_config.yaml] at time 16:22:50.482102
[INFO    ] Executing state file.managed for [/opt/so/conf/elastalert/elastalert_config.yaml]
[INFO    ] File /opt/so/conf/elastalert/elastalert_config.yaml is in the correct state
[INFO    ] Completed state [/opt/so/conf/elastalert/elastalert_config.yaml] at time 16:22:50.564387 (duration_in_ms=82.285)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 16:22:50.564821
[INFO    ] Executing state file.append for [/opt/so/conf/so-status/so-status.conf]
[INFO    ] Executing command git in directory '/root'
[INFO    ] Executing command 'grep' in directory '/root'
[INFO    ] ['unless condition is true']
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 16:22:52.755510 (duration_in_ms=2190.688)
[INFO    ] Running state [so-elasticsearch-wait] at time 16:22:52.755861
[INFO    ] Executing state cmd.run for [so-elasticsearch-wait]
[INFO    ] Executing command 'so-elasticsearch-wait' in directory '/root'
[INFO    ] {'pid': 600345, 'retcode': 0, 'stdout': "Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)\nReceived expected response; proceeding.", 'stderr': ''}
[INFO    ] Completed state [so-elasticsearch-wait] at time 16:22:52.915448 (duration_in_ms=159.587)
[INFO    ] Running state [so-elastalert] at time 16:22:52.917955
[INFO    ] Executing state docker_container.running for [so-elastalert]
[INFO    ] Executing command 'so-elasticsearch-query' in directory '/root'
[INFO    ] {'container': {'Networks': {'bridge': {'IPConfiguration': {'old': 'automatic', 'new': 'not connected'}}, 'sobridge': {'Aliases': {'old': None, 'new': ['843b8aa46ad9']}, 'IPAMConfig': {'old': None, 'new': {'IPv4Address': 'IPADDRESS'}}}}}, 'container_id': {'added': '843b8aa46ad9ae5aec9b0ba1e723226ff9586301f6843622eb5796b290ca5b74'}, 'state': {'old': None, 'new': 'running'}}
[INFO    ] Completed state [so-elastalert] at time 16:22:55.011181 (duration_in_ms=2093.226)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 16:22:55.011715
[INFO    ] Executing state file.uncomment for [/opt/so/conf/so-status/so-status.conf]
[INFO    ] Pattern already uncommented
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 16:22:55.017019 (duration_in_ms=5.304)
local:
----------
          ID: elastagroup
    Function: group.present
        Name: elastalert
      Result: True
     Comment: Group elastalert is present and up to date
     Started: 16:22:49.496244
    Duration: 9.204 ms
     Changes:
----------
          ID: elastalert
    Function: user.present
      Result: True
     Comment: User elastalert is present and up to date
     Started: 16:22:49.507218
    Duration: 80.775 ms
     Changes:
----------
          ID: elastalogdir
    Function: file.directory
        Name: /opt/so/log/elastalert
      Result: True
     Comment: The directory /opt/so/log/elastalert is in the correct state
     Started: 16:22:49.592125
    Duration: 3.087 ms
     Changes:
----------
          ID: elastalert_sbin
    Function: file.recurse
        Name: /usr/sbin
      Result: True
     Comment: Recursively updated /usr/sbin
     Started: 16:22:49.595653
    Duration: 643.382 ms
     Changes:
              ----------
              /usr/sbin:
                  ----------
                  /usr/sbin:
                      ----------
                      user:
                          933
                  user:
                      933
----------
          ID: elastarules
    Function: file.directory
        Name: /opt/so/rules/elastalert
      Result: True
     Comment: The directory /opt/so/rules/elastalert is in the correct state
     Started: 16:22:50.244221
    Duration: 7.299 ms
     Changes:
----------
          ID: elastaconfdir
    Function: file.directory
        Name: /opt/so/conf/elastalert
      Result: True
     Comment: The directory /opt/so/conf/elastalert is in the correct state
     Started: 16:22:50.251966
    Duration: 2.884 ms
     Changes:
----------
          ID: elastasomodulesdir
    Function: file.directory
        Name: /opt/so/conf/elastalert/modules/so
      Result: True
     Comment: The directory /opt/so/conf/elastalert/modules/so is in the correct state
     Started: 16:22:50.255318
    Duration: 2.818 ms
     Changes:
----------
          ID: elastacustmodulesdir
    Function: file.directory
        Name: /opt/so/conf/elastalert/modules/custom
      Result: True
     Comment: The directory /opt/so/conf/elastalert/modules/custom is in the correct state
     Started: 16:22:50.258577
    Duration: 2.866 ms
     Changes:
----------
          ID: elastasomodulesync
    Function: file.recurse
        Name: /opt/so/conf/elastalert/modules/so
      Result: True
     Comment: The directory /opt/so/conf/elastalert/modules/so is in the correct state
     Started: 16:22:50.261907
    Duration: 219.696 ms
     Changes:
----------
          ID: elastaconf
    Function: file.managed
        Name: /opt/so/conf/elastalert/elastalert_config.yaml
      Result: True
     Comment: File /opt/so/conf/elastalert/elastalert_config.yaml is in the correct state
     Started: 16:22:50.482102
    Duration: 82.285 ms
     Changes:
----------
          ID: append_so-elastalert_so-status.conf
    Function: file.append
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: unless condition is true
     Started: 16:22:50.564822
    Duration: 2190.688 ms
     Changes:
----------
          ID: wait_for_elasticsearch
    Function: cmd.run
        Name: so-elasticsearch-wait
      Result: True
     Comment: Command "so-elasticsearch-wait" run
     Started: 16:22:52.755861
    Duration: 159.587 ms
     Changes:
              ----------
              pid:
                  600345
              retcode:
                  0
              stderr:
              stdout:
                  Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)
                  Received expected response; proceeding.
----------
          ID: so-elastalert
    Function: docker_container.running
      Result: True
     Comment: Created container 'so-elastalert'
     Started: 16:22:52.917955
    Duration: 2093.226 ms
     Changes:
              ----------
              container:
                  ----------
                  Networks:
                      ----------
                      bridge:
                          ----------
                          IPConfiguration:
                              ----------
                              new:
                                  not connected
                              old:
                                  automatic
                      sobridge:
                          ----------
                          Aliases:
                              ----------
                              new:
                                  - 843b8aa46ad9
                              old:
                                  None
                          IPAMConfig:
                              ----------
                              new:
                                  ----------
                                  IPv4Address:
                                      IPADDRESS
                              old:
                                  None
              container_id:
                  ----------
                  added:
                      hiden
              state:
                  ----------
                  new:
                      running
                  old:
                      None
----------
          ID: delete_so-elastalert_so-status.disabled
    Function: file.uncomment
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: Pattern already uncommented
     Started: 16:22:55.011715
    Duration: 5.304 ms
     Changes:

Summary for local
-------------
Succeeded: 14 (changed=3)
Failed:     0
-------------
Total states run:     14
Total run time:    5.503 s

Any idea please?

[cm-ops]

Any clues in /opt/so/log/elastalert? Look at both the elastalert.log and stderr.log. Also, what are your node resources?

[dougburks]

I did not made any changes in the ElastAlert configuration, I only created new rule for function "Email - External" via so-elastalert-create script in folder /opt/so/rules/elastalert/

Have you tried removing this rule?

and I changed grid value from false to true here: Administration –> Configuration –> elastalert

Any particular reason you changed the grid value? Elastalert should have been enabled on your manager by default.

[oliverhavrila]

Hello @dougburks

Yes, I removed rules, there is nothing now, just folder "playbook":

[admin@standalone/]$ ls /opt/so/rules/elastalert/
playbook

Grid value was by default false, I had to change it to true

There is no way, how to reinstall just this one container "so-elastalert"?

[dougburks]

Grid value was by default false, I had to change it to true

Grid value is supposed to be false because if you have a distributed deployment you probably don't want elastalert running on all nodes in the grid. It should be enabled on the manager node and the bottom of your screenshot confirms that. Please return the grid value back to its default setting of false and reboot the box.

There is no way, how to reinstall just this one container "so-elastalert"?

The docker image is there and shouldn't need to be reinstalled. The reason it's not running is because of a configuration problem.

[oliverhavrila]

OK, now I change it back to default value and reboot the server.. I am using standalone version, not distributed.

After that, I do not see elastalert in status:

[admin@standalone ~]$ sudo so-status

                         Security Onion Status
                         Container │ Status  │                 Details
───────────────────────────────────┼─────────┼─────────────────────────
                        so-curator │ running │           Up 14 minutes
                 so-dockerregistry │ running │           Up 23 minutes
                  so-elastic-fleet │ running │           Up 14 minutes
 so-elastic-fleet-package-registry │ running │ Up 16 minutes (healthy)
                  so-elasticsearch │ running │           Up 17 minutes
                       so-idstools │ running │           Up 18 minutes
                       so-influxdb │ running │ Up 18 minutes (healthy)
                         so-kibana │ running │           Up 16 minutes
                         so-kratos │ running │           Up 23 minutes
                       so-logstash │ running │           Up 16 minutes
                          so-mysql │ running │ Up 18 minutes (healthy)
                          so-nginx │ running │ Up 19 minutes (healthy)
                       so-playbook │ running │           Up 14 minutes
                          so-redis │ running │           Up 16 minutes
                      so-sensoroni │ running │           Up 18 minutes
                            so-soc │ running │           Up 18 minutes
                       so-soctopus │ running │           Up 14 minutes
                so-strelka-backend │ running │           Up 15 minutes
            so-strelka-coordinator │ running │           Up 16 minutes
             so-strelka-filestream │ running │           Up 15 minutes
               so-strelka-frontend │ running │           Up 15 minutes
             so-strelka-gatekeeper │ running │           Up 16 minutes
                so-strelka-manager │ running │           Up 15 minutes
                       so-suricata │ running │           Up 16 minutes
                       so-telegraf │ running │           Up 18 minutes
                           so-zeek │ running │ Up 16 minutes (healthy)

[dougburks]

Make sure that Elastalert is still enabled for the standalone entry at the bottom of the configuration screen. This is what it should look like by default:

If that is set correctly, then try running the following and see if there are any errors:

sudo salt-call state.highstate

If all else fails, you can always download the latest ISO image and perform a fresh installation.

[oliverhavrila]

Hello @dougburks

OK, I set up elastalert like you told me:

salt-call looks like without any errors:
salt-call.txt

I am worried that fresh installation is not possible, because client will lost all alerts what are there now, also also all assets are already connected via elastagents.

Wouldn't it be possible to order a service intervention, only for this problem?

[dougburks]

I am worried that fresh installation is not possible, because client will lost all alerts what are there now, also also all assets are already connected via elastagents.

You could try a fresh installation on a separate machine (perhaps just a VM) and then compare the two systems side-by-side to help pinpoint what is wrong with your production system.

Wouldn't it be possible to order a service intervention, only for this problem?

I'm not sure what you're asking here. We do offer paid support options if that's what you're asking:
https://securityonionsolutions.com/support

[keis3cker]

Do you have any elastalert rules enabled under /opt/so/rule/elastalert/playbook?
If so I would try to move them to another folder and try the so-elastalert-restart command again.
Your stderr log, states that the rule type is missing for one rule.