Broke my Security-Onion setup

[skb50bd]

I have a SecurityOnion (standalone) setup. It has two forwarder nodes hosted on separate machines.
The installation was performed on Alma Linux 9.3, using the git repo. All three nodes have ~400GB of space on them.
The system was running fine until I was required to add all the hosts in the cluster. There were about 20 linux hosts I needed to add to the fleet. I enrolled them into the fleet using elastic agents. However, I was quickly running out of space for obvious reasons.

A couple of hours later the disk usage was at 90% and it exceeded the flood-stage watermark. I wasn't able to use the Kibana web UI to delete indices to free up space and I used curl to delete a large zeek index, and was able to use Kibana after that. I reduced the log retention period and disabled (not-unenrolled) the elastic-agents on the newly added agents to reduce the logs coming in. I found the fleet server agent to be in "Inactive" state.

I found the agent in running state when I ran sudo systemctl status elastic-agent, and so-status was also showing all greens. However, the docker container for elastic-fleet-agent (so-elastic-agent) shows some errors in the logs. Here's a snippet of the output from sudo docker logs so-elastic-agent:

Error: fail to enroll: fail to execute request to fleet-server: status code: 500, fleet-server returned an error: BadRequest, message: fail CreateAPIKey: [429 Too Many Requests] {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [.security-7] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}],"type":"cluster_block_exception","reason":"index [.security-7] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"},"status":429}
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.10/fleet-troubleshooting.html

I tried restarting the container and the machine. Finally, I tried to re-enroll the fleet server (on the standalone SecurityOnion installation), using these commands:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.10.4-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.10.4-linux-x86_64.tar.gz
cd elastic-agent-8.10.4-linux-x86_64
sudo ./elastic-agent install \
  --fleet-server-es=10.0.0.99 \
  --fleet-server-service-token=redacted \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-port=8220

That also didn't work. (I don't have the logs). It said the port is already in use which made sense as the docker container was using it.

I tried to use the so-* tools to set it up, and executed the so-elastic-fleet-setup. It ran for a while and spit out a whole bunch of logs with errors and got stuck. I cancelled it with ctrl+c and at this point the whole system was broken. I couldn't access the SecurityOnion WebUI. Everything was returning 500. However, so-status was showing everything as running. I was confused and restarted the machine. After that, nothing is working and when I run so-status it gives me this message:
⌛ System appears to be starting. No highstate has completed since the system was restarted. It is not moving anywhere and I am afraid of breaking the system even more. I need to retain the configuration that's already there (with the expense of some data/log loss).

Thanks for reading up to this point. I am very new to the whole SecurityOnion setup and seeking help on how I should go about and recover the system.

Security Onion Version: 2.4
Alma Linux 9.3

[cm-ops]

Have you tried to reset fleet with so-fleet-reset?

[skb50bd]

I was able to fix it looking at logs from salt-master and salt-minion.

I had two yaml files corrupted that I am pretty sure I messed up when I ran so-elastic-fleet-setup. opt/so/saltstack/local/pillar/minions/sec1-s-cc_standalone.sls and opt/so/saltstack/local/pillar/global/soc_global.sls. They both had duplicate keys. I manually removed the duplicate keys, and the cluster came back.

However, the fleet-server-agent was still missing (in so-status), and the docker container: so-elastic-fleet was missing. I had to run so-docker-refresh and things came back to normal.