HELP - I just lost all my data on the Search node!

[ihuxn4hi]

Hi there,

I logged into my SO console this morning and it was throwing 502 errors. I've noticed the Search node has been running high on CPU resource utilization recently so went to take a look in Grafana and noticed that the CPU utilization was 0% and that the disk utilization on the /nsm filesystem had dropped from 85% to 5%, both around the same time yesterday evening (~midnight UTC).

I rebooted the Search node so it's no longer throwing 502s and it's displaying new data but it seems all historical data is gone. It's not the end of the world as I'm currently doing a POC but if this were actually production, it would be bad.

Does anyone know what might have happened? Is there a cronjob that kicks off at midnight that could have this impact? Or a salt job that runs once storage hits 85% utilization? I saw this in yesterday's log in /opt/so/log/elasticsearch/ on the Search node:

[2023-02-09T23:50:02,494][INFO ][org.elasticsearch.cluster.metadata.MetadataDeleteIndexService] [so-kratos-2023.02.09/SFZ8gC-MQhq7BLuZ7lGEBw] deleting index
[2023-02-09T23:50:02,494][INFO ][org.elasticsearch.cluster.metadata.MetadataDeleteIndexService] [so-ids-2023.02.09/QgkQiJcDSGCj2uqPcMnvzQ] deleting index
[2023-02-09T23:50:02,494][INFO ][org.elasticsearch.cluster.metadata.MetadataDeleteIndexService] [so-zeek-2023.02.08/nUagxlmqQDuixC6U5EKbVA] deleting index
[2023-02-09T23:50:02,494][INFO ][org.elasticsearch.cluster.metadata.MetadataDeleteIndexService] [so-zeek-2023.02.09/R0mw1s0CRU6h4HkNQs33sQ] deleting index
[2023-02-09T23:50:02,494][INFO ][org.elasticsearch.cluster.metadata.MetadataDeleteIndexService] [so-elasticsearch-2023.02.09/Kg3AmOX2QQyExOg8v8v9cg] deleting index
[2023-02-09T23:50:13,719][INFO ][org.elasticsearch.cluster.metadata.MetadataCreateIndexService] [so-zeek-2023.02.09] creating index, cause [auto(bulk api)], templates [so-zeek], shards [2]/[0]
[2023-02-09T23:50:13,728][WARN ][org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDecider] after allocating [[so-zeek-2023.02.09][0], node[null], [P], recovery_source[new shard recovery], s[UNASSIGNED], unassigned_info[[reason=INDEX_CREATED], at[2023-02-09T23:50:13.721Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]] node [V2F_InLsR3-3CAdCX_b7-A] would be above the high watermark setting [98%], having less than the minimum required 5.8gb of free space (actual free: 41.8gb, actual used: 85.5%, estimated shard size: 114.5gb), preventing allocation
[2023-02-09T23:50:13,729][WARN ][org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDecider] after allocating [[so-zeek-2023.02.09][1], node[null], [P], recovery_source[new shard recovery], s[UNASSIGNED], unassigned_info[[reason=INDEX_CREATED], at[2023-02-09T23:50:13.721Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]] node [V2F_InLsR3-3CAdCX_b7-A] would be above the high watermark setting [98%], having less than the minimum required 5.8gb of free space (actual free: 41.8gb, actual used: 85.5%, estimated shard size: 118.6gb), preventing allocation
[2023-02-09T23:50:13,747][WARN ][org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDecider] after allocating [[so-zeek-2023.02.09][0], node[null], [P], recovery_source[new shard recovery], s[UNASSIGNED], unassigned_info[[reason=INDEX_CREATED], at[2023-02-09T23:50:13.721Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]] node [V2F_InLsR3-3CAdCX_b7-A] would be above the high watermark setting [98%], having less than the minimum required 5.8gb of free space (actual free: 41.8gb, actual used: 85.5%, estimated shard size: 114.5gb), preventing allocation
[2023-02-09T23:50:13,747][WARN ][org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDecider] after allocating [[so-zeek-2023.02.09][1], node[null], [P], recovery_source[new shard recovery], s[UNASSIGNED], unassigned_info[[reason=INDEX_CREATED], at[2023-02-09T23:50:13.721Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]] node [V2F_InLsR3-3CAdCX_b7-A] would be above the high watermark setting [98%], having less than the minimum required 5.8gb of free space (actual free: 41.8gb, actual used: 85.5%, estimated shard size: 118.6gb), preventing allocation
[2023-02-09T23:50:13,748][INFO ][org.elasticsearch.cluster.routing.allocation.AllocationService] current.health="RED" message="Cluster health status changed from [YELLOW] to [RED] (reason: [reconcile-desired-balance])." previous.health="YELLOW" reason="reconcile-desired-balance"

So I'm guessing it blew everything away after hitting that threshold? I just want to make sure I understand what happened so I can ensure it doesn't happen again.

Thank you very much!

[InfosecGoon]

This definitely sounds like an automated cleanup happening -- how much disk space do you have on your search nodes, and how much data are you collecting per day? You can check the current amount of data in Elasticsearch with "so-elasticsearch-indices-list" on the search node CLI.

[InfosecGoon]

Well, it looks like you're collecting several hundred gigs of Zeek data every day, so it's not surprising that your Search nodes are filling up. Is there anything you can do to reduce the amount of data you're seeing? If you're feeding your sensors with a tap infrastructure (Gigamon or Ixia or something) you may be able to do some data deduplication. Or you could use BPF settings (https://docs.securityonion.net/en/2.3/bpf.html) to block Zeek from seeing network traffic that's not useful in an investigation.

[ihuxn4hi]

Thank you so much for your response! I'm only monitoring six systems currently and it's only web app traffic for the most part (~170Mbps average). I guess I could create a BPF filter to ignore HTTPS since we can't see the payloads of that traffic anyway but is there a way to look at what the highest volume of traffic looks like so I can make a more surgical decision?

Is there a way to drop the Zeek data that is older than 24 hours to try and reclaim some of this space?

Thank you again!

[InfosecGoon]

A couple of things:

1. If you want to see what Zeek data is being recorded, you can go into Hunt and use this query:

event.module:"zeek" | groupby event.dataset

Hundreds of gigs of Zeek logs for a six system network seems very, very high. I'll be interested to see what you find.

2. A BPF won't help too much here, although one to block 443 from stenographer might help save disk space on your forward node(s).

3. You can expire the Zeek data fast by changing this stanza in /opt/so/saltstack/local/pillar/global.sls:

    so-zeek:
      index_template:
        template:
          settings:
            index:
              number_of_shards: 2
      warm: 7
      close: 45
      delete: 365

If you change it to close 2 and delete 3, that will delete your Zeek logs after a couple of days.

[ihuxn4hi]

Thank you very much for this info -- that's super helpful!

Over the last 24 hours, Zeek has collected over 126 million weird logs and over 100 million connection logs:

I'm not sure what's causing that but I'll purge the Zeek data as the new search node w/ a 2TB /nsm is already at 50%. Is there a way to determine what it's matching on?

Thank you again!

[InfosecGoon]

That is a shocking amount of Weird logs. What are they? Is it all coming from one source IP?

I'm wondering if you've got some embedded device or some other misconfigured endpoint trying to pass bad traffic and that's what's filling up the logs.

[ihuxn4hi]

It looks like it's all internal traffic (AWS). Here's the top 10 weird log types for the last 24 hours:

What would be your suggestion on tuning this? The search node whose /nsm was at 50% on Friday is now at 98%.

I've made the change to global.sls that you mentioned and ran 'salt-call state.apply' but that didn't seem to have an affect. Is there a way to have that force an immediate run?

Thank you again for your help -- I really appreciate it!

[InfosecGoon]

One quick fix might be to just us so-zeek-logs on the Manager to stop collecting the weird logs while you investigate -- it looks like there are some spots where you're only seeing half of the network connection, which is generating these events.

https://docs.securityonion.net/en/2.3/so-zeek-logs.html#so-zeek-logs

[ihuxn4hi]

Thank you! I've disabled the weird and conn logs.

And thank you very much for the link to the Utilities section of the documentation. I did a 'so-elasticsearch-query <so-zeek-*> -XDELETE' on each of the search nodes which purged all existing zeek data and it freed a huge amount of storage in /nsm:

Not ideal to lose that data but at least things are more stable now.

Thanks again!