Curator does not seem to close logs-syslog-so indices.

[Maurice-De]

Version

2.4.10

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

16

Storage for /

300gb

Storage for /nsm

1.7TB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,

I have tried to search for this, but a lot of old version topics came across, but no recent ones.

We have a distributed setup but after a couple of months we noticed that the logs where on 30days/365days with curator disabled.

So I enabled curator from the grid config and changed several 365days to 90days, but after that threshold was reached there was no flattening of the used disk space, it was still increasing. I noticed in elestic index management that the most logging was in logs-syslog-so and so I went looking for that in the curator logging but found nothing. Only a lot of logs that stated that actions where skipped because there was nothing to close. There are actions for this index in /opt/so/conf/curator/action/ but I cannot find what should run this action and why it is not running.

Does anyone know if this is a known issue or if there is a standard quick config/fix to configure curator. Or does anyone know what I can try or troubleshoot for this?

Regards,
Maurice

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[weslambert]

Let's try this:

• Edit the following files to contain logs-syslog-so-close.yml instead of logs-syslog-close.yml
  Example:

  securityonion/salt/curator/tools/sbin/so-curator-cluster-close

  Line 29 in d3802c1

  docker exec so-curator curator --config /etc/curator/config/curator.yml /etc/curator/action/logs-syslog-close.yml > /dev/null 2>&1;

  • /opt/so/saltstack/default/salt/curator/tools/sbin/so-curator-close
  • /opt/so/saltstack/default/salt/curator/tools/sbin/so-curator-cluster-close
• sudo salt-call state.apply curator queue=True
• Observe the behavior to see if it aligns with expectations

[weslambert]

You might need to do one more thing.

• Modify /opt/so/saltstack/default/salt/curator/files/action/logs-syslog-so-close.yml to include the following line under the options section.

allow_ilm_indices: True

• Restart Curator or apply the changes and see if it makes a difference.

[Maurice-De]

Hi,
with the alteration it seems the action runs, but it still throws the message "Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>"

[Maurice-De]

Sorry if my initial question/issue was with a wrong assumption from my end, let me explain where I am at the moment:
First we have installed a distributed deployment of security onion.
Second we have made Syslog work by enabling it and configure the firewall rules with the grid config page as described in the docs.
Third we have configured a couple of devices to use the security onion as syslog destination
And fourth I came to realize that it is a lot of data and we only want to keep like 90 days of data so that we don't need lots of TB's to host the logging.

So I went looking how to change the retention and came across curator. Enabled it and came to realize it did not delete or even close it for syslog data. And eventually found out that the specific syslog action was not run. And now that it runs I noticed that the logs-syslog-so is in 1 big index, and after the update to 2.4.30 I see the option that is described in the issue I referred to, but the stack management (index management) still looks the same as in the previous screenshot. So I do not see an index (with format like ".ds-logs-syslog-so-YYYY.MM.DD-00000X") with the backing datastream log-syslog-so. But there is still syslog data received and put in the big index.

So it started indeed with a different question then where we are now, but I think the single big logs-syslog-so index is an issue in our deployment at the moment. The logs-syslog-so is now around 560GB and has syslog data for around 100 days.

[kvkamin]

under Administration/Configuration/curator/elasticsearch/index_settings/logs-syslog-so
what are the values for close and delete? I had a similar problem and I found a crazy value there like 73000. I don't know if that was default or something I did.. Also, i don't think they are actually deleted for 24 hours...

K

[Maurice-De]

@kvkamin that was the first thing I did when I started with the retention settings. Close is at 60 and delete is at 90. But that only applies to the ".ds-logs-syslog-so-YYYY.MM.DD-00000X" which I cannot find in my deployment.

[weslambert]

I think you would need to re-index the logs-syslog-so index to another named index, or delete it for the new data stream to be created. Otherwise, Elastic will use the template for the old index and Logstash will continue writing to it, since it matches the logs-syslog-so name.

This script may work for you, but please test it before using it in production.

https://github.com/weslambert/securityonion-elastic-misc/blob/2.x/so-elasticsearch-reindex

For an index of that size it may take a very long time.

[Maurice-De]

Do you mean I need to reindex it so it has a different name and can be deleted when my retention is reached (and a new indices/datastream is created for the new data), or do you mean the reindex will create the indices with the datastream?

[Maurice-De]

I have tried the reindexing, unfortunately it did not go as hoped. The data from that index has been lost (it was not that of a big deal and we still have a snapshot in case we really really need it). But in the good news segment: I now see a ".ds-logs-syslog-so-YYYY.MM.DD-00000X" with a "logs-syslog-so" datastream. So I hope Curator can now properly close and delete the syslog indices on time. I have marked the script as answer, but it was a combination with the update from 2.4.10 to 2.4.30; the reindexing script; the added action in Curator.

I will close the discussion as soon as the first indices are being closed if that is ok?
Thank you very much for the help @weslambert

[Maurice-De]

Thank you very much for the help, the first index is moved to the cold phase, so I think the deletion will also work.