logs older than a day are Gzipped and not searchable in UI. Can I make them searchable again?

[Bal33p]

Standalone
2.3.91

Hello,
When I search in our UI for anything older than a day I do not return any information
We have 8TB assigned for this standalone SO box.
I found that in the /nsm dir subfolder like Zeek or Wahzu all our logs are gzipped
is there a way to reimport these logs for a specific time period?

[DebianGuru]

You say that all things over than a day or so disappeared? Sounds like something is cleaning up your data from ElasticSearch. I believe that I'd check the log_size_limit variable in your .sls config (in my case it's /opt/so/saltstack/local/pillar/minions/securityonion_standalone.sls because I have the default machine name in a standalone setup). Anyway, that is the size (In MB). It's in the documentation https://docs.securityonion.net/en/2.3/elasticsearch.html#pillar-files . I'd also check the settings for warm, close, and delete indices time in your global.sls file.
Sorry, I don't know how to re-import them. I'm just a regular schmoe. Fortunately, the SO guys on here usually answer questions each morning (I guess that's the time they lay aside) and they're awesome.

[Bal33p]

Thank you, This is helpfull

[dougburks]

Filebeat should be monitoring /nsm/import/*/zeek/logs/, so you might be able to do something like:

sudo mkdir -p /nsm/import/20220325/zeek/logs/

and then place the unzipped Zeek logs there.

[Bal33p]

Thanks, @dougburks
Does this only work for Zeek? how about Wazuh or say my checkpoint logs that are brought in from the checkpoint module?
Is there a doc on this?

[dougburks]

This functionality works for Zeek and Suricata because we built this for our Import Node. You could probably modify your Filebeat configuration to do something similar for Wazuh or other logs.

[cyb3rz3us]

I know this is probably obvious feedback but still curious, were any changes made to your logrotate config? The behavior you describe matches exactly what logrotate will do when compression is enabled. The original files will be gone and replaced with a version that has a .gz (or similar) extension.

[Bal33p]

I did not make any changes to that, where do I find that setting? Regards, Phillip Samson | I.T. Security Architect GISF, GSEC The only time success comes before work is in the dictionary!

…

On Mar 27, 2022, at 4:01 PM, cyb3rz3us ***@***.***> wrote:  I know this is probably obvious feedback but still curious, were any changes made to your logrotate config? The behavior you describe matches exactly what logrotate will do when compression sis enabled. The original files wil be gone and replaced with a version that has a .gz (or similar) extension. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[cyb3rz3us]

I don't wanna lead you down a rabbit hole so if you haven't made any changes to logrotate, then I'm pretty sure it's not your issue so please don't allow my question to distract you. Nonetheless, to understand more about logrotate, check the manpage and also the files at /etc/logrotate.conf & /etc/logrotate.d. You will also want to understand cron & anacron.