Storage offloading

[B3DTech]

So we want to expand our storage so that we can go from 2 to 4 weeks of pcap storage, and 2 to at least 4 weeks of search data storage. That's the easy part.

Wondering if:

1. there's a good way to offload pcap files to be able to query them later if needed.
2. back up the Elasticsearch data nodes that that we can pull them back in later.

Ideally, we'd like to have 30 days of online pcap and elastic data, and be able to fairly quick pull in previous, backed data to research old incidents.

What are thoughts on best practices around this? This seems like the best balance of expensive, online storage and inexpensive long term storage like S3 Glacier.

Thanks!

[linsecnet]

I am also interested in doing the same thing with our deployment. @B3DTech have you gotten any updates?

[B3DTech]

Unfortunately no. You can use normal elastic tools to back up the indicies and truncate them I guess, and when you do that, automate the migration of the associated time window of pcap data to cloud storage. There is no built in way to do it now though.

[SanibelJack]

Since SO is now at 2.3.200 Is this possible?

[InfosecGoon]

A couple of clarifying questions:

1. How much PCAP data are we talking about? Do you want to offload it in the native steno format or convert it to standard PCAP first?
2. Are your Elasticsearch indices just holding Zeek and Suricata data, or are you bringing in other logs from elsewhere? I've had success in the past with just using a crontab and the aws command line tool to copy raw Zeek and Suri logs up to S3, then lifecycling them out to Glacier from there.

[Acewiza]

Storage is relatively cheap.

[isaacgolding]

So I think the answer you are looking for is. "No". there isn't any "good" way to backup the pcap and elastic data. However this is/was a topic I spent some time on in the past and IMO the smartest thing to do would be to have a cronjob script to export the pcacp data to another mounted drive, NFS or other external share to and then store at that remote location. PCAPs can always be pulled back into the system manually and re-run and queried or even just pushed through an analyst workstation at a later time.