Troubleshooting Disk Space Issues

[BuffaloWill]

Hi!

The /nsm directory of our manager regularly hits 100% disk space used (~2x a month). I am not sure what the root cause is and I was hoping for some help troubleshooting. Thank you!

du -hd1 /nsm
86G	/nsm/docker-registry
217M	/nsm/elasticsearch
295M	/nsm/mysql
20K	/nsm/logstash
727M	/nsm/backup
4.0K	/nsm/pcapout
16K	/nsm/lost+found
4.0K	/nsm/import
95G	/nsm/wazuh
8.0K	/nsm/osquery
4.0K	/nsm/pcap
8.0M	/nsm/grafana
16G	/nsm/influxdb
41M	/nsm/thehive
84K	/nsm/soc
197G	/nsm

# df -h

Filesystem                    Size  Used Avail Use% Mounted on
udev                          9.8G     0  9.8G   0% /dev
tmpfs                         4.0G  2.8M  4.0G   1% /run
/dev/mapper/so-vg-root   72G   51G   18G  75% /
/dev/mapper/so--vg-nsm   271G  197G   60G  77% /nsm
tmpfs                         9.8G  8.0K  9.8G   1% /tmp

/opt/so/saltstack/local/pillar/minions/so_manager.sls

...
log_size_limit: 216
...

[InfosecGoon]

Good morning, Will!

It looks like you probably have a bunch of old Docker images from past upgrades in /nsm/docker-registry; if you run "docker images", does it go back several versions? You can clear those out with "sudo so-docker-prune".

You're using a lot of storage in /nsm/wazuh - you might want to consider deleting some of the old logs in /nsm/wazuh/logs/archives or migrating them off to long-term storage.

The log_size_limit is the point at which Curator will start deleting old indices to free up space, so it will kick in at 216GB of Elasticsearch data -- but with that much space being taken up by old images and Wazuh logs, it will never get there and the volume will just fill up.

Hope that helps!

--Matt

[BuffaloWill]

@InfosecGoon thank you! And thanks for the great explanation with curator and suggestions for wazuh. All helpful knowledge.

I cleaned up older images (i.e. .140 images) with:

docker images | grep -F ".140" | tr -s ' ' | cut -d' ' -f3 | xargs -I{} docker image rm {}

I also ran sudo so-docker-prune again. However, /nsm/docker-registry looks the same size. In that directory I see older files and components going back to 2021. Any additional suggestions for how to clean up /nsm/docker-registry?

[InfosecGoon]

If you run "sudo docker images", does it list a bunch of older packages? For comparison, my standalone install has .181 and .140 images and that directory is 19GB, so yours seems larger than it should be.

[BuffaloWill]

If you run "sudo docker images", does it list a bunch of older packages?

No. All images are from 2.3.180.

[BuffaloWill]

It does duplicate every image between two registries. I am not sure if that is normal. For example:

$. docker images
...
ghcr.io/security-onion-solutions/so-steno                    2.3.180   5ef4bc307744   6 weeks ago     809MB
hostname:5000/security-onion-solutions/so-steno                2.3.180   5ef4bc307744   6 weeks ago     809MB
ghcr.io/security-onion-solutions/so-elastalert               2.3.180   bdc060705faa   6 weeks ago     334MB
hostname:5000/security-onion-solutions/so-elastalert           2.3.180   bdc060705faa   6 weeks ago     334MB

[BuffaloWill]

Looking into the /nsm/docker-registry more:

# du -hd1 docker/registry/v2/

59M	docker/registry/v2/repositories
86G	docker/registry/v2/blobs
86G	docker/registry/v2/

[InfosecGoon]

Does "sudo docker system prune -a" reduce the size at all? That should remove any images that aren't actively in use.

[BuffaloWill]

No, it only removed 5Gb.

[dougburks]

@BuffaloWill Is this deployment configured for airgap, perhaps?

[BuffaloWill]

Thanks @dougburks
How would I check for this configuration? This system can access the internet.

Going off of the airgapped documentation:

• /nsm/repo/ - no such file or directory
• /nsm/repo/rules/ - no such file or directory

[dougburks]

You may be able to purge old images out of the Docker Registry in /nsm/docker-registry/ (this is separate from the normal docker images and docker system prune -a). However, looking back at your original post, 271GB is not much more than the bare minimum listed at https://docs.securityonion.net/en/2.3/hardware.html. If possible, your best option may be to increase storage.

[BuffaloWill]

Thank you!