Ephermeral Storage for AWS Sensor Nodes

[DoughnutPeach]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

16

Storage for /

500GB

Storage for /nsm

1TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hi

I have a Sensor Node running in AWS. In addition to our 500GB root volume, we have allocated an 1TB Ephemeral Storage (EBS) dedicated for /nsm.

I ran through the setup as per the documentation, but somehow my /nsm is not mapped to the right partition. Instead, it's using my 500GB root partition.

df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs            62G  8.0K   62G   1% /dev/shm
tmpfs            25G  2.5G   23G  10% /run
/dev/nvme0n1p1  296G  251G   33G  89% /
tmpfs            13G     0   13G   0% /run/user/939
tmpfs            13G     0   13G   0% /run/user/307223616
tmpfs            13G     0   13G   0% /run/user/0

Can I check how should I set it up such that /nsm can be mapped to my additional 1TB EBS?

Do I need to first do a sudo so-prepare-fs /dev/nvme1n1n1 /nsm ?

Furthermore, my setup has been running some time now.. Will I need to destroy the instance or can I just run so-setup-network again?

Thanks a lot for any help! Really appreciate it!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[DoughnutPeach]

Still unable to fix this. Anyone with experience able to help?

Thanks!

[DoughnutPeach]

Hey @cm-ops, thanks for checking this.

It says

For distributed search nodes, or an evaluation node if using ephemeral storage

So I didn’t follow this as I was setting up a distributed Sensor Node.

Should I have done this?

And given that my node has already been set up, do I now need to do a sudo so-prepare-fs /dev/nvme1n1n1 /nsm, then run so-setup-network again?

Thanks alot for your reply!

[cm-ops]

Sorry glossed over the sensor piece, looking at the script you should be able to overrirde the defaults if you pass them as arguements, so in your case sudo so-prepare-fs /dev/nvme1n1n1 /nsm I have not tested doing this on a node that is already set up though.

[DoughnutPeach]

Gotcha, thanks much for your inputs @cm-ops !

So I have tested the following steps and it seems to be working:

1. so-stop all services
2. Make a backup of /nsm (i.e. mv /nsm /nsm.bak)
3. so-prepare-fs /dev/nvm1... /nsm
4. rsync -a /nsm.bak /nsm to copy back the files and retain ownerships
5. so-start all services.

All seems to be running good for the past few days.

However, now I'm getting an error from the SOC GUI. When I go to my SOC Console > Administration, I get an Error that reads

Request failed with status code 400
See the Help section of the Security Onion documentation for additional troubleshooting guidance. 
The Hunt interface may contain additional log messages to further diagnose the issue

Initially, I see the following Errors in Hunt for /nsm/pcap.
i.e. the error

{"fields":{"error":"open /nsm/pcap: permission denied","path":"/nsm/pcap"},"level":"error","timestamp":"2024-03-11T08:52:27.235252203Z","message":"Unable to access path while updating epoch"}

I have since fixed the permissions on /nsm/pcap and the Errors in Hunt have gone away. But I'm still getting the HTTP 400 Error from SOC GUI.

drwxr-xr-x.  13 root root   226 Mar  7 04:44 nsm
drwxr-xr-x. 2 stenographer stenographer 196608 Mar 11 09:15 pcap

Any ideas how I can troubleshoot a HTTP 400 Error on SOC GUI? Thanks!

[DoughnutPeach]

Hey @cm-ops, managed to resolved the HTTP 400 Error issue. I've detailed the steps I took here.

Thanks for all your help!

TLDR

All Ephemeral nodes require you to first run so-prepare-fs

If you unfortunately did not, you can use these steps to apply it post-setup.

1. so-stop all services
2. Make a backup of /nsm (i.e. mv /nsm /nsm.bak)
3. so-prepare-fs /dev/nvm1... /nsm
4. rsync -a /nsm.bak /nsm to copy back the files and retain ownerships
5. so-start all services.