Elasticsearch Status Fault and Stenographer 90%+ packet loss after 2.4.141 upgrade #14505
-
Version2.4.140 Installation MethodSecurity Onion ISO image Descriptionupgrading Installation TypeStandalone Locationon-prem with Internet access Hardware SpecsExceeds minimum requirements CPU32 RAM128 Storage for /32 TB Storage for /nsm28 TB Network Traffic Collectionspan port Network Traffic SpeedsLess than 1Gbps StatusYes, all services on all nodes are running OK Salt StatusYes, there are salt failures (please provide detail below) LogsYes, there are additional clues in /opt/so/log/ (please provide detail below) DetailSince upgrading to Security Onion 2.4.141, my standalone installation maintains an Elasticsearch Status of Fault and Stenographer consistently has over 90% packet loss. so-status shows all containers are running OK. There is one fault out of 711 that shows when running a high state. /opt/so/log/salt/master shows the following error: Name: /usr/sbin/so-elasticsearch-templates-load - Function: cmd.run - Result: Failed - Started: 14:24:57.622941 - Duration: 20014.427 ms Is anyone else experiencing this problem? Guidelines
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 17 replies
-
|
Do you see errors in the |
Beta Was this translation helpful? Give feedback.
-
|
None that I can see. This is the /root/soup.log output: sudo cat /root/soup.log Preparing soup at Thu Apr 3 06:03:51 PM UTC 2025Checking if we can access the salt master and that it is ready at: Apr 03 18:03: 51 This is a manager, so we can proceed. Salt Master does not need unlocked. The manager's pillars can be rendered. We can proceed with SOUP. Checking to see if this is an airgap install. Found that Security Onion 2.4.141 is currently installed. Cloning Security Onion github repo into /tmp/sogh/securityonion. |
Beta Was this translation helpful? Give feedback.
-
|
I was able to figure out the Elasticsearch Status Fault issue. I had reached the 1000/1000 shard maximum. After cleaning up my indices and shards, the Fault issue has been corrected. However I am still having problems with over 90% Stenographer Packet Loss as well as 30%-40% I/O Wait times. I did not experience any of these prior to the 2.4.141 upgrade. How would I go about troubleshooting this particular matter? |
Beta Was this translation helpful? Give feedback.
-
|
How much traffic throughput on the monitoring interface? What type of disk for |
Beta Was this translation helpful? Give feedback.
-
|
Typically around 300-400 mbps on average. I have 8 4TB Samsung 870 SSDs in a RAID 5 config. No red/malfunctioning drive lights. I am sniffing a 1Gbps SPAN port with a 10GB SFP+ fiber connected network card. Even when I switch from Stenographer to Suricata for PCAP, I still get the excessive I/O wait times, as well as high packet loss. |
Beta Was this translation helpful? Give feedback.
-
|
What about |
Beta Was this translation helpful? Give feedback.
-
|
top shows fluctuations between the elasticsearch and logstash processes mostly: iostat shows: Here's a screenshot of my Grid while checking the above top and iostat. I had to unplug the SPAN sniff because Redis had millions in the queue that wasn't being processed: |
Beta Was this translation helpful? Give feedback.
-
|
These are current screenshots of Grid, top, and iostat after I plugged the SPAN sniff back in and rebooted the server at Grid request. I changed pcap to Transition to rule out Stenographer write problems, but Suricata is having the same packet loss problem:
|
Beta Was this translation helpful? Give feedback.
-
|
I am beginning to suspect this issue may be related to #14545, which is a log ingestion problem related to a template failure load. I am experiencing the same error during a high state call. |
Beta Was this translation helpful? Give feedback.
-
|
I just checked /opt/so/log/stenographer/stenographer.log, and I keep seeing this error message repeated over and over: 2025/04/28 15:30:15 Blockfile opening: "/tmp/stenographer2713390978/PKT0/1741182252539200" Is this indicative of anything? |
Beta Was this translation helpful? Give feedback.
-
|
What is your diskfreepercentage set to for Steno? Also, are there any errors in that log? What does this show? |
Beta Was this translation helpful? Give feedback.
-
Diskfreepercentage is back at 21 after I had it at 25 trying to troubleshoot these problems. The Steno logs keeps reporting this same error over and over:
And also reports this stats update:
This is what 'sudo du -hd1 /nsm` reports as of 4/29/2025. I am currently using Stenographer for packet capture. I tested out Suricata with the Transition pcap setting for a time: 45G /nsm/docker-registry |
Beta Was this translation helpful? Give feedback.
-
|
Wonder if |
Beta Was this translation helpful? Give feedback.
-
Doing this got rid of the error message in the Steno logs but I still have high Steno packet loss. This is the weirdest thing. Zeek is collecting data just fine, no packet loss. Packet loss only happens with Stenographer or Suricata packet capture. |
Beta Was this translation helpful? Give feedback.
-
|
I think I finally figured out the Stenographer issue. I believe there was some seasonal internal traffic spikes that I was not accounting for. We'll find out on the next go round lol smh. Thank you for all of your help!! |
Beta Was this translation helpful? Give feedback.
I think I finally figured out the Stenographer issue. I believe there was some seasonal internal traffic spikes that I was not accounting for. We'll find out on the next go round lol smh.
Thank you for all of your help!!