High Suricata Packet Loss #12329
High Suricata Packet Loss
#12329
-
Version2.4.40 Installation MethodSecurity Onion ISO image Descriptionother (please provide detail below) Installation TypeDistributed Locationon-prem with Internet access Hardware SpecsMeets minimum requirements CPU6 RAM64 Storage for /500gb Storage for /nsm1tb Network Traffic Collectionspan port Network Traffic SpeedsLess than 1Gbps StatusYes, all services on all nodes are running OK Salt StatusNo, there are no failures LogsNo, there are no additional clues DetailHi all, back for round two! I managed to get Security Onion running on Alma Linux 9.3 (I thought it was fine). However, I came across a really weird problem, which led to me install it via the officially supported ISO. But the exact same thing happens even after the ISO installation. Suricata after around ~10-20 minutes starts losing packets, and then it caps out at 50% on the influxDB dashboard. As can be seen here: The only way to fix this is via a container restart. But then the same thing happens again. Looking at the monitoring interface status on InfluxDB you can also see that none are showing as dropped there. I then checked the stats.log of Suricata and below is the output: Nothing jumps out at me from the log file that would cause Suricata to just get stuck at 50% packet loss. Looking at ifconfig, it only reports 1156 being dropped. I'm using an Intel I350 network card. Am I missing something here? I've tried adjusting the ring-size, max-pending-packets, and workers on Suricata but the same exact problem happens. Looking at the used system resources as well nothing jumps out at me as being wrong. CPU Usage caps out at around 50%, System Memory Usage stays at around 55% Does anyone have any idea on what might be going on here or any other troubleshooting steps I could do? Guidelines
|
Beta Was this translation helpful? Give feedback.
Answered by
technox123
Mar 1, 2024
Replies: 2 comments 2 replies
-
|
Update: So, I have done some more investigation into this issue I seem to be having and I noticed that the "Elastic Ingest TIme Spent" jumps to around 25-30s (I assume seconds) for the "rename_time" variable on the manager in InfluxDB. Is this a normal and safe value to have? The "IO-Wait" time on the sensor also jumps to between 1%-8%. Could this be causing my packet loss in Suricata? My other question would be why is Suricata packet loss capping out at 50% in InfluxDB? If I try to get raw value it just get's stuck at 500M. Is this a hard limit on InfluxDB or with Telegraf? |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
You can check |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply. I've since managed to source a new machine that is 8 cores, has a 1tb nvme and 64gb of ram (same NIC as before though). The IO-Wait has completely dropped down now. I've also updated to version 2.4.50, however, I seem to still be having the same issue of Suricata dropping more and more packets until I restart the container. Any other suggestions for troubleshooting steps I could do? |
Beta Was this translation helpful? Give feedback.
-
|
If anyone else comes across this issue, it turns out the problem was my network card. I swapped to an Intel x710-da2 and all my issues I had with packet loss have disappeared. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
technox123
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If anyone else comes across this issue, it turns out the problem was my network card. I swapped to an Intel x710-da2 and all my issues I had with packet loss have disappeared.