SOC webpage is extremely slow

[ccpaasmyser]

I have a freshly loaded security onion instance on top of a Cent OS 7 install. The issue i have, is navigating the web interface for the SOC between the alerts, dashboard, hunt, any of those things takes literal minutes to load anything. If i refresh or change the filtering in the alerts menu, literally takes minutes. According to Grafana, nothing seems pegged and in fact it seems to be running great. but I'm not seeing it on my end.

Any help would be greatly appreciated on seeing where this thing isn't running right. Unless this is normal behavior?

Also to note, we stopped the monitoring to see if that was slowing down the server/webpage and that didn't seem to make a bit of difference on processing.

Edit: So it seems its the main webpage. The tools that you can open on the left hand side seem to load and function correctly and seem to be responsive enough currently.

[isaacgolding]

They need to know your system specs and how your SO is setup to start.

Please include the following details where you can:

Security Onion version as seen in the lower left corner of SOC and in /etc/soversion. For example: 2.3.140

• Is this a cloud deployment or on-prem? If on-prem, do you have Internet access or this an airgap installation?
• Did you install from our Security Onion ISO image or did you perform a network installation?
• If network installation, did you install on CentOS 7 or Ubuntu?
• How many nodes do you have?
• What are the hardware specs of each of those nodes?
• How are each of those nodes configured? (ex. manager with 2 search nodes and 3 forward nodes)
• Are you experiencing issues monitoring network traffic? If so, are you sniffing from a tap or span port and what is the traffic volume?
• Does so-status show all services running?
• Do you get any failures when you run sudo salt-call state.highstate?
• Does the SOC Grid page show any failures?
• Explain your issue. For example: Installation fails when I select this series of options...
• Provide applicable logs. If you are having problems right after setup, provide /root/sosetup.log. If you are having problems during soup, provide /root/soup.log. If you are having problems with a specific component, provide that component's logs from /opt/so/log/.
• If you need to attach files, please make sure they are plain text format. No Word docs or PDFs please.

[ccpaasmyser]

Security Onion Version - 2.3.150

Hardware Specs:
Ram: 128 GB - (8x - 16GB RDIMM, 3200MT/s, Dual Rank)
CPU: 2x Intel Xeon Silver 4216 2.1G, 16C/32T, 9.6GT/s, 22M Cache, Turbo, HT (100W) DDR4-2400
Space - 72TB RAID 10 (12x 12TB 7.2K RPM SATA 6Gbps 512e 3.5in Hot-plug Hard Drive)

Is this a cloud deployment or on-prem? If on-prem, do you have Internet access or this an airgap installation? - Not a cloud deployment - does have internet access directly

Did you install from our Security Onion ISO image or did you perform a network installation? - Installed CentOS 7 then followed the instructions on docs.security onion to install on top of that.

If network installation, did you install on CentOS 7 or Ubuntu? - Cent OS 7

How many nodes do you have? - 1 Standalone Install

What are the hardware specs of each of those nodes? - Listed Above

Are you experiencing issues monitoring network traffic? If so, are you sniffing from a tap or span port and what is the traffic volume? - Span Port - Mirrored port (We did try stopping the traffic to security onion to see if that was overloading it, didn't seem to matter)

Does so-status show all services running? - All green OK Status
Does the SOC Grid page show any failures? As far as i can tell there are no errors and everything is under 50% utilized

[dougburks]

What browser are you using? From https://docs.securityonion.net/en/2.3/soc.html:

We recommend chromium or chromium-based browsers such as Google Chrome. Other browsers may work, but fully updated chromium-based browsers provide the best compatibility.

Do you have any plugins installed in that browser?

Have you tried incognito mode?

Have you tried a different browser?

Have you tried accessing from a different computer?

[ccpaasmyser]

Tried Chrome and Edge
Chrome Version: 104.0.5112.102
I've tried chrome versions with zero plugins, the only plugin i have is a google docs offline extension
Didn't think to try incognito - no luck, still took just a long
I've had my manager try it from his computer as well with the same issues

[dougburks]

Which specific SOC pages are slow? Is it just the pages that do searches (Alerts, Dashboards, Hunt, Cases) or is it other pages like Overview, Grid, Downloads, Administration?

If it's just the pages that do searches, what kind of time frame are you searching? Have you tried searching a smaller time frame?

For Dashboards and Hunt, there is a metric at the bottom of the window that shows how long the search took. Here is an example from one of my deployments that shows that it's less than a second for me:

The backend data fetch took 0.609 seconds. The total round trip took 0.619 seconds.

Are you able to share an example like that?

[ccpaasmyser]

Overview and downloads runs just fine. Administration takes forever to load the 2 users i have in the system. Alerts, Dashboards, Hunt, PCAP, Grid all take multiple minutes to load up.

I went from Users to Dashboard and this is the result: it still took about 2 minutes and some leftover seconds to load with this at the bottom:

The backend data fetch took 0.047 seconds. The total round trip took 0.075 seconds.

I went to Alerts: Currently set to an interval of 24 hours. It took a couple minutes, I changed the timeframe to the last minute. It took its time loading, then I went back to Dashboard. It still took multiple minutes to load, I then realized the setting doesn't follow across pages and reset the interval to 1 minute/s to see how fast it loads. Still too about the same amount of time from what I'm seeing. Bottom of the page reads:

The backend data fetch took 0.044 seconds. The total round trip took 0.543 seconds.

edit: more precise information put above.

[dougburks]

Overview and downloads runs just fine. Administration takes forever to load the 2 users i have in the system. Alerts, Dashboards, Hunt, PCAP, Grid all take multiple minutes to load up.

This is unusual and I don't believe we've had any other reports like this.

What kind of network connectivity do you have between your browser and the Security Onion manager?

Are there any firewalls, proxy servers, or any other network devices that could be interfering with the traffic?

Do you have any endpoint agent on your browser machine that could be interfering?

[ccpaasmyser]

That's interesting. So I feel a bit foolish that I didn't check that before this point. It would appear that it is our endpoint protections Website scanning AND the Endpoint IPS causing it to take forever to load. I've turned these things completely off to see if it would work fine and it does. So I just need to get it whitelisted for my machine and it should work just fine.

Note to self - Check A/V and other Endpoint Protections