Request failed with code 500 on dashboard, user

[ByteAndBits]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

512

Storage for /

15TB

Storage for /nsm

30TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,

since two days we get

Request failed with status code 500

on Dashboard, when we create a new user and sometimes in alerts. Dashboard works if we set timespan to 15min. All over the timespan the error immediately comes.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[dougburks]

Since you have a distributed deployment, we'll need more information about your manager node and each of your search nodes:

• detailed hardware specs for each
• storage type (NVME, SSD, rotational)
• check for additional clues in /opt/so/log/

[ByteAndBits]

Hi,

Hypervisor:
Proxmox: 64CPUs Intel GOLD, 512GB RAM, 2xRAID10 (15TB), 1x RAID5 (15TB), Raidcontroler (no SAN), All Discs Datacenter SSDs

VMs:
Manager: 12CPUs, 64GB RAM, 1x 500GB System, 1x 7TB (NSM) Raid5, 1xNIC
Searchnode1: 16CPUs, 128GB RAM, 1x500GB System, 1x 15TB NSM (Raid10)
Searchnode1: 16CPUs, 128GB RAM, 1x500GB System, 1x 15TB NSM (Raid10)
Forwarder: 12CPUs, 64GBRAM, 1x500GB System, 1x7TB NSM, 1x1GB NIC, 2x10GE LWL NIC
Receiver: 6CPUs, 16GB RAM, 1x 500GB for all
Fleed: 8CPUs, 16GB RAM, 1x 500GB for all

logs I first must look in.

[dougburks]

Are you saying that all of these VMs are on the same proxmox host? If so, you may be experiencing resource contention.

Go to SOC Grid:
https://docs.securityonion.net/en/2.4/grid.html

Then expand each node and click the InfluxDB button in the lower left of the Node Status section. Once InfluxDB appears, check the system health metrics and in particular the IO Wait.