Real Time Logs drop drastically after 9am EST but look normal the next day after upgrade

[Bal33p]

Version

2.4.20

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

Description

upgrading

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

64gig

Storage for /

500gig

Storage for /nsm

8tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

est

Every day when I login at 7 am everything looks normal then promptly at 9 am since the 9.4.20 upgrade logging tanks!
but when I come back the next day and look at the 24 hour trend it looks normal

I don't understand why during the day it looks like nothing is happening. Does anyone have any thoughts on this?

13:43est

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[jaypfe]

Does the SOC behave the same way?

[Bal33p]

I don't know
I don't usually look in there as much

This is the status this morning

same things is happening

[weslambert]

Version

2.4.20

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

Description

upgrading

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

64gig

Storage for /

500gig

Storage for /nsm

8tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

est

Every day when I login at 7 am everything looks normal then promptly at 9 am since the 9.4.20 upgrade logging tanks! but when I come back the next day and look at the 24 hour trend it looks normal

I don't understand why during the day it looks like nothing is happening. Does anyone have any thoughts on this?

13:43est

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

Have you checked the Redis queue?

It might be possible that traffic volume is higher during the day, and logs could be getting backed up and not processed until later in the day.

[Bal33p]

first thing I checked

Is there something in the config that would cause this type of behavior

[Bal33p]

Same today. Is it possible the forward node is queuing these up or something?

[Bal33p]

@weslambert
Any thoughts or next steps

[Bal33p]

Complete mystery. I have millions of logs that will be here tomorrow but my Redis queue is at 0 . What am I missing
Its as if the forward node is not sending the logs in
. How do I test for that??

[weslambert]

Are the date and time correct on each machine?