Need 2nd node as "Everything but Manager", how to do this?

[kzurell]

• Security Onion version as seen in the lower left corner of SOC and in /etc/soversion. For example: 2.4.10
  

2.4.110

• Is this a cloud deployment or on-prem? If on-prem, do you have Internet access or this an airgap installation?
  

On-premises, LAN-connected.

• Did you install from our Security Onion ISO image or did you perform a network installation?
  

ISO

• How many nodes do you have?
  

1, need 2

• What are the hardware specs of each of those nodes?
  

Not enough.

• How are each of those nodes configured? (ex. manager with 2 search nodes and 3 forward nodes)
  

1st as Standalone, need 2nd as "everything but Manager".

• Are you experiencing issues monitoring network traffic? If so, are you sniffing from a tap or span port and what is the traffic volume?
  

Syslog messages to 2nd IP are being dropped by IPTables.

• Does so-status show all services running?
  

On Manager, yes. On 2nd node when configured as Heavy, so-logstash is "missing".

• Does the SOC Grid page show any failures?
  

On 2nd node when configured as Heavy, so-logstash is "missing".
After reinstall as Sensor/Forward, Detections has a persistent orange exclamation point, but gives a red 500 error when clicked.

• Explain your issue. For example: Installation fails when I select this series of options...
  

2nd node does not consume all syslog messages sent to it. Was able to enable so-logstash, but it went "missing".

Our Standalone node is at capacity, esp. CPU high, Memory high, & I/O wait is in red. We have a spare machine and IP from a previous security system, that already receives syslog traffic from a number of senders.

We unwisely disregarded the doc advice and installed a Heavy node, and changed Configuration to allow receipt of traffic on syslog ports. Eventually we did get logstash to appear but never seemed to work, with lots of "IPTables-dropped" messages in /var/log/messages. EPS remained 0, though that might be a known bug. Reinstalled as Forward/Sensor node.

Still learning the architecture as documented, though we need an "everything but Manager/Kibana" node that can shoulder some of the workload from our Standalone. In particular, the documentation about logstash is confusing, seeming to conflate logs between SO components and external sources needing analysis.

Thanks for any advice.

[dougburks]

What are the hardware specs of each of those nodes?
Not enough.

What are the actual specs?

If your machines don't meet the minimum requirements, then there is not much we can do to help:
https://docs.securityonion.net/en/2.4/hardware.html

We unwisely disregarded the doc advice and installed a Heavy node

It's important to follow the advice in the documentation. What exactly are you hoping that a heavy node might help with?

What kinds of data do you want to monitor?

Do you want to monitor network traffic from a tap or span port? Or are you just collecting syslog?

[dougburks]

Having worked as a tech writer, I am well aware that nobody reads the documentation unless as a last resort.

Yeah, we do have a warning in the installer itself that heavy nodes are NOT recommended for most users:

The advice against Heavy nodes is not convincing, it just hangs out there & doesn't overcome the consideration to choose them. If they're deprecated, I can't see why.

Heavy nodes are not deprecated but they are only for very specific use cases which do not apply to most users. There's an orange warning banner in the documentation that says:

Heavy nodes are NOT recommended for most users

Please note I'm not trying to argue or beat you up about this. I'm genuinely trying to understand how we can better communicate this. I'm open for suggestions.

Cycles. We have a Standalone, so we already have a (previously working) manager. Beyond that we need "more standalone". We have one additional machine to make best use of.

I asked previously but I don't see an answer, so I'll ask again:
What are the actual specs of your machines?

We'll need that information to help you make the most of your two machines.

"Just" syslog to start, but rolling out is a process, we might add other things via integrations.

If you're just consuming syslog and you're not consuming network traffic from a tap or span port, then you probably don't want standalone, heavy node, or forward (sensor) node as all 3 of these are designed for analyzing network traffic from a tap or span port.

Without more information about the specs of your machines, I think I'd take the more powerful machine and install it as a ManagerSearch and see how that works for consuming syslog. Since it is not a Standalone it won't be wasting any CPU cycles running Suricata or Zeek to analyze network traffic that isn't there.

Another option might be for one machine to be just a Manager and the other machine to just be a search node but again we don't know enough about your machines or about the amount of syslog that you're consuming to make an educated recommendation.

[kzurell]

Specs:

Standalone: Lenovo Core i7 6700 3.4GHz, 48G DDR3 RAM, 256K SSD, 4TB nsm storage, onboard I219LM & 82752 PCIe NICs

New machine: Lenovo Core i7 6700 3.4GHz, 32G DDR3, 256K SSD, 3TB nsm storage, onboard I219LM & 82751 PCIe NICs

Re: amount of syslog, again, we're probably low-throughput from your POV, but we have six buildings w/ firewalls, so lots of things to listen to if we choose.

The Standalone is a little pokey when accessing the web interface.

[dougburks]

256K SSD

Is that supposed to be a 256GB SSD?

4TB nsm storage

Is that SSD, NVME, or spinny?

If spinny, please note this warning at https://docs.securityonion.net/en/2.4/hardware.html#elastic-stack:

We recommend placing all Elastic storage (/nsm/elasticsearch) on SSD or fast spinning disk in a RAID 10 configuration.

Since you say that you are low throughput and it seems like you're just consuming syslog and not analyzing network traffic from a tap or span port, then here's what I would try if I were you:

• wipe the current standalone and install it as a ManagerSearch (https://docs.securityonion.net/en/2.4/architecture.html#manager-search)
• monitor that for a while and see if you have better performance than you had in Standalone configuration
• if the amount of syslog is overwhelming the ManagerSearch, then you could wipe the "new machine" and join it to the ManagerSearch as a search node (https://docs.securityonion.net/en/2.4/architecture.html#search-node)
• this will give you 2 Elastic nodes joined together in an Elastic cluster so that should increase your index and search speeds compared to what you have now

[kzurell]

256K SSD

Is that supposed to be a 256GB SSD?

I used to work in embedded computing, & I'm old. :)

4TB nsm storage

Is that SSD, NVME, or spinny?

Spinny. Large SSDs are still a little thin on the ground in our world (refurbished).

... on SSD or fast spinning disk in a RAID 10 configuration.

I'm certain I did read that at one point. We go with what we've got...

Since you say that you are low throughput and it seems like you're just consuming syslog and not analyzing network traffic from a tap or span port,

The Standalone does take from a span port, but it could be either node. There might be a case for two spans but that's more arcane than I want to be on a Friday.

• wipe the current standalone and install it as a ManagerSearch

I didn't quite get the fine distinction, I'm guessing ManagerSearch is the closest parallel to Standalone for the distributed scenario.

• if the amount of syslog is overwhelming the ManagerSearch

We have two IPs receiving syslog, for reasons reasons, that's the impetus for the original question, wanting another node to take in syslog (and potentially more span) and shoulder some of the work.

• this will give you 2 Elastic nodes joined together in an Elastic cluster so that should increase your index and search speeds compared to what you have now

Thanks for the consult. I was afraid of getting bogged down in hardware discussion when I'm focused on understanding the configuration details. Also, I figured out why Alerts/Hunt/Detections weren't working: removing the Heavy node from the grid did not remove it from the Elasticsearch _cluster/settings persistent list.

I'll start building consensus w/ team on reinstalling the Standalone to become a ManagerSearch. But that means reinstalling the agent on my endpoints, doesn't it?

I don't have any brainwaves on how to characterize Heavy nodes any better than you have, other than to reiterate that it's a very tempting configuration option. It ticks more potential boxes, and some reading shows I'm not the only person who thinks that way. Maybe the best way to approach it is to explicitly guide the Standalone -> ManagerSearch migration, so that the inevitable second node comes from Search or Forward as a first preference.

Thanks again.

[dougburks]

The Standalone does take from a span port, but it could be either node. There might be a case for two spans but that's more arcane than I want to be on a Friday.

I asked before if you were analyzing network traffic from a span port and it sounded like you were just consuming syslog, so that's the direction that my recommendations went. ManagerSearch and search node do not analyze network traffic from a span port at all. You would need a third node installed as a forward node to analyze network traffic from a span port. If you don't have a third node but do indeed need to analyze network traffic from a span port, then you would need a different architecture.

I'll start building consensus w/ team on reinstalling the Standalone to become a ManagerSearch. But that means reinstalling the agent on my endpoints, doesn't it?

Yes. Depending on how many agents you have deployed, it may be worth the time to investigate your Standalone installation further to see if it can be tuned without wiping and reinstalling. You previously characterized your syslog traffic as low throughput but if you're now saying that you are actively analyzing network traffic from a span, then you might need to review how much network traffic is coming from that span port as that could cause significant performance issues on your Standalone. For example, there's a big difference in the hardware required for 100Mbps span traffic compared to 1Gbps and there's a huge difference in hardware required when going from 1Gbps to 10Gbps. For more information, please see:
https://docs.securityonion.net/en/2.4/hardware.html#sensor-hardware-considerations