SecurityOnion over Kubernetes

[l-rossetti]

Since SO is already a container based solution, is it in your plans to provide support for native installation over Kubernetes? Is there an ETA for that?
Thanks

[x86dx2]

I'll fork using k3s or harvester

[TOoSmOotH]

We have had several requests to make Security Onion work on Kubernetes over the past year or 2 and although certain parts of SO will work fine under it, there are some major hurdles technical and not technical that would need to be overcome.

On the non technical side is what benefit does a defender get by running this on K8s? Security Onion is "For Defenders by Defenders" so the goal is to make it as simple as possible to stand up an environment and start finding evil. We try and remove the administrative burden as much as possible. This is why in 2.4 we use a web interface to get users to stop having to write yaml so they can concentrate on the important things. The user would need to have a firm grasp of Kubernetes in order to do the install unless you handled that setup through the ISO as well and that adds complexity.

I have heard "auto scaling" as a benefit of Kubernetes but only certain parts of the infrastructure could benefit from that. We have users who want to run SO in shared VM environment then once they start throwing lots of data on to it they end up getting kicked off the environment due to IO usage. Sensors use a lot of sequential writes as well causing issues to shared environments. I have seen instances of nodes being booted off of SANs because of the IO usage. Our Enterprise Appliances have dedicated NVME for that reason.

So with the above in mind what is Kubertees getting me?

[l-rossetti]

@TOoSmOotH thank you for answering, let me try to elaborate on that.

On the non technical side is what benefit does a defender get by running this on K8s?

It would be beneficial to have it on Kubernetes if for example someone wants to monitor the network traffic of the pods over the K8s nodes. This actually is our use case and it is the use case of the hyper-converged infrastructure examples provided by @x86dx2 (either through harverster or kubervirt).
This is how we are dealing with that even though we are not satisfied at all: each pods' network interface traffic is collected through OpenVSwitch via OVS-cni plugin port mirroring feature. But in order to diagnose the traffic we had to provide a "network consumer" interface to SecurityOnion and so we had to run the SO VM onto Kubernetes by executing it into a container that is running a QEMU process.
Our outcome is that SO suite does surely fit our needs in terms of functional requirements, but it doesn't really fit our architecture: the stack presents too many layers cause we have a machine running k8s that is running a container that is executing a SO VM through a QEMU process. And the SO VM is running its components through docker again.
Another solution would be to bring the pods traffic out of Kubernetes and let an external SO VM deal with it, but in my opinion it would be an overkill for network performances cause the traffic can't be locally diagnosed.
Moreover one of the benefits of hyper-converged infrastructure is that you can manage all your infra through a single stack which in this case is kubernetes, so we don't really want to have SO outside of it. This makes things simpler when you want to recreate the infra cause all its components can be installed via a single tooling (e.g. Helm).

The user would need to have a firm grasp of Kubernetes in order to do the install unless you handled that setup through the ISO as well and that adds complexity.

Current SO VM installation is sufficiently simple, that's true, but I don't think it would be so much harder for someone to run it on Kubernetes. For sure you need some k8s knowledge but you have a similar problem also with the current SO VM if someone doesn't know how to deal with virtual machines and their networking. The installation may be done via a Kubernetes Operator, the same way other complex projects are doing (e.g. Kafka, Elastic, InnoDB, etc). The web interface may still be used to configure it, but having also the possibility to define an initial YAML configuration would allow SysAdmins to deploy it conveniently.

Sensors use a lot of sequential writes as well causing issues to shared environments. I have seen instances of nodes being booted off of SANs because of the IO usage.

That's true, IO would take a hit, but this is not different with respect to a pub-sub system (e.g. Kafka), a DB, Elastic stack or Prometheus. All of them may run very smoothly on K8s given that you provide sufficient HW resources and configure these systems in the correct way.

Finally, I'll give you my opinion about what stated by @x86dx2. If I correctly got it, he suggests to release it as a VM which is running Kubernetes and SO inside it. That's ok, cause basically you can transform the current installation from a VM running Docker to a VM running k8s. That installation would be as simple as currently is, and people won't really need k8s knowledge. But In my opinion you should also provide a way to install it natively on an already existing k8s cluster so that people can decide if they want to go the simple bundled way or the harder one. Moreover, to keep things a bit lighter I would go with K3S instead of Harvester.

[x86dx2]

@l-rossetti , thank you for enriching the discussion. You understood my point. That's why I suggested having different flavors of Security Onion. I recommended Harvester because it addresses architectural needs for on-premises solutions. This way, it would be very easy to install it on bare metal and have all high availability issues resolved. However, since Harvester has k3s at its core, it would be straightforward to create a fork and offer another flavor for Kubernetes. This is what I had in mind when I suggested Harvester and k3s.
I personally love the idea that Security Onion offers, and that's why I've been thinking about how to adapt it for more robust technologies.

[l-rossetti]

Got it, Harvester may help with HA as well. In that case that's a good solution for the bundled release! I'm not sure there's an opensource alternative to SO, neither a system with these features that fits on top of K8s.
SecurityOnion over Kubernetes may bring the product to a whole upper level that would help it to further expand!

[x86dx2]

That's my understanding as well. I realize there will be challenges in trying to build this, but I believe an open-source product with the Security Onion's concept and all these combined features would elevate the project to an incredibly high level of maturity.

[jpvlsmv]

I'd describe this "Running on Kubernetes" as an alternative option to the current LXD-based container architecture and orchestration via saltstack.

One (simplistic) way to approach this would be to run a full-VM virtualization as a Kubernetes workload (harvester is one full-VM stack that could support this), with no changes to the underlying saltproject structure. This would be less-than-optimal due to virtualization overhead and relative inflexibility of what gets deployed on the Search Node and/or Heavy Node.

A more flexible/dynamic approach would switch the per-component LXD containers for Kubernetes workloads, preferably with "autoscaling" support. This could leverage already-existing, published Kubernetes resources for running (for example) the Redis service, ensuring that it is available to the other nodes with auto-failover.

"Ideally", you could take a pile of hardware, configure it to run Kubernetes (or rent Infrastructure as a Service or Kubernetes as a Service), and deploy a fully-distributed Security Onion environment from a single configuration file. Kubernetes would automatically expand compute resources as needed, starting additional Kibana nodes if there is a lot of reporting going on, or logstash nodes when a lot of data needs to be processed simultaneously.

[TOoSmOotH]

Full disclosure: I do use Kubernetes for homelab stuff so I have a decent grasp on it. Originally I tried going with harvester but it was easier to roll my own with https://github.com/onedr0p/flux-cluster-template

It builds all the things and sets up the cluster for you but there is still a decent sized learning curve. When something breaks I have to go in and fix it. It's not simple and requires me to know yaml and metallb etc. The whole point of 2.4 was to remove yaml from the equation. There were so many times in 2.3 where huge problems were caused by errant spaces. There is a reason that people with extensive Kubernetes knowledge have no trouble finding work. Considering the wide variety of users Security Onion has you have to keep ongoing maintenance in mind. Simple to install doesn't mean simple to maintain.

As far as auto scaling is concerned Elastic Search is all we really need to worry about. We do all the parsing in ES so all logstash is doing is grabbing events from redis and dropping them into ES. I can't stress this enough that IO will be the main issue here and that you will need to use some sort of local non shared disk per container to get any value. Maybe that could be scripted I am not sure. Once you get above 5k-10k EPS then you are going to start to struggle with rust and have to start looking at NVME. The key to a healthy SO environment is a healthy ES environment.

From this discussion I can see the need for possibly some sort of SO sidecar that could be used to monitor the cluster etc. Maybe that is the starting position on anything Kubernetes related as I haven't been convinced on what the full stack on K3s buys me.

As far as a different version that is orchestrated differently that is pretty much a non starter. Right now there are about 36 different SO install types that kick off to test the current 2.4 codebase. These tests take 8-10 hours to complete that include fresh installs of several different install methods including distributed setups as well as soups from older versions. This helps us ensure that the changes we make on a daily basis keep the product stable and viable as an enterprise solution.

[jpvlsmv]

I agree with 90+% of what you wrote, and I've experienced a lot of the same problems and a lot of different problems. It's perfectly OK with me that the security tools have a certain amount of expected learning curve. SO challenged me to learn more about saltstack and I've been able to use some of that knowledge when dealing with the Puppet and Ansible configuration management that are imposed in the environment. Kubernetes is challenging me right now to grok it, and I'm still figuring out what it's good and bad at.

I suspect that some of the "some requests [for Kubernetes]" could be based on already-existing comfort on that platform, rather than with running multiple containers on an OS deployed on bare metal. I'm sure if I asked my Ansible-wielding colleagues, they would want to orchestrate it all as playbooks (to which the answer, obviously, would be "chef")

As I'm sure you know, ES isn't the only place where processing/bottlenecks happen. I've had security onion nodes run out of IOPS for Stenographer, CPU power for Zeek, and available memory for Suricata (I might have those switched around...) Kubernetes has reservations and limits (and I/O pools) to cause chaos^W^W ensure fair distribution to keep the different components working. If there were some way to scale out the components that need more resources, I'm sure it would have its own complexity/learning curve as well. (Actually, saltstack could be drafted into that sort of distributed resource scheduling, but it would be very complicated)

As a final thought, putting the single-point-of-failure Master node services (such as redis) into a properly-redundant cluster service environment under Kubernetes+MetalLB might be simpler than the (diverse, more-Linux-native) alternative failover setup tools. Or it might not be.

[TOoSmOotH]

@jpvlsmv To get full pipeline redundancy you can easily stand up a receiver node and everyone on the grid knows about it and will load balance events to it. If the manager goes down you will continue to receive events and index them into elastic.