Optimizing Security Onion: Solving Logstash, Elasticsearch & Redis Memory Issues

[Chizan-Dolrocks]

🛠️ Resource Optimization on a Distributed Security Onion Architecture

📌 Context

I have set up a distributed Security Onion architecture with the following configuration:

• 2 Sensor Nodes
• 1 Search Node
  • RAM: 25 GB
  • Disk: 500 GB
  • Processors: 8
• 1 Manager Node
  • RAM: 32 GB
  • Disk: 1000 GB
  • Processors: 32

The overall status of the services is OK, but several resource management issues have been encountered.

• Sensor Node 1

• Sensor Node 2

• Search Node

• Manager node

---

⚠️ Issues Encountered and Solutions Applied

1️⃣ Logstash Error:

"uncaught error in thread Ruby-0-Thread..."
"java.lang.OutOfMemoryError: Java heap space"

🔹 Solution Applied:

• Increased memory allocation for Logstash via the SOC interface:
  • 📌 SOC → Administration → Configuration → Logstash → Settings → lsheap
  • 🔧 Adjusted the value based on available resources.

---

2️⃣ Elasticsearch Error:

"Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards"

🔹 Solution Applied:

• Adjusted the number of shards based on available resources:
  • 📌 SOC → Configuration → Elasticsearch → index_settings → so-INDEX-NAME → index_template → Template → Settings → Index → number_of_shards
  • 📈 New configuration: Heap*20

---

3️⃣ Redis Error:

"OOM command not allowed when used memory > 'maxmemory'"

🔹 Solution Applied:

• Increased Redis memory and restarted services:
  • 📌 SOC → Administration → Configuration → Redis → Config → maxmemory → 2048m
  • 🔄 Restarted Redis and Elasticsearch services

🚨 Issue: Despite this optimization, the issue persists after a few days.

---

⚙️ Current Configuration After Optimization

🔹 Elasticsearch (esheap)

• Search Node: 8,381 MB
• Manager Node: 8,381 MB

🔹 Logstash (lsheap)

• Manager Node: 6 GB
• Search Node: 8 GB

🔹 Redis (maxmemory)

• 8,192 MB

🔹 Number of Elasticsearch Shards

• 120

---

📚 References Used

• 📝 Security Onion Discussion
• 📘 Security Onion Logstash Documentation
• 📖 Elasticsearch Heap Sizing
• 📙 Security Onion Elasticsearch Shards

---

🚀 Request for Optimization and Best Practices

Despite these optimizations, the Redis issue persists. I am looking for recommendations on:

1. Better Redis resource management (e.g., tuning maxmemory-policy, appendonly, save...)
2. Optimizing Elasticsearch shards (is 120 shards too high for this setup?)
3. Balancing memory allocation between Logstash, Elasticsearch, and Redis

Any help or insights would be greatly appreciated. Thank you in advance! 🙏

[rh-ops]

You can also navigate to SOC > Administration > Configuration > logstash > config > pipeline_x_batch_x_size, and reduce the value there for better optimization.