Kafka-Driven 5G Network Simulation Data Pipeline

Project Overview

A real-time monitoring and analytics system for 5G network simulations using Kafka-based event streaming architecture.

System Architecture

The following diagram illustrates the end-to-end data flow of our 5G network monitoring system:

The architecture shows:

1. NS-3 Simulator generating network simulation data
2. 5G Network Simulation environment
3. Flow Monitor generating logs
4. Kafka Producer ingesting the logs
5. Kafka Topic storing the messages
6. Kafka Consumer processing the data
7. Prometheus Metrics Exporter exposing metrics
8. Prometheus Server collecting metrics
9. Grafana Dashboard visualizing the data

Tech Stack

• Messaging System: Apache Kafka
• Backend: Node.js, Express.js
• Monitoring: Prometheus
• Simulation: ns-3
• Languages: JavaScript

Key Achievements

Data Ingestion Pipeline

• Architected a real-time data ingestion pipeline using Kafka and Node.js
• Streamed network performance metrics from the ns-3 network simulator
• Enabled continuous monitoring and analysis of network behavior

Monitoring Infrastructure

• Developed a Node.js microservice with Express.js to expose a Prometheus metrics endpoint
• Facilitated visualization and alerting on key network performance indicators:
  • Throughput
  • Latency
  • Connection quality
  • Resource utilization

Real-time Data Capture

• Implemented a custom Kafka producer to capture and transmit ns-3 simulation logs in real-time
• Provided immediate insights into network behavior
• Facilitated rapid issue diagnosis and troubleshooting

Metrics Visualization

• Integrated Prometheus with a custom Node.js exporter
• Collected and visualized network metrics through customizable dashboards
• Enhanced observability and enabled proactive performance management

Data Persistence & Reliability

• Engineered a robust data persistence layer leveraging Kafka's distributed log architecture
• Ensured high availability and fault tolerance for critical network data
• Optimized Kafka consumer configuration for at-least-once delivery
• Configured to read uncommitted messages to minimize data loss
• Ensured accurate representation of simulation results

Additional Notes

This project demonstrates a practical application of event-driven architecture and real-time data processing in the context of 5G network simulation.

Future Plans

• Explore integrating a service mesh for enhanced inter-service communication and resilience
• Investigate the use of distributed tracing to gain deeper insights into message flow
• Identify and address performance bottlenecks within the data pipeline
• Extend the system to support additional network protocols and simulation scenarios

Team Members

Name	Roll Number
Yash Bhambhani	202252310
Yatharth Chavda	202251032
Parth Sonawane	202251086
Soham Haldar	202251130