A distributed metrics processing pipeline showcasing a polyglot architecture. The system uses a Go service for API management and orchestration, while a C++ worker handles compute-intensive metric aggregation. It is fully containerized, deployable on Kubernetes, and integrated with Prometheus for comprehensive observability.
This project demonstrates a modern, scalable approach to processing metrics by combining the strengths of two powerful languages:
- Go: Manages the API, request lifecycle, and orchestrates jobs. Its concurrency model is perfect for handling numerous simultaneous connections and coordinating tasks.
- C++: Performs the heavy lifting for metric aggregation. Its high performance and efficiency are ideal for the computational core of the pipeline.
The entire stack is designed to run in containers, orchestrated by Kubernetes, and is observable via Prometheus metrics.
The system consists of two primary services:
-
Go API Service: Located in the
go-api/directory.- Exposes REST endpoints for clients to submit processing requests or query status.
- Handles orchestration, potentially managing a queue of work for the C++ worker.
- Exposes its own operational metrics (e.g., request rates, latencies) for monitoring.
-
C++ Worker: Located in the
cpp-worker/directory.- Performs the core, compute-heavy metric aggregation logic.
- Receives tasks from the Go service.
- Exports detailed performance and internal metrics (e.g., tasks processed, processing duration, memory usage) in Prometheus format.
- Core Languages: Go (for API/orchestration), C++17 (for compute worker).
- Build Tools: Go modules, CMake (for C++).
- Containerization: Docker with multi-stage builds for optimized images.
- Orchestration: Kubernetes with manifests for deployment, services, and configmaps (see
k8s/). - Observability: Prometheus for metrics collection (
prometheus/) and Grafana for visualization. - Local Development: Docker Compose for spinning up the entire stack locally (see
docker-compose.yml).
- Go (version 1.21+)
- CMake (version 3.14+)
- Docker and Docker Compose
- kubectl (for Kubernetes deployment)
- (Optional) A Kubernetes cluster (e.g., Minikube, Kind)
The easiest way to run the entire pipeline locally is using Docker Compose.
-
Clone the repository:
git clone https://github.com/monokkai/go-cpp-metrics-pipeline.git cd go-cpp-metrics-pipeline -
Configure environment (optional): Copy the example environment file and adjust variables as needed.
cp .env.example .env
-
Build and run the stack:
docker-compose up --build
This command builds the Go API and C++ worker images, and starts the services along with Prometheus and Grafana.
-
Access the services:
- Go API:
http://localhost:8080 - C++ Worker Metrics:
http://localhost:3000/metrics - Prometheus:
http://localhost:9090 - Grafana:
http://localhost:3001(default login:admin/admin)
- Go API:
Ready-to-use Kubernetes manifests are available in the k8s/ directory. You can deploy them to your cluster:
kubectl apply -f k8s/This will deploy the Go API, C++ worker, and necessary services. You may need to adjust the image names in the manifests to point to your own container registry.
.
├── .env.example # Example environment variables
├── docker-compose.yml # Local orchestration file
├── go-api/ # Source code for the Go API service
│ ├── cmd/
│ ├── internal/
│ └── Dockerfile
├── cpp-worker/ # Source code for the C++ worker
│ ├── src/
│ ├── CMakeLists.txt
│ └── Dockerfile
├── k8s/ # Kubernetes deployment manifests
├── prometheus/ # Prometheus configuration
├── scripts/ # Helper scripts for building and running
└── README.md # This file
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
This project is licensed under the MIT License - see the LICENSE file for details (once added).