Skip to content

README.md

Latest commit

 

History

History
226 lines (176 loc) · 6.08 KB

README.md

File metadata and controls

226 lines (176 loc) · 6.08 KB

🌡️ Sensor Stream Analytics System

Complete system of two microservices for sensor simulation and real-time data analysis:

  1. Rust Sensor Simulator - simulates sensors and sends data to RabbitMQ
  2. Python Analytics Engine - analyzes data in real-time

🏗️ Architecture

┌─────────────────────┐    ┌──────────────┐    ┌─────────────────────┐
│   Rust Sensor       │───▶│   RabbitMQ   │───▶│   Python Analytics  │
│   Simulator         │    │   Message    │    │   Engine            │
│   • Temperature     │    │   Queue      │    │   • Rolling Window  │
│   • Humidity        │    │              │    │   • Statistics      │
│   • Pressure        │    │              │    │   • JSON Export     │
│   • Async timers    │    │              │    │   • CLI Interface   │
└─────────────────────┘    └──────────────┘    └─────────────────────┘

🚀 Quick Start

Prerequisites

🐳 Running with Docker (recommended)

# Start entire system (in background)
docker compose up -d

# Or start with real-time logs
docker compose up

# View logs for individual services
docker compose logs -f sensor-simulator
docker compose logs -f analytics-engine

# Stop the system
docker compose down

# Rebuild after code changes
docker compose up --build

Important: Analytics Engine runs in interactive mode. To interact with CLI use:

docker exec -it analytics-engine python cli.py

🔧 Manual Setup

1. Start RabbitMQ

docker run -d --hostname my-rabbit --name rabbitmq \
  -p 5672:5672 -p 15672:15672 \
  rabbitmq:3-management

Management UI: http://localhost:15672
Login: guest / guest

2. Start Rust Sensor Simulator

# Build and run sensor simulator
cargo build
cargo run

3. Start Python Analytics Engine

# Install dependencies
cd analytics
pip install -r requirements.txt

# Run analytics
python cli.py

✨ System Features

🦀 Rust Sensor Simulator

  • ✅ Simulation of 3 sensor types (Temperature, Humidity, Pressure)
  • ✅ Different sampling intervals for each sensor
  • ✅ Realistic data (sine wave + noise)
  • ✅ Asynchronous programming with Tokio
  • ✅ Internal channels for separating generation and publishing
  • ✅ Retry logic and error handling
  • ✅ Detailed logging

🐍 Python Analytics Engine

  • ✅ Real-time data consumption from RabbitMQ
  • ✅ Rolling window (last 10 values)
  • ✅ Mean and standard deviation calculation
  • ✅ CLI with sensor type filtering
  • ✅ Statistics export to JSON
  • ✅ Retry logic and graceful error handling
  • ✅ Monitoring and logging
  • ✅ Docker containerization

📊 Message Format

{
  "sensor_id": "123e4567-e89b-12d3-a456-426614174000",
  "sensor_type": "Temperature",
  "timestamp": 1716581298,
  "value": 24.73
}

🔧 Interactive Analytics Engine Commands

When running analytics, available commands:

  • s - show current statistics
  • e - export statistics to JSON
  • f Temperature - filter by sensor type
  • c - clear filter
  • q - quit

🧪 Testing

Rust tests

cargo test

Python tests

cd analytics
python test_analytics.py

📁 Project Structure

sensors/
├── src/                    # Rust Sensor Simulator
│   ├── main.rs            # Entry point and orchestrator
│   ├── sensor.rs          # Sensor simulation logic
│   ├── publisher.rs       # RabbitMQ connection and publishing
│   └── models.rs          # Data models
├── analytics/             # Python Analytics Engine
│   ├── models.py          # Data models
│   ├── analytics_engine.py # Main analytics logic
│   ├── cli.py             # CLI interface
│   ├── test_analytics.py  # Tests
│   ├── Dockerfile         # Docker image
│   └── requirements.txt   # Python dependencies
├── docker-compose.yml     # Configuration for entire system
└── README.md             # Documentation

📈 Example Output

Real-time Analytics Engine output:

🔥 Temperature | ID: a1b2c3d4... | Value: 24.73 | Average: 24.12 | Std Dev: 1.45 | Window: 10/10 | Total messages: 150
🔥 Humidity | ID: e5f6g7h8... | Value: 62.45 | Average: 63.21 | Std Dev: 3.12 | Window: 10/10 | Total messages: 145
🔥 Pressure | ID: i9j0k1l2... | Value: 1013.25 | Average: 1012.87 | Std Dev: 2.73 | Window: 10/10 | Total messages: 140

Interactive CLI commands:

🌡️ Sensor Analytics CLI

Available commands:
  s - Show current statistics
  e - Export statistics to JSON
  f <sensor_type> - Set filter (Temperature/Humidity/Pressure)
  c - Clear filter
  q - Quit

> s
📊 Current Statistics:
Temperature: avg=24.12, count=150, window=10/10
Humidity: avg=63.21, count=145, window=10/10
Pressure: avg=1012.87, count=140, window=10/10

> f Temperature
✅ Filter set for: Temperature

> e
✅ Statistics exported to: exports/sensor_stats_20250526_105230.json

🔧 Useful Commands

System Status Check

# Status of all containers
docker compose ps

# Check tests are working
docker exec analytics-engine python test_analytics.py

# Interactive work with Analytics CLI
docker exec -it analytics-engine python cli.py

System Monitoring

# View logs in real-time
docker compose logs -f

# View RabbitMQ Management UI
# Open http://localhost:15672 (guest/guest)

# Check message queues
docker exec rabbitmq rabbitmqctl list_queues

Debugging

# Connect to container for debugging
docker exec -it analytics-engine bash
docker exec -it sensor-simulator bash

# View exported files content
docker exec analytics-engine ls -la exports/