Skip to content

Latest commit

 

History

History
634 lines (441 loc) · 21.1 KB

README.md

File metadata and controls

634 lines (441 loc) · 21.1 KB
CMNW Logo

🎯 CMNW

Intelligence Always Wins

Website License Version

NestJS TypeScript PostgreSQL MongoDB Redis BullMQ Docker

📖 About

CMNW is a sophisticated NestJS-based microservices platform specializing in Open Source Intelligence (OSINT) and Data Market Analysis (DMA) for World of Warcraft. Built on a distributed architecture with 13 microservices and 7 shared libraries, CMNW provides comprehensive intelligence gathering, real-time market analytics, and advanced financial valuations using XVA-based pricing models.

🎯 Key Features

  • 🕵️ Advanced OSINT Capabilities: Comprehensive character and guild intelligence gathering with Battle.net API integration
  • 📊 Real-Time Market Analysis: Auction house monitoring with historical price tracking and trend detection
  • 💰 XVA Financial Valuations: Sophisticated pricing engine with risk assessment and portfolio analytics
  • 🔄 Distributed Job Processing: BullMQ-based message queuing with Redis, priority queues, and retry mechanisms
  • 🐳 Cloud-Native Architecture: Full Docker containerization with multi-platform support (ARM64 & x64)
  • 🔒 Enterprise Authentication: Battle.net OAuth and Discord OAuth integration via Passport
  • 📈 Monitoring & Observability: Prometheus metrics, Loki logging, and Bull Board queue monitoring
  • 🧪 Comprehensive Testing: Jest unit tests, Playwright E2E tests, and extensive mock data

🏗️ Architecture

CMNW Ignition

Microservices Architecture Overview

🧩 Microservices

CMNW operates as a pnpm monorepo with 13 specialized microservices:

Service Port Description Key Features
🎯 Core 3000 Central management service Realms & Keys management, Authentication, System configuration
🌐 API Gateway 8080 REST API gateway Request routing, Swagger docs at /api/docs, Rate limiting
🕵️ OSINT 3000 Intelligence gathering 3 workers (characters, guilds, profile), Battle.net integration
📊 DMA 3004 Data Market Analysis 2 workers (auctions, items), Real-time AH monitoring
💰 Market 3002 Market operations XVA calculations, Contracts, Gold tracking, Evaluation
👤 Characters - Character management Player profiles, Statistics tracking, Entity indexing
🏰 Guilds - Guild analytics Member tracking, Roster management, Activity logs
📈 Analytics - Metrics aggregation Character/Guild/Market/Contract metrics services
🏆 Ladder - Ranking system Competitive rankings, Leaderboards
🧪 Tests - Testing infrastructure E2E tests, Mock data, Benchmarking
📊 Warcraft Logs - Raid analytics Combat log parsing, Performance metrics
🌍 WoW Progress - Progress tracking Guild progression, Raid completion tracking
💎 Valuations - Financial modeling XVA calculations, Risk assessments, Portfolio analytics

📚 Shared Libraries

7 shared libraries provide common functionality across microservices:

🛠️ Technology Stack

🎯 Backend Framework

  • NestJS 11.1.9 - Progressive Node.js framework
  • TypeScript 5.9.3 - Type-safe JavaScript development
  • Node.js >=24.0.0 - JavaScript runtime
  • RxJS 7.8.2 - Reactive programming with observables

🗄️ Data Layer

  • PostgreSQL 17.4 - Primary relational database with SnakeNamingStrategy
  • MongoDB - Document store with Mongoose 9.0.1
  • Redis 7.4.3 - In-memory caching and session storage
  • TypeORM 0.3.28 - Database ORM with migration support

🔄 Message Queuing

  • BullMQ 5.66.0 - Redis-based job queue with priority support and retry mechanisms
  • Bull Board 6.15.0 - Queue monitoring dashboard

🔌 External Integrations

  • @alexzedim/blizzapi 2.7.0 - Battle.net API client
  • Warcraft Logs API - Combat log analytics
  • Raider.IO API - Mythic+ and raid progress
  • AWS S3 - Object storage with @aws-sdk/client-s3@3.948.0

🔐 Authentication & Security

  • Passport 0.7.0 - Authentication middleware
  • Battle.net OAuth - Blizzard authentication
  • Discord OAuth - Discord integration via passport-discord@0.1.4

🐳 DevOps & Deployment

  • Docker - Containerization with multi-platform builds
  • GitHub Actions - CI/CD pipeline with self-hosted runners
  • pnpm 10.28.1 - Fast, disk space efficient package manager

🔧 Development Tools

  • Jest 30.2.0 - Testing framework with comprehensive coverage
  • Playwright 1.57.0 - End-to-end testing automation
  • ESLint & Prettier - Code quality and formatting
  • Swagger/OpenAPI - API documentation with @nestjs/swagger@11.2.3

📊 Monitoring & Observability

  • Prometheus - Metrics collection with @willsoto/nestjs-prometheus@6.0.2
  • Loki - Log aggregation
  • Bull Board - Queue monitoring and management

🚀 Quick Start

Prerequisites

  • Node.js >=24.0.0
  • pnpm >=10.0.0 (managed via corepack)
  • Docker & Docker Compose
  • PostgreSQL 17+
  • MongoDB (latest)
  • Redis 7.4+

Note: This project uses pnpm 10.28.1. The correct version is managed via corepack and specified in package.json.

🔧 Installation

# Clone the repository
git clone https://github.com/alexzedim/cmnw.git
cd cmnw

# Enable corepack for pnpm version management
corepack enable

# Install dependencies
pnpm install

# Copy environment configuration
cp .env.example .env
cp .env.docker.example .env.docker

🐳 Docker Development Setup

Start infrastructure services and microservices using Docker Compose:

# Start database services (PostgreSQL, MongoDB, Redis)
docker-compose -f docker-compose.db.yml up -d

# Start core services
docker-compose -f docker-compose.core.yml up -d

# Start OSINT services
docker-compose -f docker-compose.osint.yml up -d

# Start DMA services
docker-compose -f docker-compose.dma.yml up -d

# Start analytics services
docker-compose -f docker-compose.analytics.yml up -d

# Or start all development services
docker-compose -f docker-compose.dev.yml up -d

🛠️ Local Development

# Build all microservices
pnpm build:all

# Start development server with watch mode
pnpm start:dev

# Run specific microservice
nest start core --watch
nest start api --watch
nest start osint --watch
nest start dma --watch
nest start market --watch

# Start with debug mode
pnpm start:debug

🌐 Access Points

After starting the services:

💼 Core Modules

🕵️ OSINT (Open Source Intelligence)

Comprehensive character and guild intelligence gathering with Battle.net API integration:

Architecture:

Key Features:

  • Character Analysis: Player profiles, equipment, achievements, professions
  • Guild Monitoring: Member tracking via guild-member.service.ts, roster management, activity logs
  • Raid Progress: Warcraft Logs and Raider.IO integration
  • Collection System: Character collection management via character-collection.service.ts

Services:

📊 DMA (Data Market Analysis)

Real-time auction house monitoring and market intelligence:

Architecture:

  • 2 Specialized Workers: auctions.worker.ts, items.worker.ts
  • High-Frequency Updates: Real-time auction house snapshots
  • Historical Analysis: Price trend detection and forecasting

Key Features:

  • Price Tracking: Historical price analysis with technical indicators
  • Market Charts: Interactive price visualization
  • Cross-Realm Analysis: Server population and pricing comparisons
  • Commodity Monitoring: Server-wide commodity price tracking
  • Item Intelligence: Item metadata and pricing calculations

💰 Market & Valuations Engine

Valuations Schema

XVA-based Pricing Engine Schema

Advanced financial modeling with XVA (X-Value Adjustments) calculations:

Services:

Key Features:

  • Risk Assessment: Credit, funding, and capital value adjustments
  • Price Discovery: Dynamic pricing based on market conditions
  • Portfolio Analytics: Multi-asset risk and return analysis
  • Crafting Economics: Disenchanting, milling, prospecting calculations via disenchanting.libs.ts, milling.libs.ts, prospecting.libs.ts

📈 Analytics & Monitoring

Comprehensive metrics aggregation and monitoring:

Services:

Monitoring:

  • Prometheus Metrics: Custom metrics collection
  • Queue Monitoring: Bull Board dashboard at /queues
  • Worker Statistics: Real-time worker performance tracking via worker-stats.listener.ts

🧪 Testing

🎯 Test Suite

Comprehensive testing infrastructure with Jest 30.2.0 and Playwright 1.57.0:

# Run all tests
pnpm test

# Run tests with coverage
pnpm test:cov

# Run tests in watch mode
pnpm test:watch

# Run tests with debugging
pnpm test:debug

📋 Test Coverage

Extensive test suites with mock data:

🧪 Test Services

Dedicated testing microservice with:

🚀 Deployment

🔄 CI/CD Pipeline

Automated deployment via GitHub Actions with self-hosted runners:

Workflows:

  • ✅ Docker Image Builds (ARM64 & x64)
  • ✅ Automated Testing (Jest + Playwright)
  • ✅ Security Scanning
  • ✅ Multi-environment Deployment
  • ✅ Container Registry Publishing

🐳 Production Deployment

1. Environment Setup:

# Configure production environment
cp .env.production .env

# Set deployment secrets
export ENC_PASSWORD="your_encryption_key"
export BNET_CLIENT_ID="your_client_id"
export BNET_CLIENT_SECRET="your_client_secret"

2. Container Registry:

# Build multi-platform images
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ghcr.io/alexzedim/cmnw:6.10.9 \
  -t ghcr.io/alexzedim/cmnw:latest \
  --push .

# Pull images on production server
docker pull ghcr.io/alexzedim/cmnw:latest

3. Service Orchestration:

# Deploy infrastructure
docker-compose -f docker-compose.db.yml up -d

# Deploy core services
docker-compose -f docker-compose.core.yml up -d

# Deploy OSINT services
docker-compose -f docker-compose.osint.yml up -d

# Deploy DMA services
docker-compose -f docker-compose.dma.yml up -d

# Deploy analytics services
docker-compose -f docker-compose.analytics.yml up -d

4. Health Checks:

# Verify service health
curl http://localhost:8080/health
curl http://localhost:3000/health
curl http://localhost:3002/health
curl http://localhost:3004/health

📚 API Documentation

🌐 REST API Endpoints

The API Gateway (port 8080) provides comprehensive REST API access:

OSINT Endpoints:

  • GET /osint/character/:realm/:name - Character intelligence
  • GET /osint/guild/:realm/:name - Guild intelligence
  • GET /osint/realm/:slug - Realm information

DMA Endpoints:

  • GET /dma/auctions/:connectedRealmId - Auction house data
  • GET /dma/items/:itemId - Item market data
  • GET /dma/commodities - Commodity prices

Market Endpoints:

  • GET /market/contracts - Active contracts
  • GET /market/gold/:region - Gold prices
  • GET /market/evaluation/:itemId - Item evaluation

Queue Monitoring:

  • GET /queues - Bull Board dashboard
  • GET /queue/metrics - Queue metrics
  • GET /queue/workers - Worker status

📖 Interactive Documentation

Swagger/OpenAPI documentation available at:

The API documentation is automatically generated from NestJS decorators and includes:

  • 📝 Request/Response schemas
  • 🔐 Authentication requirements
  • 🧪 Interactive API testing
  • 📊 Model definitions

🔧 Development Workflow

📦 Package Management

# Install dependencies
pnpm install

# Add dependency to specific workspace
pnpm add <package> --filter @app/osint

# Update dependencies
pnpm update

# Clean install
pnpm clean-install

🏗️ Building

# Build all microservices
pnpm build:all

# Build specific service
nest build core
nest build api
nest build osint
nest build dma
nest build market

🔍 Code Quality

# Lint and fix code
pnpm lint

# Format code
pnpm format

# Type checking
tsc --noEmit

🐛 Debugging

# Start with debugger
pnpm start:debug

# Debug specific service
nest start osint --debug --watch

# Debug tests
pnpm test:debug

📊 Monitoring & Observability

📈 Metrics Collection

Prometheus Integration:

  • Custom metrics via @willsoto/nestjs-prometheus@6.0.2
  • Service health metrics
  • Queue performance metrics
  • Database connection pool metrics

Metrics Endpoints:

  • GET /metrics - Prometheus metrics
  • GET /health - Health check endpoint

📝 Logging

Structured Logging:

  • Centralized logging via @app/logger
  • Loki integration for log aggregation
  • Contextual logging with request tracing
  • Log levels: error, warn, info, debug, verbose

🔍 Queue Monitoring

Bull Board Dashboard:

  • Real-time queue monitoring at /queues
  • Job status tracking
  • Failed job inspection
  • Queue metrics and statistics
  • Worker performance monitoring

🎯 Worker Statistics

Real-time worker performance tracking:

  • Job processing rates
  • Success/failure ratios
  • Average processing times
  • Queue depth monitoring

🤝 Contributing

We welcome contributions from developers experienced in:

  • 🎯 Microservices Architecture - NestJS, distributed systems
  • 🕵️ Intelligence Gathering Systems - OSINT, data aggregation
  • 📊 Financial Data Analysis - XVA calculations, risk modeling
  • 🎮 Gaming API Integration - Battle.net, Warcraft Logs, Raider.IO
  • 🔒 Security & Authentication - OAuth, JWT, API security
  • 🐳 DevOps & Infrastructure - Docker, Kubernetes, CI/CD

📋 How to Contribute

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

🐛 Issues & Feature Requests

📝 Development Guidelines

  • Follow TypeScript best practices
  • Write comprehensive tests for new features
  • Update documentation for API changes
  • Use conventional commit messages
  • Ensure all tests pass before submitting PR

📄 License

This project is licensed under the Mozilla Public License 2.0 - see the LICENSE file for details.

🌟 Built with ❤️ by @alexzedim

🌐 Website • 🐛 Issues • 💬 Discussions • 🐦 Twitter

"Intelligence Always Wins" 🎯