Kafka NestJS Microservices

A microservices architecture built with NestJS and Apache Kafka, demonstrating event-driven communication between multiple services with a hybrid HTTP/Kafka communication pattern.

📋 Project Overview

This project implements a distributed system using microservices pattern with the following services:

• API Gateway (Port: 3000) - HTTP entry point that orchestrates parallel requests to multiple HTTP services
• Post Service (Port: 3001) - Hybrid HTTP/Kafka service for post operations with event publishing and DLQ support
• Assets Service (Port: 3002) - HTTP service for media and asset processing with in-memory storage
• Analytics Service - Pure Kafka consumer microservice for processing analytics data from multiple event streams
• Notification Service - Kafka consumer/producer microservice for handling notifications and emitting secondary events

The architecture uses a hybrid approach: HTTP for synchronous request-response patterns and Kafka for asynchronous event-driven communication with proper error handling and retry mechanisms.

🚀 Quick Start

Prerequisites

• Node.js (v18 or higher)
• npm or yarn
• Docker and Docker Compose

1. Start Infrastructure (Kafka)

First, start the Kafka broker using Docker Compose:

docker-compose up -d

This will start:

• Apache Kafka broker on localhost:9092

2. Install Dependencies

Install dependencies for all services:

# Install dependencies for each service
cd api-gateway && npm install && cd ..
cd post-service && npm install && cd ..
cd analytics-service && npm install && cd ..
cd assets-service && npm install && cd ..
cd notification-service && npm install && cd ..

3. Start Services

Start each service in development mode (in separate terminals):

# Terminal 1 - API Gateway
cd api-gateway
npm run start:dev

# Terminal 2 - Post Service
cd post-service
npm run start:dev

# Terminal 3 - Analytics Service
cd analytics-service
npm run start:dev

# Terminal 4 - Assets Service
cd assets-service
npm run start:dev

# Terminal 5 - Notification Service
cd notification-service
npm run start:dev

📦 Services

API Gateway

• Port: 3000
• Type: HTTP Application
• Purpose: Entry point for all client requests, orchestrates parallel HTTP calls to multiple services
• Technology: NestJS with Express, Axios HTTP client, and TypeScript interfaces
• Architecture: Service orchestration pattern with parallel request processing

Endpoints:

• POST /posts - Creates posts by calling both Post Service (/api/posts) and Assets Service (/api/assets/upload) in parallel

cd api-gateway

# Development
npm run start:dev

# Production
npm run build
npm run start:prod

# Testing
npm run test
npm run test:e2e

Post Service

• Port: 3001
• Type: Hybrid HTTP Application + Kafka Producer/Consumer
• Purpose: Handles post creation logic, publishes domain events, and processes published posts with DLQ support
• Technology: NestJS with Kafka integration and Dead Letter Queue pattern
• Consumer Group: post-consumer-group

HTTP Endpoints:

• POST /api/posts - Creates new posts and triggers Kafka events

Kafka Integration:

• Producer: Emits post_published events
• Consumer: Processes post_published events via DLQ service
• DLQ Controller: Separate controller for handling failed message processing

cd post-service

# Development
npm run start:dev

# Production
npm run build
npm run start:prod

# Testing
npm run test
npm run test:e2e

Assets Service

• Port: 3002
• Type: HTTP Application
• Purpose: Handles media and asset processing with in-memory storage
• Technology: NestJS with Express
• Architecture: RESTful API with stateful asset management

HTTP Endpoints:

• GET /api/assets - Retrieves all processed assets
• POST /api/assets/upload - Processes and stores new assets

Asset Processing:

• Generates unique asset IDs with timestamp-based naming
• Creates CDN-style URLs for asset access
• Tracks processing timestamps and status
• Stores processed assets in memory for retrieval

cd assets-service

# Development
npm run start:dev

# Production
npm run build
npm run start:prod

# Testing
npm run test
npm run test:e2e

Analytics Service

• Port: Kafka microservice (no HTTP port)
• Type: Pure Kafka Consumer Microservice
• Purpose: Processes analytics data from multiple event streams for comprehensive tracking
• Technology: NestJS Kafka microservice with multi-event consumption
• Consumer Group: analytics-consumer-group

Event Handlers:

• post_published - Tracks post creation analytics and user engagement metrics
• notification_sent - Tracks notification delivery analytics and success rates

cd analytics-service

# Development
npm run start:dev

# Production
npm run build
npm run start:prod

# Testing
npm run test
npm run test:e2e

Notification Service

• Port: Kafka microservice (no HTTP port)
• Type: Kafka Consumer/Producer Microservice
• Purpose: Sends notifications based on events and emits secondary events for analytics
• Technology: NestJS Kafka microservice with dual consumer/producer capabilities
• Consumer Group: notification-consumer-group

Event Handlers:

• post_published - Processes new post events and sends notifications to users

Event Emission:

• notification_sent - Emits events after successful notification processing for analytics tracking

Notification Flow:

1. Receives post_published events from Post Service
2. Processes notification logic (simulated notification sending)
3. Emits notification_sent events for Analytics Service consumption
4. Provides comprehensive logging for notification tracking

cd notification-service

# Development
npm run start:dev

# Production
npm run build
npm run start:prod

# Testing
npm run test
npm run test:e2e

🔄 Event Flow

The system uses a hybrid communication pattern combining HTTP and Kafka:

HTTP Flow (Synchronous)

1. Client Request → API Gateway receives POST /posts request
2. Parallel Service Orchestration → API Gateway makes simultaneous HTTP calls to:
   • Post Service (POST /api/posts) on port 3001
   • Assets Service (POST /api/assets/upload) on port 3002
3. Response Aggregation → API Gateway combines both responses and returns unified result to client
4. Error Handling → Comprehensive error handling with service-specific error messages

Kafka Flow (Asynchronous Event Chain)

1. Primary Event Publishing → Post Service publishes post_published event to Kafka after successful HTTP processing
2. Multi-Consumer Processing → Multiple consumers process the event simultaneously:
   • Notification Service → Processes notifications and emits secondary notification_sent event
   • Analytics Service → Tracks post creation metrics and logs analytics data
3. Secondary Event Processing → Analytics Service also consumes notification_sent events for delivery tracking
4. DLQ Processing → Post Service handles failed post_published events with comprehensive retry logic and exponential backoff

Event Chain Visualization

POST /posts → API Gateway
    ├── POST /api/posts → Post Service (HTTP)
    │   └── post_published → Kafka Event
    │       ├── Notification Service → notification_sent → Analytics Service
    │       └── Analytics Service (direct consumption)
    └── POST /api/assets/upload → Assets Service (HTTP)

DLQ Flow: post_published → Post Service DLQ Controller → Retry Logic

🛠️ Development

Project Structure

kafka-nestjs-without-nx/
├── api-gateway/          # HTTP API Gateway service (Port: 3000)
├── post-service/         # HTTP Post service with Kafka producer + DLQ (Port: 3001)
├── assets-service/       # HTTP Assets service (Port: 3002)
├── analytics-service/    # Kafka consumer for analytics
├── notification-service/ # Kafka consumer for notifications with Kafka producer
└── docker-compose.yml    # Kafka infrastructure

Testing the API

Create a Post

curl -X POST http://localhost:3000/posts \
  -H "Content-Type: application/json" \
  -d '{
    "title": "My First Post",
    "content": "This is the content of my first post",
    "author": "John Doe"
  }'

Expected Response:

{
  "message": "Post and assets created successfully via API Gateway",
  "data": {
    "post": {
      "message": "Post created successfully",
      "post": {
        "title": "My First Post",
        "content": "This is the content of my first post",
        "author": "John Doe",
        "id": "post_1234567890",
        "createdAt": "2024-01-01T12:00:00.000Z"
      }
    },
    "asset": {
      "message": "Asset processed successfully",
      "asset": {
        "title": "My First Post",
        "content": "This is the content of my first post",
        "author": "John Doe",
        "id": "asset_1234567890",
        "processedAt": "2024-01-01T12:00:00.000Z",
        "status": "processed",
        "url": "https://cdn.example.com/assets/1234567890"
      }
    }
  }
}

Get Assets

curl -X GET http://localhost:3002/api/assets

Event Monitoring

Monitor the event flow and service interactions in real-time by watching the console logs of each service. You'll see:

1. API Gateway: Request orchestration and parallel service calls
2. Post Service: HTTP processing and Kafka event emission
3. Assets Service: Asset processing and storage
4. Notification Service: Event consumption and secondary event emission
5. Analytics Service: Multi-event consumption and analytics processing

Monitoring Events

You can monitor Kafka events and service logs in real-time:

# Monitor all services
docker-compose logs -f

# Monitor specific service
docker-compose logs -f broker