Nova Learn Video Transcoder

A microservice for transcoding videos into multiple formats and bitrates.

This is a microservice component of the NovaLearn E-Learning Platform

The NovaLearn Video Transcoder is a dedicated microservice that processes uploaded videos into multiple resolutions and bitrates to support adaptive streaming (HLS).
This ensures an optimized viewing experience across different devices and varying network conditions by leveraging asynchronous processing, distributed transcoding, and CDN-based streaming.

Reference: System Design - YouTube (ByteByteGo)

---

Getting Started

Prerequisites

• Node.js 16+
• Docker & Docker Compose
• Apache Kafka
• Google Cloud Storage or AWS S3 account

Installation

1. Clone the repository:

   git clone https://github.com/KhoiMaiDinh/ELearning_VideoTranscoder.git

2. Install dependencies:

   pnpm install

3. Configure environment variables:

   cp .env.example .env
   # Edit .env with your configuration

4. Run the application:

   pnpm run start:dev

System Overview

NovaLearn's video handling system includes the following core components:

• Original Storage → Stores raw uploaded videos.
• Message Broker (Kafka) → Manages transcoding job queues asynchronously.
• Transcoding Servers → Convert raw videos into multiple formats and bitrates.
• Transcoded Storage → Stores HLS playlists (.m3u8) and video segments (.ts).
• API Servers → Manage uploads, track video statuses, and serve metadata.
• CDN → Distributes transcoded videos globally for low-latency streaming.
• Client → Students and instructors accessing video lectures across devices.

---

Upload and Transcoding Workflow

Steps

1. Upload Request
   The client uploads the video via the API server.

2. Store Original Video
   The uploaded file is saved in Original Storage (e.g., GCS, S3).

3. Publish Transcoding Job
   API server sends a Kafka event to notify transcoding servers.

4. Distributed Transcoding
   Transcoding servers process videos into multiple resolutions:

   • 1080p, 720p, 480p, 360p
   • Generate HLS playlists and segmented video chunks.

5. Store Transcoded Output
   Processed files are stored in Transcoded Storage.

6. Update Status
   Video metadata and status are updated in the database.

7. Notify Client
   Users receive notifications when the video is ready for streaming.

---

Video Streaming Workflow

Steps

1. Client Requests Video
   The frontend fetches the streaming URL from the API server.

2. CDN Caching
   HLS .m3u8 playlists and .ts segments are cached at the CDN layer.

3. Adaptive Streaming
   Based on network speed and device capabilities, the player automatically switches between available resolutions.

4. Low-Latency Delivery
   Transcoded videos are delivered directly via the CDN for better scalability.

---

Storage Strategy

Type	Purpose	Technology
Original Storage	Stores raw uploaded videos	GCS / S3
Transcoded Storage	Stores HLS playlists and segments	GCS / S3
CDN	Delivers transcoded content	Cloudflare / Cloud CDN

Message Broker (Kafka)

• Decouples API servers from transcoding servers.
• Ensures asynchronous and scalable video processing.
• Supports retry mechanisms and dead-letter queues for failed jobs.
• Allows horizontal scaling of transcoding workers for heavy workloads.

---

Adaptive Streaming (HLS)

• Uses HTTP Live Streaming (HLS) for delivering videos in multiple qualities.
• Each video is transcoded into multiple TS chunks with corresponding M3U8 playlists.
• The player automatically adjusts quality based on:
  • Device capabilities
  • Network conditions

---

Scalability and Performance

Aspect	Strategy
Horizontal Scaling	Dynamically add more transcoding servers to handle load spikes.
Chunked Processing	Split large videos into smaller chunks to parallelize transcoding.
CDN Integration	Offload streaming traffic from backend servers.
Kafka Partitioning	Distribute transcoding jobs across multiple consumers.

---

Notifications

• Trigger: When transcoding completes.
• Recipients: Instructors.
• Delivery: WebSocket, email, or push notifications.