Skip to content

Latest commit

 

History

History
217 lines (155 loc) · 5.63 KB

README.md

File metadata and controls

217 lines (155 loc) · 5.63 KB

🎬 StreamForge - HLS Video Transcoding System

A scalable, cloud-native video transcoding system that automatically converts uploaded videos into adaptive bitrate HLS streams with multiple quality levels.

🏗️ Architecture Overview

StreamForge is built with a microservices architecture designed for scalability and reliability:

Core Components

  1. 🐳 HLS Transcoder Service (hls-transcoder/)

    • Dockerized FFmpeg-based transcoding service
    • Converts videos to adaptive bitrate HLS streams
    • Generates multiple quality levels (240p, 360p, 720p)
    • Automatically uploads processed content to S3
  2. ⚡ API Service (api/)

    • Node.js/TypeScript API server
    • SQS message consumer
    • Docker container orchestration
    • Process monitoring and logging
  3. 🎥 Test Player (hls-transcoder/test-player.html)

    • Modern HTML5 video player with HLS.js
    • Quality selection controls
    • Adaptive bitrate streaming support
    • Real-time quality monitoring

🔄 Processing Flow

image

Step-by-Step Process

  1. Video Upload: User uploads video to S3 bucket
  2. Event Trigger: S3 triggers notification to SQS queue
  3. Message Processing: API service polls SQS for new messages
  4. Container Launch: API spawns Docker container with video key
  5. Transcoding: FFmpeg converts video to multiple HLS qualities
  6. Upload: Processed HLS files uploaded back to S3
  7. Cleanup: Container removed, SQS message deleted

🎯 Key Features

📹 Advanced Video Processing

  • Adaptive Bitrate Streaming: Multiple quality levels for optimal viewing
  • HLS Protocol: Industry-standard HTTP Live Streaming
  • Smart Quality Selection: Automatic quality switching based on bandwidth
  • Segment-based Delivery: Efficient streaming with 4-second segments

🔧 Quality Levels

Quality Resolution Video Bitrate Audio Bitrate Use Case
240p 426×240 400 kbps 64 kbps Mobile/Slow connections
360p 640×360 800 kbps 96 kbps Standard mobile viewing
720p 1280×720 2.5 Mbps 128 kbps HD desktop viewing

🚀 Scalability Features

  • Containerized Processing: Isolated, scalable transcoding jobs
  • Queue-based Architecture: Handle high volumes with SQS
  • Stateless Design: Easy horizontal scaling
  • Cloud-native: AWS S3, SQS integration

🛡️ Reliability & Monitoring

  • Error Handling: Comprehensive error catching and logging
  • Process Timeout: 10-minute timeout protection
  • Real-time Logging: Live transcoding progress monitoring
  • Graceful Cleanup: Automatic container removal

📁 Output Structure

For each processed video, the system creates:

s3://your-bucket/hls/video-name/
├── master.m3u8                 # Master playlist (entry point)
├── 240p/
│   ├── prog.m3u8              # 240p playlist
│   ├── segment_000.ts         # Video segments
│   ├── segment_001.ts
│   └── ...
├── 360p/
│   ├── prog.m3u8              # 360p playlist
│   ├── segment_000.ts
│   └── ...
└── 720p/
    ├── prog.m3u8              # 720p playlist
    ├── segment_000.ts
    └── ...

🚀 Quick Start

Prerequisites

  • Docker & Docker Compose
  • Node.js 18+
  • AWS Account (S3, SQS access)
  • FFmpeg (included in Docker image)

1. Environment Setup

# Clone repository
git clone <repository-url>
cd streamforge

# Set up HLS Transcoder
cd hls-transcoder
cp .env.example .env
# Configure AWS credentials in .env

# Set up API
cd ../api
npm install
cp .env.example .env
# Configure SQS queue URL and AWS credentials

2. Build Docker Image

cd hls-transcoder
docker build -t hls-transcoder .

3. Configure AWS Services

# Create S3 bucket
aws s3 mb s3://your-video-bucket

# Create SQS queue
aws sqs create-queue --queue-name video-processing-queue

# Set up S3 event notifications to SQS
# (Configure in AWS Console or via CLI)

4. Start Services

# Start API service
cd api
npm run dev

# The API will automatically poll SQS and process videos

5. Test with Video Player

# Open test player
open hls-transcoder/test-player.html

# Enter your HLS master playlist URL:
# https://your-bucket.s3.region.amazonaws.com/hls/video-name/master.m3u8

📝 Configuration

Environment Variables

HLS Transcoder (.env):

AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
S3_BUCKET_NAME=your-video-bucket
AWS_REGION=us-east-1

API Service (.env):

SQS_QUEUE_URL=https://sqs.region.amazonaws.com/account/queue-name
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=us-east-1

FFmpeg Configuration

The system uses optimized FFmpeg settings:

  • Preset: veryfast for speed
  • GOP Size: 48 frames for efficient streaming
  • Segment Length: 4 seconds
  • Format: HLS with VOD playlist type

📊 Performance

Typical Processing Times

  • 5-minute 1080p video: ~2-3 minutes
  • 10-minute 720p video: ~3-4 minutes
  • 30-minute 4K video: ~15-20 minutes

Resource Requirements

  • CPU: 2+ cores recommended
  • Memory: 2GB+ RAM
  • Storage: Temporary space for processing
  • Network: High bandwidth for S3 uploads

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.