H2V (Horizontal to Vertical Video Converter)

H2V is an intelligent video conversion pipeline that transforms horizontal (landscape) videos into vertical (portrait) format while maintaining the main point of interest in frame. Unlike simple cropping solutions, H2V uses computer vision to track faces, detect speech, and follow motion to ensure the most important content remains visible.

Features

• Intelligent Focus Tracking: Automatically identifies and follows faces, speakers, and motion
• Scene-Aware Processing: Detects scene changes to maintain context across cuts
• Speech Detection: Prioritizes speakers during dialogue scenes
• Smooth Transitions: Applies temporal smoothing to prevent jarring camera movements
• API Option: Use via command line, or API server.

Design Choices

Architecture

H2V uses a modular pipeline architecture with specialized components:

1. Scene Detection: Identifies scene boundaries to process each scene independently
2. Face Detection: Uses MediaPipe to locate and track faces
3. Speech Detection: Analyzes audio to identify when someone is speaking
4. Motion Tracking: Falls back to optical flow tracking when no faces are detected
5. Focus Point Selection: Combines all signals to determine the optimal focus point
6. Temporal Smoothing: Reduces jitter by smoothing focus point transitions
7. Video Creation: Crops and renders the final vertical video

Technical Decisions

• Batch Processing: Processes frames in batches to manage memory usage
• Keyframe Analysis: Analyzes keyframes and interpolates between them for efficiency
• Downscaling: Optionally downscales frames during analysis for faster processing
• Configurable Parameters: Adjustable settings for detection thresholds, performance, etc.

Setup Instructions

Prerequisites

• Python 3.10+
• mediapipe (for face detection)
• moviepy (for video processing)
• scenedetect (for scene detection)
• librosa (for speech detection)
• opencv-python (for optical flow tracking)
• uvicorn (for API server)
• fastapi (for API server)
• python-multipart (for API server)
• aiofiles (for API server)

Installation

1. Clone the repository:

git clone https://github.com/felixLandlord/h2v.git
cd h2v

2. Create a virtual environment: macOS/Linux:

python -m venv .venv
source .venv/bin/activate

Windows:

python -m venv .venv
.venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

Usage

Run the pipeline:

1. Through the Command Line:

python main.py input_video.mp4 --output output_video.mp4

2. Through an API Server:

python main.py --api

OR

uvicorn api:app --reload

This will start a local server at http://localhost:8000.

API Endpoints

The H2V API provides the following endpoints for video processing:

Upload Video

• URL : /upload/
• Method : POST
• Content-Type : multipart/form-data
• Parameters :
  • file : The video file to be processed (MP4, MOV, AVI, MKV)
• Response :

{
"job_id": "11cee74f-4af5-4850-8385-8c419a615ecd",
"status": "processing",
"message": "Video upload successful. Processing started."
}

Uploads a horizontal video and starts the conversion process. Returns a job ID for tracking the process.

Get Job Status

• URL : /status/{job_id}
• Method : GET
• Parameters :
  • job_id : The ID of the job to check the status for
• Response :

{
  "status": "completed",
  "input_file": "temp/uploads/11cee74f-4af5-4850-8385-8c419a615ecd.mp4",
  "output_file": "temp/outputs/11cee74f-4af5-4850-8385-8c419a615ecd.mp4",
  "metadata_file": "temp/metadata/11cee74f-4af5-4850-8385-8c419a615ecd_focus_points.json",
  "original_filename": "example.mp4"
}

Checks the current status of a video processing job. Possible status values: "processing", "completed", "failed".

Download Processed Video

• URL : /download/{job_id}
• Method : GET
• Parameters :
  • job_id : The ID of the job to download the processed video for
• Response : The processed video file in MP4 format.

Download Metadata

• URL : /metadata/{job_id}
• Method : GET
• Parameters :
  • job_id : The ID of the job to download the metadata for
  • Response : The focus points metadata for the processed video file in JSON format.

Sample Results

Check out this sample conversion result: Sample Vertical Video

This sample demonstrates how H2V intelligently tracks the main subject while converting from horizontal to vertical format.

Additionally, you can view the focus points metadata used in the conversion: Sample Focus Points

This JSON file provides detailed information about the focus points tracked during the video conversion process.