TwitchVision 🎬

AI-powered system that automatically creates vertical TikTok clips from Twitch VODs.

Features

• Automated Detection: Uses NLTK/VADER sentiment analysis to detect funny moments from Twitch chat
• AI Verification: Google Gemini 1.5 Pro watches clips to verify they're actually funny
• Vertical Format: Automatically crops to 9:16 TikTok format (1080x1920)
• LangGraph Pipeline: State machine orchestration with 4 nodes: ingest → detect_hype → verify_with_ai → render_video

Requirements

System Dependencies

• Python 3.9+
• FFmpeg and ffprobe
• twitch-archiver

API Keys

• Google Gemini API key (get one at https://ai.google.dev/)

Installation

1. Clone and navigate to the repository

cd /path/to/Agent

2. Install Python dependencies

pip install -r requirements.txt

3. Install system dependencies

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install -y ffmpeg

macOS:

brew install ffmpeg

4. Configure API key

cp .env.example .env
# Edit .env and add your GEMINI_API_KEY

Usage

python main.py https://www.twitch.tv/videos/1234567890

Or with just the VOD ID:

python main.py 1234567890

Example Output

======================================================================
🎬 TwitchVision - AI TikTok Clip Generator
======================================================================
📺 Processing VOD: https://www.twitch.tv/videos/1234567890

🚀 Starting pipeline...

📥 Downloading VOD: https://www.twitch.tv/videos/1234567890
   VOD ID: 1234567890
✅ Downloaded video: 1234567890.mp4 (1234.5 MB)
✅ Downloaded chat: 1234567890_chat.json
   Duration: 7825.0s (130.4 min)

📊 Analyzed 45231 chat messages
🎯 Found 23 hype moments
⭐ Top candidate: 923.0s - 952.0s
   Sentiment: 0.784, Velocity: 3.21 msg/s, Score: 0.712

🤖 Verifying clip with Gemini AI...
   Extracted clip: /tmp/tmpxyz123.mp4
   Uploading to Gemini...
   Analyzing clip...
   🎬 Gemini Verdict: ✅ APPROVED
   Reason: Hilarious streamer fail with perfect comedic timing

🎬 Rendering vertical TikTok clip...
   Time range: 920.0s - 950.0s (30.0s)
   Input: 1920x1080
   Crop: 607x1080 at offset (656, 0)
   Output: 1080x1920
✅ Rendered: tiktok_clip_20250101_123456_920.mp4
   Size: 12.34 MB

======================================================================

✅ SUCCESS! TikTok clip created

📁 Output Details:
   File: ./output/tiktok_clip_20250101_123456_920.mp4
   Duration: 30.0s
   Format: 1080x1920 (9:16 vertical)

📊 Hype Metrics:
   Sentiment Score: 0.784 (-1 to 1)
   Chat Velocity: 3.21 msgs/sec
   Peak Sentiment: 0.912
   Message Count: 96

🤖 AI Verification:
   Status: ✅ Approved
   Reason: Hilarious streamer fail with perfect comedic timing

🎉 Ready to upload to TikTok!
======================================================================

How It Works

1. Ingest Node

• Downloads Twitch VOD and chat using twitch-archiver
• Extracts video duration with ffprobe

2. Detect Hype Node (Sentiment Analysis)

• Parses chat JSON and runs VADER sentiment analysis on each message
• Uses sliding window algorithm (30-second windows, 5-second steps)
• Calculates metrics per window:
  • Average sentiment (VADER compound -1 to 1)
  • Chat velocity (messages/second)
  • Peak sentiment (highest individual message)
• Filters by thresholds:
  • Sentiment > 0.3 (positive)
  • Velocity > 0.5 msgs/sec (active chat)
  • Minimum 15 messages
• Merges overlapping windows
• Ranks by composite score: sentiment*0.4 + velocity*0.4 + peak*0.2

3. Verify with AI Node

• Extracts temporary clip for the top candidate
• Uploads to Google Gemini 1.5 Pro
• AI watches the clip and evaluates:
  • Is there a clear funny moment?
  • Does visual content match chat excitement?
  • Would it be engaging without chat overlay?
• Returns APPROVED or REJECTED with reasoning
• Filters false positives (off-screen events, inside jokes, etc.)

4. Render Video Node

• Adjusts clip to 15-30 second duration
• Uses FFmpeg to:
  • Extract time range
  • Center crop to 9:16 aspect ratio
  • Scale to 1080x1920 (TikTok format)
  • Export with high quality (5Mbps video, 128k audio)

Configuration

Edit config.py to tune parameters:

# Sentiment Detection Thresholds
SENTIMENT_WINDOW_SIZE = 30  # seconds
SENTIMENT_STEP_SIZE = 5     # seconds
MIN_MESSAGES_PER_WINDOW = 15
SENTIMENT_THRESHOLD = 0.3   # VADER compound score
MIN_CHAT_VELOCITY = 0.5     # messages/second

# Clip Duration
MIN_CLIP_DURATION = 15      # seconds
MAX_CLIP_DURATION = 30      # seconds

# Hype Score Weights (should sum to 1.0)
SCORE_SENTIMENT_WEIGHT = 0.4
SCORE_VELOCITY_WEIGHT = 0.4
SCORE_PEAK_WEIGHT = 0.2

Project Structure

twitchvision/
├── main.py                    # CLI entry point
├── graph.py                   # LangGraph state machine
├── config.py                  # Configuration
├── nodes/
│   ├── ingest.py             # VOD download
│   ├── detect_hype.py        # Sentiment analysis
│   ├── verify_with_ai.py     # Gemini verification
│   └── render_video.py       # FFmpeg cropping
├── requirements.txt
├── .env.example
├── .gitignore
├── downloads/                 # Downloaded VODs (gitignored)
└── output/                    # Generated clips (gitignored)

Troubleshooting

"No hype moments detected in chat"

• Try a VOD with more active chat
• Lower SENTIMENT_THRESHOLD in config.py
• Reduce MIN_MESSAGES_PER_WINDOW

"Clip rejected by AI"

• The chat was excited about something not visible in the video
• Try a different VOD
• Check the reasoning in the output

"twitch-archiver not found"

pip install twitch-archiver

"ffmpeg not found"

• Install FFmpeg for your platform (see Installation section)

"GEMINI_API_KEY not set"

• Create .env file with your API key
• Or export it: export GEMINI_API_KEY='your-key'

License

MIT

Contributing

Contributions welcome! Focus areas:

• Improve sentiment detection accuracy
• Add face/character tracking for smarter crop
• Support multiple clips per VOD
• Add chat overlay option
• Optimize processing speed