DEEPEDGE is a modular, production-style computer vision pipeline that integrates multiple state-of-the-art models into a single workflow:
-> Text-to-Image Generation (Stable Diffusion)
-> Image–Text Alignment Scoring (CLIP)
-> Instance Segmentation (SAM2 – Segment Anything Model 2)
The project focuses not only on model inference, but also on system design, fault tolerance, and reproducibility, closely resembling how real-world ML pipelines are built.
i)Build an end-to-end vision pipeline using multiple models
ii)Support both CPU and GPU execution
iii)Design a modular and extensible architecture
iv)Persist pipeline results for traceability and debugging
v)Handle third-party library failures gracefully
The project is organized using a modular and scalable structure to clearly separate model logic, pipeline orchestration, testing, and documentation. This design improves readability, maintainability, and makes it easy to extend or replace individual components.
Each major model (Stable Diffusion, CLIP, SAM 2) is encapsulated in its own module, while a central pipeline coordinates execution and data flow.
## 4)Pipeline Architecture
Text Prompt
↓
Stable Diffusion (Image Generation)
↓
CLIP (Image–Text Similarity Scoring)
↓
SAM2 (Instance Segmentation)
↓
Persist Results (Database + Artifacts)
## 5)Environment Setup
i)Create Virtual Environment
python -m venv venv
source venv/bin/activate # Linux / macOS
venv\Scripts\activate # Windows
ii)Install Dependencies
pip install -r requirements.txt
## 6)Model Components
i)Stable Diffusion
Converts text prompts into images
Primary generative component of the pipeline
ii)CLIP
Computes semantic similarity between text and generated images
Used for validation and ranking
iii)SAM2 (Segment Anything Model 2)
Performs instance-level segmentation
Operates on generated images
Optional step due to current external limitations
## Q.Why a Database Is Used??(PostgreSQL)
The pipeline connects to a database to make the system stateful and reproducible, rather than a one-time script.
The database stores:
1.Text prompts
2.Image file paths
3.CLIP similarity scores
4.Segmentation status and metadata
## Benefits:
1.Reproducibility of results
2.Debugging past runs
3.Graceful handling of partial failures
4.Decoupling between pipeline stages
5.Required persistence for containerized execution
## Known Issue: SAM2 Library Bug
Problem Description:
The current SAM2 library contains a bug in build_sam.py.
If the required checkpoint file is missing:
sam2/checkpoints/sam2_hiera_s.pt
the library:
Fails to load the model
Attempts to call .to(device) on a configuration object
Produces a misleading error such as:
Missing key to
## 7. Documentation
### 7.1 API Documentation
#### Base URL
#### Health Check Endpoint
**GET** `/`
Check API status and availability.
**Response:**
```json
{
"status": "running"
}
POST /generate
Generates an image from a text prompt, computes CLIP similarity score, and optionally performs instance segmentation.
Request Body:
{
"prompt": "A serene landscape with mountains and a lake at sunset"
}Response Model:
{
"image_path": "images/2026/01/generated_image_uuid.png",
"clip_analysis": {
"concepts": ["serene landscape with mountains and lake"],
"confidence": 0.85
},
"segmentation": "completed"
}Status Codes:
200 OK: Image generated successfully400 Bad Request: Invalid prompt (empty or null)500 Internal Server Error: Pipeline failure (model loading, generation error, etc.)
Example Using cURL:
curl -X POST "http://localhost:8000/generate" \
-H "Content-Type: application/json" \
-d '{"prompt": "A futuristic city with flying cars"}'Example Using Python:
import requests
url = "http://localhost:8000/generate"
payload = {
"prompt": "A serene landscape with mountains and a lake at sunset"
}
response = requests.post(url, json=payload)
result = response.json()
print(f"Image saved at: {result['image_path']}")
print(f"CLIP Score: {result['clip_analysis']['confidence']}")POST /analyze
Analyzes an uploaded image for segmentation and feature extraction.
Note: Currently under implementation. Will support:
- Instance segmentation via SAM2
- Feature extraction
- Detailed mask analysis
Requirements:
- NVIDIA GPU with CUDA support
- CUDA Toolkit 11.8+
- cuDNN 8.x
Steps:
- Clone the repository:
git clone https://github.com/your-username/deepedge.git
cd deepedge- Create virtual environment:
python -m venv venv
venv\Scripts\activate # Windows
source venv/bin/activate # Linux/macOS- Install PyTorch for GPU:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118- Install dependencies:
pip install -r requirements.txt- Verify GPU availability:
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"- Run API:
uvicorn app.api.main:app --reload --host 0.0.0.0 --port 8000Requirements:
- Python 3.10+
- 8GB+ RAM recommended
- ~20GB disk space for models
Steps:
- Clone the repository:
git clone https://github.com/your-username/deepedge.git
cd deepedge- Create virtual environment:
python -m venv venv
venv\Scripts\activate # Windows
source venv/bin/activate # Linux/macOS- Install PyTorch for CPU:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu- Install dependencies:
pip install -r requirements.txt- Configure environment variables (optional):
# Force CPU usage
set CUDA_VISIBLE_DEVICES=-1 # Windows
export CUDA_VISIBLE_DEVICES=-1 # Linux/macOS- Run API:
uvicorn app.api.main:app --reload --host 0.0.0.0 --port 8000Build Docker Image:
docker build -t deepedge:latest .Run Container (CPU):
docker run -p 8000:8000 \
-v $(pwd)/images:/app/images \
-v $(pwd)/logs:/app/logs \
deepedge:latestRun Container (GPU):
docker run --gpus all -p 8000:8000 \
-v $(pwd)/images:/app/images \
-v $(pwd)/logs:/app/logs \
deepedge:latestAPI will be available at: http://localhost:8000
File: app/models/stable_diffusion.py
Key Parameters:
{
"model_id": "runwayml/stable-diffusion-v1-5",
"device": "cuda" or "cpu", # Auto-detected
"dtype": torch.float16, # GPU: float16, CPU: float32
"num_inference_steps": 50, # More steps = better quality (slower)
"guidance_scale": 7.5, # Text-image alignment strength
"height": 512,
"width": 512,
"seed": None # Set for reproducibility
}Customization Example:
from app.models.stable_diffusion import StableDiffusionModel
# Initialize with custom config
model = StableDiffusionModel()
image = model.generate_image(
prompt="A cyberpunk city street at night",
num_inference_steps=75, # Slower but higher quality
guidance_scale=10.0 # Stronger text adherence
)File: app/models/clip_model.py
Key Parameters:
{
"model_name": "ViT-B/32", # Vision Transformer variant
"device": "cuda" or "cpu",
"normalize": True, # Normalize embeddings
"batch_size": 32
}Output:
- CLIP Score (0.0 to 1.0): Measure of text-image alignment
- Confidence: How well the generated image matches the prompt
Example:
from app.models.clip_model import compute_clip_score
from PIL import Image
image = Image.open("generated_image.png")
prompt = "A serene mountain landscape"
score = compute_clip_score(image, prompt)
print(f"Alignment Score: {score:.4f}") # Output: 0.8450File: app/models/sam2.py
Key Parameters:
{
"model_cfg": "sam2_hiera_s", # Model size: s, b, l
"checkpoint_path": "sam2/checkpoints/sam2_hiera_s.pt",
"device": "cuda" or "cpu",
"image_size": 1024
}Model Variants:
| Variant | Size | Memory | Speed | Accuracy |
|---|---|---|---|---|
| tiny | 100M | 2GB | Fast | Good |
| small | 200M | 4GB | Medium | Better |
| base | 400M | 8GB | Slow | Best |
| large | 700M | 12GB+ | V.Slow | Best |
Checkpoint Download:
# Auto-downloaded on first use or manually:
wget https://dl.fbaipublicfiles.com/segment_anything_2/checkpoints/sam2_hiera_s.pt \
-O sam2/checkpoints/sam2_hiera_s.ptUsage Example:
from app.models.sam2 import SAM2Model
from PIL import Image
# Initialize
sam2_model = SAM2Model(
model_cfg="sam2_hiera_s",
checkpoint_path="sam2/checkpoints/sam2_hiera_s.pt"
)
# Segment image
image = Image.open("generated_image.png")
masks, logits = sam2_model.segment(image)
print(f"Number of segments: {len(masks)}")
print(f"Mask shape: {masks[0].shape}")File: src/db/postgres.py
Connection Setup:
import os
DB_CONFIG = {
"host": os.getenv("DB_HOST", "localhost"),
"port": int(os.getenv("DB_PORT", 5432)),
"database": os.getenv("DB_NAME", "deepedge_db"),
"user": os.getenv("DB_USER", "postgres"),
"password": os.getenv("DB_PASSWORD", "your_password")
}Environment Variables (.env):
DB_HOST=localhost
DB_PORT=5432
DB_NAME=deepedge_db
DB_USER=postgres
DB_PASSWORD=secure_password
DB_POOL_SIZE=10
PostgreSQL Installation (Windows):
- Download and install PostgreSQL from https://www.postgresql.org/download/windows/
- Create database:
createdb deepedge_db - Test connection:
psql -U postgres -d deepedge_db
PostgreSQL via Docker:
docker run --name deepedge-postgres \
-e POSTGRES_DB=deepedge_db \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=secure_password \
-p 5432:5432 \
-d postgres:15-alpinedeepedge/
├── app/
│ ├── api/ # FastAPI routes and schemas
│ │ ├── main.py # FastAPI app initialization
│ │ ├── routes.py # API endpoints
│ │ ├── schemas.py # Request/response models
│ │ └── __init__.py
│ ├── models/ # ML model wrappers
│ │ ├── stable_diffusion.py # Text-to-image generation
│ │ ├── clip_model.py # Text-image alignment scoring
│ │ ├── sam2.py # Instance segmentation
│ │ └── __init__.py
│ ├── pipeline/ # End-to-end pipeline logic
│ │ ├── generation_pipeline.py
│ │ └── __init__.py
│ └── __init__.py
├── src/
│ ├── db/ # Database operations
│ │ ├── postgres.py # PostgreSQL connection
│ │ ├── image_repo.py # Image metadata CRUD
│ │ └── __init__.py
│ ├── storage/ # File storage operations
│ │ ├── image_store.py # Save generated images
│ │ ├── mask_store.py # Save segmentation masks
│ │ └── __init__.py
│ ├── exception.py # Custom exception handling
│ ├── logger.py # Logging configuration
│ └── __init__.py
├── tests/ # Unit and integration tests
│ ├── test_api.py
│ ├── test_pipeline_errors.py
│ ├── test_clip_errors.py
│ └── __init__.py
├── images/ # Generated images (organized by date)
├── logs/ # Application logs
├── Dockerfile # Multi-stage Docker build
├── requirements.txt # Python dependencies
├── setup.py # Package configuration
├── pytest.ini # Pytest configuration
└── README.md # This file
Custom Exception Handling:
# File: src/exception.py
class CustomException(Exception):
"""Centralized exception handler with traceback preservation"""
def __init__(self, error_message, error_details):
self.error_message = error_message
self.error_details = error_detailsImplemented Error Handling:
-
Empty Prompt Validation:
- Prevents pipeline execution with empty text
- Returns
400 Bad Requestwith meaningful message
-
Model Loading Errors:
- Graceful fallback for SAM2 failures
- Logs detailed error context for debugging
-
Storage Failures:
- Image save failures don't crash pipeline
- Logged for manual intervention
-
Database Connection Errors:
- Connection pooling prevents resource exhaustion
- Retry logic for transient failures
-
Device Management:
- Auto-detection of GPU availability
- Fallback to CPU if CUDA unavailable
Unit Tests:
-
API Tests (
tests/test_api.py)- Health check endpoint
- Response schema validation
- HTTP status code verification
-
Pipeline Error Tests (
tests/test_pipeline_errors.py)- Empty prompt handling
- Pipeline failure recovery
- Exception propagation
-
Model Tests (
tests/test_clip_errors.py)- CLIP score computation
- Image-text alignment validation
Running Tests:
# Run all tests
pytest
# Run with coverage report
pytest --cov=app --cov=src
# Run specific test file
pytest tests/test_api.py -v
# Run with detailed output
pytest -v --tb=shortTest Configuration (pytest.ini):
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = -v --strict-markersFile: src/logger.py
Log Levels:
DEBUG: Detailed information for diagnosing problemsINFO: Confirmation that pipeline is working as expectedWARNING: Warning about potential issuesERROR: Error occurred but pipeline recoveredCRITICAL: Critical error requiring immediate attention
Log Output:
- Console: Real-time monitoring
- File:
logs/deepedge.log- Persistent record
Example Log Entries:
[2026-01-18 10:45:30] INFO - DEEPEDGE API started successfully
[2026-01-18 10:45:35] INFO - PIPELINE STARTED
[2026-01-18 10:45:40] INFO - Loading Stable Diffusion model...
[2026-01-18 10:46:20] INFO - Image generated: 512x512 pixels
[2026-01-18 10:46:25] INFO - CLIP score computed: 0.8450
[2026-01-18 10:46:30] INFO - Attempting to load SAM2 model
[2026-01-18 10:46:35] INFO - PIPELINE COMPLETED
Image Storage:
images/
├── 2026/
│ ├── 01/
│ │ ├── 550e8400-e29b-41d4-a716-446655440000.png
│ │ ├── 550e8400-e29b-41d4-a716-446655440001.png
│ │ └── ...
│ └── 02/
│ └── ...
└── [year]/
└── [month]/
└── [uuid].png
Segmentation Masks:
masks/
├── 2026-01-18/
│ ├── 550e8400-e29b-41d4-a716-446655440000/
│ │ ├── mask_0.npy
│ │ ├── mask_1.npy
│ │ └── metadata.json
│ └── ...
Step-by-Step Execution:
1. API Request
├─> Validate prompt (non-empty)
├─> Generate UUID for request tracking
└─> Pass to pipeline executor
2. Image Generation
├─> Load Stable Diffusion model (GPU/CPU)
├─> Tokenize prompt
├─> Run diffusion process (50 steps default)
└─> Return PIL Image object
3. CLIP Scoring
├─> Load CLIP model (ViT-B/32)
├─> Encode image features
├─> Encode text features
└─> Compute cosine similarity (0.0-1.0)
4. Image Persistence
├─> Convert PIL Image to PNG
├─> Save to: images/[YYYY]/[MM]/[UUID].png
└─> Record file path for metadata
5. Optional Segmentation
├─> Attempt to load SAM2 model
├─> If successful:
│ ├─> Segment image into instances
│ ├─> Save mask files (NPY format)
│ └─> Log mask count
└─> If failed:
├─> Log warning
└─> Continue without masks
6. Metadata Storage
├─> Connect to PostgreSQL
├─> Insert record with:
│ ├─> Prompt text
│ ├─> Image file path
│ ├─> CLIP score
│ ├─> Mask directory
│ ├─> Timestamp
│ └─> Request ID
└─> Return response with all results
7. Response
├─> Image path
├─> CLIP analysis (concepts + confidence)
└─> Segmentation status
Request:
curl -X POST "http://localhost:8000/generate" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A majestic eagle soaring over snow-capped mountains at golden hour"
}'Response (200 OK):
{
"image_path": "images/2026/01/a7f1d8c9-e4b2-4f3a-9d6c-2b8e5f1c3d4a.png",
"clip_analysis": {
"concepts": [
"majestic eagle soaring over snow-capped mountains at golden hour"
],
"confidence": 0.8632
},
"segmentation": "completed"
}Request:
curl -X POST "http://localhost:8000/generate" \
-H "Content-Type: application/json" \
-d '{
"prompt": ""
}'Response (400 Bad Request):
{
"detail": "Prompt cannot be empty"
}Request:
curl -X GET "http://localhost:8000/"Response (200 OK):
{
"status": "running"
}- SAM2 model only loads if explicitly used
- Prevents startup delays for image-only generation
- Graceful handling if checkpoint unavailable
- Image generation runs in thread pool
- Prevents blocking of other requests
- Supports concurrent pipeline runs
- Unique UUID for each request
- Complete audit trail in logs
- Enables debugging of specific requests
- Reduces final image size by separating build and runtime stages
- Only runtime dependencies in final image
- Optimized for production deployment
- Automatic GPU detection via PyTorch
- Fallback to CPU if CUDA unavailable
- Precision optimization (float16 on GPU, float32 on CPU)
| Configuration | Device | Steps | Time |
|---|---|---|---|
| Stable Diffusion | NVIDIA A100 | 50 | ~5 seconds |
| Stable Diffusion | NVIDIA V100 | 50 | ~12 seconds |
| Stable Diffusion | CPU (16 cores) | 50 | ~120 seconds |
| SAM2 Segmentation | NVIDIA A100 | - | ~2 seconds |
| Component | GPU Memory | RAM |
|---|---|---|
| Stable Diffusion | ~6GB | 2GB |
| CLIP Model | ~1.5GB | 500MB |
| SAM2 | ~4GB | 2GB |
| Total (concurrent) | ~11.5GB | 4.5GB |
-
Reduce inference steps:
model.generate_image(prompt, num_inference_steps=30) # Faster, lower quality
-
Use smaller SAM2 variant:
SAM2Model(model_cfg="sam2_hiera_t") # Tiny variant
-
Enable model quantization (coming in future release)
-
Batch requests when possible
Error: RuntimeError: CUDA out of memory
Solutions:
# Reduce batch size
# Use smaller model variant
# Clear GPU cache
python -c "import torch; torch.cuda.empty_cache()"
# Force CPU execution
export CUDA_VISIBLE_DEVICES=-1Error: FileNotFoundError: sam2/checkpoints/sam2_hiera_s.pt
Solutions:
# Manually download checkpoint
mkdir -p sam2/checkpoints
wget https://dl.fbaipublicfiles.com/segment_anything_2/checkpoints/sam2_hiera_s.pt \
-O sam2/checkpoints/sam2_hiera_s.pt
# Or let it auto-download on first useError: psycopg2.OperationalError: could not connect to server
Solutions:
# Verify PostgreSQL is running
psql -U postgres -h localhost
# Check environment variables
echo $DB_HOST
echo $DB_USER
# Use Docker PostgreSQL if local install fails
docker run --name deepedge-db \
-e POSTGRES_PASSWORD=password \
-p 5432:5432 \
-d postgres:15Error: urllib.error.URLError: urlopen error [Errno 11001]
Solutions:
# Check internet connection
ping huggingface.co
# Use local model caching
export HF_HOME=/path/to/cache
# Download models manually and point to local pathdeepedge/
├── README.md # Complete documentation
├── SETUP_GUIDE.md # Detailed setup instructions
├── API_DOCUMENTATION.md # API reference
├── requirements.txt # All dependencies
├── setup.py # Package info
├── pytest.ini # Test configuration
├── Dockerfile # Production-ready image
├── .dockerignore # Docker build optimization
├── .gitignore # Git ignore rules
├── app/ # Source code
│ ├── api/
│ ├── models/
│ ├── pipeline/
│ └── __init__.py
├── src/ # Core utilities
│ ├── db/
│ ├── storage/
│ ├── exception.py
│ ├── logger.py
│ └── __init__.py
├── tests/ # Test suite
│ ├── test_api.py
│ ├── test_pipeline_errors.py
│ ├── test_clip_errors.py
│ └── __init__.py
├── examples/ # Usage examples
│ ├── basic_generation.py
│ ├── batch_processing.py
│ └── docker_deployment.md
├── logs/ # Application logs (git ignored)
└── images/ # Generated outputs (git ignored)# Virtual environments
venv/
env/
ENV/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# IDEs
.vscode/
.idea/
*.swp
*.swo
*~
# Generated files
logs/
*.log
images/
masks/
.DS_Store
# Environment variables
.env
.env.local
# Database
*.db
*.sqlite
# IDE settings
.pylintrc
.flake8
# Initialize git repository
git init
git add .
git commit -m "Initial commit: DEEPEDGE - Multi-model vision pipeline"
# Add remote repository
git remote add origin https://github.com/your-username/deepedge.git
# Push to GitHub
git branch -M main
git push -u origin mainname: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.10, "3.11"]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: pytest --cov=app --cov=src# 1. Clone repository
git clone https://github.com/your-username/deepedge.git
cd deepedge
# 2. Create environment
python -m venv venv
venv\Scripts\activate # Windows
# 3. Install dependencies
pip install -r requirements.txt
# 4. Run API
uvicorn app.api.main:app --reload
# 5. Test endpoint
curl -X POST "http://localhost:8000/generate" \
-H "Content-Type: application/json" \
-d '{"prompt": "A beautiful sunset"}'# 1-4: Same as above, but install GPU PyTorch
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
# 5. Run API
uvicorn app.api.main:app --reload
# 6. Verify GPU detection
python -c "import torch; print('GPU:', torch.cuda.is_available())"# Build and run in one command
docker build -t deepedge:latest . && \
docker run -p 8000:8000 deepedge:latest
# API available at http://localhost:8000Author: Ritvik
Email: ritvikk92@gmail.com
GitHub: https://github.com/your-username/deepedge
For issues, questions, or contributions:
- Check existing GitHub Issues
- Review Troubleshooting Section
- Create new issue with detailed error logs
- Submit pull requests for improvements
This project is provided as-is for educational and portfolio purposes. Some components use pre-trained models from Meta (SAM2) and OpenAI (CLIP), which have their respective licenses.
Last Updated: January 18, 2026
Project Status: Production-Ready (Internship Submission)
Version: 1.0.0