memory-optimization-recommendations.md

Memory Optimization Recommendations for Qdrant RAG

Quick Fixes (Environment Variables)

# Reduce number of models in memory
export QDRANT_MAX_MODELS_IN_MEMORY=1

# Reduce memory limit to force more aggressive eviction
export QDRANT_MEMORY_LIMIT_GB=8.0

# Disable progressive context to reduce caching
export QDRANT_PROGRESSIVE_CONTEXT_ENABLED=false

# Use CPU instead of MPS to avoid MPS memory issues
export QDRANT_DEVICE=cpu

Code Changes for Better Memory Management

1. More Aggressive Memory Cleanup in index_directory

# Add after each file group processing
if files_processed > 0:
    embeddings_manager = get_embeddings_manager_instance()
    if hasattr(embeddings_manager, 'use_specialized') and embeddings_manager.use_specialized:
        # Force model eviction between file types
        if hasattr(embeddings_manager, 'clear_all_models'):
            embeddings_manager.clear_all_models()
        
        # Aggressive garbage collection
        import gc
        gc.collect()
        gc.collect()  # Run twice for thorough cleanup
        
        # On macOS, also try to return memory to system
        import ctypes
        import sys
        if sys.platform == 'darwin':
            libc = ctypes.CDLL('libc.dylib')
            libc.malloc_trim(0)

2. Process Files in Smaller Batches

Instead of grouping all files by type, process in smaller batches:

BATCH_SIZE = 50  # Process 50 files at a time

for group_name, file_list, index_func in file_groups:
    if not file_list:
        continue
    
    # Process in batches
    for i in range(0, len(file_list), BATCH_SIZE):
        batch = file_list[i:i + BATCH_SIZE]
        console_logger.info(f"Processing batch {i//BATCH_SIZE + 1} of {group_name} files...")
        
        for file_path in batch:
            # Process file
            
        # Force cleanup after each batch
        if i % BATCH_SIZE == 0:
            gc.collect()

3. Fix CodeRankEmbed Memory Estimate

Update the memory estimate in specialized_embeddings.py:

memory_estimates = {
    'nomic-ai/CodeRankEmbed': 8.0,  # More realistic estimate
    'jinaai/jina-embeddings-v3': 5.0,  # Also underestimated
    # ... other models
}

4. Add Model-Specific Memory Limits

# In SpecializedEmbeddingManager.__init__
self.model_memory_limits = {
    'code': 8.0,  # Don't load code model if less than 8GB available
    'config': 5.0,
    'documentation': 3.0,
    'general': 1.0
}

5. Use Streaming/Chunked Processing

Instead of loading all files at once, use a generator:

def get_files_to_process(directory, patterns, recursive):
    """Generator that yields files one at a time"""
    for pattern in patterns:
        glob_func = directory.rglob if recursive else directory.glob
        for file_path in glob_func(pattern):
            if should_process_file(file_path):
                yield file_path

Alternative Solutions

1. Use Docker with Memory Limits

   docker run --memory=8g --memory-swap=8g qdrant-rag

2. Use Smaller Models

   export QDRANT_CODE_EMBEDDING_MODEL=microsoft/codebert-base
   export QDRANT_CONFIG_EMBEDDING_MODEL=sentence-transformers/all-mpnet-base-v2

3. Disable Specialized Embeddings

   export QDRANT_SPECIALIZED_EMBEDDINGS_ENABLED=false

Monitoring Memory Usage

Add memory tracking to help identify issues:

import psutil
import os

def log_memory_usage(context):
    process = psutil.Process(os.getpid())
    memory_info = process.memory_info()
    logger.info(f"Memory usage at {context}: "
                f"RSS={memory_info.rss / 1024**3:.2f}GB, "
                f"VMS={memory_info.vms / 1024**3:.2f}GB")

Long-term Fixes

1. Implement model quantization to reduce model sizes
2. Use model sharding to load only necessary parts
3. Implement disk-based caching instead of memory caching
4. Use separate processes for different models (process isolation)