Skip to content

release.md

Latest commit

 

History

History
646 lines (482 loc) · 16.8 KB

release.md

File metadata and controls

646 lines (482 loc) · 16.8 KB
Raw

KnowledgeVault Release Documentation

This document provides comprehensive guidance for deploying, maintaining, and operating the KnowledgeVault RAG application in production environments.

Table of Contents

  1. Pre-Deployment Checklist
  2. Encrypted Backup & Rollback Process
  3. Maintenance Flag Usage
  4. Connection Test Guidance
  5. Observability & Monitoring

Pre-Deployment Checklist

Before deploying KnowledgeVault to production, ensure the following:

Infrastructure Requirements

  • SQLite database directory (/data/knowledgevault) with appropriate permissions
  • LanceDB vector store directory with write access
  • Vault directories (/data/knowledgevault/vaults/{vault_id}/uploads/) with sufficient disk space for each vault
  • Redis instance running for CSRF token storage (default: redis://localhost:6379/0)
  • Ollama or OpenAI-compatible embedding service accessible
  • LLM chat service accessible (Ollama, OpenAI, or compatible API)

Configuration

  • Environment variables configured in .env file: 
    # Required settings
DATA_DIR=/data/knowledgevault
OLLAMA_EMBEDDING_URL=http://host.docker.internal:11434
OLLAMA_CHAT_URL=http://host.docker.internal:11434
EMBEDDING_MODEL=bge-m3
CHAT_MODEL=qwen2.5:32b

# Security settings
ADMIN_SECRET_TOKEN=<secure-random-token>
AUDIT_HMAC_KEY_VERSION=v1

# Rate limiting
ADMIN_RATE_LIMIT=10/minute
CSRF_TOKEN_TTL=900

# Feature flags
ENABLE_MODEL_VALIDATION=false

Health Verification

  • Run connection tests (see Connection Test Guidance)
  • Verify /health endpoint returns {"status": "ok"}
  • Verify /api/health returns detailed health status including LLM and models

Encrypted Backup & Rollback Process

Backup Strategy

KnowledgeVault requires backing up two primary data stores:

  1. SQLite Database (/data/knowledgevault/app.db)
  2. LanceDB Vector Store (/data/knowledgevault/lancedb/)

Automated Backup Script

#!/bin/bash
# backup-knowledgevault.sh

BACKUP_DIR="/backup/knowledgevault"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DATA_DIR="/data/knowledgevault"
ENCRYPTION_KEY_FILE="/secure/backup-key.pub"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Backup SQLite database (hot backup with WAL checkpoint)
sqlite3 "$DATA_DIR/app.db" ".backup '$BACKUP_DIR/app_$TIMESTAMP.db'"

# Backup LanceDB (copy with timestamp)
tar -czf "$BACKUP_DIR/lancedb_$TIMESTAMP.tar.gz" -C "$DATA_DIR" lancedb/

# Encrypt backups
gpg --encrypt --recipient-file "$ENCRYPTION_KEY_FILE" \
    --output "$BACKUP_DIR/app_$TIMESTAMP.db.gpg" \
    "$BACKUP_DIR/app_$TIMESTAMP.db"

gpg --encrypt --recipient-file "$ENCRYPTION_KEY_FILE" \
    --output "$BACKUP_DIR/lancedb_$TIMESTAMP.tar.gz.gpg" \
    "$BACKUP_DIR/lancedb_$TIMESTAMP.tar.gz"

# Remove unencrypted files
rm "$BACKUP_DIR/app_$TIMESTAMP.db"
rm "$BACKUP_DIR/lancedb_$TIMESTAMP.tar.gz"

# Retention: Keep last 30 days of backups
find "$BACKUP_DIR" -name "*.gpg" -mtime +30 -delete

echo "Backup completed: $TIMESTAMP"

Restore Process

#!/bin/bash
# restore-knowledgevault.sh

BACKUP_DIR="/backup/knowledgevault"
DATA_DIR="/data/knowledgevault"
TIMESTAMP=$1  # Pass timestamp as argument

if [ -z "$TIMESTAMP" ]; then
    echo "Usage: $0 <timestamp>"
    echo "Available backups:"
    ls -1 "$BACKUP_DIR"/*.gpg | xargs -n1 basename
    exit 1
fi

# Stop application
docker-compose down  # or systemctl stop knowledgevault

# Decrypt backups
gpg --decrypt \
    --output "$DATA_DIR/app.db" \
    "$BACKUP_DIR/app_$TIMESTAMP.db.gpg"

gpg --decrypt \
    --output "/tmp/lancedb_$TIMESTAMP.tar.gz" \
    "$BACKUP_DIR/lancedb_$TIMESTAMP.tar.gz.gpg"

# Restore LanceDB
rm -rf "$DATA_DIR/lancedb"
tar -xzf "/tmp/lancedb_$TIMESTAMP.tar.gz" -C "$DATA_DIR"
rm "/tmp/lancedb_$TIMESTAMP.tar.gz"

# Restart application
docker-compose up -d  # or systemctl start knowledgevault

echo "Restore completed from: $TIMESTAMP"

Verification After Restore

  1. Check database integrity:

    sqlite3 /data/knowledgevault/app.db "PRAGMA integrity_check;"

  2. Verify document count matches expected:

    sqlite3 /data/knowledgevault/app.db "SELECT COUNT(*) FROM files;"

  3. Test search functionality via API:

    curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "test", "history": [], "stream": false}'

Maintenance Flag Usage

The maintenance flag system allows graceful degradation of service during deployments or maintenance windows.

How It Works

When maintenance mode is enabled:

  • GET requests continue to work normally (read-only access)
  • POST, PUT, PATCH, DELETE requests return HTTP 503 with "maintenance" message
  • RAG queries fallback to memory-only mode (no vector search)

Enabling Maintenance Mode

# Via API (requires admin scope)
curl -X POST http://localhost:8080/api/admin/maintenance \
  -H "Content-Type: application/json" \
  -H "X-CSRF-Token: <csrf-token>" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{"enabled": true, "reason": "Database migration in progress"}'

Checking Maintenance Status

curl http://localhost:8080/api/admin/maintenance

Response:

{
  "enabled": true,
  "reason": "Database migration in progress",
  "version": 5,
  "updated_at": "2026-02-06T12:00:00Z"
}

Disabling Maintenance Mode

curl -X POST http://localhost:8080/api/admin/maintenance \
  -H "Content-Type: application/json" \
  -H "X-CSRF-Token: <csrf-token>" \
  -H "Authorization: Bearer <admin-token>" \
  -d '{"enabled": false}'

Maintenance Mode Best Practices

  1. Before maintenance:

    • Enable maintenance mode 5 minutes before starting
    • Communicate maintenance window to users
    • Verify no active uploads or processing jobs

  2. During maintenance:

    • Monitor logs for any blocked write requests
    • Check /api/health for service status

  3. After maintenance:

    • Disable maintenance mode
    • Verify all endpoints return 200
    • Run smoke tests (upload, search, chat)

Connection Test Guidance

Automated Connection Testing

Run the following tests to verify all external dependencies:

1. Database Connection Test

#!/bin/bash
# test-database.sh

echo "Testing SQLite database..."
sqlite3 /data/knowledgevault/app.db "SELECT 1;" && echo "✓ Database OK" || echo "✗ Database FAILED"

echo "Testing required tables..."
for table in files chunks memories document_actions admin_toggles system_flags; do
    sqlite3 /data/knowledgevault/app.db "SELECT COUNT(*) FROM $table;" > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        echo "✓ Table $table exists"
    else
        echo "✗ Table $table missing"
    fi
done

2. Embedding Service Test

#!/bin/bash
# test-embedding.sh

EMBEDDING_URL="${OLLAMA_EMBEDDING_URL:-http://localhost:11434}/api/embeddings"
EMBEDDING_MODEL="${EMBEDDING_MODEL:-bge-m3}"

echo "Testing embedding service at $EMBEDDING_URL..."

response=$(curl -s -X POST "$EMBEDDING_URL" \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"$EMBEDDING_MODEL\", \"prompt\": \"test\"}" \
  -w "\n%{http_code}")

http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | sed '$d')

if [ "$http_code" = "200" ]; then
    echo "✓ Embedding service OK"
    # Verify embedding dimension
    dim=$(echo "$body" | python3 -c "import sys,json; print(len(json.load(sys.stdin)['embedding']))")
    echo "  Embedding dimension: $dim"
else
    echo "✗ Embedding service FAILED (HTTP $http_code)"
    echo "  Response: $body"
fi

3. LLM Chat Service Test

#!/bin/bash
# test-llm.sh

CHAT_URL="${OLLAMA_CHAT_URL:-http://localhost:11434}/api/chat"
CHAT_MODEL="${CHAT_MODEL:-qwen2.5:32b}"

echo "Testing LLM service at $CHAT_URL..."

response=$(curl -s -X POST "$CHAT_URL" \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"$CHAT_MODEL\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}], \"stream\": false}" \
  -w "\n%{http_code}")

http_code=$(echo "$response" | tail -n1)

if [ "$http_code" = "200" ]; then
    echo "✓ LLM service OK"
else
    echo "✗ LLM service FAILED (HTTP $http_code)"
fi

4. Redis Connection Test

#!/bin/bash
# test-redis.sh

REDIS_URL="${REDIS_URL:-redis://localhost:6379/0}"

echo "Testing Redis at $REDIS_URL..."

# Extract host and port from URL
redis_host=$(echo "$REDIS_URL" | sed -E 's|redis://([^:]+):.*|\1|')
redis_port=$(echo "$REDIS_URL" | sed -E 's|redis://[^:]+:([0-9]+).*|\1|')

if redis-cli -h "$redis_host" -p "$redis_port" ping | grep -q "PONG"; then
    echo "✓ Redis OK"
else
    echo "✗ Redis FAILED"
fi

5. Full Integration Test

#!/bin/bash
# test-integration.sh

API_BASE="${API_BASE:-http://localhost:8080}"

echo "Running integration tests against $API_BASE..."

# Test health endpoint
echo -n "Health check... "
if curl -sf "$API_BASE/health" > /dev/null; then
    echo "✓"
else
    echo "✗ FAILED"
    exit 1
fi

# Test detailed health
echo -n "Detailed health... "
health=$(curl -sf "$API_BASE/api/health")
if echo "$health" | grep -q '"status": "ok"'; then
    echo "✓"
    echo "  LLM: $(echo "$health" | python3 -c "import sys,json; print('OK' if json.load(sys.stdin)['llm']['ok'] else 'FAIL')")"
    echo "  Models: $(echo "$health" | python3 -c "import sys,json; d=json.load(sys.stdin)['models']; print(f\"embed={d['embedding_model']['available']}, chat={d['chat_model']['available']}\")")"
else
    echo "✗ FAILED"
fi

# Test document upload (if auth available)
echo -n "Document upload... "
upload_response=$(curl -sf -X POST "$API_BASE/api/documents/upload" \
  -F "file=@/tmp/test_doc.txt" 2>/dev/null)
if [ $? -eq 0 ]; then
    echo "✓"
else
    echo "⚠ SKIPPED (auth required)"
fi

# Test memory operations
echo -n "Memory operations... "
if curl -sf "$API_BASE/api/memories" > /dev/null; then
    echo "✓"
else
    echo "✗ FAILED"
fi

echo "Integration tests complete"

Python Test Suite

Run the comprehensive Python test suite:

# Run all tests
cd backend
python -m pytest tests/ -v

# Run specific test categories
python -m pytest tests/test_integration.py -v
python -m pytest tests/test_rag_engine.py -v
python -m pytest tests/test_document_processor.py -v

# Run with coverage
python -m pytest tests/ --cov=app --cov-report=html

Observability & Monitoring

Request ID Tracking

Every HTTP request is assigned a unique X-Request-ID header for distributed tracing:

  • Request: Client may provide X-Request-ID header (optional)
  • Response: Server always returns X-Request-ID header
  • Logs: All request logs include the request_id field

Example Usage

# Make request with custom request ID
curl -H "X-Request-ID: req-123-abc" http://localhost:8080/api/health

# Response includes the same ID
# X-Request-ID: req-123-abc

Correlating Logs

# Find all logs for a specific request
grep "request_id.*req-123-abc" /var/log/knowledgevault/app.log

# Or using structured logging (JSON)
jq 'select(.request_id == "req-123-abc")' /var/log/knowledgevault/app.log

Log Redaction Fields

Sensitive fields are automatically redacted in logs:

Field Redaction Reason
message May contain user input with PII
content Chat messages may contain sensitive data
user_input Direct user queries
token Authentication tokens
email User email addresses
username User identifiers
file_path Internal file system paths
ip Client IP addresses (privacy)
session_id Session identifiers
api_key API credentials
authorization Auth headers
secret Any secret values

Redaction Behavior

Values are replaced with [redacted] in logs:

# Before redaction
{"message": "My password is secret123", "user": "john"}

# After redaction
{"message": "[redacted]", "user": "john"}

Structured Logging Format

All HTTP requests are logged in structured JSON format:

{
  "timestamp": "2026-02-06T12:34:56.789Z",
  "level": "INFO",
  "logger": "http",
  "message": "http_request",
  "request_id": "550e8400-e29b-41d4-a716-446655440000",
  "method": "POST",
  "path": "/api/chat",
  "query": "",
  "status_code": 200,
  "client_ip": "[redacted]",
  "duration_ms": 145.32
}

Key Metrics to Monitor

Application Metrics

Metric Description Alert Threshold
http_request_duration_ms Request latency P99 > 5000ms
http_requests_total Request rate Baseline ± 50%
http_errors_total Error rate > 1% of total
rag_query_duration_ms RAG query latency P99 > 10000ms
embedding_request_duration_ms Embedding latency P99 > 3000ms
llm_request_duration_ms LLM latency P99 > 30000ms

Database Metrics

Metric Description Alert Threshold
sqlite_connection_pool_size Active connections > 80% of max
lancedb_query_duration_ms Vector search latency P99 > 1000ms
files_total Total indexed documents Track growth
chunks_total Total indexed chunks Track growth

External Service Metrics

Metric Description Alert Threshold
ollama_embedding_availability Embedding service up < 99%
ollama_chat_availability LLM service up < 99%
redis_availability Redis connection < 99%

Log Analysis Queries

Find Slow Requests

# Using jq with structured logs
jq 'select(.duration_ms > 5000) | {path, method, duration_ms, request_id}' app.log

Error Rate by Endpoint

# Count errors by endpoint
jq -r 'select(.status_code >= 400) | "\(.status_code) \(.method) \(.path)"' app.log | \
  sort | uniq -c | sort -rn | head -20

RAG Query Performance

# Extract RAG query timings from logs
grep "rag_query" app.log | \
  jq -s 'map(.duration_ms) | {count: length, avg: (add/length), max: max}'

Health Check Endpoints

Endpoint Purpose Frequency
GET /health Basic liveness Every 30s
GET /api/health Full health with dependencies Every 60s
GET /api/settings Configuration verification On demand

Alerting Rules

Example Prometheus alerting rules:

groups:
  - name: knowledgevault
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          
      - alert: SlowRAGQueries
        expr: histogram_quantile(0.99, rate(rag_query_duration_ms_bucket[5m])) > 10000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "RAG queries are slow"
          
      - alert: EmbeddingServiceDown
        expr: up{job="ollama-embedding"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Embedding service is down"
          
      - alert: MaintenanceModeActive
        expr: maintenance_mode_enabled == 1
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "Maintenance mode has been active for over 1 hour"

Support & Troubleshooting

Common Issues

1. Document Upload Fails

Symptoms: 413 Payload Too Large or 500 Server Error

Resolution:

  • Check max_file_size_mb setting (default: 50MB)
  • Verify uploads directory has disk space
  • Check document processor logs for parsing errors

2. Search Returns No Results

Symptoms: Chat responses don't reference documents

Resolution:

  • Verify documents show as "indexed" in database
  • Check vector store connectivity
  • Verify rag_relevance_threshold setting (default: 0.1)
  • Test embedding service is generating vectors

3. Chat Stream Interrupted

Symptoms: Streaming responses cut off mid-response

Resolution:

  • Check LLM service timeout settings
  • Verify network stability between app and LLM
  • Review llm_request_duration_ms metrics

Getting Help

  1. Check application logs with request ID
  2. Run connection test scripts
  3. Verify health endpoint responses
  4. Review this documentation for relevant section

Release Notes

Version 0.1.0 (Phase 6)

  • Added comprehensive integration test suite
  • Implemented encrypted backup/rollback procedures
  • Added maintenance mode flag system
  • Enhanced observability with request ID tracking
  • Structured logging with sensitive field redaction
  • Rate limiting on admin endpoints
  • CSRF protection for state-changing operations
  • HMAC audit logging for document operations