K8s Sentinel 🛡️

K8s Sentinel is an intelligent, autonomous agent that watches your Kubernetes cluster for pod failures, analyzes their logs using a local LLM (Ollama), and performs Root Cause Analysis (RCA) to send actionable alerts to workload owners.

🏗 Architecture

The system consists of two main components:

1. Sentinel (The Eyes): A DaemonSet running on every node that watches for CrashLoopBackOff, OOMKilled, and stuck Pending pods. It collects logs and events intelligently.
2. Brain (The Mind): A centralized FastAPI service that processes the logs using an LLM (Mistral/Llama) and generates an RCA report.

graph TD
    subgraph Cluster [Kubernetes Cluster]
        subgraph Node [Worker Node]
            App[Failing App] -->|Logs/Events| Sentinel[Sentinel Agent]
        end
        Mailpit[Mailpit SMTP]
    end
    
    subgraph Host [Host Machine / AI Server]
        Brain[Brain Service API]
        Ollama[Ollama LLM]
    end

    Sentinel -->|POST /analyze| Brain
    Brain -->|Inference| Ollama
    Brain -->|SMTP| Mailpit
    Mailpit -->|Email| Developer

Loading

✨ Key Features

• 🛡️ Comprehensive Failure Detection: Detects OOMKilled, CrashLoopBackOff, ImagePullBackOff, CreateContainerConfigError, and stuck Pending pods.
• 📩 Smart Routing: Routes alerts to the correct team using the sentinel.owner/email annotation on workloads.
• 🔍 Intelligent Log Capture: Automatically finds the "Error" or "Exception" line in logs and captures relevant context (before/after) to reduce noise.
• 📄 Event Fallback: If logs are unavailable (e.g., ContainerCreating or stuck Pending), it automatically fetches Kubernetes Events instead.
• 🔇 Noise Reduction: Includes built-in throttling to prevent alert storms from the same failing workload.

🚀 Prerequisites

• Docker Desktop (with Kubernetes enabled or Kind)
• Python 3.10+ (for local Brain execution)
• Ollama (running locally with mistral or llama3 model)
• kubectl

🛠️ Setup Guide

This guide assumes a Local-Hybrid setup where the Brain runs on your host machine (to leverage GPU/Ollama easily) and Sentinel runs inside the Kind/K8s cluster.

Phase 1: Prepare the Environment

1. Start Ollama: Ensure Ollama is running and the model is pulled.

   ollama pull mistral:7b-instruct
   ollama serve

2. Create Kubernetes Cluster (if not exists):

   kind create cluster --config kind-config.yaml

3. Setup Python Environment:

   python -m venv venv
   source venv/bin/activate
   pip install -r brain/requirements.txt

Phase 2: Deploy Infrastructure

1. Deploy Mailpit (Email Server):

   kubectl create namespace messaging
   kubectl apply -f deploy/demo/mailpit.yaml

2. Port Forward Mailpit (to view emails):

   # Open http://localhost:8025 in your browser
   kubectl port-forward -n messaging svc/mailpit 8025:8025 1025:1025

Phase 3: Start the Brain (Host)

1. Configure Environment: Create a .env file in the root directory:

   OLLAMA_HOST=http://localhost:11434
   SMTP_HOST=127.0.0.1
   SMTP_PORT=1025
   LOG_LEVEL=INFO

2. Run the Server:

   python -m brain.app.main

   The Brain is now listening on http://0.0.0.0:8000

Phase 4: Deploy Sentinel (Cluster)

1. Build Sentinel Image: We need to build the image and load it into Kind so Kubernetes can find it.

   docker build -t sentinel:v2.0.0 ./sentinel
   kind load docker-image sentinel:v2.0.0

2. Deploy RBAC & DaemonSet:

   kubectl create namespace sentinel
   kubectl apply -f deploy/sentinel/rbac.yaml
   kubectl apply -f deploy/sentinel/daemonset.yaml

🧪 Usage & Demo

To test the system, deploy the provided "broken" demo workloads.

# Deploy all demo workloads (4 total)
kubectl apply -f deploy/demo/workloads/

What happens next?

1. The pod fails.
2. Sentinel detects the failure instantly.
3. Sentinel collects logs (smartly filtering for errors).
4. Sentinel sends data to the Brain.
5. Brain asks Ollama for an RCA.
6. Brain sends an HTML email to the owner (defined in sentinel.owner/email annotation).
7. Check http://localhost:8025 to see the RCA report!

Screenshots

⚙️ Configuration

Sentinel (Env Vars in DaemonSet)

Variable	Default	Description
AI_ENDPOINT	http://host.docker.internal:8000/analyze	Address of the Brain service.
NODE_NAME	(Downward API)	Node the agent is watching.
PATIENCE_THRESHOLD	60	Seconds to wait before re-analyzing a crash.
LOG_TAIL_LINES	1000	Number of log lines to fetch from the container.
CONTEXT_LINES_BEFORE	30	Lines of context to include before the detected error line.
CONTEXT_LINES_AFTER	50	Lines of context to include after the detected error line.

Brain (Env Vars in .env)

Variable	Default	Description
APP_NAME	K8s Log Analyzer Brain	Service name used in OpenAPI docs and logs.
LOG_LEVEL	INFO	Logging verbosity.
HOST	0.0.0.0	Bind host for the Brain HTTP server.
PORT	8000	Bind port for the Brain HTTP server.
OLLAMA_HOST	http://localhost:11434	URL of Ollama API.
OLLAMA_MODEL	mistral:7b-instruct	LLM Model to use.
PROMPT_FILE_PATH	brain/prompts/default_sre.txt	Path to custom system prompt.
LLM_TEMPERATURE	0.2	Model temperature for analysis responses.
SMTP_HOST	127.0.0.1	SMTP Server Host.
SMTP_PORT	1025	SMTP Server Port.
SMTP_SENDER	sentinel@k8s-cluster.local	From address used when sending notifications.
SMTP_TIMEOUT	15	SMTP connection timeout in seconds.

Future scope

This project is extensible and open ended. Some possible extensions:

• Replace Ollama with enterprise LLMs (private cloud / managed providers) via a pluggable model backend.
• Add a RAG-based “internal solutions” knowledge base:
  • Automatically store resolved incidents, RCAs, runbooks, and operator feedback in an internal DB/vector store.
  • Ground future analyses on company-specific history and known fixes for common failure modes.
• Add integrations for notification + workflow:
  • Slack/Teams alerts, and optional ticket creation (Jira/ServiceNow) with the RCA attached.
• Add an “action” service for safe auto-remediation (with human oversight):
  • Propose actions (restart/rollback/scale/apply known fix), require approval, then execute with audit logs and guardrails.

📄 License

MIT License.