No eval harness to measure candidate-LLM RCA accuracy / confidence calibration against Kubernaut's own demo scenarios

[jordigilh]

Summary

Kubernaut has no automated way to measure how well a candidate LLM (self-hosted or hosted) actually performs on Kubernaut's own investigation workload, or whether its self-reported confidence score (see below) is a meaningfully predictive signal. This gap surfaced while documenting a v1.5.2 local/self-hosted pilot setup for gpt-oss-120b (see docs/development/getting-started/setup/LOCAL_SELF_HOSTED_LLM_v1.5.2.md §5).

Why this matters: the confidence threshold is the only auto-approval gate

The Rego approval policy's confidence_threshold (rego.confidenceThreshold, default 0.8) is, in practice, the only confidence-based control standing between an LLM's remediation suggestion and unattended execution (pkg/aianalysis/testdata/policies/approval.rego):

is_high_confidence if {
    input.confidence >= confidence_threshold
}

input.confidence is entirely LLM self-report — a byproduct of the model narrating its own uncertainty in the "Pre-Submit Adversarial Due Diligence" step of the RCA prompt (internal/kubernautagent/prompt/templates/incident_investigation.tmpl, dimension 8, "Confidence Calibration": "Start at 1.0 and list each factor that reduced it..."). There is no log-prob-based score, no independent verifier model, no self-consistency/ensemble check, and no historical calibration curve behind it. The JSON schema (internal/kubernautagent/parser/schema.go) only enforces that the calibration field is present as a string, not that it's truthful or substantive.

This means: for any given model (especially a self-hosted/weaker one being evaluated for a pilot), nobody currently knows whether its self-reported confidence correlates with actual RCA correctness. Tuning confidence_threshold up or down for such a model is a guess, not a measurement.

Existing asset that's not wired up for this

Demo scenarios already exist as a specification (docs/requirements/BR-PLATFORM-002-demo-scenario-specification.md), but nothing today replays them against a candidate model and compares self-reported confidence against human-verified ground-truth correctness (right root cause? right remediation target? right severity?).

Proposed fix

Build a lightweight eval harness that:

1. Takes the BR-PLATFORM-002 demo scenarios (or an equivalent curated incident corpus with known-correct RCA/remediation-target/severity ground truth) as fixed input.
2. Runs each scenario through KA's investigation loop against a configurable model/provider.
3. Records the model's root_cause_analysis, remediation_target, severity, and self-reported confidence for each scenario.
4. Compares against ground truth to produce a calibration report: for each confidence bucket (e.g. 0.5-0.6, 0.6-0.7, ..., 0.9-1.0), what fraction of RCAs in that bucket were actually correct?
5. Surfaces this as a reusable command (e.g. make eval-model MODEL=...) so any candidate model (self-hosted or hosted) can be benchmarked before a pilot, and rego.confidenceThreshold can be set from measured calibration data rather than a guess.

Impact if left unaddressed

Every self-hosted/local-model pilot (and any hosted-model swap, for that matter) proceeds on an unvalidated assumption that a chosen confidence threshold provides real safety margin. The threshold either passes bad RCAs that happen to be confidently wrong, or blocks good RCAs that happen to be modestly self-doubting — and nobody can tell which is happening without this harness.

Related

• Local self-hosted setup guide: docs/development/getting-started/setup/LOCAL_SELF_HOSTED_LLM_v1.5.2.md
• BR-PLATFORM-002 (demo scenario specification): docs/requirements/BR-PLATFORM-002-demo-scenario-specification.md
• Approval policy: pkg/aianalysis/rego/evaluator.go, pkg/aianalysis/testdata/policies/approval.rego