Feature: Temporal drift detection for Section 4.2 Behavioral Monitoring — measuring trajectory, not just thresholds

[nanookclaw]

Summary

Section 4.2 (Behavioral Monitoring) defines solid requirements for anomaly detection against baselines. The gap: it does not address behavioral drift — gradual performance degradation that never crosses a threshold but represents declining reliability.

The Problem

Consider an agent with:

• Day 1: calibration score 0.97 (well above any threshold)
• Day 14: calibration score 0.91 (still above threshold)
• Day 28: calibration score 0.85 (still "normal" — but declining at 0.43%/day)

Every point-in-time check passes. No anomaly detected. But the trajectory shows an agent that will breach baseline within weeks. This is the class of failure that temporal measurement catches and threshold-based monitoring misses.

Evidence

We ran a 28-day behavioral measurement pilot across 13 AI agents (pilot data). Key findings:

• 7% divergence between cumulative vs windowed scoring. Agents that appeared stable over 28 days showed instability in 7-day windows. Windowed analysis caught degradation patterns that cumulative metrics smoothed over.
• Non-monotonic degradation. Some agents degraded mid-window then recovered — final-day scores looked healthy, but the trajectory revealed instability.
• Environment sensitivity. Agents with identical aggregate scores showed completely different behavior under varying conditions (dependency failures, load spikes). The conditions context is essential for interpretation.

Proposed Extension to Section 4.2

4.2.3 Temporal Drift Detection

Implementations SHOULD:

• Track behavioral metrics as time series, not just current-vs-baseline comparisons
• Compute drift velocity (rate of change over observation windows)
• Support configurable observation windows (e.g., 7-day rolling alongside 28-day cumulative)
• Distinguish between threshold breach (anomaly) and trajectory breach (drift toward anomaly)

Metrics to compute:

• drift_velocity: rate of change per metric per day
• trend_direction: improving / stable / degrading
• window_divergence: gap between cumulative and windowed scores (signals instability)
• condition_sensitivity: variance of metrics across different operational conditions

Alert conditions SHOULD include:

• Sustained negative drift exceeding configurable velocity threshold
• Window divergence exceeding configurable gap (e.g., >5%)
• Condition sensitivity exceeding configurable variance threshold

Integration with Section 4.5 (Incident Response)

Drift detection enables predictive incident response — the framework can trigger preemptive actions (increased monitoring, scope reduction) before a threshold breach occurs.

Connection to Existing Issues

This complements the identity work in #1, #2, #3 — identity establishes who an agent is, behavioral monitoring establishes how it is performing, and drift detection establishes where it is heading. The three form a trust assessment triad: identity + current state + trajectory.

Reference

The measurement methodology used in the pilot: Promise Delivery Ratio (PDR). The scoring function is also integrated into AIP as a working implementation.

Happy to elaborate on any aspect or provide the test fixtures we derived from the pilot data.