feat(monitor): Sprint 21 β continuous monitoring mode (P3-5) (v1.11.0) - #13
Merged
Conversation
P3-5, the last strategic backlog item, unblocked by the P3-2 compliance work. Safety drift happens in production, not at review time β a model behind an endpoint can regress after a deploy or template change. This wires the Sprint 11 regression gate to a live target: probe on a cadence, diff against a frozen baseline, and alert the moment any category regresses beyond tolerance. - toki.monitor: AlertSink ABC + LogSink/CollectingSink/WebhookSink (urllib POST, failures logged not raised); SafetyMonitor.probe runs the generator battery through the real RobustnessEvaluator, establish_baseline freezes a trusted run, check() diffs via toki.regression.compare and dispatches alerts on regression, run(cycles) for cron-driven cadence; ProbeResult/MonitorReport with save/load/to_json; monitor_once wrapper - CLI: python -m toki monitor --model --reference --baseline --tolerance --webhook - toki.__init__ exports; version 1.10.0 -> 1.11.0; pyproject bumped - 25 new tests (22 module + 3 CLI); 741/741 passing; module 100% covered Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WRE1YLhT6aNP4GZT8zbw6q
The Sprint 21 monitor CLI tests were accidentally written to a stray repo-root tests/test_main.py (wrong working directory) where they were never collected and lacked the `main` import. Move them into the real python/tests/test_main.py and remove the stray file. 744/744 tests passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WRE1YLhT6aNP4GZT8zbw6q
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sprint 21 β Continuous Monitoring Mode (P3-5) (v1.11.0)
Motivation
P3-5 β the last strategic backlog item, unblocked by the P3-2 compliance work. Safety drift happens in production, not at review time: a model behind an endpoint can regress after a deploy, a prompt-template change, or a dependency bump. This wires the Sprint 11 regression gate to a live target β probe on a cadence, diff against a frozen baseline, and alert the moment any category regresses beyond tolerance.
What's new β
toki.monitor(zero external deps)AlertSinkABC +LogSink(WARNING),CollectingSink(in-memory),WebhookSink(stdliburllibJSON POST; delivery failures logged, never raised β a flaky webhook can't crash the loop)MonitorConfigΒ·ProbeResultΒ·MonitorReport(to_json/saveno-overwrite/load)SafetyMonitorβprobe()runs the generator battery through the realRobustnessEvaluator;establish_baseline()freezes a trusted run;check()diffs viatoki.regression.compareand dispatches alerts on regression;run(cycles)for cron-driven cadence;monitor_once()wrapperIntegration
python -m toki monitor --model safe|unsafe|mixed --reference --baseline --tolerance --webhook --seed --output-dir [--json]toki.__init__exports +__version__β1.11.0;pyproject.tomlbumpedLive demo
Baseline established from the
safereference; the endpoint then regresses tounsafeβ monitor reports overall Ξ = β0.90 across 4 categories (worst:boundary) and dispatches the alert. An unchanged endpoint reportsstatus: okwith no alert.Verification
toki.monitor100% covered β theWebhookSinkfailure path is exercised offline via a refused127.0.0.1:1connection (no external network)ruff check/formatclean; vulture clean; all functions β€ grade B;cargo testgreenpython/tests/test_main.py(they were briefly written to a stray repo-root file)π€ Generated with Claude Code
https://claude.ai/code/session_01WRE1YLhT6aNP4GZT8zbw6q
Generated by Claude Code