ci: one runner for the WCM demos, callable from the SDK repository - #92
Merged
Conversation
weight-custody-manifest 0.27.0 shipped two correct security changes and both broke demos here: release began refusing manifests whose identity was not pinned out of band, and the memory-fingerprint challenge began requiring a signed sweep. Six demos that passed on 0.26.0 failed on 0.27.0, and nothing caught it. Nothing could have. The list of demos to run lived inside this repository's workflow file, where the SDK's CI cannot reach it, and this repository only ever tests against a version already on PyPI. So the earliest possible detection was after publishing, which is after the point where a version number can be taken back. run_demos.py moves the list next to the demos, where both repositories can call it. This one runs it against the published package, catching a demo somebody broke. The SDK repository runs it against a wheel built from the branch under review, catching an SDK change that breaks the demos, on the pull request that causes it. Demos are discovered rather than listed. Every top-level module that is not a test runs, so a new demo is covered the day it lands, which matters because the gap being closed is a change nobody happened to exercise. A demo that cannot run offline goes in REQUIRES_NETWORK with a reason and is reported as skipped on every run, so the exclusions stay visible instead of living in a comment nobody re-reads. Failures print after the summary rather than interleaved, because six demos failing the same way is a different problem from one failing alone and that is the first thing worth knowing. Both streams are captured: a demo printing its refusal to stdout and a trace to stderr is the normal shape here. Verified by reverting one demo to its pre-fix state and watching the runner single it out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak
imran-siddique
added a commit
to agentrust-io/weight-custody-manifest
that referenced
this pull request
Aug 27, 2026
0.27.0 shipped two correct security changes and both broke demos in agentrust-io/examples: release began refusing manifests whose identity was not pinned out of band (#98), and the memory-fingerprint challenge began requiring a signed sweep (#95). Six demos that passed on 0.26.0 failed on 0.27.0. Nothing caught it and nothing could have. That repository only ever tests against a version already on PyPI, so the earliest possible detection was after the upload, which is after a version number can be taken back. And the list of demos to run lived inside its workflow file, where this repository's CI cannot reach it. The list now lives beside the demos as run_demos.py (agentrust-io/examples#92), and this job calls it against a wheel built from the branch under review. A change here that breaks them fails on the pull request that causes it. A failure is not automatically a reason to revert. A security fix SHOULD break a demo that was relying on the hole; #98 broke six because six were building brokers that would release against any manifest reusing a held weights hash, which is exactly what it closed. It is a reason to know, and to land the demo fix alongside rather than discover it afterwards. The job comment says so, since whoever meets a red check here will be deciding that under time pressure. The install order is deliberate: the examples' requirements first, to bring in the extras the demos need, then the local wheel with --force-reinstall --no-deps to replace the SDK with the one under review. The pinned floor is usually already satisfied by the version being replaced, so a plain install is a no-op. RELEASING.md gains two steps that were learned the expensive way: diff the changelog against git log before dating a section, and note that the downstream check has already run by the time a release PR is merged. Verified locally end to end: wheel built from this branch, installed over the examples' pin, 13 demos and the unit tests pass. Claude-Session: https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap that let 0.27.0 break six demos here without anything noticing.
Why nothing caught it
The list of demos to run lived inside
.github/workflows/ci.yml, where the SDK repository's CI cannot reach it. And this repository only ever tests against a version already on PyPI.So the earliest possible detection was after publishing, which is after the point where a version number can be taken back.
What changes
run_demos.pymoves the list next to the demos, where both repositories call it:weight-custody-manifestThe SDK-side job is a follow-up PR there; this one has to land first so the script exists on
main.Discovered, not listed
Every top-level module that is not a test runs. A new demo is covered the day it lands, which matters because the gap being closed is a change nobody happened to exercise.
A demo that cannot run offline goes in
REQUIRES_NETWORKwith a reason and is reported asskipon every run, so exclusions stay visible rather than living in a comment nobody re-reads. Two today:real_open_model.pydownloads a model,real_lora_custody.pytrains a LoRA adapter.Two small output choices
Failures print after the summary rather than interleaved. Six demos failing the same way is a different problem from one failing alone, and that is the first thing worth knowing.
Both streams are captured. A demo printing its refusal to stdout and a trace to stderr is the normal shape here, and reading only one has sent people looking in the wrong place.
Verified
And, more usefully, verified it fails correctly: reverting
refuse_and_wipe.pyto its pre-fix state makes the runner single it out and exit non-zero.🤖 Generated with Claude Code
https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak