Skip to content

test(python): assert agent identity survives a restart - #31

Merged
andyjsbell merged 3 commits into
mainfrom
phase/3-identity-regression-test
Aug 5, 2026
Merged

test(python): assert agent identity survives a restart#31
andyjsbell merged 3 commits into
mainfrom
phase/3-identity-regression-test

Conversation

@andyjsbell

Copy link
Copy Markdown
Contributor

Closes the last unchecked item under §3 Persistent agent identity (Persona-B load-bearing) in docs/PRODUCTION_CHECKLIST.md.

§3's substance was already proven manually and marked [x]. The one remaining item was a regression test guarding the SDK's load_or_create default against a future swap to the ephemeral generate() — a settlement system can't use receipts whose signing keys rotate on restart.

What changed

  • crates/attestly-python/tests/test_identity_restart.py (new) — two tests:
    • test_identity_survives_restart — two separately spawned interpreters sharing a tmp_path identities dir must print the same pubkey.
    • test_restart_identity_is_per_agent — a third process with a different agent_id must print a different pubkey, ruling out a degenerate constant-key implementation passing the first test.
  • docs/PRODUCTION_CHECKLIST.md — ticked the §3 item, pointing at the test.

Placement

Binding-level (the recommended option in the dispatch prompt). The seam the checklist names is wrapper.py:120, but the property that actually has to hold is that AgentIdentity.load_or_create returns a disk-backed key in a fresh process — testable without standing up a StateGraph.

Existing coverage doesn't reach this: test_api.py::test_load_or_create_persists_across_calls is same-process repeated calls, and the only other subprocess use (cross_language.py) spawns the attestly-verify binary for the encoding contract. No cross-process identity test existed.

The guard bites

Verified empirically, not just by reasoning — temporarily swapping the subprocess snippet to AgentIdentity.generate(agent_id) fails the test, at two independent assertions (the identity file is never persisted, and the two pubkeys differ). The temporary edit was reverted; no production code was touched.

Notes

  • Hermetic: identities go to tmp_path only; the real data/identities/ is never read or written.
  • No Makefile change — picked up by the existing python-test target (pytest crates/attestly-python/tests).
  • Production code untouched: no edits to identity.rs, crates/attestly-python/src/lib.rs, wrapper.py, or services/attestly-cloud/.

Implements Step 3 of the production checklist — §3 Persistent agent identity.

🤖 Generated with Claude Code

andyjsbell and others added 3 commits August 5, 2026 18:01
docs/PRODUCTION_CHECKLIST.md §3 marks the substance of persistent agent
identity proven, leaving one unchecked item: a regression test that
identity survives a restart, guarding the SDK's `load_or_create` default
against a future swap to the ephemeral `generate()`.

Placed at the binding level rather than the SDK level. The named seam is
`wrapper.py:120`, but what actually has to hold is that
`AgentIdentity.load_or_create` returns a disk-backed key in a *fresh*
process — that is the property a settlement system depends on, and it is
testable without standing up a StateGraph. Each pubkey comes from a
separately spawned `sys.executable`, so an in-memory-only key cannot pass;
existing coverage (test_api.py::test_load_or_create_persists_across_calls)
is same-process and would not catch the swap.

Two tests: identity is stable across two fresh processes sharing a
tmp_path identities dir, and distinct agent_ids get distinct keys (so a
constant-key implementation can't pass the first). Hermetic — tmp_path
only, with an explicit assertion that the real data/identities/ is
untouched. Verified the guard bites: swapping the subprocess snippet to
generate() fails both the persistence and the pubkey-equality assertion.

Picked up by the existing `python-test` pytest run; no Makefile change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last unchecked item under §3 Persistent agent identity is now covered
by crates/attestly-python/tests/test_identity_restart.py. Records where
the test lives and that the guard was verified to bite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review flagged that the "did not touch the real data/identities/" assertion
used a cwd-relative Path("data"), so it inspected whatever happened to be
below the invocation directory rather than the real store — a check that
read stronger than it was. Derive the path from __file__ instead, matching
the REPO_ROOT convention in cross_language.py, so it holds no matter where
pytest is invoked from.

Actual hermeticity was never at risk: load_or_create is passed the absolute
tmp_path, so test identities are structurally confined there. This makes the
belt-and-braces assertion do what it claims.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@andyjsbell

Copy link
Copy Markdown
Contributor Author

Agent Run Report

Implementation Commits

  • 5cf1b4a test(python): assert agent identity survives a restart
  • e0ae5ae docs: tick production checklist §3 restart regression test
  • 11d2625 fix(python): anchor the hermeticity check at the repo root

Review Report

Requirements Checked

  • Regression test asserts identity survives a restart (the single unchecked §3 item) — PASS. crates/attestly-python/tests/test_identity_restart.py::test_identity_survives_restart spawns two fresh interpreters against a shared tmp_path store and asserts equal pubkeys.
  • The guard actually bites — PASS (empirically verified, not just reasoned). Temporarily swapped the subprocess snippet to AgentIdentity.generate(agent_id); the test FAILED. Because it first tripped on the intermediate persistence assertion (restart-agent.json never written), a second probe with that assertion temporarily disabled confirmed the core equality assertion independently catches the swap — it did, with two differing pubkeys (ca9f5566… != 3d3028b1…). So the guard bites at two independent layers. Both temporary edits were reverted; working tree verified clean.
  • Cross-process reality (not same-process repeat) — PASS. Each pubkey comes from a separate subprocess.run([sys.executable, "-c", ...]). Independently confirmed distinct PIDs (7797, 7798, both ≠ parent 7796) returning an identical key loaded off the persisted JSON.
  • Hermeticity — never touches real data/identities/ — PASS. The store dir is the absolute tmp_path, so placement is structurally guaranteed regardless of cwd. The real data/identities/ does not exist on disk even after full runs.
  • Binding-level placement (one of the two acceptable options) — PASS. Landed in crates/attestly-python/tests/, the recommended option.
  • Comment tying the test to the checklist item — PASS. Module docstring cites PRODUCTION_CHECKLIST.md §3 and wrapper.py:120, and states that a generate() swap makes it fail.
  • Per-agent key distinctness — PASS. test_restart_identity_is_per_agent rules out a degenerate constant-key load_or_create.
  • Never asserts against non-deterministic generate() output — PASS.
  • Do Not Touch compliance — PASS. Zero edits to services/attestly-cloud/, crates/attestly-core/src/identity.rs, crates/attestly-python/src/lib.rs, or wrapper.py.
  • Makefile: nothing new wired, picked up by existing python-test — PASS. Makefile untouched; python-test runs pytest crates/attestly-python/tests and the test_*.py name matches the existing python_files config. Both tests appear in the make check output.
  • Checklist item ticked — PASS.

Gaps Found

None blocking.

One minor, non-blocking observation: the secondary hermeticity assertion checked a cwd-relative Path("data") / "identities" / .... Since pytest's rootdir is crates/attestly-python while invocation cwd is the repo root, that path was not guaranteed to point at the real store, making it a weak belt-and-braces check. Real hermeticity is structurally guaranteed by passing the absolute tmp_path, so the test was correct as written — the assertion just did less than it appeared to.

Fixes Made

  • 11d2625 fix(python): anchor the hermeticity check at the repo root — applied after the review, addressing the observation above. Derives REPO_ROOT from __file__ (matching the cross_language.py convention) so the assertion inspects the real data/identities/ regardless of invocation cwd. Re-verified the guard still bites when run from a foreign cwd (/tmp).

Quality Gate

make check: PASS (exit code 0 — core Rust workspace, Python bindings, LangGraph suite, and the services/attestly-cloud sub-workspace; re-run after the follow-up fix).

@andyjsbell
andyjsbell marked this pull request as ready for review August 5, 2026 16:17
@andyjsbell
andyjsbell merged commit deded20 into main Aug 5, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant