Skip to content

feat: attest legacy AWS scope for retained reconciliation - #2861

Merged
steipete merged 4 commits into
mainfrom
feat/retained-reconcile-legacy-aws-scope
Oct 16, 2026
Merged

steipete merged 4 commits into
mainfrom
feat/retained-reconcile-legacy-aws-scope

Conversation

@steipete

@steipete steipete commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Legacy terminal AWS leases created before scope persistence could never leave the retained index because their original account was missing. Add an explicit operator attestation for AWS records created strictly before 2026-09-13T00:00:00Z, preserving every existing ownership and cleanup check.

The optional CRABBOX_RETAINED_RECONCILE_AWS_LEGACY_ACCOUNT runtime setting accepts exactly 12 digits. It never overrides a present scope. STS must match the attested account; exact-ID and companion reads, credential pinning, and two complete observations at least ten minutes apart still gate completion. The attested scope is written only with cleanup completion through putLeaseRecord, with cleanupConfirmedBy: retained-reconcile-legacy-attested. Attested leases cannot enter the deletion path even when deletion is enabled.

Admin status exposes the legacy_scope_attested counter and a configured boolean without the account value. Documentation covers the cutover and the risk of an incorrect attestation. No account value, credential, deployment, or runtime configuration change is included.

The Go admin CLI preserves the configured boolean and attestation counter in both JSON and human-readable output.

Validation on Node 24:

  • Worker format, lint, TypeScript, and Node TypeScript checks passed.
  • Retained reconciliation, fleet, and lease-record-store invariant tests: 3 files and 1,677 tests passed.
  • Worker dry-run and Node builds passed.
  • Documentation link check: 262 Markdown files passed.
  • Focused Go admin CLI tests passed: go test ./internal/cli -run '^TestAdminRetained' -count=1.
  • Isolated Codex autoreview at P0/P1: no accepted/actionable findings.

@clawsweeper

clawsweeper Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T05:57:09.405828Z 122f9e0 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d28ada9997

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +34 to +38

function attestedAWSLegacyScope(lease: LeaseRecord, account?: string): string | undefined {
if (
lease.provider === "aws" &&
lease.providerScope === undefined &&

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move legacy AWS attestation behind the provider adapter

This adds AWS-specific account formatting, region and instance-ID eligibility, and rollout-cutover policy directly to coordinator reconciliation. The repository requires provider-specific reconciliation, resource identity, and rollout compatibility to live behind provider adapters, so this logic should be exposed through an AWS adapter capability while core consumes a provider-neutral result.

AGENTS.md reference: AGENTS.md:L24-L28

Useful? React with 👍 / 👎.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Oct 16, 2026
@clawsweeper

clawsweeper Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge.

What this changes

This PR adds optional operator attestation of the original AWS account so eligible legacy lease records can complete retained reconciliation without resource deletion.

Example: A released AWS lease created on 2026-09-12T23:59:59Z has no providerScope, and the operator attests account 123456789012.

  • Before: The lease remains unresolved as missing_original_scope.
  • After: Matching STS identity and two complete absence observations at least ten minutes apart allow cleanup completion and removal from the current index.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) Useful maintainer-directed feature with focused state-transition tests, an actionable provider-boundary violation, and unqualified authority and upgrade behavior.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is necessary before merge. See Before merge.
Patch quality 🦐 gold shrimp (3/6) Security review found an item that needs attention.

Product

Kind: Feature · Worth it: Yes · Fix scope: Complete
User problem: Legacy terminal AWS records without recorded account scope remain in current listings and maintenance after their resources are gone.
Reason: The supplied maintainer-authored direction authorizes this optional audit-only exception, and existing recovery has stricter eligibility that leaves this use case uncovered.

Merge readiness

⛔ Blocked before merge - 5 items remain

Keep open: current main lacks this maintainer-directed capability, but the new AWS policy violates the repository’s provider boundary.

Priority: P2
Reviewed head: facbd4efb5e539c026a1bed32986a1501249ae0c

Before merge

  • Add real behavior proof - Authority-chain proof required: exercise coordinator maintenance through the real AWS transport with matching and mismatched accounts, showing allowed cleanup publication and rejection before forbidden publication or deletion; change or remove attestation between persisted observations and restart to show stale evidence cannot complete cleanup. The complete body and discussion provide checks and mocked-session tests, without these traces. Supply redacted logs or terminal output. Stored scope, completion attribution, and observation changes also lack a latest-stable upgrade/restart demonstration with existing state intact. After adding proof, update the PR body, then request a new review with @clawsweeper re-review. The PR author or someone with repository write access can request it.
  • Move AWS attestation policy behind the provider adapter (P2) - The new helper places AWS account formatting, instance-ID eligibility, Region validation, and rollout-cutover policy directly in shared coordinator reconciliation. AGENTS.md explicitly requires provider-specific reconciliation, resource identity, regions, and rollout compatibility behind provider adapters, and docs/architecture.md identifies aws.ts as that boundary. Move this policy to an AWS-owned capability and let reconciliation consume a provider-neutral eligibility result. This also addresses the still-applicable discussion at feat: attest legacy AWS scope for retained reconciliation #2861 (comment).
  • Resolve security concern: Qualify attestation across persisted authority changes - Source rejects mismatched STS identity before exact reads, binds absence fingerprints to the attested account, and excludes attested candidates from deletion; supplied coverage substitutes the AWS operation session and does not demonstrate those boundaries through real transport and durable restart.
  • Resolve merge risk - Attested scope is persisted in the same providerScope field used for recorded authority; its audit-only boundary across restart and attestation changes remains unqualified at the final effects.
  • Add data-model compatibility proof - The review found that existing stored data may not work after upgrade. Show that existing data still loads and works with this change.

Findings

  • [P2] Move AWS attestation policy behind the provider adapter — worker/src/retained-lease-reconcile.ts:35-45
  • [medium] Qualify attestation across persisted authority changes — worker/src/retained-lease-reconcile.ts:599

Tests

  • Missing end-to-end proof: Provide changed-path authority traces through the real provider transport and an upgrade/restart run from v0.78.1 preserving scoped records, settings, and persisted observations.
Agent review details

How this fits together

Coordinator maintenance reads retained terminal leases from the current index, verifies AWS resource absence, and transactionally publishes cleanup facts and index changes.

flowchart TD
  A[Coordinator maintenance] --> B[Current lease index]
  B --> C[Legacy eligibility and attestation]
  C --> D[AWS identity and exact resource reads]
  D --> E[Two absence observations]
  E --> F[Transactional lease publication]
  F --> G[Cleanup facts and current index]
Loading

Technical review

Best possible solution:

Support narrowly scoped audit-only legacy attestation through an AWS-owned capability while preserving recorded authority and deletion fences.

Do we have a high-confidence way to reproduce the issue?

Current-main source deterministically classifies an otherwise eligible AWS lease without providerScope as missing_original_scope; this is an intentional safety restriction that the feature narrowly relaxes.

Is this the best way to solve the issue?

The audit-only approach addresses records excluded by existing CloudTrail recovery, but AWS eligibility and cutover policy should live behind the provider adapter.

Full review comments:

  • [P2] Move AWS attestation policy behind the provider adapter — worker/src/retained-lease-reconcile.ts:35-45
    The new helper places AWS account formatting, instance-ID eligibility, Region validation, and rollout-cutover policy directly in shared coordinator reconciliation. AGENTS.md explicitly requires provider-specific reconciliation, resource identity, regions, and rollout compatibility behind provider adapters, and docs/architecture.md identifies aws.ts as that boundary. Move this policy to an AWS-owned capability and let reconciliation consume a provider-neutral eligibility result. This also addresses the still-applicable discussion at feat: attest legacy AWS scope for retained reconciliation #2861 (comment).
    Confidence: 0.99
    Late finding: first raised on code an earlier review cycle already covered.

Overall correctness: patch is incorrect
Overall confidence: 0.95

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 05eded31b4d8.

Merge-risk options

Maintainer options:

  1. Qualify the audit-only authority boundary (recommended)
    Exercise matching and mismatched AWS accounts through the real provider transport, then remove or change attestation between persisted observations and restart; verify forbidden cleanup publication and provider deletion never occur.

Provenance checked

Testing

Proof path: unit tests only.

Security

Needs attention: No reachable unauthorized deletion was found, but the new persisted authority path needs final-effect qualification.

Evidence

Security concerns:

  • [medium] Qualify attestation across persisted authority changes — worker/src/retained-lease-reconcile.ts:599
    Source rejects mismatched STS identity before exact reads, binds absence fingerprints to the attested account, and excludes attested candidates from deletion; supplied coverage substitutes the AWS operation session and does not demonstrate those boundaries through real transport and durable restart.
    Confidence: 0.96

What I checked:

Labels

Label changes:

  • add P2: This is an optional operator capability for retained legacy records, with limited immediate user impact.
  • add merge-risk: 🚨 security-boundary: Operator-supplied account attestation becomes persisted scope and authorizes cleanup publication; the accepted audit-retirement tradeoff does not establish final-effect rejection for mismatched or changed authority.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Label justifications:

  • P2: This is an optional operator capability for retained legacy records, with limited immediate user impact.
  • merge-risk: 🚨 security-boundary: Operator-supplied account attestation becomes persisted scope and authorizes cleanup publication; the accepted audit-retirement tradeoff does not establish final-effect rejection for mismatched or changed authority.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add final-effect proof for mismatched AWS identity and prove changed or removed attestation invalidates persisted observations before cleanup publication.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 16, 2026, 1:01 AM ET / 05:01 UTC.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a1f4d797b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lease.region &&
sanitizeAWSRegion(lease.region) === lease.region &&
/^i-[a-f0-9]{8,17}$/.test(lease.cloudID) &&
Date.parse(lease.createdAt) < awsLegacyScopeCutoverMs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject noncanonical creation timestamps

When a stored legacy record has a malformed createdAt such as "0", "1", or "2026-09-12junk", JavaScript's permissive Date.parse still returns a pre-cutover date, so the record becomes eligible for attestation without valid evidence that it predates the scope-persistence rollout. This can incorrectly publish cleanup and persist the attested account for malformed records; validate the expected timestamp format before comparing it with the cutoff.

Useful? React with 👍 / 👎.

@steipete
steipete force-pushed the feat/retained-reconcile-legacy-aws-scope branch from c68d993 to 122f9e0 Compare October 16, 2026 05:54

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 122f9e0ff3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +284 to +287
(lease.providerScope ??
(!retainedUnresolvedReason(lease, legacyAccount)
? attestedAWSLegacyScope(lease, legacyAccount)
: undefined)) !== `aws:account:${identity.account}` ||

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid globally backing off AWS on attestation mismatch

When the configured legacy account differs from the STS identity, this classifies every eligible legacy record as scope_mismatch; maintainRetainedLeases then increments the shared state.backoff.aws for each record, and subsequent passes skip observations for every AWS scope while that backoff is active. A typo in this optional rollout setting can therefore repeatedly suspend reconciliation of correctly scoped AWS records for up to the one-hour cap, so legacy-attestation mismatches should not activate the provider-wide backoff.

AGENTS.md reference: AGENTS.md:L26-L26

Useful? React with 👍 / 👎.

@steipete
steipete merged commit 8f73749 into main Oct 16, 2026
37 checks passed
@steipete
steipete deleted the feat/retained-reconcile-legacy-aws-scope branch October 16, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant