Repository navigation
Check snapshot isolation of signature transactions #6188
Description
We may break snapshot isolation because signature transactions are special.
There is a delay between booking a version for a signature, and producing the corresponding write. What happens if a separate transaction tries to read from the signature table in that gap? Does it read an inaccurate value, or do we separate the booked-version and readable-version? Should you be allowed to read from the signature table in other transactions?
Activity
achamayou commentedon Oct 1, 2026
Investigation: the reservation is exposed as a readable version
Investigated the published revision
f656489f2fa2, on 2026-10-01.There is no separate complete/readable-version frontier in the default transaction read path. Reserving signature sequence number
Sadvances the same store version that a new transaction uses as its read version. Before the signature is materialised, that transaction can capture the preceding signature, or absence if this is the first one. The more serious cases are inconsistent cross-table snapshots and successfully applied writes whose inputs do not correspond to any prefix of the ledger.This is not a non-repeatable
get()on one handle, and is not, by itself, a forged signature, a broken Merkle signature, or divergent replica state. Reading a valid older signature in isolation can be acceptable for an explicitly "latest available signature" API. The defect is treating the allocated version as an immutable, consistent KV snapshot when a lower/equal-version write can still arrive.1. Why the gap is real
The relevant sequence is:
emit_signature()callsstore.next_txid()before submitting the pending signature transaction.next_txid()callsnext_version_unsafe(), incrementingStore::version.current_txid_and_commit_term()returns that version to readers.Store::commit()queues pending transactions and only invokes a signature'scall()once the preceding replication entries are available. Even without a missing predecessor, there is a window between allocation and applying the write, including signing/serialisation work.MerkleTreeHistoryPendingTx::call()constructs the reserved transaction, computes the signatures, and callscommit_reserved().BaseTxcaptures a read TxID at the first KV handle acquisition, not at transaction construction. Each table's change set is acquired lazily.Map::create_change_set()selects the newest currently present table state at or below that read version.last_replicated, the replicated history's end index, andcompactedexist separately, but none is used as the default readable frontier here. Also, the reservation does not hold the signature-table locks throughout the gap. The store's replicationcommit_lockdoes not cover default read-version acquisition or ordinary transactions' earlier local write application.In dual-signing mode the delayed transaction writes three tables:
public:ccf.internal.signatures,public:ccf.internal.cose_signatures, andpublic:ccf.internal.tree. COSE-only mode still delays the latter two. The raw-signature table deliberately stops advancing in COSE-only mode; that separate, intentional staleness must not be confused with this race.2. Old/absent data under a version that will contain a newer signature
Here
Pis the preceding signature, and the signature table already exists:sequenceDiagram participant E as Signature emitter participant K as KV store participant A as Reader A participant B as Reader B E->>K: next_txid() reserves S Note over K: Store version is S, signature table still contains P A->>K: First handle pins R = S and snapshots signatures K-->>A: Signature P E->>K: Pending signature applies its writes at S B->>K: First handle pins the same R = S K-->>B: Signature S A->>K: get() using its original signature handle K-->>A: Still P A->>K: commit() with no writes K-->>A: SUCCESS, read TxID is S Note over A,B: Same read TxID, different signature contentsThe old handle is stable:
BaseTxreuses its per-table change set, andget()reads that captured state. Creating another typed handle over the same named table in the same transaction does not refresh it. The state is captured at actual KV handle acquisition, even if the firstget()happens later;has(), iteration/size, and previous-write-version queries can also observe the old captured state.Read-only endpoint transactions do not ordinarily repair this at commit:
apply_changes()callsprepare()only when there are writes, andget_txid()reports the read TxID for a no-write transaction.First-signature qualification: if the table does not yet exist, the reader creates a transaction-local empty map and observes absence. If the real table is created before an endpoint's
CommittableTx::commit(), the separate new-map conflict check can reject that transaction, even without writes. A plainReadOnlyTxhas no such commit step, and a transaction that finishes before materialisation can still return or act on absence. So "all read-only transactions always bypass all validation" would be too strong.3. A torn signature/tree/COSE bundle
For a reader whose read version includes
S, opening different tables on opposite sides of materialisation can yield a combination which was never an atomic KV state. This diagram assumes dual signing, where all three tables change atS:sequenceDiagram participant R as Reader at R = S participant K as KV store participant E as Pending signature S R->>K: Open signatures table before S is applied K-->>R: Capture signature P and root P E->>K: Atomically apply raw, COSE, and tree writes at S R->>K: Open tree table after S is applied K-->>R: Capture tree S R->>K: Open COSE table after S is applied K-->>R: Capture COSE signature S Note over R: Cached signature/root P plus tree/COSE S R->>K: No-write commit K-->>R: SUCCESS for existing tablesThis is not a partially applied signature transaction: the writes themselves take all affected map locks. The inconsistent assembly comes from lazy, separately cached table snapshots at a read version that was published before its signature write was present. The reverse old/new mixture is possible if the tables are opened in a different order. Keys within a single captured table remain consistent with that table's captured state.
In COSE-only mode, the equivalent tear is old COSE/new tree or new COSE/old tree. Old raw/new COSE is not sufficient evidence of a tear there: leaving the raw table unchanged is intentional.
An application building its own proof from such a bundle can obtain a signature/root and proof/tree from different signature transactions, leading to verification failure, a wrong coverage decision, or an early error. It cannot thereby make an invalid proof pass correct cryptographic verification. Commit-time conflicts also cannot undo incorrect handler branching or an error response returned before commit.
Safe boundary: a transaction pinned to
R < Sbefore reservation excludes the new signature from every later table handle, so this particular gap does not tear its snapshot. A transaction that captures the relevant tables after all writes through its read version are present is also safe.4. Dependent writes: conflict detection helps only in one ordering
For a write transaction,
prepare()compares observed key versions with the current table state:P;Smaterialises before the reader prepares its writeP; reader prepares/applies its write beforeSmaterialisesP, so the reader can succeed at a sequence number greater thanS. Filling the earlier signature slot does not revalidate that already-applied transaction.The second ordering can affect persisted application or governance decisions, not just a transient GET. Normal application transactions can already have applied later-version writes while replication is held behind the missing signature:
sequenceDiagram participant E as Signature emitter participant W as Independent writer participant R as Dependent reader/writer participant K as Local KV participant L as Ordered ledger Note over K,L: Initial prefix 2 has signature 1 and application data 0 E->>K: Reserve signature slot 3, leave it unfilled W->>K: Apply application data 4 at version 4 K-->>W: SUCCESS, replication waits for slot 3 R->>K: Pin read version 4, read signature and application data K-->>R: Signature 1 and application data 4 R->>K: Prepare and apply derived writes at version 5 K-->>R: SUCCESS, signature dependency still matches version 1 E->>K: Materialise signature 3 K->>L: Replicate entries 3, 4, 5 in order E->>K: Emit covering signature 6 K->>L: Signature 6 can commit the preceding writes Note over L: Prefix 2 has sig 1/data 0, prefix 3 has sig 3/data 0 Note over L: Prefixes 4 and 5 have sig 3/data 4, not sig 1/data 4The read inputs
(signature 1, application data 4)match no ledger prefix, yet derived writes can be replicated and subsequently committed. They can therefore disagree with decisions an auditor would reproduce from the preceding ledger state. Replicas still agree on the recorded write sets: CCF replicates writes, not re-execution of the endpoint's read logic. There is no demonstrated consensus divergence or authorization bypass in this investigation.This violates the documented
consistent, opaque view of the KV/ version-to-ledger-prefix expectation.Governance nuance: proposal creation has a special replicated-history-root read TxID, rather than the default allocated version. That avoids a reservation which is not yet in that prefix. It is specific to proposal creation, not a blanket protection for governance: ballot submission can resolve and apply a proposal using the ordinary transaction path.
5. How many readers?
Counts below are direct, table-specific KV handle acquisition locations, not text matches, declarations, writers, or every caller of a shared helper.
Current CCF source: five production raw-signature reader locations in four files.
history.h:601node_state.h:117node_state.h:2156historical_queries.h:73node_frontend.h:833GET /node/state, includinglast_signed_seqno/node/statetakes the maximum raw/COSE sequence number, so a mixed pair may still give a sensible latest-available sequence number. It does not construct a receipt. The demonstrated snapshot defect should not be inflated into a claim that this endpoint emits invalid proofs.There are also five production COSE-table reader locations, alongside the same five raw-table locations, and four production tree-table reader locations:
history.h:542: restore history from a snapshot.historical_queries.h:109: historical receipt extraction.internal_tables_access.h:390: preserve the previous service root during service creation/recovery.gov/handlers/acks.h:142: live read-write/gov/members/state-digests/{memberId}:update, deriving and storing a member state digest from the tree.Thus banning only reads of the raw table would miss a related live governance consumer and would leave the same mechanism on the COSE/tree tables.
Two additional direct raw-table readers are in C++ tests (history and snapshot). Other test matches are writers/fixtures. The Python SDK's signature parser and Python e2e readers inspect serialised ledger entries, not live KV snapshots.
No table-specific live readers of these three tables were found in the current repository's sample/template applications, sample constitutions, JS SDK, or test application code. That is not proof that generic access cannot reach them: for example, the permissions test endpoint accepts an arbitrary table name.
Built-in receipts are a separate, safe path for this issue: historical receipts use per-entry reconstructed stores, and on-commit receipts use a complete, version-keyed signature-cache tuple, consumed here. Neither builds proofs by combining the primary's live signature-table snapshots.
External applications: a ban is not usage-free
A bounded public GitHub search verified five statically configured live raw-signature read sites in five external repositories. All are governance constitutions deriving a new member's acknowledgement
state_digestfromsignature.root; these are copies/adaptations of the same default-governance logic, not five distinct proof-building algorithms.Azure/azure-cleanroomchinenyeokafor/minimal_ccf_imagepeterms2021/cbdcmaceip/stewardmicrosoft/ccfdnsThree more live-style expressions were found in copied KMS helper modules, but their action closures were not verified reachable, so they are not included in the five configured consumers: microsoft/azure-privacy-sandbox-kms, datakaveri/DEPA-kms-gcp, and mayankgiri90/DEPA-kms-gcp-OIDC.
No bespoke external application endpoint reader, live COSE/tree reader, or non-core C++/Rust consumer was verified in this search. Offline ledger parsers, recorded fixtures, and copied CCF distributions were excluded. This is a lower bound, not an ecosystem census or evidence of deployed exploitation: the search covered the first 100 results for each of the three literal table names plus
Tables::SIGNATURESandccf::Signatures, excluding microsoft/CCF, and inspected up to ten non-fork candidate repositories. Private/unindexed code and dynamically constructed names are not covered.6. Is preventing reads an option?
Yes, as an API-policy mitigation, but not as a compatible blanket ban on every transaction.
Today JS permissions explicitly grant read access to public internal tables in application and governance contexts. C++ transactions and the Rust bridge also allow named-table reads. A JS-only restriction would not fix the native paths.
A defensible restriction would deny live transactional reads of the whole delayed signature-table family, while retaining narrowly defined framework replay/restore access and historical/offline reads. It would require migrating
/node/stateand the live member state-digest helper, and replacing or adapting the external constitution reads above. Banning only application endpoint reads leaves governance exposed; exempting all governance would retain the configured external readers.Do not simply blacklist the names in
check_kv_map_access(): historical JS access calls the same helper, so that would also forbid safe historical reads. Native/framework distinctions would likewise need an explicit access mechanism rather than removing the table types.7. Possible solutions and tradeoffs
commit_lockor signature-emitter lock is insufficient.get()alone can be too late if the handle already captured old state. Waiting while holding locks or occupying the worker needed to finish the signature risks deadlock/starvation; an immediate retry loop can also exhaust retries before progress.prepare()alone is not a full fix: it still passes if the signature has not yet materialised, and writers can still validate/apply in the gap. Checking returnedsig.seqno <= Ris also insufficient. Validation after execution does not prevent erroneous branching/errors on an inconsistent snapshot.The first three are correctness fixes if implemented end-to-end. Restricting live reads is a viable mitigation or alternative API design, not a claim that the underlying version-frontier invariant is sound.
Taking the three signature-table snapshots together would prevent torn bundles, but would not by itself fix future read-version labels or old-signature/new-application inputs. Switching to COSE-only removes the raw-table update, not the COSE/tree race. Moving allocation until after signing is not a free reorder either: the COSE signature includes the TxID, and the signature/tree must cover exactly the ledger prefix ending at
S - 1, so the version/history boundary must remain coordinated.Using the replicated-history index as the read version looks tempting, but it should not be a one-line substitution: the existing proposal-root path warns about session consistency, and session read-your-writes is a documented guarantee.
8. Reproduction and regression cases
A session-local C++ probe was built against the unmodified
ccf_kvat the revision above, usingNullTxEncryptor,StubConsensus, and the actualnext_txid(),ReservedTx::commit_reserved(), normal transaction commit, and ordered deserialisation paths. Synthetic raw-signature values and COSE/tree byte markers identify table generations; this probes KV isolation, not cryptographic validity or a deployed consensus network. No production or repository test source was modified.The deterministic observations were:
1.3returned raw signature sequence numbers1and3, respectively.1but obtained tree/COSE generation3through later table handles; its no-write commit succeeded.FAIL_CONFLICT, with no write applied.5, storing inputs(signature 1, application data 4). Those writes survived ordered replay; no replayed prefix contained that input pair, including after a subsequent covering signature.CommittableTxcommitted after real table creation hit the new-map conflict check.These are the core regression schedules for any fix. Extend them to COSE-only mode, multiple pending signature slots, rollback/view changes, first-signature/table creation, compaction, arbitrary JS/native access, early handler error paths, and sequential requests on one session when later application writes are already locally successful.