feat(sdk-fixtures): prove the Android SDK lane and make the SDK receipts generic - #151
Merged
Conversation
github-actions
Bot
added
ci
rldyourmnd
force-pushed
the
fixtures/sdk-android-closure
branch
from
August 14, 2026 08:31
d31f4b3 to
9ab4049
Compare
github-actions
Bot
added
ci
rldyourmnd
changed the title
github-actions
Bot
added
ci
Applies the PR #145 checkpoint (7cbd1e8) on top of the track-M fixes merged as PR #146. Conflicts were unions: the Python execution surface carries both track-M's subjects and this branch's. This is the unverified WIP baseline; every defect it carries is fixed in the commits that follow. One fix is folded in here rather than applied after: `gitleaks` flags the sample cache identity on line 238 as `generic-api-key`, and `secret-scan.yml` scans the history of the checked-out ref, so leaving the literal in an ancestor commit would fail the scan on `main` for good. The value is a Flutter action cache key in a negative-self-test sample, not a credential, so nothing needs rotating -- but the line really did read like one, and the fix is to stop the sample looking like a secret rather than to teach the scanner to skip the file. The identities are built from parts, with the version read from `sdk-runtime-spec.yml` instead of written out three more times.
…ot the shell
`check_sdk_runtime_fixtures` failed on a clean tree for any contributor whose
umask is 002: `_sdk_environment_selftests` built its *valid* case out of a real
temporary directory, and `mkdir` under that umask yields 0o775, which the trust
rule correctly rejects as group-writable. A blocking validator was a function of
the developer's shell -- green on runners (umask 022), red locally.
The same construction inverted the negative case in the other direction. The
`unowned` probe passed `os.getuid() + 1` against files owned by the executing
user, so under root the files are uid 0, the rule sees `0 in {0, 1}`, no
exception is raised, and the test reports a *failure to reject* something it
never managed to make unowned.
`ownership_problem(OwnershipFacts(uid, mode), trusted_uid=...)` is now a pure
function of two integers, exercised on ten stated pairs covering both trusted
owners, both write bits, and the root-owned-but-writable combination the old
tests could not express at all. The live-filesystem cases keep only what needs
a filesystem -- symlink aliases, alternate ancestors, missing and mismatched
JDKs -- and state their modes instead of inheriting them.
The production rule is unchanged and still strict. Proven by mutation: dropping
the world-write bit, trusting every uid, and short-circuiting root each fail
with the exact case that catches them.
… one fixture
Two defects in the evidence step the three SDK reusables gained, both of which
would have reached consumers the moment this surface was released.
**The receipt described the caller's tree.** Each step hashed
`${github.workspace}/.github/workflows/<callee>.yml` and published the result
as `workflow_sha256`. Inside a called reusable every `github.*` value belongs
to the *caller*, so that path is the caller's checkout: an external consumer
either has no such file and the step dies, or has a same-named file of their
own and its bytes are published as the provenance of a workflow they never
edited. `job.workflow_repository` / `job.workflow_sha` / `job.workflow_file_path`
are bound to the callee and supplied by the runner, and a commit SHA already is
the cryptographic identity of the content at a path -- so the triple pins the
exact bytes, needs no second checkout or token, works for a private callee, and
cannot be forged by the caller. Proven rather than read off the documentation:
a probe reusable pinned at an absolute commit and called from a different
commit reported the pinned SHA in `job.workflow_sha` and the caller's head in
`github.sha` --
https://github.com/NDDev-it-com/ci-workflows/actions/runs/31779014883
actionlint v1.7.12, the current release, still models `job` as
`{check_run_id, container, services, status}`, so `.github/actionlint.yaml`
carries one suppression naming those three properties in those three files. A
typo such as `job.workflow_shaa` is still reported.
**The reusables demanded one repository's fixture.** The evidence step ran
unconditionally and required things no generic caller owes anyone. Qt documents
`configure_command`, `build_command` and `test_command` as "Empty to skip", yet
the step read the test log's first line regardless, so a caller taking the
documented option got an unhandled `FileNotFoundError` instead of a receipt.
Android's step required a Gradle wrapper, `ANDROID_HOME` (absent on macOS,
Windows and most self-hosted images when `setup_android` is false, its default),
a `gradle/verification-metadata.xml` most projects do not have, a module
literally named `app`, and an APK -- in a workflow whose own header promises it
"works for pure-JVM Kotlin and Android", which by definition produces no APK.
Each receipt is now a discriminated record: `kind` selects the vocabulary and
`sections` lists what this run actually produced, so a skipped lane is absent
rather than fabricated or fatal. Android additionally reports `untrusted_roots`,
which keeps "no Android SDK here" distinct from "an Android SDK I refuse to
vouch for" instead of collapsing both into a hard exit. The fixture-specific
expectations move to the observer, where they belong.
None of this breaks compatibility: `main` has no `outputs:` on these three
workflows at all, so the entire surface is unreleased.
…gest grammar The observer's validator now owns both halves the reusables gave up. **Typed by discriminator.** `_shape_problems` checks the receipt against the vocabulary its own `kind` selects: every required field of a declared section must be present, a field may not appear without its section, and a field outside the vocabulary is rejected outright. The leak and unknown-field rules are deliberately disjoint -- computed against the same set, one masks the other and the weaker of the two can be deleted without a single test noticing. **One grammar for every digest.** `_is_sha` used to be applied to exactly two fields. `pubspec_lock_sha256`, `apk_sha256`, `lock_sha256` and `verification_metadata_sha256` were accepted on truthiness, so `["nope"]` was a valid list of APK digests. Scalars are now exactly 64 lowercase hex; lists are non-empty and repeat nothing; mappings are non-empty and keyed by relative POSIX paths in lexical order, which rejects an absolute path, a `..` traversal and a Windows separator. The negative suite is driven off the `DIGEST_*` sets, so a field added to the grammar is negatively tested as soon as a sample carries it. **Canonical assertions, in one place.** Required sections, the `:app:` task graph, the SDK platform and build-tools, the JVM identity coherence and the "vouched for every root" rule are assertions about `tests/fixtures/**` and now live only here. Shape negatives are asserted against `_shape_problems` directly. Run through the whole pipeline they passed for the wrong reason -- the canonical rule also requires those sections, so dropping one was rejected before the shape rule was ever consulted, and the check under test was never exercised. That was found by mutation: disabling the leak rule left the suite green. Proven by mutation. Truthiness-only digests, an unordered lock map, a duplicated APK digest, tolerated unknown fields, an undeclared section, a hollow section and an unchecked callee path each fail, naming the case that catches them.
…fest The fixture pins five facts about one Flutter release -- channel, version, Dart version, framework revision, and the Linux x64 archive digest -- and nothing checked that they name the same release. They were verified by eye, and an external audit reading them by eye concluded the pin was unresolvable and asked for it to be re-pinned. It was not unresolvable: `check_flutter_pin` fetches Google's `releases_linux.json` and the five fields resolve to exactly one row. Acting on that audit item would have broken a correct pin, which is precisely the class of mistake a machine check removes. Advisory tier. Whether a third party still publishes a row is a calendar fact about someone else, and `AGENTS.md` is explicit that those never sit in the blocking tier. An unreachable manifest is reported rather than silently passed; `--offline` exists for running the full local tier without network and says outright that it proved nothing. Two drift sources removed while here. Every opaque identifier in `sdk-runtime-spec.yml` is now quoted: they parse as strings today only because they happen to contain letters, and a mutation test that substituted an all-numeric revision had it silently arrive as the integer `0` -- the same retyping that produced seven unquoted `last_verified` dates in the catalog. And `generate_android_fixture_provenance.py` had its own second copy of the wrapper JAR digest, the distribution digest, the Gradle version, the JDK major, the compile SDK and the build-tools version; it reads all six from the spec now, so the wrapper contract the generator enforces cannot diverge from the one the validator enforces. Proven by mutation: a wrong Dart version, a wrong framework revision, an unpublished version and a wrong channel each fail with the field that differs.
…archive refs `docs/15` and two catalog risk lines still described the pre-split model, and one of them stated the defect as if it were the design: "Android evidence requires a checksum-pinned wrapper plus strict dependency locks and verification metadata for the full default build graph" is a fact about `tests/fixtures/android`, not something `kotlin-android-ci.yml` may demand of a consumer. Both now describe the discriminated receipt, where the callee provenance comes from, and the fact that fixture-specific assertions live in the observers. `docs/08` gains the convention for the two long-lived ref classes the tag rules do not cover. Two archive refs resolve to the same object; the canonical name is recorded and the other is documented as its alias, so neither gets deleted for looking redundant and a reader meeting the alias in an old comment can tell what it is.
Generates the dependency locks and Gradle verification closure on a hosted runner and publishes them, so the committed fixture inputs come from the same environment the fixture lane runs in rather than from a maintainer's laptop.
Running them for the first time found three reasons they could not, none of
which any static check could have seen.
**The observers never installed `uv`.** Every other provisioning site in this
repository runs `astral-sh/setup-uv` before `uv pip install`; the three SDK
observer jobs went straight to `uv pip install`, so all three failed in
`Provision validator` before reaching the evidence they exist to check.
**Flutter and Qt cannot start under a SHA-pinning policy.** This repository
requires every action to be pinned to a full-length commit SHA, and both
`subosito/flutter-action` and `jurplel/install-qt-action` call
`actions/cache@v5` *by tag* on their caching path. The job is rejected during
`Set up job`, before a single step runs:
The action actions/cache@v5 is not allowed in NDDev-it-com/ci-workflows
because all actions must be pinned to a full-length commit SHA.
This is a real constraint on the library, not on the fixture: any consumer whose
organisation enforces the same policy hits it the first time they call either
workflow. Both now expose `enable_cache` (default `true`, so nothing changes for
consumers who can cache) and the fixture callers set it `false`. The Flutter
canonical receipt therefore no longer requires the `cache` section, and the
section stays in the vocabulary for consumers who can produce it.
**Android needs its lock state before it can build.** The fixture sets
`LockMode.STRICT`, so `:app:debugRuntimeClasspath` fails with "locked but does
not have lock state" until the closure is generated. That is what the new
reproduction lane produces.
`tools.yml` gains the four action usages the new lane introduces, which the
registry contract caught immediately.
…plicitly Running the generator on `ubuntu-latest` for the first time failed with `JAVA_HOME is group/world writable`. That is not a fixture bug: `setup-java` installs the JDK into the hosted tool cache, and the hosted tool cache is group writable, so the strict rule refuses **every** GitHub-hosted runner -- the one environment the committed closure should be generated in, because it is where the fixture lane runs. `ownership_problem` now separates the two bits it used to conflate. World-write is fatal under every flag. Group-write is fatal by default and relaxable through `allow_group_write`, which the generator sets with the reason written next to it: the runner VM is single-tenant and destroyed with the job, so the group that can write is the job itself. An unowned uid is still refused however the mode reads. Six further pure cases cover the relaxed rule, including `unowned-and-group-writable`, so the relaxation cannot widen past the one bit. Failure messages now carry the octal mode, because "group/world writable" did not say which bit or what the mode actually was, and that cost a runner round trip to find out. `enable_cache`, added an hour ago, is reverted. It was meant to let a consumer avoid the `actions/cache@v5` call inside `subosito/flutter-action` and `jurplel/install-qt-action`, and it cannot: the rejection happens in `Set up job`, where GitHub resolves the whole nested action graph before any input is evaluated, so a runtime `false` never gets a chance. Shipping an input whose description claims it solves a problem it cannot solve is the exact defect this repository keeps removing, so it is gone and the real finding is recorded instead.
The hosted tool cache is mode **0777**, not group-writable as the previous round guessed from a message that did not print the mode. Relaxing one bit was therefore both wrong and, more importantly, the wrong shape: what varies between a shared build host and a hosted runner is not which write bit is acceptable, it is whether the mode bits carry information at all. `ownership_problem` now takes a named trust model. `exclusive-filesystem`, the default, is the strict rule unchanged: any group or world write bit disqualifies a root, because on a shared or long-lived host those bits name other principals who can swap the toolchain between validation and use. `ephemeral-single-tenant-runner` stops reading them, because on a VM that is destroyed with the job there is no other principal for them to name. Ownership is enforced identically under both -- that invariant never depended on the host. The generator declares the ephemeral model with the reason beside it. Refusing to run there would not have made anything safer; it would only have moved generation of the committed closure onto a maintainer's laptop, which is less reproducible and no better guarded. An unrecognised model is itself a finding, so a typo fails closed instead of selecting the permissive branch. Proven by mutation: deleting the unknown-model guard, dropping ownership enforcement from the ephemeral model, and stopping the strict model reading mode bits each fail with the cases that catch them.
… on a runner 557 verified artifacts, two Gradle 9.5 lockfiles, and the provenance receipt for the exact default `./gradlew build` task graph (116 tasks), produced by the hosted reproduction lane rather than by a laptop. The fixture sets `LockMode.STRICT`, so until this landed `:app:debugRuntimeClasspath` could not resolve at all and the Android lane could never have run. The lane is now a standing check rather than a one-off generator: it regenerates the closure, publishes it, and fails if it differs from what is committed. That is only a stable check because the receipt no longer records host identity -- otherwise every runner-image JDK bump would have reported drift in a file nothing had touched. `validate_all --tier core` is green.
… it ran on Three defects, all visible only once the Android lane got far enough to emit a receipt. **The launcher stripped the receipt.** `check_sdk_runtime_fixtures --receipt` reads `SDK_RUNTIME_EVIDENCE`, and the hermetic execution boundary replaces the child environment with the base allowlist, which does not contain it. The observer therefore died with `invalid/missing JSON: 'SDK_RUNTIME_EVIDENCE'` before looking at anything. It is now registered in `surface_environment`, the same mechanism `verify_scorecard_runtime.py` uses for `GH_TOKEN` — an explicit per-tool inheritance rather than a hole in the allowlist. **The reusable applied the strict root rule on a hosted runner.** All five roots came back "group/world writable", so the receipt honestly reported five `untrusted_roots` and omitted the `jdk`, `gradle` and `android_sdk` sections — the degradation worked exactly as designed, and the canonical observer correctly refused the result. The workflow now selects its trust model from `RUNNER_ENVIRONMENT`, which the runner sets and a caller cannot forge: hosted runners get the ephemeral model, everything else stays strict. The chosen model is recorded in the receipt, so a reader can see which rule produced it rather than having to infer it. **The task graph ran past the end of each line.** `^> Task (:[^ ]+)` — `[^ ]` matches a newline, so entries arrived as `":app:generateDebugResources\\n>"` and `CANONICAL_TASKS.issubset` could never hold. `:\\S+` stops at the line end. Verified by replaying the receipt from run 31782520515 through the observer locally: it now passes.
…me the real barrier The reproduction lane reported drift on a build that had not changed: Gradle schedules independent tasks concurrently, so two runs of the identical exact build emit the same 116 tasks in a different sequence, and the committed manifest recorded the sequence. The evidence is *which* tasks the default command ran, not the order the scheduler happened to pick, so the graph is stored sorted. The same `[^ ]` bug fixed in the workflow was also here, where it would have let a task name run past the end of its line. `runtime_debt.barrier` gains `dependency-policy-conflict`. Neither existing term fits: `heavy-toolchain` says provisioning is expensive, which is what `dart-flutter-ci` and `qt-ci` claimed before anyone tried to run them, and `external-authority` says someone else has to act. The truth is narrower and more useful — a third-party action names nested actions by tag, this repository requires full-length commit SHAs, and the job is refused before it starts. The fix is ours to make and no amount of fixture work reaches it. Ledger: `kotlin-android-ci` is `runtime-proven` on linux with run 31782942981 and digest 25229a37. `dart-flutter-ci` and `qt-ci` move from `unverified` to `blocked` against issue #150 — they were never merely unfixtured, and saying so was itself inaccurate. 39 runtime-proven, 5 blocked, 2 unverified, 1 partial. All three tiers, actionlint clean.
Replaces the checkpoint's one-line placeholder with what actually changed: callee provenance, the generic-versus-canonical split, the digest grammar, the ambient-independent trust rule, the proven Android lane and its committed closure, the honest blocked status for Flutter and Qt, and the three defects that stopped the estate running.
…y scanners Committing the Android dependency locks handed three whole-tree scanners a dependency graph this library does not own. `tests/fixtures/android` vendors the Android Gradle Plugin's transitive closure -- 187 locked packages, 319 pinned in verification metadata -- purely so the fixture can build under `LockMode.STRICT`. osv-scanner reported 159 known vulnerabilities across 35 packages, Trivy 34 at CRITICAL/HIGH, and Grype enough to trip its critical cutoff. None of them are actionable here: the versions are AGP's to choose, not ours, and nothing in that tree is shipped, imported, or executed by a consumer. Left alone this makes `fixture / osv-scan`, `fixture / container-ci` and `fixture / grype scan` permanently red, which is worse than noisy — three lanes that always fail cannot report the finding that would matter. This is the same trap as a deliberately broken fixture colliding with a repository-wide lane, and it wants the same answer: scope the exclusion narrowly and say why. Each tool needed a different mechanism, so each is expressed where that tool looks: - `.grype.yaml` — two patterns, because `exclude` matches relative to the scan root and this closure is reachable from both `tests/fixtures` (the estate lane) and the repository root. - `trivy.yaml` — `scan.skip-dirs`. Verified 34 CRITICAL/HIGH to 0. - `osv-scan` — a scan argument, not a config file: 2.4.0 rejects `PackageOverrides.path` as an unknown key, and `--experimental-exclude` is the only path scoping it has. Verified 159 findings to none. Everything else under `tests/fixtures` is still scanned by all three.
rldyourmnd
force-pushed
the
fixtures/sdk-android-closure
branch
from
August 14, 2026 14:24
e516591 to
250809a
Compare
github-actions
Bot
added
ci
`dart-flutter-ci` and `qt-ci` are refused during `Set up job` (#150), so their four estate jobs failed on every run, and `render_runtime_evidence` — correctly fail-closed, since a lane that did not succeed is not evidence — turned `evidence summary (languages)` red with them. An estate that is always red reports nothing about the next real regression, which is the same defect as a gate that never fails. The four lanes are removed. A workflow that cannot start does not belong in the estate; it belongs in `catalog/runtime-coverage.yml` as `blocked`, with the barrier and the handoff, which is where it now is and where `validate_runtime_coverage` already forces it to carry both. To stop the two drifting apart, `check_sdk_runtime_fixtures.py` now derives the expected wiring from the ledger instead of assuming all three kinds are live: `runtime-proven` requires an exactly-wired caller and observer, `blocked` requires that neither exists, and any other status is itself a finding. Before this, the contract demanded a live caller for every kind, so recording the truth in the ledger would have failed the blocking tier — the two contracts could not both be satisfied. Proven by mutation in both directions: marking Qt `runtime-proven` while its lane is gone fails, and marking Android `blocked` while its lane exists fails.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the work checkpointed in #145 and the findings in #147, and closes #129
for the one pack that can be proven here.
What this proves
kotlin-android-ci.ymlis runtime-proven onubuntu-latest(run 31783511309):
fixture green, observer accepted the receipt, and the committed dependency
closure re-derived and diffed byte for byte in the same run. Ledger: 39
runtime-proven, 5 blocked, 2 unverified, 1 partial.
What changed, and why
Callee provenance (#147). The receipts hashed
${github.workspace}/.github/workflows/<callee>.yml. Inside a reusable that isthe caller's checkout, so an external consumer either crashed or published
their own file's bytes as our provenance. Now the runner-supplied
job.workflow_repository/job.workflow_sha/job.workflow_file_pathtriple.Proven by a probe pinned at an absolute commit:
job.workflow_shareported thecallee,
github.shathe caller —run 31779014883.
.github/actionlint.yamlcarries one suppression for those three property namesin those three files, because actionlint v1.7.12 does not model them yet; a typo
is still reported, and
check_actionlint_contractwill say when it can go.Generic reusable, canonical observer (#147). Qt died on a documented "Empty
to skip"; Android demanded a wrapper,
ANDROID_HOME, verification metadata, amodule named
appand an APK — in a workflow that advertises pure-JVM support.Receipts are now discriminated records (
kind+sections); fixture-specificassertions moved to the observers.
One digest grammar. Four of six digest-bearing fields were checked only for
truthiness.
The trust rule no longer depends on the machine. It failed on any
umask 002developer box and inverted under root. Now a pure(uid, mode)predicate with a named trust model — strict by default, explicit
ephemeral-single-tenant-runnerfor hosted runners whose tool cache is 0777.Flutter and Qt are
blocked, notunverified— see #150. Their vendoredsetup actions name nested actions by tag, so the job is refused at
Set up jobunder the SHA-pinning control this library recommends. That is a consumer-facing
defect, and no input can reach a setup-time graph resolution.
Three defects that stopped the estate running: observers never installed
uv; the hermetic launcher strippedSDK_RUNTIME_EVIDENCE; and the task-graphregex matched across newlines.
Corrections to the external review
check_flutter_pin.py. Acting on that item would have broken it.sample. Fixed by lowering its entropy, not by an allowlist; re-proven with the
same digest-pinned scanner.
catalog/runner-routing.ymldoes not exist. The real file isworkflow-routing.yml.Verification
validate_allall three tiers,actionlint, andzizmor@1.26.1 --persona pedantic --min-severity lowwith a token: all clean. Every new rule is proven bymutation, not only by passing — see the commit messages.
Supersedes #145, which stays open until this merges and is then closed as
checkpointed work.