Repository navigation
ci(shorebird): precache Android engine artifacts before Shorebird builds - #7988
Conversation
shorebird release android dies in Gradle with "Flutter debug embedding JAR not found ... Run 'flutter precache --android'". Shorebird ships its own Flutter and precaches only what the host needs, which on a macOS runner is iOS and desktop; the engine artifact directory holds common, darwin-x64 and the three ios entries, and no android ones. Reproduced on a local install of the same pinned version, so it is Shorebird's default rather than a CI fluke. Android releases 1.0.20+714/716/719 predate this because shorebird_install rm -rf's ~/.shorebird whenever the pin does not match, which discarded the artifacts those builds had. Nothing recreated them: codemagic.yaml contained no precache at all. Since ~/.shorebird is also cached, the failure looks intermittent across runs, which is worse to diagnose under incident pressure. Precaches in both Shorebird Android workflows, after their gates so a rejected build does not pay for the download first, and not in e2e-smoke-android, which builds with Codemagic's Flutter. The revision comes from shorebird --version rather than a second hardcoded pin that could drift from EXPECTED_SHOREBIRD_REVISION. Fixes #7987.
rabble
left a comment
There was a problem hiding this comment.
Blocking test gap in .github/scripts/tests/test_codemagic_shorebird_config.py:361-390: the new test does not exercise the behavior that fixes the release. It checks that the alias appears before the build and that the whole file contains FLUTTER_REVISION=$(shorebird --version, but it never asserts that the anchor invokes the derived Flutter binary or runs precache --android. I mutation-tested this by replacing "$SHOREBIRD_FLUTTER" precache --android with a no-op; the new test still passed. It also does not require shorebird_install < precache < build, so moving the step before Shorebird exists would remain green.
This is the same whole-file false-positive shape that let the missing patch-key wiring pass before #7936, and the expensive Codemagic release is currently the only end-to-end proof. Please scope the assertions to the &precache_shorebird_android_artifacts definition, require the revision parse, derived executable path, and actual precache --android invocation there, and assert shorebird_install < precache < build in both Android workflows.
The production configuration itself matches the diagnosed failure, and the revision parser/path worked against a real local Shorebird installation. This test coverage is the only blocker I found.
Review found the new test verified the alias's placement but never its body: replacing "$SHOREBIRD_FLUTTER" precache --android with a no-op left it green. It also allowed precache to sit before shorebird_install, so the step could run before the CLI it derives its Flutter path from exists. Adds _definition_block so assertions scope to the anchor rather than the whole file — the same whole-file false-positive shape that hid the missing patch-key wiring until #7936. The definition must now parse the revision from the CLI, must not hardcode a 40-char SHA, must build the executable path from that revision, must invoke precache --android through it, and must keep both fail-loud guards. Ordering is asserted both ways: shorebird_install < precache < build, in both Android workflows. Mutation-tested: no-op precache, precache-before-install, a hardcoded revision, and a disabled -x guard each turn it red.
mbradley
left a comment
There was a problem hiding this comment.
Verified first-hand: step ordering is correct in both android-build and android-patch (shorebird_install populates ~/.shorebird before precache, nothing wipes it between, and precache lands right before build_aab / patch_android), and the new test passes at head and goes red when the wiring is removed, with e2e-smoke-android correctly excluded.
One thing to confirm before relying on this: the fix rests entirely on shorebird --version printing a Flutter ... revision <40-hex> line on stdout (the step reads it with 2>/dev/null). Nothing covers that. The test is a string match, and the PR reproduces the failure rather than the fix. If the format differs, the line goes to stderr, or the sha isn't 40 chars, FLUTTER_REVISION is empty and the step exits 1, blocking every Android release/patch. It fails loud and can't regress an already-broken path, so I'm not blocking on it, but since this is the first provenance-recorded Android release there's no prior green run to fall back on. Can you paste the actual shorebird --version output from the pinned CLI, or confirm the assignment resolves the revision locally?
Minor: the step resolves FLUTTER_REVISION silently. An echo before precache would help the incident diagnosis this change is aimed at:
echo "Precaching Android engine artifacts for Flutter $FLUTTER_REVISION"
"$SHOREBIRD_FLUTTER" precache --androidThanks, as always.
|
@rabble — fixed in You were right, and the mutation test is the part that made it undeniable: I asserted the alias's placement and never its body, so the step could have done nothing and stayed green. Both items were the same underlying mistake — asserting where the change sits rather than what it does. Fixed — the anchor's body is now the assertion target. Added Fixed — ordering asserted both ways. Mutation-tested, including yours. Each of these turns the test red, verified individually with the YAML restored between runs:
The No production changes in this push; |
rabble
left a comment
There was a problem hiding this comment.
Re-reviewed the current head. The prior test gap is fixed: assertions are scoped to the precache anchor, require the derived Shorebird Flutter path and actual precache --android invocation, reject a hardcoded revision, require both fail-loud guards, and enforce shorebird_install < precache < build in release and patch workflows. The production YAML remains the diagnosed fix. I also confirmed the installed Shorebird CLI emits the expected Flutter … revision <40-hex> stdout shape; the author reports validating the pinned release locally. All current checks are green.
Fixes #7987. Unblocks the Android half of #7843 (epic #7841).
The failure
Android Buildstep 16,Build Android AAB (Shorebird release), exit 70:build_apkpasses just before it because that is a plainflutter buildon Codemagic's own precached Flutter.build_aabrunsshorebird release android, which builds through Shorebird's bundled Flutter — and that copy has no Android engine artifacts.Reproduced on a local install of the same pinned version, so it is Shorebird's default on a macOS host rather than a CI fluke:
Why it worked before and doesn't now
Android releases
1.0.20+714,+716and+719exist, so this path used to work.shorebird_install— from the pinned-revision hardening in #7202 — wipes the cache whenever the pin does not match:That discards
bin/cache/flutter/*/bin/cache/artifactsalong with everything else, and nothing recreates it —grep -c precache codemagic.yamlreturned0. Because$HOME/.shorebirdis also incache_paths, the failure looks intermittent across runs, which is worse to diagnose under incident pressure.Changes
A
precache_shorebird_android_artifactsstep in both Shorebird Android workflows.Placed after each workflow's gates, so a build rejected by the release preflight or by provenance/patch-source verification does not pay for a multi-hundred-MB download first: after
build_apkinandroid-build, afterwrite_shorebird_private_keyinandroid-patch.Not in
e2e-smoke-android— it builds with Codemagic's Flutter and neither needs the artifacts nor should pay for them.The revision is derived, not hardcoded.
shorebird flutterexposes onlyconfigandversions, so there is no CLI proxy for precache and the Flutter binary inside Shorebird's cache has to be invoked directly. Parsing it fromshorebird --versionavoids a second pin that could silently drift fromEXPECTED_SHOREBIRD_REVISION. The step fails loudly if either the revision or the binary is missing, rather than skipping quietly and letting Gradle produce the original confusing error.android-patchis fixed too. It runsshorebird patch androidthrough the same bundled Flutter and would hit the identical wall the moment a patchable Android release exists.Verification
.github/scripts/testspass, including the newtest_android_shorebird_builds_precache_android_engine_artifacts.codemagic.yaml:'- *precache_shorebird_android_artifacts' not found in ' android-build: …'.codemagic.yamlstill parses with aliases; all 8 workflows resolve.The test asserts per workflow that the step exists and precedes the actual build step, asserts
e2e-smoke-androiddoes not get it, and pins that the revision comes fromshorebird --version.Real proof is the next
Android Buildreachingshorebird release android— which is also the first provenance-recorded Android release Divine will have, sincedivinevideo/divine-release-provenancecurrently has noreleases/android/directory at all.🤖 Generated with Claude Code