Skip to content

ci(shorebird): precache Android engine artifacts before Shorebird builds - #7988

Merged
NotThatKindOfDrLiz merged 2 commits into
mainfrom
fix/shorebird-android-precache
Aug 20, 2026
Merged

NotThatKindOfDrLiz merged 2 commits into
mainfrom
fix/shorebird-android-precache

Conversation

@NotThatKindOfDrLiz

Copy link
Copy Markdown
Member

Fixes #7987. Unblocks the Android half of #7843 (epic #7841).

The failure

Android Build step 16, Build Android AAB (Shorebird release), exit 70:

Flutter debug embedding JAR not found at
/Users/builder/.shorebird/bin/cache/flutter/0cae2fc0a4b0c190d9f9090ad731b28cc6fb85df/bin/cache/artifacts/engine/android-arm64/flutter.jar.
Run 'flutter precache --android' for the FLUTTER_ROOT toolchain.

build_apk passes just before it because that is a plain flutter build on Codemagic's own precached Flutter. build_aab runs shorebird release android, which builds through Shorebird's bundled Flutter — and that copy has no Android engine artifacts.

Reproduced on a local install of the same pinned version, so it is Shorebird's default on a macOS host rather than a CI fluke:

$ ls ~/.shorebird/bin/cache/flutter/0cae2fc0a4b0c190d9f9090ad731b28cc6fb85df/bin/cache/artifacts/engine/
common  darwin-x64  ios  ios-profile  ios-release

Why it worked before and doesn't now

Android releases 1.0.20+714, +716 and +719 exist, so this path used to work. shorebird_install — from the pinned-revision hardening in #7202 — wipes the cache whenever the pin does not match:

if ! printf '%s\n' "$SHOREBIRD_VERSION_OUTPUT" | grep -Fq "$EXPECTED_SHOREBIRD_VERSION" || \
   [ "$SHOREBIRD_REVISION" != "$EXPECTED_SHOREBIRD_REVISION" ]; then
  rm -rf "$HOME/.shorebird"

That discards bin/cache/flutter/*/bin/cache/artifacts along with everything else, and nothing recreates it — grep -c precache codemagic.yaml returned 0. Because $HOME/.shorebird is also in cache_paths, the failure looks intermittent across runs, which is worse to diagnose under incident pressure.

Changes

A precache_shorebird_android_artifacts step in both Shorebird Android workflows.

Placed after each workflow's gates, so a build rejected by the release preflight or by provenance/patch-source verification does not pay for a multi-hundred-MB download first: after build_apk in android-build, after write_shorebird_private_key in android-patch.

Not in e2e-smoke-android — it builds with Codemagic's Flutter and neither needs the artifacts nor should pay for them.

The revision is derived, not hardcoded. shorebird flutter exposes only config and versions, so there is no CLI proxy for precache and the Flutter binary inside Shorebird's cache has to be invoked directly. Parsing it from shorebird --version avoids a second pin that could silently drift from EXPECTED_SHOREBIRD_REVISION. The step fails loudly if either the revision or the binary is missing, rather than skipping quietly and letting Gradle produce the original confusing error.

android-patch is fixed too. It runs shorebird patch android through the same bundled Flutter and would hit the identical wall the moment a patchable Android release exists.

Verification

  • 70/70 .github/scripts/tests pass, including the new test_android_shorebird_builds_precache_android_engine_artifacts.
  • New test confirmed red against pre-fix codemagic.yaml: '- *precache_shorebird_android_artifacts' not found in ' android-build: …'.
  • codemagic.yaml still parses with aliases; all 8 workflows resolve.

The test asserts per workflow that the step exists and precedes the actual build step, asserts e2e-smoke-android does not get it, and pins that the revision comes from shorebird --version.

Real proof is the next Android Build reaching shorebird release android — which is also the first provenance-recorded Android release Divine will have, since divinevideo/divine-release-provenance currently has no releases/android/ directory at all.

🤖 Generated with Claude Code

shorebird release android dies in Gradle with "Flutter debug embedding JAR
not found ... Run 'flutter precache --android'". Shorebird ships its own
Flutter and precaches only what the host needs, which on a macOS runner is
iOS and desktop; the engine artifact directory holds common, darwin-x64 and
the three ios entries, and no android ones. Reproduced on a local install of
the same pinned version, so it is Shorebird's default rather than a CI fluke.

Android releases 1.0.20+714/716/719 predate this because shorebird_install
rm -rf's ~/.shorebird whenever the pin does not match, which discarded the
artifacts those builds had. Nothing recreated them: codemagic.yaml contained
no precache at all. Since ~/.shorebird is also cached, the failure looks
intermittent across runs, which is worse to diagnose under incident pressure.

Precaches in both Shorebird Android workflows, after their gates so a
rejected build does not pay for the download first, and not in
e2e-smoke-android, which builds with Codemagic's Flutter. The revision comes
from shorebird --version rather than a second hardcoded pin that could drift
from EXPECTED_SHOREBIRD_REVISION.

Fixes #7987.
@NotThatKindOfDrLiz
NotThatKindOfDrLiz requested a review from a team as a code owner August 20, 2026 22:43
@NotThatKindOfDrLiz
NotThatKindOfDrLiz requested review from dcadenas, mbradley, rabble and realmeylisdev and removed request for a team, dcadenas and realmeylisdev August 20, 2026 22:43

@rabble rabble left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking test gap in .github/scripts/tests/test_codemagic_shorebird_config.py:361-390: the new test does not exercise the behavior that fixes the release. It checks that the alias appears before the build and that the whole file contains FLUTTER_REVISION=$(shorebird --version, but it never asserts that the anchor invokes the derived Flutter binary or runs precache --android. I mutation-tested this by replacing "$SHOREBIRD_FLUTTER" precache --android with a no-op; the new test still passed. It also does not require shorebird_install < precache < build, so moving the step before Shorebird exists would remain green.

This is the same whole-file false-positive shape that let the missing patch-key wiring pass before #7936, and the expensive Codemagic release is currently the only end-to-end proof. Please scope the assertions to the &precache_shorebird_android_artifacts definition, require the revision parse, derived executable path, and actual precache --android invocation there, and assert shorebird_install < precache < build in both Android workflows.

The production configuration itself matches the diagnosed failure, and the revision parser/path worked against a real local Shorebird installation. This test coverage is the only blocker I found.

Review found the new test verified the alias's placement but never its
body: replacing "$SHOREBIRD_FLUTTER" precache --android with a no-op left
it green. It also allowed precache to sit before shorebird_install, so the
step could run before the CLI it derives its Flutter path from exists.

Adds _definition_block so assertions scope to the anchor rather than the
whole file — the same whole-file false-positive shape that hid the missing
patch-key wiring until #7936. The definition must now parse the revision
from the CLI, must not hardcode a 40-char SHA, must build the executable
path from that revision, must invoke precache --android through it, and
must keep both fail-loud guards. Ordering is asserted both ways:
shorebird_install < precache < build, in both Android workflows.

Mutation-tested: no-op precache, precache-before-install, a hardcoded
revision, and a disabled -x guard each turn it red.

@mbradley mbradley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified first-hand: step ordering is correct in both android-build and android-patch (shorebird_install populates ~/.shorebird before precache, nothing wipes it between, and precache lands right before build_aab / patch_android), and the new test passes at head and goes red when the wiring is removed, with e2e-smoke-android correctly excluded.

One thing to confirm before relying on this: the fix rests entirely on shorebird --version printing a Flutter ... revision <40-hex> line on stdout (the step reads it with 2>/dev/null). Nothing covers that. The test is a string match, and the PR reproduces the failure rather than the fix. If the format differs, the line goes to stderr, or the sha isn't 40 chars, FLUTTER_REVISION is empty and the step exits 1, blocking every Android release/patch. It fails loud and can't regress an already-broken path, so I'm not blocking on it, but since this is the first provenance-recorded Android release there's no prior green run to fall back on. Can you paste the actual shorebird --version output from the pinned CLI, or confirm the assignment resolves the revision locally?

Minor: the step resolves FLUTTER_REVISION silently. An echo before precache would help the incident diagnosis this change is aimed at:

        echo "Precaching Android engine artifacts for Flutter $FLUTTER_REVISION"
        "$SHOREBIRD_FLUTTER" precache --android

Thanks, as always.

@NotThatKindOfDrLiz

Copy link
Copy Markdown
Member Author

@rabble — fixed in 4fa5ce8425. All checks green (8 passed, 6 skipped for a CI-config-only change).

You were right, and the mutation test is the part that made it undeniable: I asserted the alias's placement and never its body, so the step could have done nothing and stayed green. Both items were the same underlying mistake — asserting where the change sits rather than what it does.

Fixed — the anchor's body is now the assertion target. Added _definition_block(anchor), mirroring the existing _workflow_block, so assertions scope to &precache_shorebird_android_artifacts instead of the whole file. Within that definition the test now requires: the revision parsed from shorebird --version, no hardcoded 40-char SHA, the executable path built from that revision, "$SHOREBIRD_FLUTTER" precache --android actually invoked, and both fail-loud guards with their two exit 1s.

Fixed — ordering asserted both ways. shorebird_install < precache < build in both android-build and android-patch. Your point stands on its own: precache derives its path from shorebird --version, so running before the CLI exists would fail at the revision parse rather than doing anything useful.

Mutation-tested, including yours. Each of these turns the test red, verified individually with the YAML restored between runs:

Mutation Result
"$SHOREBIRD_FLUTTER" precache --android → true # no-op FAILED
precache moved before shorebird_install FAILED
revision hardcoded to the current 40-char SHA FAILED
if [ ! -x "$SHOREBIRD_FLUTTER" ] → if false FAILED
unmutated OK

The #7936 comparison is apt and worth keeping in mind beyond this PR — that one passed because the release commands carried the flag the patch commands were missing. Same shape here: whole-file assertIn proves a string exists somewhere, never that it exists where it does the work.

No production changes in this push; codemagic.yaml is byte-identical to what you reviewed. 70/70 .github/scripts/tests pass. Re-requesting review.

@rabble rabble left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head. The prior test gap is fixed: assertions are scoped to the precache anchor, require the derived Shorebird Flutter path and actual precache --android invocation, reject a hardcoded revision, require both fail-loud guards, and enforce shorebird_install < precache < build in release and patch workflows. The production YAML remains the diagnosed fix. I also confirmed the installed Shorebird CLI emits the expected Flutter … revision <40-hex> stdout shape; the author reports validating the pinned release locally. All current checks are green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Shorebird Android builds fail — bundled Flutter has no Android engine artifacts

3 participants