task: prove the Shorebird release-backport runbook on iOS and Android

Description

Exercise the release-specific Shorebird patch runbook end to end on non-production iOS and Android releases after the implementation in the sibling child issue lands. Prove first patch, cumulative second patch, and rollback with captured evidence.

Context

Static workflow tests cannot prove that operators can build, install, validate, promote, observe, and roll back a patch for the exact targeted store artifact. Parent epic #7841 is not complete until both supported platforms pass a controlled drill.

Preconditions

The release-specific patch workflow from the sibling implementation task is merged.
Test releases are built with signed shorebird release workflows and valid immutable provenance.
The drill uses staging/non-production configuration and must not promote a test patch to the production stable track.

Drill procedure

Cut identifiable non-production Shorebird releases for iOS and Android.
Advance main with representative Dart, native, asset, and version changes so it no longer qualifies as a patch source.
Create each canonical release patch line from its recorded baseline.
Backport one minimal Dart behavior change plus its test.
Confirm provenance, branch identity, blocked-path, configuration, and signing checks pass.
Publish each patch to staging.
Preview/install the exact targeted release and verify the first fix applies.
Add a second patch on the same release line and prove that both the first and second fixes remain present.
Exercise rollback and verify the device returns to the expected prior behavior.
Capture duration, permissions, commands, console steps, logs/Crashlytics patch identity, and any runbook corrections.

Acceptance criteria

Testing evidence

Attach Codemagic build links, Shorebird release/patch identifiers, device or preview verification, rollback evidence, and the final clean runbook checklist. Do not include credentials, signing material, dart-define values, or other secrets.

Drill progress — 2026-08-19 (iOS prepared, Android blocked)

iOS: patch line is live and validated

Target release is 1.0.21+851, whose private provenance record is intact and patchable: true, with patch_baseline_commit d6ae3c0a00becd3e15679a771e5da0d4af4865a7.

Drill step 2 ("advance main so it no longer qualifies as a patch source") needed no synthetic work — main is already 18 commits past that baseline, including native, asset, and dependency changes. That is the real condition the epic claims to survive, so the drill exercises it rather than simulating it.

Patch line shorebird-patch/ios/1.0.21+851 created from the recorded baseline and pushed.
Backport is 57a3acc8dd (fix(library): drop the grid-size button from the editor clip picker, fix(library): drop the grid-size button from the editor clip picker #7790) → cherry-picked as 80a25a44b8. Two Dart files, no blocked paths, ships its own test, and is visually obvious on device.
Chosen deliberately as a genuine merged main fix rather than a throwaway: because the line is immutable and cumulative, whatever lands on it must be retained by every future patch for this release. A real fix stays legitimate; a synthetic one would be permanent debt.
Repository ruleset Shorebird patch line immutability confirmed active on refs/heads/shorebird-patch/** (deletion + non-fast-forward prohibited).

Validator evidence (`main`'s copy, run against the real line)

Accepted:

Verified patch branch: shorebird-patch/ios/1.0.21+851
Patch source commits after release baseline d6ae3c0a00becd3e15679a771e5da0d4af4865a7:
80a25a44b8 fix(library): drop the grid-size button from the editor clip picker (#7790)

Dart files changed by the complete patch:
lib/screens/library_screen.dart
test/screens/library_screen_test.dart

Rejected, all fail-closed with exit 1:

Case	Result
Current `main` as patch source	`patch source contains commits already on main`
Wrong release version (`1.0.21+850`)	`patch branch must be exactly "shorebird-patch/ios/1.0.21+850"`
Wrong platform (`android`)	`patch branch must be exactly "shorebird-patch/android/1.0.21+851"`
Native change (`ios/Runner/AppDelegate.swift`) in payload	`refusing to patch: … cut a normal store release instead`

The first row is the epic's central claim, proven against the actual advanced main rather than a fixture.

As a side effect, the pre-push hook ran the backported screen's suite against the release baseline tree: 28 passed. That is a useful extra signal the runbook does not currently ask for — the backport is green on the code it will actually patch, not only on main.

Android: blocked on provenance

divinevideo/divine-release-provenance contains only releases/ios/. The three Android Shorebird releases (1.0.20+714, +716, +719) all predate automatic provenance recording, so they fail closed exactly like iOS 1.0.21+849.

The Android half of this drill cannot start until an android-build runs and mints a release with a valid record. That is a scheduling dependency, not a defect.

Runbook corrections

Two gaps found while preparing the line, fixed in #7917:

A fresh patch worktree has no .dart_tool/, so the pre-push hook's analyzer fails on the missing package config rather than on the backport. Needs mise trust + flutter pub get first, then a clean git status so no pubspec.lock churn joins the payload (it would be a blocked path).
The validator is absent from any patch line whose baseline predates ci(shorebird): keep emergency patches isolated from ongoing development #7844 — by design, since the workflow copies it from main. Documented, with the command to dry-run it locally.

Remaining for this issue

Run ios-patch in Codemagic from main (CONFIRM_PATCH=YES, RELEASE_VERSION=1.0.21+851, PATCH_BRANCH=shorebird-patch/ios/1.0.21+851, DEFAULT_ENV=PRODUCTION to match the recorded fingerprint).
Verify on device via shorebird preview --track staging --release-version 1.0.21+851.
Second cumulative patch on the same line; prove both fixes survive.
Rollback and verify.
Android release with provenance, then repeat.
Record timings, permissions, and named operator roles.

No drill patch will be promoted to the production stable track; patch_ios hardcodes --track=staging.

docs(shorebird): record the two patch-line gotchas the drill hit

Drill finding — the patch workflow could never sign a patch

The first ios-patch run failed at step 19, Patch iOS (Shorebird), exit 64:

Both public and private keys must be provided.

Not an input mistake and not a regression from #7844. shorebird patch needs both halves — --private-key-path signs the patch, --public-key-path validates that signature against the key baked into the release by shorebird release --public-key-path. Both patch commands passed only the private key, and neither patch workflow ran *write_shorebird_public_key, so the public key was missing from the command line and from disk.

This has been broken since code push was wired up in #7202. Nothing caught it because no patch had ever actually been built. Every safety mechanism the epic added works — provenance verification, branch identity, ancestry, blocked paths, the not-already-on-main check all passed on this run — the workflow simply could never reach a successful signing step behind them.

Both ios-patch and android-patch were affected identically, so the Android half would have hit the same wall.

Fix in #7936 (green).

Why the config tests missed it

self.assertIn("--public-key-path=build/shorebird/patch_public_key.pem", self.contents)

Whole-file substring. The release commands carry that flag, so the assertion passed on their strength while neither patch command had it. The replacement asserts per extracted patch command and per workflow block; both new tests are confirmed red against pre-fix codemagic.yaml.

What this says about the drill

This is the drill doing its job. Static configuration tests pinned every property the implementation was written to have, and the one property nobody had ever exercised — that the command actually runs — was the one that was broken. Worth carrying into the epic's conclusion: the value here was not confirming the guards work, it was reaching the step past them.

Blocked until #7936 merges

Re-run ios-patch with the same inputs once it lands:

CONFIRM_PATCH = YES
RELEASE_VERSION = 1.0.21+851
PATCH_BRANCH = shorebird-patch/ios/1.0.21+851
RELEASE_COMMIT = empty
DEFAULT_ENV = PRODUCTION

The patch line itself is unchanged and still validates; nothing about shorebird-patch/ios/1.0.21+851 needs redoing.

Drill finding — iOS staged-patch validation is not executable as written

The signing fix (#7937) worked: ios-patch completed and patch #1 is published on the staging track for 1.0.21+851. Step 7 of the drill procedure — "Preview/install the exact targeted release and verify the first fix applies" — is where it stops, and the reason is structural rather than incidental.

`shorebird preview` cannot work on iOS for Divine

First attempt failed on wireless transport (NWError 54 — Connection reset by peer over .coredevice.local). Over USB it failed differently and permanently:

Failed to install embedded profile for co.openvine.app : 0xe800801f
(Attempted to install a Beta profile without the proper entitlement.)

The artifact's embedded.mobileprovision:

Key	Value
`beta-reports-active`	`true`
`get-task-allow`	`false`
`ProvisionedDevices`	absent

That is an App Store distribution profile. No device list exists, so no device can be provisioned for it directly; it installs through the App Store or TestFlight and nowhere else. This follows from ios_signing: distribution_type: app_store in codemagic.yaml, which is correct for a shipping build. shorebird preview is simply not available on this platform for this product.

The documented fallback is also closed

The runbook offers "or an equivalent installed build" — the TestFlight copy, which genuinely is the same binary. But an installed build only polls stable: auto_update is on, the shorebird_code_push default is UpdateTrack.stable, and lib/main.dart calls only isAvailable and readCurrentPatch(), never update(track:). The package exposes UpdateTrack.staging; the app does not use it.

So the only way to make a staged patch visible on iOS is to promote it to stable — the production deploy that is supposed to follow validation, not enable it.

Tracked as #7979.

Unblocking this drill: promote on `851` is safe today, and only today

The live App Store build is 1.0.21+849. Patches are scoped to a release version, so a patch on 851 can only reach installs running 851 — TestFlight testers. There is no path from a 851 patch to a device running 849.

Promoting patch #1 to stable on 851 therefore has zero production blast radius, and reaches exactly the cohort we want to validate with.

This bends the letter of the acceptance criterion "No drill patch is promoted to the production stable track" while preserving its intent, which is that no drill patch reaches production users. Recording the deviation explicitly rather than quietly:

shorebird patches set-track --release 1.0.21+851 --patch 1 --track stable

The shortcut expires on success. Once #7980 puts a patchable build in the App Store, promoting to stable is a production deploy. From that release onward this substitution is invalid and #7979 is the only correct validation path — which is why #7979 must ship in that binary rather than after it.

Revised iOS procedure for this drill

Promote patch 🏗️ [Infrastructure] Configure R2 Storage Buckets for Media Processing #1 to stable on 1.0.21+851.
Verify on a TestFlight install of 1.0.21+851: the grid-size button is gone from the editor clip picker. Record the patch number from logs/Crashlytics.
Add the second cumulative backport to the patch line, build, and confirm both fixes are present.
Exercise rollback and confirm the device returns to prior behaviour.
Record timings, permissions, and named operator roles.

Runbook corrections this produces

The iOS validation step must stop recommending shorebird preview, state why it cannot work against app_store-signed artifacts, and point at task: add a staging-track patch validation affordance to developer options #7979.
Note that a patch's blast radius is the set of installs on that exact release version — which is what makes a TestFlight-only release safe to promote and a shipping one not.

Android is unaffected by the preview problem and can use it normally, once a provenance-recorded release exists.

iOS drill evidence — first patch and cumulative patch verified on device

Both verified on a TestFlight install of 1.0.21+851, which is the same store-signed binary App Store users would receive if that build were promoted.

Patches

Patch	ID	Contents	Track
#1	`608115`	`80a25a44b8` — backport of `57a3acc8dd` (#7790, drop grid-size button from editor clip picker)	`stable`
#2	`609899`	adds `7963afddfa` — backport of `5c9f747b8a` (#7846, Divine Family / Divine Kids links in Support Center)	`stable`

Patch line shorebird-patch/ios/1.0.21+851, rooted at the provenance-recorded baseline d6ae3c0a00becd3e15679a771e5da0d4af4865a7. Both backports are genuine merged main fixes, chosen deliberately: the line is immutable and cumulative, so anything placed on it must be retained by every later patch for this release.

Result

First patch applies and a later patch preserves the active one's fixes. Patch #2 carries both backports, and after promotion the Divine Family and Divine Kids rows appeared in Settings → Support Center on device. Since #2 is cumulative, its verification satisfies both criteria in one observation.

Verification-signal correction, worth recording

The grid-size button (#7790) turned out to be a poor go/no-go signal. It is a small gridNine icon button that renders only in the editor's clip-picker bottom sheet — not the Library tab — and it was reported as "not there" on both builds while patch #1 was already promoted to stable, meaning absence was the success state being read as a failure. Ambiguous either way.

The Support Center links are a much better signal and are what the drill should use going forward: two full-width rows with title and subtitle, on a screen reachable without entering an editor flow, either visibly present or visibly absent. Establish the "before" state before promoting.

Deviation from the acceptance criteria

Both patches were promoted to stable rather than validated on staging. This was necessary and is safe only under today's conditions:

shorebird preview cannot install an app_store-signed artifact at all (see the previous comment), so staging is unreachable on iOS.
The live App Store build is 1.0.21+849. Patches are scoped to an exact release version, so a patch on 851 has no path to a device running 849. Production blast radius was zero.

This satisfies the intent of "no drill patch is promoted to the production stable track" — no drill patch reached production users — while breaking its letter. The route expires when #7980 lands; once a patchable build is in the App Store, stable means production and #7979 becomes the only correct validation path.

Timings

Xcode archive: 278.7s
IPA build: 19.2s
Linking AOT files: the dominant cost. Still running past the 14-minute mark of the overall build before completing.

That last figure matters operationally: ios-patch carries max_build_duration: 60, and nobody previously knew where an app this size lands against it. Worth capturing the exact total from the Codemagic build page for the runbook, so the next operator knows whether 20 minutes is normal or alarming.

Verifying patch can be applied to release passed before linking, which is the gate that would have caught a patch/release mismatch.

Still open

iOS rollback — move patch 🎨 [Backend] Implement GIF Creation Engine with gifenc #2 back to staging and confirm the device reverts to patch 🏗️ [Infrastructure] Configure R2 Storage Buckets for Media Processing #1 behaviour (Support Center rows gone, grid-size button still gone).
Android, all of it — still blocked on any provenance-recorded release. Unlike iOS, Android can use shorebird preview normally once one exists.
Exact build duration, named operator roles, and permissions used.

Android is unblocked — first patchable Android release exists

Android Build succeeded after #7988. Gradle bundleRelease ran to completion in 304.4s, where the pre-fix run died at 12s on the missing flutter.jar.

✅ Published Release 1.0.22+739!

The precache step did exactly what it was added for — [4/12] Android Internal Build Artifacts pulled all ten Android engine entries including android-arm64, plus the six android-*-{profile,release}/darwin-x64 toolchains, in about 20 seconds.

The record that has never existed before

divinevideo/divine-release-provenance now has releases/android/1.0.22+739.json:

Field	Value
`platform`	`android`
`release_version`	`1.0.22+739`
`build_source_commit`	`b34e84b6dc94c26afd1ac983c80ff2a0b4d5cf88`
`patch_baseline_commit`	`b34e84b6dc94c26afd1ac983c80ff2a0b4d5cf88`
`flutter_version`	`3.44.9`
`patchable`	`true`

Before this, the provenance repository had no releases/android/ directory at all, so the Android patch workflow had never had a valid input in its life. It does now.

Uploaded to Play as a draft (submit_as_draft: true), so nothing reached users.

What now blocks the Android drill: nothing to backport

The baseline is b34e84b6, which is main's tip — and git log b34e84b6..origin/main is empty. Every merged fix is already inside the release, so cherry-picking one produces an empty diff, and the validator refuses:

abort_with('no Dart changes found between the release baseline and patch source') if dart_paths.empty?

The iOS half never hit this: 1.0.21+851 was cut from a commit main had already moved 18 commits past, so there were merged fixes available to backport. A release cut from current main has no such gap by construction.

So the Android drill needs a Dart-only fix that merges after b34e84b6, and it has to be device-visible, since the drill requires confirming the fix actually applied. Normal merge velocity should supply one shortly. Deliberately manufacturing a change for the drill would be worse than waiting — the whole reason both iOS backports were genuine merged fixes is that the patch line is immutable and cumulative.

Ruled out as a candidate: #7986 (deleting the unused AppConfig.appVersion). Dart-only and real, but nothing renders it, so there would be no way to tell whether the patch applied.

Once a candidate lands

Android verification is far simpler than iOS — shorebird preview --track staging --release-version 1.0.22+739 works, emulator included. No promotion to stable, no TestFlight, and none of the app_store-signing obstacles that forced the iOS deviation.

Unrelated observation from the build log

libapp.so.sym or libapp.so.dbg not present when checking final appbundle for debug symbols.
Play Console will not receive Dart crash symbols for this build.

Shorebird suggests removing packaging.jniLibs.keepDebugSymbols.add("**/libapp.so") from android/app/build.gradle.kts, but that line is not in the file — the suggestion does not match this configuration. The effect is real though: Android Dart crashes reach Play Console unsymbolicated, and unlike iOS (upload_dsyms_to_crashlytics) the Android workflow has no symbol-upload step. Pre-existing and unrelated to this epic; noting it here so it is not lost.

Drill complete — manual verification passed

Liz confirmed on 2026-08-21 that the remaining manual drill was completed on both platforms and all checks passed, including:

first isolated patch;
cumulative second patch preserving the first fix; and
rollback to the expected prior behavior.

The earlier comments retain the release, provenance, signing, build, and iOS device evidence. This confirmation closes the remaining manual Android and rollback work, so the two-platform drill is complete.

Closing as completed. Parent epic: #7841.

docs(shorebird): record the two patch-line gotchas the drill hit (div…

ci(shorebird): pass both signing keys to the patch commands (divinevi…

task: prove the Shorebird release-backport runbook on iOS and Android #7843

Description

Description

Context

Preconditions

Drill procedure

Acceptance criteria

Testing evidence

Activity

NotThatKindOfDrLiz commented on Aug 19, 2026

Drill progress — 2026-08-19 (iOS prepared, Android blocked)

iOS: patch line is live and validated

Validator evidence (main's copy, run against the real line)

Android: blocked on provenance

Runbook corrections

Remaining for this issue

NotThatKindOfDrLiz commented on Aug 20, 2026

Drill finding — the patch workflow could never sign a patch

Why the config tests missed it

What this says about the drill

Blocked until #7936 merges

NotThatKindOfDrLiz commented on Aug 20, 2026

Drill finding — iOS staged-patch validation is not executable as written

shorebird preview cannot work on iOS for Divine

The documented fallback is also closed

Unblocking this drill: promote on 851 is safe today, and only today

Revised iOS procedure for this drill

Runbook corrections this produces

NotThatKindOfDrLiz commented on Aug 20, 2026

iOS drill evidence — first patch and cumulative patch verified on device

Patches

Result

Verification-signal correction, worth recording

Deviation from the acceptance criteria

Timings

Still open

NotThatKindOfDrLiz commented on Aug 21, 2026

Android is unblocked — first patchable Android release exists

The record that has never existed before

What now blocks the Android drill: nothing to backport

Once a candidate lands

Unrelated observation from the build log

NotThatKindOfDrLiz commented on Aug 21, 2026

Drill complete — manual verification passed

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Validator evidence (`main`'s copy, run against the real line)

`shorebird preview` cannot work on iOS for Divine

Unblocking this drill: promote on `851` is safe today, and only today