Skip to content

Commit 7564317

Browse files
authored
[visual testing] Make the platform checks neutral on differences
The `Visual regression (android)` and `(ios)` checks fail on any difference, and applying `visual-change-approved` only passes the `Visual regression approval` gate, so an approved PR still shows two red checks - eg #59787, where both platforms only differ by the new test it adds. This makes the platform checks neutral when there are differences, so the gate is the only visual check that fails, and only until the differences are approved. None of the visual checks are required, and a neutral check doesn't fail a PR's overall status. Removing the label, or a push that changes the differences, fails the gate again as before. Changelog: [Internal] Test plan: New unit test. Also ran the compare step's `report` against #59787's real report with the GitHub API mocked: both platform checks post `neutral`, and the gate posts `failure` without the label and `success` ("approved by @javache") with it. Differential Revision: D132905555 Pull Request resolved: #60110
1 parent f4f8b6a commit 7564317

3 files changed

Lines changed: 30 additions & 9 deletions

File tree

‎.github/workflow-scripts/__tests__/visual-compare-test.js‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
const {compareCapture, compareImages, policyFor} = require('../visual/compare');
1212
const {
1313
comparePlatform,
14+
platformConclusion,
1415
resolveContext,
1516
stageBaseline,
1617
} = require('../visual/compare-run');
@@ -389,6 +390,16 @@ describe('visual comparison', () => {
389390
});
390391
});
391392

393+
describe('platformConclusion', () => {
394+
test.each([
395+
['passes without differences', [{status: 'unchanged'}], null, 'success'],
396+
['is neutral with differences', [{status: 'changed'}], null, 'neutral'],
397+
['is neutral without a comparison', [], {title: 'no capture'}, 'neutral'],
398+
])('%s', (_, results, error, conclusion) => {
399+
expect(platformConclusion({results, error})).toBe(conclusion);
400+
});
401+
});
402+
392403
describe('resolveContext', () => {
393404
const ENV = {
394405
CAPTURE_HEAD_SHA: SHA_A,

‎.github/workflow-scripts/visual/compare-run.js‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -367,6 +367,17 @@ async function prepare() {
367367
}
368368
}
369369

370+
/**
371+
* A platform check reports what it found without failing: differences are
372+
* neutral, and the approval gate is the only check that fails until they're
373+
* approved, so an approved PR has no failing visual checks.
374+
*/
375+
function platformConclusion(platform) {
376+
return platform.error == null && !summarize(platform.results).hasDifferences
377+
? 'success'
378+
: 'neutral';
379+
}
380+
370381
async function report() {
371382
const github = createGitHubClient();
372383
const outDir = requireEnv('OUT_DIR');
@@ -402,16 +413,10 @@ async function report() {
402413
`${prefix}/${platform.platform}/${kind}/${id}.png`,
403414
);
404415
const output = renderCheckOutput({...platform, imageUrl});
405-
const {hasDifferences} = summarize(platform.results);
406416
const checkRun = await github.createCheckRun({
407417
name: `${CHECK_NAME_PREFIX} (${platform.platform})`,
408418
headSha: context.headSha,
409-
conclusion:
410-
platform.error != null
411-
? 'neutral'
412-
: hasDifferences
413-
? 'failure'
414-
: 'success',
419+
conclusion: platformConclusion(platform),
415420
output,
416421
detailsUrl,
417422
});
@@ -511,4 +516,9 @@ if (require.main === module) {
511516
});
512517
}
513518

514-
module.exports = {comparePlatform, resolveContext, stageBaseline};
519+
module.exports = {
520+
comparePlatform,
521+
platformConclusion,
522+
resolveContext,
523+
stageBaseline,
524+
};

‎packages/rn-tester/visual-tests/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ Each test is a [Maestro](https://maestro.dev) flow in `flows/`. Maestro only ope
77
## How it runs in CI
88

99
1. Test All builds RNTester Release for Android and iOS. Once a build is ready, the `test_visual_android_rntester` and `test_visual_ios_rntester` jobs (`.github/workflows/visual-capture-rntester.yml`) run every flow in this directory on one canonical device per platform: an API 35 Pixel 7 emulator and an iPhone 17 Pro simulator on iOS 26.2. Any screenshot that differs from the baseline is retaken once, and the retake is kept if it matches, so a notification or other system UI that briefly covers the app doesn't cause a failure. A new test is taken twice too, and if the two captures differ, the report marks it unstable and it isn't published as the baseline. They upload the screenshots and a manifest of the device environment as `visual-capture-android` and `visual-capture-ios` artifacts.
10-
2. When Test All completes, Visual Compare RNTester (`.github/workflows/visual-compare-rntester.yml`) compares the screenshots with the baseline and reports a `Visual regression (android)` and `Visual regression (ios)` check and a `Visual regression approval` gate, plus a single PR comment with the expected, captured and diff images for anything that changed.
10+
2. When Test All completes, Visual Compare RNTester (`.github/workflows/visual-compare-rntester.yml`) compares the screenshots with the baseline and reports a `Visual regression (android)` and `Visual regression (ios)` check, which are neutral when there are differences, and a `Visual regression approval` gate, which fails until a maintainer approves them, plus a single PR comment with the expected, captured and diff images for anything that changed.
1111
3. On pushes to `main`, Visual Compare RNTester also publishes the captures as the baseline for that commit. There's no separate workflow for this: the same run publishes them once it has compared them.
1212

1313
Only `main` and pull requests into it are compared. Stable branches have no baselines, so Visual Compare RNTester skips them.

0 commit comments

Comments
 (0)