[Tool] Fix failing UI tests on GitHub Action runners

[pylapp]

Description

Today we use swift-snapshot-testing to make assertions on UI rendering of views.
We have Fastlane running these tests and it work as expected.
We have our internal GitLab CI runner, running the same Fastlane lanes, and it works like expected.
However in GitHub Actions some tests do not pass, even if they pass with the other environment, without further details.

It seems the provided environment with GitHub runners is unstable making some test fail (even if they pass on our GitLab CI and local computers), e.g. testAllTypographies*, testAllSizes*, testAllElevations* don't pass in GitHub Actions level but pass in our internal levels.

We do not know if there are hardware resources issues, shadow tomeouts or other configurations trobules witht he Mac runners or the in-use simulators.

Thus we will have checks failed for UI tests and must disable requirement son them to preven pull request to be merged.

This is a bit sad and we should find a workaround or a fix for this issue.
This is not mandatory as we plan to move the demo app, with the UI tests, in out internal GitLab software forge in the future.

But could be nice to have a fix befoire.

Details

Below are the outputs of our last job:

 [16:40:24]: ▸ Test Suite OUDSTokensElevationUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:40:39]: ▸ ::error ::    testAllElevationsInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (15.131 seconds)
[16:40:40]: ▸ ::error ::    testAllElevationsInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.325 seconds)
[16:40:54]: ▸ ::error ::    testAllElevationsOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.344 seconds)
[16:41:28]: ▸ ::error ::    testAllElevationsOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (13.717 seconds)
[16:41:28]: ▸ Test Suite OUDSTokensOpacityUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:41:33]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.511 seconds)
[16:41:33]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.119 seconds)
[16:41:33]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.139 seconds)
[16:41:34]: ▸     ✔ [OUDSTokensOpacityUITests] testAllOpacitiesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.135 seconds)
[16:41:34]: ▸ Test Suite OUDSTokensBorderUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:41:34]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.386 seconds)
[16:41:35]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.291 seconds)
[16:41:38]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.370 seconds)
[16:41:38]: ▸     ✔ [OUDSTokensBorderUITests] testAllBordersOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (0.253 seconds)
[16:41:39]: ▸ Test Suite OUDSTokensSpaceUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:41:43]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpaceInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.751 seconds)
[16:41:43]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpaceInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.348 seconds)
[16:41:44]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpacesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.428 seconds)
[16:42:42]: ▸     ✔ [OUDSTokensSpaceUITests] testAllSpacesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.564 seconds)
[16:42:43]: ▸ Test Suite OUDSTokensTypographyUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:44:09]: ▸ ::error ::    testAllTypographiesInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (104.090 seconds)
[16:44:18]: ▸ ::error ::    testAllTypographiesInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (42.247 seconds)
[16:44:23]: ▸ ::error ::    testAllTypographiesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (8.834 seconds)
[16:44:41]: ▸ ::error ::    testAllTypographiesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.303 seconds)
[16:44:42]: ▸ Test Suite OUDSTokensColorUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:44:43]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.323 seconds)
[16:44:43]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.951 seconds)
[16:44:44]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.311 seconds)
[16:44:44]: ▸     ✔ [OUDSTokensColorUITests] testAllColorsOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (4.065 seconds)
[16:44:44]: ▸ Test Suite OUDSTokensSizeUITests started on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)'
[16:44:47]: ▸ ::error ::    testAllSizeInverseThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (2.872 seconds)
[16:44:50]: ▸ ::error ::    testAllSizesInverseThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (3.435 seconds)
[16:44:52]: ▸ ::error ::    testAllSizesOrangeThemeDark on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (2.221 seconds)
[16:44:54]: ▸ ::error ::    testAllSizesOrangeThemeLight on 'Clone 1 of iPhone 16 Pro - OUDS Showcase (11989)' (1.749 seconds)

In attachments the logs.

logs_31306604165.zip
job-logs.txt

[pylapp]

Idea of @ludovic35: check the l10n of the views in the used simulators. Maybe the simulator is not in the same locale as our, and the rendring of the view with the displayed token text are not the same.

[pylapp]

Some details after investigations:

• runner (i.e. GitHub Action runners), internal runner (GitLab) and local computer use all Xcode 16
• they use all iPhone 16 Pro simulator
• the iOS version of the simulator according to the GitHub documentation is 18.0
• it cannot be an issue with orientation of the simulator (portrait or landscape) because, in that case, all tests must fail and not only a part
• it cannot be an issue with a11y settings like text sizes because, in that case, all tests must fail and not only a part
• it cannot be an issue related to color scheme or theme because we override these values for the tests
• it might be a timeout issue but the threshold value has not been found
• maybe an l10n issue with string formatting (like suggested here [Tool] Fix failing UI tests on GitHub Action runners #305 (comment)), but tests pass in our GitLab CI/CD chain and locally
• maybe an issue with the SoC architcture of the runners impling differences of behaviour with the lib?

Note the branch (ci/305-fix-UI-tests-failure-on-GitHub-Actions) has an interesting commit (7e8f7c7) freezing the Xcode version to exactly 16 with a new Xcode manager. Can be cherry-picked.

[pylapp]

So, an issue has been created in swift-snapshot-testing GitHub project: pointfreeco/swift-snapshot-testing#932

The only difference I spot today is the architecture in use: GitHub Actions macos-15 runner is based on Apple Silicon, i.e. ARM-powered SoC and not our computers no internal runners with Intel-powered SoC.

It could mean if we upgrade our ahrdware tests won't pass anymore.

I recommend the following:

• not upgrade our hardware yet
• follow the evolution of the issue Discrepencies between tests results depending to Intel and ARM64 architectures (issue with Apple Silicon SoC on GitHub Action runner) pointfreeco/swift-snapshot-testing#932 and help the project team
• remove the UI tests from GitHub Action workflows because, for this CI/CD environment, we can't rely on them anymore, or at least, make their success status not mandatory to merge

cc @ludovic35 @B3nz01d

[pylapp]

On hold until we got fix for this or any unblocking details from the lib team.

[pylapp]

FYI now we accept to have UI tests failing on GitHub Actions workflow.
Thus the tests will still be run, and won't prevent us to merge any PR of they fail, assuming they fail because of the issue mentioned above.

Of course dev team must run all the tests locally.

Commit: b0c8bdd

[pylapp]

Some solutions to dig:

• Register images through run of tests with Fastlane: Snapshot taken on CI has slight color differences compared to local pointfreeco/swift-snapshot-testing#313 (comment)
• Change precision even if it implies to not be pixel perfect: https://gist.github.com/Tovkal/0eeb8169b4b1f53a5c0896e5645b7656
• Custom assertSnpashot implementation depending to SoC architecture: Snapshots on Apple Silicon devices pointfreeco/swift-snapshot-testing#424 (comment)
• Change shadow opacity: Snapshots on Apple Silicon devices pointfreeco/swift-snapshot-testing#424 (comment)
• Force snapshots to sRGB: Snapshots on Apple Silicon devices pointfreeco/swift-snapshot-testing#424 (comment)
• Swizzling to get rid of shadows: Snapshots on Apple Silicon devices pointfreeco/swift-snapshot-testing#424 (comment)
• Change subpixel threshold: https://github.com/finn-no/FinniversKit/blob/b16baf22b75d3ae41faf37439d5e3bac105039ac/Demo/SnapshotTests/XCTestCastExtensions.swift, https://github.com/pointfreeco/swift-snapshot-testing/pull/571/files#diff-635d5262e3de3758ccee19b27b90f23c03f33dc7bbbe350736c966a3902aa7b1

[pylapp]

Tested first solution with references images generated through tests started with Fastlane.
Not Git diff, the assets are the same.
This solution does not work.