Discrepencies between tests results depending to Intel and ARM64 architectures (issue with Apple Silicon SoC on GitHub Action runner)

[pylapp]

Context

We use the swift-snapshot-testing library to make UI tests on our components.

All the tests pass locally in our MacBook Pro computers, based on Intel architecture (e.g. Intel Core i9 and MacBook Pro 2019 16"). Our CI/CD internal chain, using also such computers with the same architecture, keeps all the tests green.

However, using the GitHub Action runners, some tests fail. Always the same. We ensure the macOS version was the same between each computer (i.e. macOS 15), and also the Xcode version (16.0), the simulator in use (iPhone 16 Pro), and its OS verison (iOS 18.0). The code base is the same, the tests base is the same, the only difference remains in the architecture of the computer.

Indeed, GitHub Actions exposes a macOS 15 runner (named macos-15) based on ARM architecture according to their documentation with M1 SoC and only on this environment some tests fail. It seems GitHub exposes, for free plans, only arm64-absed environments and not Intel-based anymore.
The runners based on Intel architecture are not available for us (i.e. "large runners"). It seems they did not plan to expose Intel-based macOS 15 now.

In few words, we are pretty sure some UI tests do not work anymore if the architecture behind is arm64, i.e. Apple SIlicon, and not Intel 😱

Expected behavior

Having our tests passing for whatever the architecture is, i.e. for both Intel-based and arm64-based architectures.

Environment

• Xcode 16
• Swift 6
• iPhone 16 Pro simulator with iOS 18.0
• swift-snapshot-testing version 1.17.6

Additional context

• Our iOS project is open source here: https://github.com/Orange-OpenSource/ouds-ios
• We created an issue about that topic: [Tool] Fix failing UI tests on GitHub Action runners Orange-OpenSource/ouds-ios#305
• You can get the logs of the test here
• We use Fastlane to start the tests (lane test_ui)
• All the tests are in a dedicated Xcode scheme
• We need this tool to prevent any regression (as explained here)

It seems some users got the same troubles:

• Intermittent Snapshot Test Failures Due to Minor Rendering Differences #926
• Snapshot tests crashes on CircleCI using Xcode 14.2 #707
• Snapshots on Apple Silicon devices #424
• Snapshot taken on CI has slight color differences compared to local #313

Today we enable a condition accepting the tests fail in GitHub Action but that's not the expect workaround 😅

[pylapp]

@stephencelis @mbrandonw I hope these details help :)
BTW your library is awesome 🤩