Comparing screenshots on Linux and MacOS

[pettero]

Our developers have are running Macs and the CI is running on Linux. When generating screenshots the images are different.

The difference however is not that big...

For instance we have an input field that looks like this:

Above the blue line in the bottom the emulator is adding a blueish line and that one differs between the mac and linux:

How ever the "distance" between the colors is quite small (using Pythagoras)

sqrt((163-162)^2+(185-184)^2+(203-203)^2)=sqrt(2)

Would it be a good idea to complement the tolerance with a distance configuration to allow colors that are very similar to be treated as the same and keep the tolerance set to 0?

Feels like the change should be quite easy to make -

Shot/core/src/main/scala/com/karumi/shot/screenshots/ScreenshotsComparator.scala

Line 54 in 64f5162

val differentPixels =

I can help out if it's a good idea.

[tdrhq]

Btw, it turns out that it's faster and easier if the screenshots are always generated on CI, that way you always get identical screenshots and you don't have to do any image processing in your CI job which is quite expensive.

You could either write a script generate the generate the screenshots on CI automatically which is commonly done, or you could consider a tool like https://screenshotbot.io (which also solves the issue of image storage). We have a gradle plugin that integrates with Shot.

[pettero]

Currently we doing as you suggested but the CI takes a long time to run all the tests and I would like to be able to run all the tests locally to see that they are passing before I go to the next task at hand.

Looks like Paparazzi implemented a similar approach in their lib recently:

Relax image comparisons with OffByTwo differ to work around cross-platform rendering issues

https://cashapp.github.io/paparazzi/changelog/#135-2024-11-06

[tdrhq]

@pettero Screenshotbot does support running tests locally, even though the screenshots are not stored in Git/Git LFS: https://screenshotbot.io/documentation/developer-tools/local-run . So you can effectively get the best of both worlds.

[pettero]

We have a similar setup using firebase test lab.

A challenge for us is that it takes time to run the tests remotely and we also loose the possibility to debug the tests.

I think we are leaning towards trying to verify colors using Euclidean distance for now.

https://en.wikipedia.org/wiki/Color_difference#sRGB