Allow tests to differentiate between iterations when ran repeatedly

[Stannieman]

Motivation

When running a test repeatedly, it is currently impossible for code inside the test to differentiate between different iterations within the same test run.

A concrete example where this poses an issue is with the SnapshotTesting library. It generates a unique name for each snapshot used in a test. E.g. when a test checks 3 snapshots it generates 3 unique names for those. When a test is ran multiple times it needs to "reset" these names between test iterations for this to work. However, it currently has no way to know the difference between 3 snapshots within a single test or 1 snapshot in a test that's ran 3 times repeated.

To solve this it needs to be able to detect that a test it has seen before is actually a different iteration. The issue is being tracked on there side here: pointfreeco/swift-snapshot-testing#963.

Proposed solution

A solution could be to add an iteration: UInt property to Test that gets incremented for each iteration. Something like this is actually already printed to the console in Xcode. SnapshotTesting could use that to "reset" the unique snapshot names when the iteration changes for the same test.

I feel like this somewhat relates to #671, where uniquely identifying parametrized cases are discussed. They are not entirely the same though.
Snapshot names should not be reset for different parametrized cases, only for different iterations when running repeatedly. Similarly, you probably want to be able to still identify a parametrized case across different iterations.
So while both problems and solutions are very similar, they do have different use cases and probably ask for separate fields.

Alternatives considered

No response

Additional information

No response

[stmontgomery]

Can you clarify what mechanism you are using to run a test repeatedly? Are you using the Xcode feature "Run Test Repeatedly…", for example, or do you have a parameterized test function, or is it using some other technique?

[Stannieman]

It is indeed through the Xcode UI.

Checking "Relaunch Tests for Each Repetition" does not change the behavior either. (Is this a bug? I'd expect global variables to be reset when this is checked, but they are not.)

[grynspan]

The testing library reports which iteration is running (and it can be computed trivially in the event stream handler anyway), so I suspect this needs to be tracked on Apple's side.

[stmontgomery]

The request here is to introduce a mechanism by which an actively-running test could inspect which iteration it's running. I do think that would require internal modeling and likely API additions in the testing library itself. One way this could be conveyed is via a property on Test.Case; then, querying Test.Case.current would allow a test function to access it. We'd need to think through whether Test.Case or Test is the right place, though; it's possible suites may want access to the iteration count, too.

Then separately, Xcode would able to adopt that information once available (perhaps replacing the mechanism it currently uses), and that adoption task should be tracked separately.

[grynspan]

Fair enough.

[Stannieman]

The issue in SnapshotTesting was fixed in a different way. This now becomes a bit less relevant, but if it's something that will still be useful in the grand scheme of things it can remain open of course.

[stephencelis]

For context, we simply require a new test scoping trait to reset this state at a task-local level.

It still would be nice for libraries to be able to detect this kind of thing and provide a nice default behavior (or warning that a necessary trait isn't applied).