[FLAKY] ScimIntegrationTest

[exoego]

ScimIntegrationTest fails on PRs where only RDS and DynamoDB are touched.

https://github.com/floci-io/floci/actions/runs/34540474254/job/103081759224?pr=3382#step:8:14946
https://github.com/floci-io/floci/actions/runs/34544524043/job/103094160658?pr=3360

[ERROR] Failures: 
[ERROR]   ScimIntegrationTest.createGroupUsesAwsScimShapeAndPersistsToIdentityStore:46 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.createGroupValidatesRequiredDisplayNameAndMemberLimit:309 1 expectation failed.
Expected status code <400> but was <401>.

[ERROR]   ScimIntegrationTest.createUserUsesAwsScimShapeAndPersistsToIdentityStore:82 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.createUserValidatesAwsUnsupportedAttributesAndRequiredFields:120 1 expectation failed.
Expected status code <400> but was <401>.

[ERROR]   ScimIntegrationTest.deleteGroupUsesAwsScimStatusAndRemovesIdentityStoreResource:258 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.deleteUserUsesAwsScimStatusAndRemovesIdentityStoreResource:230 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.getGroupUsesAwsScimRepresentation:194 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.getSchemaReturnsAwsScimSchema:712 1 expectation failed.
Expected status code <200> but was <401>.

[ERROR]   ScimIntegrationTest.getUserUsesAwsScimRepresentation:158 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.listGroupsSupportsAwsFiltersAndCursorPagination:830 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.listResourceTypesReturnsAwsScimCatalog:672 1 expectation failed.
Expected status code <200> but was <401>.

[ERROR]   ScimIntegrationTest.listSchemasReturnsAwsScimCatalog:695 1 expectation failed.
Expected status code <200> but was <401>.

[ERROR]   ScimIntegrationTest.listUsersSupportsAwsFiltersAndCursorPagination:738 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.patchGroupRejectsUnsupportedOrBulkMembershipChanges:617 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.patchGroupUpdatesAttributesAndMemberships:544 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.patchUserRejectsUnsupportedAndRepeatedProtectedChanges:496 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.patchUserUpdatesSupportedAttributesAndReturnsUser:463 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.putUserOverwritesExistingUserAndPreservesResourceIdentity:345 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.putUserRejectsInvalidReplacementAndConflictingUserName:407 1 expectation failed.
Expected status code <201> but was <401>.

[ERROR]   ScimIntegrationTest.serviceProviderConfigMatchesAwsScimCapabilities:650 1 expectation failed.
Expected status code <200> but was <401>.

[okinaka]

Fixed on main by #3387 (b87ee5bd4), which #3386 tracked as the same failure.

It is not flaky in the usual sense, it is deterministic per shard layout. EmulatorInfoController.performReset() clears every Resettable and then calls storageFactory.clearAll(), which wipes the bootstrap identity store that SsoAdminService.clear() had just re-seeded. Any class that hits /_floci/state/reset earlier in the same fork, such as EmulatorInfoControllerIntegrationTest, leaves hasIdentityStore("d-9067f2a3c1") false for the rest of that JVM, so every later SCIM call 401s. That is why it follows the shard layout rather than the diff: partition.sh puts the two classes in one fork only sometimes, which is also why PRs touching only RDS or DynamoDB were affected.

Reproducible in two classes before the fix: ./mvnw test -Dtest='EmulatorInfoControllerIntegrationTest,ScimIntegrationTest' gave 20 of 22 failures, and 0 after.

[exoego]

I think #3387 fixed the symptom for Identity Center.
The cause is still in performReset() in which every Resettable.clear() runs before storageFactory.clearAll().
So anything a service re-seeds in clear() is wiped right after.

#3388 fixes that order, so future services with default state do not hit the same bug, and adds a regression test on the reset endpoint.
Both fix can coexist.

[okinaka]

Update: the symptom is fixed, the cause is not. A branch rebased onto main with #3387 still fails, in a different class.

https://github.com/floci-io/floci/actions/runs/34549984251/job/103110649274 is shard 3 on such a branch. ScimIntegrationTest passes there, but SsoAdminIntegrationTest fails 4 of 41, all of them 200 expected and 400 received:

SsoAdminIntegrationTest.updateInstancePersistsNameAndEncryptionConfiguration:97
SsoAdminIntegrationTest.createApplicationReturnsAwsIdentifiersAndIsIdempotent:296
SsoAdminIntegrationTest.listApplicationAccessScopesReturnsAwsScopeDetailsWithPagination:607
SsoAdminIntegrationTest.deleteApplicationGrantValidatesGrantTypeAndMissingGrant:918

Same shape as before: EmulatorInfoControllerIntegrationTest posts the reset earlier in that fork. Reproducible in two classes on b87ee5bd4:

./mvnw test -Dtest='EmulatorInfoControllerIntegrationTest,SsoAdminIntegrationTest'
Tests run: 58, Failures: 4

#3387 self-heals inside hasIdentityStore, which is the path SCIM authorises on. UpdateInstance, CreateApplication and the application APIs reach the instance by other paths that have no such recovery, so they still see an empty store after a reset. Adding ensureBootstrapInstance one call site at a time will keep leaving gaps like this.

#3388 changes the order in performReset() instead, so whatever clear() re-seeds survives and every path sees the instance. Cherry-picked onto b87ee5bd4 locally, EmulatorInfoControllerIntegrationTest, SsoAdminIntegrationTest and ScimIntegrationTest run together at 81 tests with no failures. Not verified in CI, but it does look like the direction that closes this class of failure.