[backport v2.12] Support HA & slow proliferation of cached value - Scenario 2 + 3

[rancher-ui-project-bot]

This is a backport issue for #15414, automatically created via GitHub Actions workflow initiated by @richard-cox

Original issue body:

Scenario 2 - Stale cache in replicate that handles http request

1. ui watches a resource
2. ui receives a resource.changes socket message provided by up-to-date replicate A
3. ui makes http request.
   • it's handled by stale replica B
   • response contains stale resources
4. ui shows stale resources, not the ones that caused the resource.changes event

Solution: Following rancher/rancher#51254 the revision will be included in resource.changes events. The UI will then include this in the resulting http request. If this is received by a stale replicate it will reject the request. The UI will then try again (but backoff if there are failures)

Scenario 3 - Old revision returned from replicate with stale cache, request also handled by replica with stale cache

This is a gap in scenario 2

1. ui makes a http request & watches the resource for changes
2. ui receives a resource.changes socket message provided by lagging replicate A
   • for example revision 5
3. ui makes http request with the lagging revision (5)
   • it's handled by up-to-date replica B
   • response contains latest revision (10)
4. ui receives a resource.changes socket message provided by lagging replicate A
   • for example revision 6
5. ui makes http request with the lagging revision (5)
   • it's handled by lagging replica A
   • response contains stale data from 6 (10 has not made it to replica 'A')

Solution: When the UI receives a resource.changes message it will check that it's revision is higher than the revision the UI has locally (assigned from the previous http request)

[richard-cox]

Blocked on 2.12 backport of #15414

[gaktive]

Moved to 2.12.5 (newly created). If there's additional backports required for 2.13.x, move that one up a version too.