[zastrowm]

Human Overview

Caching is region dependent and testing indicated and this test ended up flaky; to reduce flakiness pin the region for this test only + re-enable tests. See the description for the trials ran to determine flakiness.

Description

The litellm prompt-cache tests are unreliable in some regions, and test_cache_read_tokens_multi_turn is currently skipped because of it. This pins the fixtures to a region where the cache behaves, and re-enables the test with reruns instead of a skip.

The test writes a cache entry on turn 1 and asserts a read of it on turn 2. Bedrock serves that read back consistently in us-east-1 and us-east-2, but only about 88% of the time in us-west-2 — measured over 130 trials there against 162 in us-east-1 and 61 in us-east-2, with all three sampled in the same two-minute window. So the region, not the code path, decides whether the assertion holds. Raw Converse and the LiteLLMModel path behave identically within a region.

conftest.py only setdefault()s AWS_REGION, so a developer with their own AWS_REGION exported keeps it — which is how this surfaced. Pinning in the fixtures makes the tests behave the same for everyone.

Read-back is best effort even in a good region, so the multi-turn cache test reruns rather than staying skipped. It uses pytest.mark.flaky and not the existing retry_on_flaky helper: that helper wraps the test in a sync function, so pytest-asyncio stops recognising an async def test and collection fails with "async def functions are not natively supported". Adding async support to retry_on_flaky would be a reasonable follow-up, and would let the retry be scoped to AssertionError instead of any exception.

us-east-2 measured equally clean, so it's noted in the comment as the alternative if us-east-1 ever has trouble.

Related Issues

Refs #5315, which tracks the underlying cache-read behaviour. That is not resolved here — the test is made usable, not the behaviour explained.

Documentation PR

None needed.

Type of Change

Bug fix (test reliability)

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

Assessment: Approve

Small, well-scoped test-reliability change: the region pin via client_args is the right lever (conftest only setdefault()s AWS_REGION), and the inline comment plus the PR description capture the empirical rationale and the retry_on_flaky-vs-flaky reasoning clearly. pytest-rerunfailures is already a bounded dev dependency, so the @pytest.mark.flaky marker is backed and valid, and retry_on_flaky is still used elsewhere so no import is orphaned.

Notes

• Correctness: Marker is registered (pytest-rerunfailures >=15.0,<16.0); async collection is fine since @pytest.mark.asyncio is retained and reruns apply per parametrized item.
• Suggestion (non-blocking): reruns=4 retries on any exception, not just AssertionError, so a transient non-cache failure (throttling, creds) would be silently retried. Low risk since persistent failures still surface after reruns, and you already flagged scoped-async retry_on_flaky as a follow-up — just noting it so the scoping intent isn't lost.

Nicely documented investigation — the measured read-back rates make the region pin easy to trust.