Skip to content

Commit 92806ee

Browse files
committed
test(inference): tolerate model non-determinism in bare-string embed test
test_embed_bare_string_auto_wrapped_{rest,async} called embed() twice with the same input (once as a bare string, once wrapped in a list) and asserted bit-exact equality of the returned values. The multilingual-e5-large model isn't bit-deterministic across replicas — two consecutive calls regularly drift in the 3rd–4th decimal. That flake just took down release-readiness on run 25746418051. The SDK contract being verified here is that the bare-string path doesn't transform the input differently from the single-element-list path, not that the model itself is deterministic. Switch the value comparison to pytest.approx(abs=1e-3), matching the existing convention from test_query_namespaces_tie_breaking_rest. Length check is retained so a shape-changing regression still fails.
1 parent 17390dc commit 92806ee

2 files changed

Lines changed: 14 additions & 4 deletions

File tree

‎tests/integration/test_inference.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -234,8 +234,13 @@ def test_embed_bare_string_auto_wrapped_rest(client: Pinecone) -> None:
234234
parameters={"input_type": "passage", "truncate": "END"},
235235
)
236236
assert len(result_as_list) == 1
237-
# Same string → same embedding values (deterministic model)
238-
assert result.data[0].values == result_as_list.data[0].values
237+
# Same string → same embedding shape and (approximately) same values.
238+
# Element-wise tolerance absorbs server-side non-determinism (e.g. mixed
239+
# precision across replicas); the SDK contract being verified here is
240+
# that the bare-string path doesn't transform the input differently from
241+
# the single-element-list path, not that the model is bit-deterministic.
242+
assert len(result.data[0].values) == len(result_as_list.data[0].values)
243+
assert result.data[0].values == pytest.approx(result_as_list.data[0].values, abs=1e-3)
239244

240245

241246
# ---------------------------------------------------------------------------

‎tests/integration/test_inference_async.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -190,8 +190,13 @@ async def test_embed_bare_string_auto_wrapped_async(async_client: AsyncPinecone)
190190
parameters={"input_type": "passage", "truncate": "END"},
191191
)
192192
assert len(result_as_list) == 1
193-
# Same string → same embedding values (deterministic model)
194-
assert result.data[0].values == result_as_list.data[0].values
193+
# Same string → same embedding shape and (approximately) same values.
194+
# Element-wise tolerance absorbs server-side non-determinism (e.g. mixed
195+
# precision across replicas); the SDK contract being verified here is
196+
# that the bare-string path doesn't transform the input differently from
197+
# the single-element-list path, not that the model is bit-deterministic.
198+
assert len(result.data[0].values) == len(result_as_list.data[0].values)
199+
assert result.data[0].values == pytest.approx(result_as_list.data[0].values, abs=1e-3)
195200

196201

197202
# ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)