Repository navigation
[BUG] DictTransformer has non-deterministic serialization causing unexpected cache misses #6775
Description
Flyte & Flytekit version
v1.15.3
Describe the bug
Flytekit now uses msgpack for serialization. The DictTransformer specifically uses the MessagePackEncoder which uses msgpack.packb. A major problem with this is that msgpack does not enforce any ordering when serializing dictionaries.
Expected behavior
Flyte Propeller should not produce different cache keys for different orderings of a dictionary. If possible, Flyte Propeller should unmarshal the literal like it does for attribute resolution before computing the cache keys.
Additional context to reproduce
Using the repro described in #2924, if you reorder the dictionary, you get different cache keys:
>>> calculate_cache_key_multiple_times(dict(a=1, b=2, c=3))
task_name-cache_version-4301e138bbdea0ddd2bf116844d2e6f9 1000
Name: count, dtype: int64
>>> calculate_cache_key_multiple_times(dict(b=2, a=1, c=3))
task_name-cache_version-4de598fac8ead0a6caa30b3881caebb9 1000
Name: count, dtype: int64
It's more common that we see Flyte inputs reordering dictionaries when they are large. But we've found that the ordering is non-deterministic (as expected of Python dictionaries).
In combination with this, when Flyte Propeller calculates a hash for the cache key, it uses the raw input literals which are likely the msgpack-serialized artifacts instead of the actual value of the object, which ends up causing us cache misses non-deterministically for the same inputs.
We have a workaround to override the DictTransformer to use ormsgpack as shown below, but there are some limitations to ormsgpack since it's a bit slower and may not support non-string keys.
class DictTransformer(FlyteDictTransformer):
@staticmethod
async def dict_to_binary_literal(
ctx: FlyteContext, v: dict, python_type: Type[dict], allow_pickle: bool
) -> Literal:
"""
Converts a Python dictionary to a Flyte-specific ``Literal`` using _sorted_ MessagePack encoding.
Falls back to Flyte's default dictionary encoding if encoding fails.
"""
try:
# Handle dictionaries with non-string keys (e.g., Dict[int, Type])
msgpack_bytes = ormsgpack.packb(v, option=ormsgpack.OPT_SORT_KEYS)
return Literal(scalar=Scalar(binary=Binary(value=msgpack_bytes, tag=MESSAGEPACK)))
except TypeError:
log.error("Error converting dictionary to binary literal using ormsgpack")
return await FlyteDictTransformer.dict_to_binary_literal(
ctx, v, python_type, allow_pickle
)
Screenshots
No response
Are you sure this issue hasn't been raised already?
- Yes
Have you read the Code of Conduct?
- Yes
Activity
runllm commentedon Dec 2, 2025
This is a known issue: Flyte's DictTransformer uses msgpack for serialization, but msgpack does not guarantee key order in dictionaries, leading to non-deterministic serialization and thus non-deterministic cache keys. This causes unexpected cache misses when the same dictionary is passed with different key orders, as confirmed by the reproducible example in issue #2924. The Flyte backend currently hashes the raw msgpack-serialized bytes, so different key orders result in different cache keys, even if the logical content is the same.
A workaround is to override the DictTransformer to use a serialization method that sorts keys (e.g., ormsgpack with OPT_SORT_KEYS), but this is not the default and may have limitations. There is no built-in fix in Flyte as of v1.15.3; the problem is acknowledged in the community and tracked in multiple issues, but a deterministic, built-in solution is not yet implemented. See issue #2924 and msgpack issue #215 for more details.
Would you like more detail on possible workarounds or the technical background?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @RunLLM to ask follow-up questions. See runllm.com for more!