Repository navigation
[BUG] DictTransformer has non-deterministic serialization causing unexpected cache misses #6776
Description
Flyte & Flytekit version
v1.15.3
Describe the bug
Flytekit now uses msgpack for serialization. The DictTransformer specifically uses the MessagePackEncoder which uses msgpack.packb. A major problem with this is that msgpack does not enforce any ordering when serializing dictionaries.
Expected behavior
Flyte Propeller should not produce different cache keys for different orderings of a dictionary. If possible, Flyte Propeller should unmarshal the literal like it does for attribute resolution before computing the cache keys.
Additional context to reproduce
Using the repro described in #2924, if you reorder the dictionary, you get different cache keys:
>>> calculate_cache_key_multiple_times(dict(a=1, b=2, c=3))
task_name-cache_version-4301e138bbdea0ddd2bf116844d2e6f9 1000
Name: count, dtype: int64
>>> calculate_cache_key_multiple_times(dict(b=2, a=1, c=3))
task_name-cache_version-4de598fac8ead0a6caa30b3881caebb9 1000
Name: count, dtype: int64
It's more common that we see Flyte inputs reordering dictionaries when they are large. But we've found that the ordering is non-deterministic (as expected of Python dictionaries).
In combination with this, when Flyte Propeller calculates a hash for the cache key, it uses the raw input literals which are likely the msgpack-serialized artifacts instead of the actual value of the object, which ends up causing us cache misses non-deterministically for the same inputs.
We have a workaround to override the DictTransformer to use ormsgpack as shown below, but there are some limitations to ormsgpack since it's a bit slower and may not support non-string keys.
class DictTransformer(FlyteDictTransformer):
@staticmethod
async def dict_to_binary_literal(
ctx: FlyteContext, v: dict, python_type: Type[dict], allow_pickle: bool
) -> Literal:
"""
Converts a Python dictionary to a Flyte-specific ``Literal`` using _sorted_ MessagePack encoding.
Falls back to Flyte's default dictionary encoding if encoding fails.
"""
try:
# Handle dictionaries with non-string keys (e.g., Dict[int, Type])
msgpack_bytes = ormsgpack.packb(v, option=ormsgpack.OPT_SORT_KEYS)
return Literal(scalar=Scalar(binary=Binary(value=msgpack_bytes, tag=MESSAGEPACK)))
except TypeError:
log.error("Error converting dictionary to binary literal using ormsgpack")
return await FlyteDictTransformer.dict_to_binary_literal(
ctx, v, python_type, allow_pickle
)
Screenshots
No response
Are you sure this issue hasn't been raised already?
- Yes
Have you read the Code of Conduct?
- Yes
Activity
welcome commentedon Dec 2, 2025
Thank you for opening your first issue here! 🛠
runllm commentedon Dec 2, 2025
This is a known issue: Flyte's DictTransformer uses msgpack for serialization, but msgpack does not guarantee key order in dictionaries, leading to non-deterministic serialization and thus spurious cache misses for the same logical input with different key orders. This has been reported and discussed in detail in Flyte issue #2924 and is acknowledged as a limitation of both msgpack and Flyte's current cache key computation, which uses the raw serialized bytes rather than a normalized value.
A workaround is to use a serialization method that sorts dictionary keys (e.g., ormsgpack with OPT_SORT_KEYS), but this is not natively supported in Flyte and may have its own limitations. The Flyte team has discussed that a proper fix would require Flyte to unmarshal and normalize dictionaries before computing cache keys, but this is not yet implemented. For more, see the related discussion and workaround.
Would you like more detail on the technical background or possible workarounds?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @RunLLM to ask follow-up questions. See runllm.com for more!
machichima commentedon Dec 23, 2025
Hi,
Just to confirm, if you pass
calculate_cache_key_multiple_times(dict(a=1, b=2, c=3)), without reordering, it will always be the same and cache will not miss right?Could you provide the use case when you will change the ordering of dictionary, but still want to trigger cache hit? I think in most cases we will have the same dictionary rather than re-ordering them?
swarupsrini commentedon Jan 11, 2026
@machichima Hi, thanks for replying here. So we have some pretty complex logic in prior tasks that end up generating large config dictionaries by providing nested class outputs, etc. So we essentially don't have a lot of strict control over keeping everything sorted all the time. I realize that in later Python versions that dictionaries are insertion-ordered, but there doesn't seem to be a way to sort dictionaries. So it feels like extremely strange behavior to assume the input is completely different just because the order in which you inserted items into a dictionary varied slightly.
swarupsrini commentedon Jan 11, 2026
As per the use-case, it's pretty non-deterministic so I have a hard time figuring out exactly where the variation comes from, but at a high level it's when we generate configs and pass around some deeply nested objects. The more you nest, the more frequently this occurs, and it actually costs us a lot of time and money in wasted runs because we don't hit cache.
machichima commentedon Jan 12, 2026
Hi @swarupsrini,
Thank you for the reply! It make sense to me.
We actually use
mashumaroand makeFlyteFileandFlyteDirectorytheSerializableTypeso that we can serialize it. If we want to useormsgpackinstead, we also need change those parts.https://github.com/flyteorg/flytekit/blob/330a1e8cf8f220aad04a320637d0d90f799bf628/flytekit/types/file/file.py#L177C1-L187C1
https://github.com/flyteorg/flytekit/blob/330a1e8cf8f220aad04a320637d0d90f799bf628/flytekit/types/directory/types.py#L162-L170
I think there's two options here:
ormsgpackyou purposed. It will be fine forDictTransformeras we only support string keys, but we need to deal withFlyteFileandFlyteDirectoryIt will be great if you are willing to contribute. We can discuss further on how we can implement this
majiayu000 commentedon Mar 22, 2026
Hi, I've been looking into this and traced the root cause.
Proposed approach: Fix is in flyteorg/flytekit repo. Option 2 (sort utility) is safer and avoids new dependency. Implement a recursive dict key-sorting utility, call it in DictTransformer.dict_to_binary_literal() before msgpack.packb. Must handle nested dicts and lists containing dicts. Add tests verifying that different key orderings produce identical serialized bytes. Also need to consider FlyteFile/FlyteDirectory serialization paths if using ormsgpack approach instead.
I'll open a draft PR shortly. Happy to adjust based on your preference.