Skip to content

fix: align dictionary coercion across typed signatures#23549

Merged
alamb merged 3 commits into
apache:mainfrom
lyne7-sc:fix/dictionary-coercion-semantics
Jul 16, 2026
Merged

fix: align dictionary coercion across typed signatures#23549
alamb merged 3 commits into
apache:mainfrom
lyne7-sc:fix/dictionary-coercion-semantics

Conversation

@lyne7-sc

@lyne7-sc lyne7-sc commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

#22905 introduced explicit dictionary encoding preservation for coercible function signatures, but dictionary inputs were still handled differently across TypeSignatureClass variants:

Signature category Before After
Native(...) Materialized by default; preserved when explicitly requested Same default/opt-in contract
Typed non-Native (e.g. Integer, Numeric, Binary) Retained the physical dictionary type by default Materialized by default; preserved when explicitly requested
Any Passed through the original physical input type Unchanged

This PR makes the encoding preservation contract consistent across all typed signature classes: coercion operates on the dictionary value type, and the dictionary encoding is restored only when EncodingPreservation::dictionary() is enabled.

An audit of the affected built-ins found two functions, Spark hex and bitmap_count, that intentionally handle dictionary inputs; both now opt in explicitly. Other affected functions generally expect materialized value arrays, so the new default also avoids cases where signature matching accepted a dictionary but the function implementation rejected it at execution time. Functions that continue to materialize dictionary inputs do not gain dictionary-aware execution efficiency yet, but they can opt in later if they add support for encoded inputs.

What changes are included in this PR?

  • Align dictionary coercion across typed signature classes and preserve dictionary encoding when explicitly requested.
  • Explicitly enable dictionary preservation for Spark bitmap_count and the binary variant of Spark hex.
  • Document the behavior change and migration guidance in the DataFusion 55.0.0 upgrade guide.

Are these changes tested?

  • Unit tests cover materialization and preservation for Native, non-Native, and Any inputs.
  • SLTs cover to_hex materialization and verify that bitmap_count preserves its dictionary input without an additional cast to Binary.
  • Existing Spark hex dictionary tests cover its opt-in preservation behavior.

Are there any user-facing changes?

This is a behavioral API change for UDFs using typed non-Native classes such as Integer or Binary. UDFs relying on implicit dictionary preservation must now enable EncodingPreservation::dictionary() explicitly.

TypeSignatureClass::Any is unaffected. The upgrade guide has been updated, and this PR should carry the api change label.

@github-actions github-actions Bot added documentation Improvements or additions to documentation logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) spark labels Jul 14, 2026
@Jefffrey Jefffrey added the api change Changes the API exposed to users of the crate label Jul 15, 2026
@alamb
alamb added this pull request to the merge queue Jul 16, 2026
Merged via the queue into apache:main with commit 5063883 Jul 16, 2026
42 checks passed
@lyne7-sc
lyne7-sc deleted the fix/dictionary-coercion-semantics branch July 17, 2026 01:54
Omega359 pushed a commit to Omega359/arrow-datafusion that referenced this pull request Jul 18, 2026
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Follow-up to apache#22905.
- Part of apache#19458

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

apache#22905 introduced explicit dictionary encoding preservation for
coercible function signatures, but dictionary inputs were still handled
differently across `TypeSignatureClass` variants:

| Signature category | Before | After |
| --- | --- | --- |
| `Native(...)` | Materialized by default; preserved when explicitly
requested | Same default/opt-in contract |
| Typed non-Native (e.g. `Integer`, `Numeric`, `Binary`) | Retained the
physical dictionary type by default | Materialized by default; preserved
when explicitly requested |
| `Any` | Passed through the original physical input type | Unchanged |

This PR makes the encoding preservation contract consistent across all
typed signature classes: coercion operates on the dictionary value type,
and the dictionary encoding is restored only when
`EncodingPreservation::dictionary()` is enabled.

An audit of the affected built-ins found two functions, Spark hex and
bitmap_count, that intentionally handle dictionary inputs; both now opt
in explicitly. Other affected functions generally expect materialized
value arrays, so the new default also avoids cases where signature
matching accepted a dictionary but the function implementation rejected
it at execution time. Functions that continue to materialize dictionary
inputs do not gain dictionary-aware execution efficiency yet, but they
can opt in later if they add support for encoded inputs.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

- Align dictionary coercion across typed signature classes and preserve
dictionary encoding when explicitly requested.
- Explicitly enable dictionary preservation for Spark `bitmap_count` and
the binary variant of Spark `hex`.
- Document the behavior change and migration guidance in the DataFusion
55.0.0 upgrade guide.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

- Unit tests cover materialization and preservation for Native,
non-Native, and `Any` inputs.
- SLTs cover `to_hex` materialization and verify that `bitmap_count`
preserves its dictionary input without an additional cast to `Binary`.
- Existing Spark `hex` dictionary tests cover its opt-in preservation
behavior.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

This is a behavioral API change for UDFs using typed non-Native classes
such as `Integer` or `Binary`. UDFs relying on implicit dictionary
preservation must now enable `EncodingPreservation::dictionary()`
explicitly.

`TypeSignatureClass::Any` is unaffected. The upgrade guide has been
updated, and this PR should carry the `api change` label.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants