chore: deprecate record_batch macro in favor of upstream one by buraksenn · Pull Request #23295 · apache/datafusion

buraksenn · 2026-07-02T16:27:34Z

Which issue does this PR close?

Closes Remove record_batch! macro once upstream updates #13037.

Rationale for this change

This was a long standing thing in the backlog. record_batch! was added in arrow; however, it did not support vectors. Then I've created a PR in arrow a while ago apache/arrow-rs#9522. Once that was released this macro can be deprecated

What changes are included in this PR?

Mark record_batch! macro deprecated and use arrow one in relevant places.

Are these changes tested?

Yes

Are there any user-facing changes?

Users will see a deprecation notice if they are using this macro

github-actions · 2026-07-02T16:39:08Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 105.443s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.037s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [ 105.344s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.042s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 214.759s] datafusion
    Building datafusion-common v54.0.0 (current)
       Built [  31.554s] (current)
     Parsing datafusion-common v54.0.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-common v54.0.0 (baseline)
       Built [  31.716s] (baseline)
     Parsing datafusion-common v54.0.0 (baseline)
      Parsed [   0.065s] (baseline)
    Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.102s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure macro_marked_deprecated: macro #[deprecated] added ---

Description:
A `macro_rules!` declarative macro is now #[deprecated]. Downstream crates will get a compiler warning when using this macro.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/macro_marked_deprecated.ron

Failed in:
  macro record_batch in /home/runner/work/datafusion/datafusion/datafusion/common/src/test_util.rs:382

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  66.059s] datafusion-common
    Building datafusion-datasource v54.0.0 (current)
       Built [  35.681s] (current)
     Parsing datafusion-datasource v54.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-datasource v54.0.0 (baseline)
       Built [  35.630s] (baseline)
     Parsing datafusion-datasource v54.0.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion-datasource v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.398s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  72.937s] datafusion-datasource
    Building datafusion-datasource-parquet v54.0.0 (current)
       Built [  41.363s] (current)
     Parsing datafusion-datasource-parquet v54.0.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-datasource-parquet v54.0.0 (baseline)
       Built [  41.414s] (baseline)
     Parsing datafusion-datasource-parquet v54.0.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-datasource-parquet v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.232s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  84.517s] datafusion-datasource-parquet
    Building datafusion-ffi v54.0.0 (current)
       Built [  58.480s] (current)
     Parsing datafusion-ffi v54.0.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-ffi v54.0.0 (baseline)
       Built [  58.874s] (baseline)
     Parsing datafusion-ffi v54.0.0 (baseline)
      Parsed [   0.065s] (baseline)
    Checking datafusion-ffi v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.374s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 119.545s] datafusion-ffi
    Building datafusion-physical-expr-adapter v54.0.0 (current)
       Built [  29.991s] (current)
     Parsing datafusion-physical-expr-adapter v54.0.0 (current)
      Parsed [   0.010s] (current)
    Building datafusion-physical-expr-adapter v54.0.0 (baseline)
       Built [  29.898s] (baseline)
     Parsing datafusion-physical-expr-adapter v54.0.0 (baseline)
      Parsed [   0.010s] (baseline)
    Checking datafusion-physical-expr-adapter v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.122s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  61.088s] datafusion-physical-expr-adapter

…tream-one

Jefffrey · 2026-07-03T23:56:09Z

+arrow-schema = { workspace = true }
 datafusion = { workspace = true }
 datafusion-ffi = { workspace = true }
 ffi_module_interface = { path = "../ffi_module_interface" }

+[package.metadata.cargo-machete]
+ignored = ["arrow-schema"]


this seems pretty unfortunate, that its kind of a hidden dependency of the macro 🙁

i suppose we cant do much about it unless we fix it upstream (and even then would require arrow-array to reexport arrow-schema i think?)

Yes, I was kinda surprised this was required as well. If you think this is a blocker I can close and just leave record_batch! as it is I'm not sure what is best

yeah i think it's a bit awkward to need to import two dependencies in order to use this macro 🤔

Using use arrow::datatypes as arrow_schema; import fixed the issue with this there is no need for adding these imports and machete ignores. Can you check again @Jefffrey ?

this does solve the machete issue, though i still worry for any downstream users of the df version of the macro this would be a non-obvious fix 🤔

buraksenn · 2026-07-09T13:20:28Z

I've deleted the branch and closed it while cleaning up branches sorry

alamb

This PR looks good to me now. Am I missing anything here? I don't see any sort of cargo dependency shenanigans

buraksenn · 2026-07-16T13:38:54Z

This PR looks good to me now. Am I missing anything here? I don't see any sort of cargo dependency shenanigans

When I've opened the PR I've added arrow-schema to some crates and since its only macro usage I've needed to ignore from cargo-machete. Then I've changed it to current state as you've said there is no cargo dependency change at the moment

alamb · 2026-07-16T17:36:36Z

When I've opened the PR I've added arrow-schema to some crates and since its only macro usage I've needed to ignore from cargo-machete. Then I've changed it to current state as you've said there is no cargo dependency change at the moment

I am trying to figure out if there is a still a problem we need to fix --- like for example should we re-export the record_batches macro into the arrow crate?

alamb · 2026-07-16T17:36:54Z

Thank you for this PR

buraksenn · 2026-07-16T17:58:41Z

I think #23400 happened since every check passed but merge queue removed it

alamb · 2026-07-16T19:43:09Z

I'll just throw it back in the queue!

alamb · 2026-07-16T20:00:43Z

🤔

…23295) ## Which issue does this PR close? - Closes apache#13037. ## Rationale for this change This was a long standing thing in the backlog. record_batch! was added in arrow; however, it did not support vectors. Then I've created a PR in arrow a while ago apache/arrow-rs#9522. Once that was released this macro can be deprecated ## What changes are included in this PR? Mark record_batch! macro deprecated and use arrow one in relevant places. ## Are these changes tested? Yes ## Are there any user-facing changes? Users will see a deprecation notice if they are using this macro

buraksenn added 2 commits July 2, 2026 16:34

chore: deprecate record_batch macro and use arrow::array::record_batch

62ca8c9

bump version for deprecation notice

56c46ba

github-actions Bot added core Core DataFusion crate common Related to common crate datasource Changes to the datasource crate ffi Changes to the ffi crate labels Jul 2, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jul 2, 2026

buraksenn added 3 commits July 2, 2026 22:37

fix false positive cargo machete

b9f3b64

Merge branch 'main' into deprecate-record-batch-macro-in-favor-of-ups…

db7386f

…tream-one

Merge branch 'main' into deprecate-record-batch-macro-in-favor-of-ups…

80b0efe

…tream-one

ByteBaker mentioned this pull request Jul 3, 2026

Use upstream arrow-rs record_batch! and create_array! macros #18252

Closed

Jefffrey reviewed Jul 3, 2026

View reviewed changes

buraksenn closed this Jul 9, 2026

buraksenn deleted the deprecate-record-batch-macro-in-favor-of-upstream-one branch July 9, 2026 13:17

buraksenn restored the deprecate-record-batch-macro-in-favor-of-upstream-one branch July 9, 2026 13:20

buraksenn reopened this Jul 9, 2026

avoid direct arrow-schema dependencies for record_batch!

5786f7d

buraksenn force-pushed the deprecate-record-batch-macro-in-favor-of-upstream-one branch from 507510e to 5786f7d Compare July 14, 2026 11:56

alamb reviewed Jul 15, 2026

View reviewed changes

alamb approved these changes Jul 15, 2026

View reviewed changes

alamb added this pull request to the merge queue Jul 16, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 16, 2026

alamb added this pull request to the merge queue Jul 16, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 16, 2026

Jefffrey added this pull request to the merge queue Jul 16, 2026

Merged via the queue into apache:main with commit f151c10 Jul 16, 2026
40 checks passed

Uh oh!

Conversation

buraksenn commented Jul 2, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jefffrey Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buraksenn Jul 4, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Jul 4, 2026

Choose a reason for hiding this comment

Uh oh!

buraksenn Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

buraksenn commented Jul 9, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

buraksenn commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Uh oh!

buraksenn commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 2, 2026 •

edited

Loading