Migrate datasource tests to insta #15258

shruti2522 · 2025-03-16T07:39:18Z

Which issue does this PR close?

related to Migrate datasource tests to insta #15246 .

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb · 2025-03-16T10:59:36Z

Thanks @shruti2522 !

It looks like there are some CI failures on this one so marking as draft for now

blaginin · 2025-03-18T16:08:36Z

datafusion/core/Cargo.toml

@@ -126,6 +126,7 @@ datafusion-physical-plan = { workspace = true }
 datafusion-sql = { workspace = true }
 flate2 = { version = "1.1.0", optional = true }
 futures = { workspace = true }
+insta = "1.42.2"


you should be use to use the workspace version

shruti2522 · 2025-03-22T06:54:19Z

I have migrated all the tests for datasource to insta, except these two, I’m still working on these and will open a separate PR for it.

physical_plan::json::tests::test_json_with_repartitioning::case_1_uncompressed : for this one, on using batches_to_string, only one case is failing and on using batches_to_sort_string all of them are failing.

---- datasource::physical_plan::json::tests::test_json_with_repartitioning::case_1_uncompressed stdout ----
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot: json_with_repartitioning
Source: datafusion/core/src/datasource/physical_plan/json.rs:582
────────────────────────────────────────────────────────────────────────────────
Expression: batches_to_string(&res)
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
    0     0 │ +-----+------------------+---------------+------+
    1     1 │ | a   | b                | c             | d    |
    2     2 │ +-----+------------------+---------------+------+
          3 │+| 2   | [2.0, , -6.1]    | [false, ]     | text |
          4 │+|     |                  |               |      |
    3     5 │ | 1   | [2.0, 1.3, -6.1] | [false, true] | 4    |
    4     6 │ | -10 | [2.0, 1.3, -6.1] | [true, true]  | 4    |
    5       │-| 2   | [2.0, , -6.1]    | [false, ]     | text |
    6       │-|     |                  |               |      |
    7     7 │ +-----+------------------+---------------+------+
────────────┴───────────────────────────────────────────────────────────────────

this one is failing because of LF (\n) and CR (\r) diff

---- datasource::physical_plan::csv::tests::test_create_external_table_with_terminator_with_newlines_in_values stdout ----
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot: create_external_table_with_terminator_with_newlines_in_values
Source: datafusion/core/src/datasource/physical_plan/csv.rs:591
────────────────────────────────────────────────────────────────────────────────
Expression: batches_to_string(&df)
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
    0     0 │ +-------+-----------------------------+␊
    1     1 │ | col1  | col2                        |␊
    2     2 │ +-------+-----------------------------+␊
    3       │-| 1     | hello␊
          3 |+| 1     | hello␍
    4     4 │ world                 |␊
    5       │-| 2     | something␊
          5 │+| 2     | something␍
    6     6 │ else              |␊
    7       │-| 3     | ␊
    8       │-many␊
    9       │-lines␊
   10       │-make␊
   11       │-good test␊
          7 │+| 3     | ␍
          8 │+many␍
          9 │+lines␍
         10 │+make␍
         11 │+good test␍
   12    12 │  |␊
   13    13 │ | 4     | unquoted                    |␊
   14    14 │ | value | end                         |␊
   15    15 │ +-------+-----------------------------+
────────────┴───────────────────────────────────────────────────────────────────

cc @blaginin @alamb

xudong963 · 2025-03-24T13:53:19Z

datafusion/core/src/datasource/file_format/csv.rs

-            "| 781          |",
-            "+--------------+"];
-        assert_batches_eq!(expected, &query_result);
+        println!("csv compressed:{}", batches_to_string(&query_result));


Do we need to keep it?

and above - https://github.com/apache/datafusion/pull/15258/files#diff-8e2e37a8f04acca01f2ec6045987c6e5a6d22d75260cd53090bba71b0c71daaeR677

datafusion/core/src/datasource/physical_plan/parquet.rs

Co-authored-by: Dmitrii Blaginin <[email protected]>

…nto insta_datasource

alamb

Thank you @shruti2522 @blaginin and @xudong963

I took the liberty of pushing a few of @blaginin 's suggestions and merging this branch up from main to get the tests passing

xudong963

Thank you all, let's go

* insta migrate * Update datafusion/core/src/datasource/physical_plan/parquet.rs Co-authored-by: Dmitrii Blaginin <[email protected]> * More suggestions from code review * Update snapshots * fmt --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Dmitrii Blaginin <[email protected]>

github-actions bot added the core Core DataFusion crate label Mar 16, 2025

alamb marked this pull request as draft March 16, 2025 10:59

blaginin reviewed Mar 18, 2025

View reviewed changes

shruti2522 force-pushed the insta_datasource branch 5 times, most recently from 1615551 to 394d444 Compare March 20, 2025 13:52

github-actions bot added substrait Changes to the substrait crate datasource Changes to the datasource crate and removed substrait Changes to the substrait crate datasource Changes to the datasource crate labels Mar 20, 2025

shruti2522 force-pushed the insta_datasource branch from 442a582 to 35ae483 Compare March 20, 2025 13:57

github-actions bot added substrait Changes to the substrait crate datasource Changes to the datasource crate and removed substrait Changes to the substrait crate datasource Changes to the datasource crate labels Mar 20, 2025

shruti2522 force-pushed the insta_datasource branch from 885a49c to e87e938 Compare March 20, 2025 23:06

github-actions bot added substrait Changes to the substrait crate datasource Changes to the datasource crate and removed substrait Changes to the substrait crate datasource Changes to the datasource crate labels Mar 20, 2025

insta migrate

dc84505

shruti2522 force-pushed the insta_datasource branch from b88b827 to dc84505 Compare March 22, 2025 06:25

shruti2522 marked this pull request as ready for review March 22, 2025 06:54

xudong963 reviewed Mar 24, 2025

View reviewed changes

blaginin reviewed Mar 24, 2025

View reviewed changes

datafusion/core/src/datasource/physical_plan/parquet.rs Outdated Show resolved Hide resolved

Merge remote-tracking branch 'apache/main' into insta_datasource

bd5ba88

alamb and others added 4 commits March 25, 2025 16:35

Update datafusion/core/src/datasource/physical_plan/parquet.rs

105cdbd

Co-authored-by: Dmitrii Blaginin <[email protected]>

More suggestions from code review

242fc61

Update snapshots

202cae3

Merge branch 'insta_datasource' of github.com:shruti2522/datafusion i…

737fbd9

…nto insta_datasource

alamb approved these changes Mar 25, 2025

View reviewed changes

blaginin approved these changes Mar 26, 2025

View reviewed changes

alamb added 2 commits March 26, 2025 16:25

Merge remote-tracking branch 'apache/main' into insta_datasource

ae4901b

fmt

1a7c05c

xudong963 approved these changes Mar 27, 2025

View reviewed changes

xudong963 merged commit 3af31ca into apache:main Mar 27, 2025
27 checks passed

qstommyshu mentioned this pull request Mar 27, 2025

Migrate subtrait tests to insta #15398

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate datasource tests to insta #15258

Migrate datasource tests to insta #15258

Uh oh!

shruti2522 commented Mar 16, 2025 •

edited

Loading

Uh oh!

alamb commented Mar 16, 2025

Uh oh!

blaginin Mar 18, 2025

Uh oh!

shruti2522 commented Mar 22, 2025 •

edited

Loading

Uh oh!

xudong963 Mar 24, 2025

Uh oh!

blaginin Mar 24, 2025

Uh oh!

Uh oh!

alamb left a comment

Uh oh!

xudong963 left a comment

Uh oh!

Uh oh!

Uh oh!

Migrate datasource tests to insta #15258

Migrate datasource tests to insta #15258

Uh oh!

Conversation

shruti2522 commented Mar 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Mar 16, 2025

Uh oh!

blaginin Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

shruti2522 commented Mar 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xudong963 Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

blaginin Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

xudong963 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shruti2522 commented Mar 16, 2025 •

edited

Loading

shruti2522 commented Mar 22, 2025 •

edited

Loading