Regression: Parsing List(Int64) results in nullable list in 57.0.0 and a non-nullable list in 57.1.0 #8883
Description
Describe the bug
In 57.0.0, parsing the string List(Int64) results in a nullable List, but on main it results in a non nullable list
I think this is a breaking API change that we should not release in a minor release
To Reproduce
Add these tests to
diff --git a/arrow-schema/src/datatype_parse.rs b/arrow-schema/src/datatype_parse.rs
index 0bb8d629e24..9da63703b6d 100644
--- a/arrow-schema/src/datatype_parse.rs
+++ b/arrow-schema/src/datatype_parse.rs
@@ -1385,13 +1385,15 @@ mod test {
"FixedSizeList(4, Int64)",
FixedSizeList(Arc::new(Field::new_list_field(Int64, true)), 4),
),
+ ("List(Int64)", List(Arc::new(Field::new_list_field(Int64, true)))),
+ ("LargeList(Int64)'", List(Arc::new(Field::new_list_field(Int64, true)))),
];
for (data_type_string, expected_data_type) in cases {
let parsed_data_type = parse_data_type(data_type_string).unwrap();
assert_eq!(
parsed_data_type, expected_data_type,
- "Parsing '{data_type_string}', expecting '{expected_data_type}'"
+ "Parsing '{data_type_string}', expecting '{expected_data_type}', got '{parsed_data_type}'"
);
}
}This fails like this
thread 'datatype_parse::test::test_parse_data_type_backwards_compatibility' (40090518) panicked at arrow-schema/src/datatype_parse.rs:1394:13:
assertion `left == right` failed: Parsing 'List(Int64)', expecting 'List(nullable Int64)', got 'List(Int64)'
left: List(Field { data_type: Int64 })
right: List(Field { data_type: Int64, nullable: true })
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/std/src/panicking.rs:698:5
1: core::panicking::panic_fmt
at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/panicking.rs:75:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /Users/andrewlamb/.rustup/toolchains/1.91-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/panicking.rs:394:5
4: arrow_schema::datatype_parse::test::test_parse_data_type_backwards_compatibility
at ./src/datatype_parse.rs:1394:13
5: arrow_schema::datatype_parse::test::test_parse_data_type_backwards_compatibility::{{closure}}
at ./src/datatype_parse.rs:1218:54
6: core::ops::function::FnOnce::call_once
at /Users/andrewlamb/.rustup/toolchains/1.91-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /rustc/ed61e7d7e242494fb7057f2657300d9e77bb4fcb/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Expected behavior
I expect the parsing to be backwards compatible
Additional context
The behavior was changed in this PR