Skip to content

GH-47859: [C++] Fix creating union types without type_codes for fields.size() == 128 #47815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 6 commits into from
Nov 23, 2025
Merged

GH-47859: [C++] Fix creating union types without type_codes for fields.size() == 128 #47815

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
78bfbc0
Fix overflow bug with child_fields.size() = 128 (kMaxTypeCode + 1)
dahbka-lis Oct 14, 2025
3e8f571
Impl test for union types with fields.size() = 128 (kMaxTypeCode + 1)
dahbka-lis Oct 14, 2025
6624e61
Fix number of possible types for unions
dahbka-lis Nov 18, 2025
0f87d4b
Try to fix test
dahbka-lis Nov 22, 2025
a6e076f
Try to change iota
dahbka-lis Nov 22, 2025
6062dc4
Merge branch 'apache:main' into union-typecode-fix
dahbka-lis Nov 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions cpp/src/arrow/type.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3270,15 +3270,15 @@ std::shared_ptr<DataType> run_end_encoded(std::shared_ptr<DataType> run_end_type
std::shared_ptr<DataType> sparse_union(FieldVector child_fields,
std::vector<int8_t> type_codes) {
if (type_codes.empty()) {
type_codes = internal::Iota(static_cast<int8_t>(child_fields.size()));
type_codes = internal::Iota<int8_t>(0, child_fields.size());
}
return std::make_shared<SparseUnionType>(std::move(child_fields),
std::move(type_codes));
}
std::shared_ptr<DataType> dense_union(FieldVector child_fields,
std::vector<int8_t> type_codes) {
if (type_codes.empty()) {
type_codes = internal::Iota(static_cast<int8_t>(child_fields.size()));
type_codes = internal::Iota<int8_t>(0, child_fields.size());
}
return std::make_shared<DenseUnionType>(std::move(child_fields), std::move(type_codes));
}
Expand Down Expand Up @@ -3310,7 +3310,7 @@ std::shared_ptr<DataType> sparse_union(const ArrayVector& children,
std::vector<std::string> field_names,
std::vector<int8_t> type_codes) {
if (type_codes.empty()) {
type_codes = internal::Iota(static_cast<int8_t>(children.size()));
type_codes = internal::Iota<int8_t>(0, children.size());
}
auto fields = FieldsFromArraysAndNames(std::move(field_names), children);
return sparse_union(std::move(fields), std::move(type_codes));
Expand All @@ -3320,7 +3320,7 @@ std::shared_ptr<DataType> dense_union(const ArrayVector& children,
std::vector<std::string> field_names,
std::vector<int8_t> type_codes) {
if (type_codes.empty()) {
type_codes = internal::Iota(static_cast<int8_t>(children.size()));
type_codes = internal::Iota<int8_t>(0, children.size());
}
auto fields = FieldsFromArraysAndNames(std::move(field_names), children);
return dense_union(std::move(fields), std::move(type_codes));
Expand Down
30 changes: 30 additions & 0 deletions cpp/src/arrow/type_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2191,6 +2191,36 @@ TEST(TestUnionType, Basics) {
ASSERT_EQ(ty6->child_ids(), child_ids2);
}

TEST(TestUnionType, MaxTypeCode) {
std::vector<std::shared_ptr<Field>> fields;
for (int32_t i = 0; i <= UnionType::kMaxTypeCode; i++) {
fields.push_back(field(std::to_string(i), int32()));
}

std::vector<int8_t> type_codes(fields.size());
std::iota(type_codes.begin(), type_codes.end(), 0);

auto t1 = checked_pointer_cast<UnionType>(dense_union(fields, type_codes));
ASSERT_EQ(t1->type_codes().size(), UnionType::kMaxTypeCode + 1);
ASSERT_EQ(t1->child_ids().size(), UnionType::kMaxTypeCode + 1);

auto t2 = checked_pointer_cast<UnionType>(dense_union(fields));
ASSERT_EQ(t2->type_codes().size(), UnionType::kMaxTypeCode + 1);
ASSERT_EQ(t2->child_ids().size(), UnionType::kMaxTypeCode + 1);

AssertTypeEqual(*t1, *t2);

auto t3 = checked_pointer_cast<UnionType>(sparse_union(fields, type_codes));
ASSERT_EQ(t3->type_codes().size(), UnionType::kMaxTypeCode + 1);
ASSERT_EQ(t3->child_ids().size(), UnionType::kMaxTypeCode + 1);

auto t4 = checked_pointer_cast<UnionType>(sparse_union(fields));
ASSERT_EQ(t4->type_codes().size(), UnionType::kMaxTypeCode + 1);
ASSERT_EQ(t4->child_ids().size(), UnionType::kMaxTypeCode + 1);

AssertTypeEqual(*t3, *t4);
}

TEST(TestDictionaryType, Basics) {
auto value_type = int32();

Expand Down
8 changes: 8 additions & 0 deletions cpp/src/arrow/util/range.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,14 @@ std::vector<T> Iota(T length) {
return Iota(static_cast<T>(0), length);
}

/// Create a vector containing the values from start with length elements
template <typename T>
std::vector<T> Iota(T start, size_t length) {
std::vector<T> result(length);
std::iota(result.begin(), result.end(), start);
return result;
}

@kou kou Nov 11, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call this in

arrow/cpp/src/arrow/util/range.h

Lines 36 to 39 in 8c16a08

std::vector<T> result(static_cast<size_t>(stop - start));
std::iota(result.begin(), result.end(), start);
return result;
}
?

@dahbka-lis dahbka-lis Nov 11, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call this in

The stop border not included, so for fields.size() == 128, Iota<int8_t>(0, 128) is called and 128 is converted to -128.

@kou kou Nov 22, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to confirm whether we can do the following:

diff --git a/cpp/src/arrow/util/range.h b/cpp/src/arrow/util/range.h
index 2055328798..7ccf3d154d 100644
--- a/cpp/src/arrow/util/range.h
+++ b/cpp/src/arrow/util/range.h
@@ -33,9 +33,7 @@ std::vector<T> Iota(T start, T stop) {
   if (start > stop) {
     return {};
   }
-  std::vector<T> result(static_cast<size_t>(stop - start));
-  std::iota(result.begin(), result.end(), start);
-  return result;
+  return Iota(start, static_cast<size_t>(stop - start));
 }
 
 /// Create a vector containing the values from 0 up to length


/// Create a range from a callable which takes a single index parameter
/// and returns the value of iterator on each call and a length.
/// Only iterators obtained from the same range should be compared, the
Expand Down
2 changes: 1 addition & 1 deletion docs/source/format/Columnar.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -880,7 +880,7 @@ each value. Its physical layout is as follows:
* One child array for each type
* Types buffer: A buffer of 8-bit signed integers. Each type in the
union has a corresponding type id whose values are found in this
buffer. A union with more than 127 possible types can be modeled as
buffer. A union with more than 128 possible types can be modeled as
a union of unions.
* Offsets buffer: A buffer of signed Int32 values indicating the
relative offset into the respective child array for the type in a
Expand Down
Loading