fix(arrow/array): honor field nullability in comparison and JSON marshaling

[zeroshade]

Rationale for this change

This is the inner-nullability parity work that was previously bundled into #558 (the TimestampWithOffset canonical extension type). #558 has now been reduced to just the extension type, and this PR carries the comparison and JSON-marshaling changes on their own so they can be reviewed independently and more carefully.

What changes are included in this PR?

• arrow/array/compare.go: thread each field's Nullable flag through Equal/RecordEqual/TableEqual/ChunkedEqual (and the Approx variants) so that non-nullable fields ignore validity-bitmap differences; add a WithNullable functional option; compare record/table fields against the right-hand schema (not the left against itself) and ignore field metadata during record/table comparison.
• arrow.Array interface: keep the single-argument GetOneForMarshal(i int) and move the field-local behavior to a new optional NullableMarshaler interface with GetOneForMarshalNullable(i int, nullable bool), so external callers and custom Array implementations are not broken. Containers (struct, union, dictionary, run-end-encoded, extension) and RecordToJSON propagate each field's nullability through an unexported helper that falls back to GetOneForMarshal when the optional interface is not implemented.
• Struct/record: honor field.Nullable when serializing to JSON, and handle explicit null for a non-nullable field when reading.
• internal/json: add a helper to detect a null message.
• Test fixtures updated to declare inner struct field nullability explicitly.

Relationship to #833

#833 pursues the same goal with a different strategy: it drops the comparison leniency (WithNullable/equalOption) in favor of schema validation in Validate() to match Arrow C++, and checks IsValid() in the struct/record builders rather than introducing GetOneForMarshalNullable(). This PR preserves the earlier comparison-based approach. Reviewers should decide which direction to land; opening as a draft pending that decision.

Are these changes tested?

Yes. go test ./arrow/array/ ./arrow/extensions/ ./arrow/internal/arrjson/ ./arrow/compute/ ./arrow/ipc/... pass.

Are there any user-facing changes?

Yes: comparison functions gain a WithNullable option and the array marshaling path gains an optional NullableMarshaler interface. The existing GetOneForMarshal(i int) signature is preserved.

[Copilot]

Pull request overview

This PR updates Arrow Go’s array comparison and JSON (un)marshaling logic to honor field-level nullability from schemas (inner-nullability parity), so non-nullable fields ignore validity-bitmap differences and do not serialize JSON null when a slot is marked null.

Changes:

• Thread schema field Nullable through Equal/RecordEqual/TableEqual/ChunkedEqual (+ Approx variants) via a new WithNullable option and nullable-aware equality plumbing.
• Introduce optional arrow.NullableMarshaler and propagate field nullability through containers and RecordToJSON, with safe fallback behavior.
• Adjust JSON decoding for struct/record builders to handle explicit null in non-nullable fields, and update fixtures/tests to declare inner field nullability explicitly.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/json/json.go	Add IsNullMessage helper for raw JSON null detection (go-json build).
internal/json/json_stdlib.go	Add IsNullMessage helper for raw JSON null detection (stdlib/tinygo build).
arrow/ipc/metadata_test.go	Adjust extension metadata expectations; explicitly assert metadata preservation since RecordEqual ignores metadata.
arrow/ipc/cmd/arrow-ls/main_test.go	Update expected schema formatting to reflect explicit inner-field nullability.
arrow/internal/arrjson/arrjson_test.go	Update JSON schema fixtures to mark struct children as nullable.
arrow/internal/arrdata/arrdata.go	Update struct-record fixtures to declare inner fields nullable and align validity masks.
arrow/extensions/variant.go	Implement nullable-aware marshaling via GetOneForMarshalNullable.
arrow/extensions/uuid.go	Implement nullable-aware marshaling via GetOneForMarshalNullable.
arrow/extensions/uuid_test.go	Update schema to set UUID field nullable for tests.
arrow/extensions/json.go	Make JSON extension marshaling nullable-aware without breaking ValueJSON API.
arrow/extensions/bool8.go	Implement nullable-aware marshaling via GetOneForMarshalNullable.
arrow/compute/vector_sort_test.go	Update test schemas to explicitly set fields nullable where JSON inputs include nulls.
arrow/array/util.go	Propagate field nullability in RecordToJSON; add helpers for nullable-aware marshaling including list element handling.
arrow/array/util_test.go	Exercise JSON round-trip with both nullable and non-nullable schemas.
arrow/array/union.go	Propagate child field nullability through union marshaling and equality.
arrow/array/timestamp.go	Add nullable-aware marshaling and thread nullability through timestamp equality.
arrow/array/struct.go	Propagate child field nullability through struct marshaling and equality; handle explicit null on non-nullable fields during decoding.
arrow/array/struct_test.go	Add coverage for explicit null in required nested struct fields.
arrow/array/string.go	Add nullable-aware marshaling and thread nullability through string equality functions.
arrow/array/record.go	Handle explicit null for non-nullable fields during record decoding.
arrow/array/record_test.go	Expand record builder test for nullable/non-nullable fields and JSON round-trip behavior.
arrow/array/numeric_generic.go	Add nullable-aware marshaling for numeric/date/time arrays.
arrow/array/null.go	Add GetOneForMarshalNullable to align with optional nullable marshaler pattern.
arrow/array/map.go	Thread nullability option through map equality.
arrow/array/list.go	Add nullable-aware list marshaling; thread child element nullability through list equality.
arrow/array/interval.go	Add nullable-aware marshaling and thread nullability through interval equality.
arrow/array/float16.go	Add nullable-aware marshaling for Float16 arrays.
arrow/array/fixedsize_binary.go	Add nullable-aware marshaling and thread nullability through equality.
arrow/array/fixed_size_list.go	Add nullable-aware marshaling; thread child element nullability through equality.
arrow/array/extension.go	Thread nullability through extension equality; document why base doesn’t implement NullableMarshaler.
arrow/array/encoded.go	Add nullable-aware marshaling and thread value-nullability through run-end-encoded equality.
arrow/array/dictionary.go	Add nullable-aware marshaling and thread nullability through dictionary equality.
arrow/array/decimal.go	Add nullable-aware marshaling and thread nullability through decimal equality.
arrow/array/compare.go	Core change: introduce schema-level nullability/field checks for record/table comparisons; add WithNullable; plumb nullable through equality stack and special-case types without top-level validity.
arrow/array/compare_test.go	Add tests for equality semantics when treating fields as non-nullable.
arrow/array/boolean.go	Add nullable-aware marshaling and thread nullability through boolean equality.
arrow/array/binary.go	Add nullable-aware marshaling and thread nullability through binary equality.
arrow/array.go	Introduce optional NullableMarshaler interface on arrays (non-breaking).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeroshade]

How Arrow C++ handles run-end-encoded value nullability

For context on the REE value-nullability discussion here, comparing against the C++ reference implementation:

C++ has no configurable value nullability. RunEndEncodedType hard-wires the values child to nullable=true (type.cc#L1307):

children_ = {std::make_shared<Field>("run_ends", std::move(run_end_type), false),
             std::make_shared<Field>("values",   std::move(value_type),   true)};

• DataType equality compares child data types only — the values field's nullability is not part of REE type identity (compare.cc#L883):

  result_ = left.value_type()->Equals(right.value_type()) &&
            left.run_end_type()->Equals(right.run_end_type());

• Array equality merges runs and compares the decoded values (validity included, since values are always nullable); it never consults Field::nullable.
• IPC reconstructs RunEndEncodedType(children[0]->type(), children[1]->type()) (metadata_internal.cc#L423) — the values-field nullable bit is dropped and re-fixed to nullable, so a non-nullable-values REE does not round-trip.
• Spec (Columnar.rst): run_ends cannot be null; values may be null.

Implications:

1. Excluding ValueNullable from arrow.TypeEqual (as it is today) matches C++. Making it part of type identity would be stricter than the reference and breaks REE record/IPC construction in arrow-go (schema-vs-array type mismatch, since ValueNullable isn't set consistently and doesn't survive IPC).
2. More broadly, C++ never makes Equal() lenient based on a field's Nullable flag — it keeps equality strict and enforces schema nullability via Validate(). That is the direction taken in fix(arrow/array): Parity with Arrow C++ when reading/writing null to/from JSON #833, and it's why the equalOpts mechanism in this PR diverges from the reference. This branch remains useful as the record of the alternative approach, but the C++-aligned path is Validate()-based (fix(arrow/array): Parity with Arrow C++ when reading/writing null to/from JSON #833).

[zeroshade]

Closing in favor of #833.

As detailed in the comparison above, the equalOpts leniency in this PR diverges from Arrow C++, which keeps Equal() strict and enforces schema nullability via Validate() instead. #833 implements that C++-aligned approach, so the inner-nullability parity work should land there.

The TimestampWithOffset canonical extension type that originally motivated this work has been decoupled into #558 and can merge independently.