Null-aware anti joins can be swapped or repartitioned by scheduler planning

[phillipleblanc]

Bug

SQL NOT IN predicates are planned as null-aware LeftAnti hash joins. These joins have three distributed-planning requirements:

• They must remain LeftAnti; swapping to RightAnti is invalid.
• They must use PartitionMode::CollectLeft so every probe sees the complete build side.
• The probe side must run in one task. DataFusion's visited_indices_bitmap, report_probe_completed(), and probe_side_has_null state is shared only within one process.

Ballista's scheduler can currently violate these requirements in static, unbounded-input, and adaptive query execution (AQE) planning.

An invalid swap fails during stage resolution with an error such as:

null_aware can only be true for LeftAnti joins, got RightAnti

The distributed query can then remain running until the client times out. A multi-task CollectLeft or Partitioned plan is worse: it can silently return duplicated or incorrect rows.

For a 20-row outer table and four-partition inputs, a null-aware CollectLeft join produced 56 rows when the correct result was 0, and 75 rows when the correct result was 15.

Expected behavior

Static and adaptive planning should:

1. preserve null-aware joins as unswapped LeftAnti(CollectLeft) joins;
2. collect the build side and coalesce the probe side so exactly one task executes the join; and
3. reject an explicitly disabled or known oversized build side instead of producing an unsafe plan.

Proposed fix

• Match DataFusion's null-aware guards in statistical and unbounded-input join swapping.
• Lower null-aware joins to a broadcast build plus a single probe task in both static and AQE stage planning.
• Use ballista.optimizer.broadcast_join_threshold_bytes as the safety limit when the build size is known. A value of 0 rejects null-aware anti joins.
• Add end-to-end NOT IN tests with four input partitions, with and without a NULL in the subquery.

Apache #2188 implements this fix against current main.

Ballista's default sort-merge planning can discard the null-aware flag before these scheduler rules run. Apache #2193 tracks that separate path.

This was originally observed and fixed partially downstream in spiceai#58.