[BUG] from_json: nested schema-category mismatch should null depth-1 ancestor (SPARK-33134 parity)

[wjxiz1992]

Description

JSONUtils.fromJSONToStructs (the GPU implementation behind Spark's from_json for struct schemas) does not match Spark's CPU behavior when a JSON record contains a nested value whose type category disagrees with the requested schema. Spark's JacksonParser.convertObject runs with isRoot=false for nested fields and propagates the conversion exception up, nulling the entire depth-1 ancestor field for that row; the GPU path silently returns the partial result.

Concrete repro from the SPARK-33134 test (org.apache.spark.sql.JsonFunctionsSuite):

val st = new StructType().add("c1", LongType).add("c2",
            ArrayType(new StructType().add("c3", LongType).add("c4", StringType)))
val df2 = Seq("""{"data": {"c2": [19], "c1": 123456}}""").toDF("c0")
withSQLConf(SQLConf.JSON_ENABLE_PARTIAL_RESULTS.key -> "false") {
  checkAnswer(df2.select(from_json($"c0", new StructType().add("data", st))),
              Row(Row(null)))  // CPU: data is NULL
}
// GPU returns Row(Row(Row(123456, null))): "data" stays populated, "data.c2" is nulled.

In RapidsJsonFunctionsSuite this is exposed as an .exclude(...) entry in RapidsTestSettings.scala, pointing at this issue. The mismatch is on c2 (array of struct elements that are scalars in the JSON), which according to Spark should null the whole data field.

Root cause

cuDF's read_json detects the category mismatch in host_tree_algorithms::mark_is_pruned and prunes the offending subtree, but it does not surface that information to the caller. spark-rapids-jni's from_json_to_structs.cu therefore has no signal that the parent should be nulled and returns a partial result.

Proposed fix

A two-layer split, so cuDF stays Spark-agnostic:

1. Upstream (cuDF): expose a per-top-level-column diagnostic column_name_info::had_schema_mismatch (std::optional<bool>) set whenever the JSON tree under that column was pruned for category mismatch against the user-supplied schema. Tracked at Expose per-column had_schema_mismatch diagnostic from JSON reader rapidsai/cudf#22450 (closes [BUG] read_json: nested struct type mismatch keeps parent struct populated; Spark expects parent NULL rapidsai/cudf#22423).
2. spark-rapids-jni: consume the diagnostic. In from_json_to_structs.cu, after read_json returns and before convert_data_type runs, replace each flagged top-level parsed_columns[i] with an ALL_NULL mask. The downstream conversion path propagates the NULL through descendants, so Spark consumers observe every field of the depth-1 column as NULL — matching JacksonParser.convertObject with isRoot=false.

The jni-side patch is a self-contained 11-line addition; PR to follow.

Validation

Local mvn package -pl tests -am -Dbuildver=330 -DwildcardSuites=org.apache.spark.sql.rapids.suites.RapidsJsonFunctionsSuite against a worktree that consumes the cuDF #22450 branch: Tests: succeeded 66, failed 0, canceled 0, ignored 0, pending 0 (66/66 incl. SPARK-33134, both PARTIAL_RESULTS=true and false branches).

[wjxiz1992]

re open, the patch has been reverted.