[BUG] nds2-parquet-3k-snappy-gh: 468 incomplete queries across 5 test iterations (all queries Failed / CompletedWithTaskFailures)

[parthosa]

Describe the bug
Build: nds2-parquet-3k-snappy-gh/616

The NDS 2.0 Parquet 3K Snappy test on Grace Hopper ran 5 iterations across 5 Spark applications. check_query_result.sh found 468 incomplete queries. Almost every query reports queryStatus='Failed' (a handful are 'CompletedWithTaskFailures', e.g. query3/13/16/32/84/85/88/90/95). The uniform failure pattern across all iterations and all queries (including query1/query2) indicates a cluster/environment-wide issue rather than per-query regressions.

Error logs:

+ ./scripts/nds/check_query_result.sh test_results
Error: found query not completed in test202605010807-...-query1-*.json
+ [[ Failed != Completed ]]
+ REASON+='Failed, '
Error: found query not completed in test202605010807-...-query13-*.json
+ [[ CompletedWithTaskFailures != Completed ]]
+ REASON+='CompletedWithTaskFailures, '
... (repeated across ~468 query result files over 5 test runs / 5 app ids
 app-20260501081059-0000, app-20260501091934-0002,
 app-20260501104334-0004, app-20260501113926-0006,
 app-20260501124201-0008)
Found 468 incomplete queries and 0 failed queries.
app-20260501081059-0000  query1   Failed
app-20260501081059-0000  query3   CompletedWithTaskFailures
app-20260501081059-0000  query13  CompletedWithTaskFailures
app-20260501081059-0000  query16  CompletedWithTaskFailures
app-20260501081059-0000  query32  CompletedWithTaskFailures
app-20260501081059-0000  query84  CompletedWithTaskFailures
app-20260501081059-0000  query85  CompletedWithTaskFailures
app-20260501081059-0000  query88  CompletedWithTaskFailures
app-20260501081059-0000  query90  CompletedWithTaskFailures
... (nearly all queries Failed across all 5 apps)
script returned exit code 1
Finished: FAILURE

Complete Logs

26/05/01 08:11:25 ERROR Executor: Exception in task 23.0 in stage 44.0 (TID 12520)
ai.rapids.cudf.CudaFatalException: CUDA error at: _deps/cuco-src/include/cuco/detail/open_addressing/open_addressing_impl.cuh884: cudaErrorIllegalAddress an illegal memory access was encountered
        at ai.rapids.cudf.Table.groupByAggregate(Native Method)
        at ai.rapids.cudf.Table.access$3300(Table.java:30)
        at ai.rapids.cudf.Table$GroupByOperation.aggregate(Table.java:4023)
        at com.nvidia.spark.rapids.AggHelper.$anonfun$performGroupByAggregation$2(GpuAggregateExec.scala:577)
        at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
        at com.nvidia.spark.rapids.AggHelper.$anonfun$performGroupByAggregation$1(GpuAggregateExec.scala:564)
        at com.nvidia.spark.rapids.NvtxId.apply(NvtxRangeWithDoc.scala:84)
        at com.nvidia.spark.rapids.AggHelper.performGroupByAggregation(GpuAggregateExec.scala:564)
        at com.nvidia.spark.rapids.AggHelper.aggregate(GpuAggregateExec.scala:479)
        at com.nvidia.spark.rapids.AggHelper.$anonfun$aggregateWithoutCombine$4(GpuAggregateExec.scala:497)
        at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
        at com.nvidia.spark.rapids.AggHelper.$anonfun$aggregateWithoutCombine$3(GpuAggregateExec.scala:495)
        at com.nvidia.spark.rapids.NvtxId.apply(NvtxRangeWithDoc.scala:84)
        at com.nvidia.spark.rapids.NvtxIdWithMetrics$.apply(NvtxWithMetrics.scala:67)
        at com.nvidia.spark.rapids.AggHelper.$anonfun$aggregateWithoutCombine$2(GpuAggregateExec.scala:495)
        at com.nvidia.spark.rapids.RmmRapidsRetryIterator$AutoCloseableAttemptSpliterator.next(RmmRapidsRetryIterator.scala:581)
        at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryIterator.next(RmmRapidsRetryIterator.scala:744)
        at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryAutoCloseableIterator.next(RmmRapidsRetryIterator.scala:629)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:496)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at com.nvidia.spark.rapids.CloseableBufferedIterator.next(CloseableBufferedIterator.scala:65)
        at com.nvidia.spark.rapids.CloseableBufferedIterator.head(CloseableBufferedIterator.scala:47)
        at com.nvidia.spark.rapids.AggregateUtils$$anon$1.$anonfun$next$1(GpuAggregateExec.scala:170)
        at com.nvidia.spark.rapids.Arm$.closeOnExcept(Arm.scala:126)
        at com.nvidia.spark.rapids.AggregateUtils$$anon$1.next(GpuAggregateExec.scala:167)
        at com.nvidia.spark.rapids.AggregateUtils$$anon$1.next(GpuAggregateExec.scala:162)
        at com.nvidia.spark.rapids.AggregateUtils$.iterateAndRepartition(GpuAggregateExec.scala:259)
        at com.nvidia.spark.rapids.GpuMergeAggregateIterator.$anonfun$next$3(GpuAggregateExec.scala:1009)
        at scala.Option.getOrElse(Option.scala:189)
        at com.nvidia.spark.rapids.GpuMergeAggregateIterator.next(GpuAggregateExec.scala:946)
        at com.nvidia.spark.rapids.GpuMergeAggregateIterator.next(GpuAggregateExec.scala:896)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at com.nvidia.spark.rapids.DynamicGpuPartialAggregateIterator.$anonfun$next$16(GpuAggregateExec.scala:2303)
        at scala.Option.map(Option.scala:230)
        at com.nvidia.spark.rapids.DynamicGpuPartialAggregateIterator.next(GpuAggregateExec.scala:2303)
        at com.nvidia.spark.rapids.DynamicGpuPartialAggregateIterator.next(GpuAggregateExec.scala:2147)
        at com.nvidia.spark.rapids.GpuOpTimeTrackingRDD$$anon$1.$anonfun$next$1(GpuExec.scala:80)
        at com.nvidia.spark.rapids.GpuMetric.ns(GpuMetrics.scala:466)
        at com.nvidia.spark.rapids.GpuOpTimeTrackingRDD$$anon$1.next(GpuExec.scala:80)
        at com.nvidia.spark.rapids.GpuTopN$$anon$1.next(limit.scala:318)
        at com.nvidia.spark.rapids.GpuTopN$$anon$1.next(limit.scala:301)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at com.nvidia.spark.rapids.GpuOpTimeTrackingRDD$$anon$1.$anonfun$next$1(GpuExec.scala:80)
        at com.nvidia.spark.rapids.GpuMetric.ns(GpuMetrics.scala:466)
        at com.nvidia.spark.rapids.GpuOpTimeTrackingRDD$$anon$1.next(GpuExec.scala:80)
        at org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$anon$1.partNextBatch(GpuShuffleExchangeExecBase.scala:456)
        at org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$anon$1.hasNext(GpuShuffleExchangeExecBase.scala:488)
        at com.nvidia.spark.rapids.GpuOpTimeTrackingRDD$$anon$1.$anonfun$hasNext$1(GpuExec.scala:68)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
        at com.nvidia.spark.rapids.GpuMetric.ns(GpuMetrics.scala:466)
        at com.nvidia.spark.rapids.GpuOpTimeTrackingRDD$$anon$1.hasNext(GpuExec.scala:68)
        at org.apache.spark.sql.rapids.RapidsShuffleThreadedWriterBase.write(RapidsShuffleInternalManagerBase.scala:589)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)

Environment details

• Platform: Grace Hopper cluster
• Spark: 3.5.7, Hadoop: 3.3.6, CUDA classifier: cuda12-arm64
• Benchmark: NDS 2.0 Parquet 3000 (decimal=true, AQE=on, UCX=off, 16 cores, GPU)
• 5 iterations scheduled, all produced failing/incomplete queries

[orbitwebsites-cloud]

I am currently working on a fix for this bounty. Will submit a PR shortly.

Planned approach:
Analyze error logs and cluster/environment configuration to identify the root cause of the issue, then implement code changes to resolve it if necessary.

[sameerz]

@orbitwebsites-cloud this issue does not have a bounty associated with it. The proposed PR #259 does not address the problem. I will close the PR.

[sameerz]

Subsequent runs succeeded. Closing this issue.