Support DST timezones conversion for ORC

[res-life]

Fixes #13437.

Depends on:

• part 1: [Part1] [ORC-timezone]: Build ORC timezone metadata at runtime, drop pregenerated binary cudf-spark-jni#4539
• part 2: TODO
• part 3: TODO
• part 4: TODO
• part 5: TODO

Description

Context

ORC OSS uses java.util.TimeZone to do rebase, it does not use java.time.ZoneId API.
The java.util.TimeZone and java.time.ZoneId have inconsistent behavior.
cuDF have java.time compatible impl, but does not have java.util compatible impl
java.util.TimeZone.getOffset and java.time.ZoneId.getOffset are not always consistent.

For more details, click to expand

static void testDiffBehaviorBetweenTwoAPIs() {
  // diff in `Africa/Casablanca` timezone at 6424721300000: 0 vs 3600000
  String tzId = "Africa/Casablanca";
  long epochMillis = 6424721300000L;
  int offsetMillis_1 = java.util.TimeZone.getTimeZone(tzId).getOffset(epochMillis);
  int offsetMillis_2 = java.time.ZoneId.of(tzId, ZoneId.SHORT_IDS).getRules().getOffset(Instant.ofEpochMilli(epochMillis)).getTotalSeconds() * 1000;
  if (offsetMillis_1 != offsetMillis_2) {
    // print: get diff!! 0 vs 3600000
    System.out.println("get diff!! " + offsetMillis_1 + " vs " + offsetMillis_2);
  }
}

Solution 1 [not feasible], use cuDF with ignoreTimezoneInStripeFooter=False.

cuDF manages the ORC writer timezone decode.
Problem: for far-future timestamps projected into the synthetic 400-year cycle, dates before the first synthetic DST transition were incorrectly using the first cycle entry, which is the DST offset. That causes exactly the +1 hour winter drift you saw for America/Los_Angeles with years like 8770.
cuDF has java.time compatible impl instead of java.util. We can not get correct result.
So this solution is not feasible.

Solution 2, develope kernel, use cuDF with ignoreTimezoneInStripeFooter=True

cuDF does not manage the ORC writer timezone decode. Decode as UTC in cuDF.
All time rebasing logic is handled by customized JNI kernel which is compatible to java.util.

changes

• Remove the orc_timezone_info.data file, get the timezone info dynamically.
  Why: A pre-built timezone info file can not handle for all Java versions, in future the timezone info may change.
• Implements java.util.Timezone logic
• Add OrcTimezoneSuite to test reader/writer timezone combinations.

Related cuDF issue

• ORC reader returns incorrect timestamp for epoch boundary values rapidsai/cudf#21993

perf number

Test	CPU avg (ms)	GPU avg (ms)	Speedup
cross-tz (LA→UTC)	23,662	13,815	1.71x
same-tz-baseline (LA→LA)	79,699	13,766	5.79x

How to run, refer to `OrcTimezonePerfSuite.scala`, click to expand

argLine="-DenableOrcTimeZonePerf=true \
         -DorcPerfWriterTZ=America/Los_Angeles \
         -DorcPerfReaderTZ=UTC \
         -DorcPerfRows=1073741824" \
mvn test -Dbuildver=350 \
  -DwildcardSuites=com.nvidia.spark.rapids.timezone.OrcTimezonePerfSuite

Checklists

• This PR has added documentation for new or modified features or behaviors.
• This PR has added new tests or modified existing tests to cover new code paths.
  OrcTimezonePerfSuite covers the ORC timezone read path end-to-end (CPU vs GPU correctness + performance).
• Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.
  yes, perf number refer to the above.

Checklists

• I have reviewed my PR using AI tools and by myself.
• This PR has added documentation for new or modified features or behaviors.
• This PR has added new tests or modified existing tests to cover new code paths.
  OrcTimezoneSuite covers the ORC reader/writer timezones conversion.
• Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.
  yes, for the perf number, please refer to the above section.

Signed-off-by: Chong Gao chongg@nvidia.com

[greptile-apps]

Greptile Summary

This PR adds full DST-aware cross-timezone support for ORC reads by implementing a java.util.TimeZone-compatible JNI kernel path instead of relying on cuDF's java.time-based (and incompatible) rebasing. It also removes the previous UTC-only restriction on ORC reads and writes.

• Core conversion logic (GpuOrcTimezoneUtils): A new rebaseOrcTimestamps dispatcher routes to a fast same-TZ base-offset path or a cross-TZ JNI kernel path (GpuTimeZoneDB.convertOrcTimezones), correctly using ZoneId.of + SHORT_IDS to canonicalise ORC footer timezone IDs and throwing explicitly for truly unrecognised IDs.
• Writer timezone plumbing (GpuOrcScan): writerTimezone is extracted from ORC stripe footers and threaded through OrcPartitionReaderContext, OrcExtraInfo, and OrcBlockMetaForSplitCheck, with a new writerTimezonesShareRules split-check guard.
• Config / scan guard cleanup: Removes the experimental orcReadIgnoreWriterTimezone config, the UTC-only write-side guard, and the UTC-only read-side guard; adds OrcTimezoneSuite matrix tests and an optional perf benchmark.

Confidence Score: 4/5

The ORC cross-timezone conversion path is well-structured and safe to merge; the one concern is a minor inconsistency in the batch split-check helper that only affects non-standard timezone IDs.

The conversion logic in GpuOrcTimezoneUtils correctly avoids the silent-GMT fallback by using ZoneId.of. The new writerTimezonesShareRules function in GpuOrcScan uses TimeZone.getTimeZone for the batch split-check, which can silently map unrecognised timezone IDs to GMT, creating an inconsistency that could yield wrong timestamps for non-standard ORC writer timezone strings. Practical risk is low since standard IANA/SHORT_IDS timezone IDs are handled correctly by both paths.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala — the writerTimezonesShareRules helper at line 998 should use the same ZoneId-based comparison as hasSameTimezoneRules in GpuOrcTimezoneUtils.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcTimezoneUtils.scala	Adds rebaseOrcTimestamps dispatcher and rebaseWithWriterTimezone cross-TZ path; correctly uses ZoneId.of+SHORT_IDS for ORC footer IDs and builds per-table JNI context; memory management looks correct.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala	Propagates writerTimezone through OrcPartitionReaderContext / OrcExtraInfo / OrcBlockMetaForSplitCheck; adds split-check writerTimezonesShareRules that uses TimeZone.getTimeZone (silent GMT fallback) inconsistently with hasSameTimezoneRules in GpuOrcTimezoneUtils.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala	Removes the now-superseded ORC_READ_IGNORE_WRITE_TIMEZONE experimental config and its accessor; clean removal with no remaining references.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuOrcFileFormat.scala	Removes the UTC-only write-side timezone guard; correct since the GPU ORC writer uses UTC internally and the reader now handles cross-TZ conversion.
tests/src/test/scala/com/nvidia/spark/rapids/timezone/OrcTimezoneSuite.scala	New correctness test for all writer x reader TZ combinations including legacy IDs; fixed seed avoids non-determinism; uses withTempPath for cleanup.
tests/src/test/scala/com/nvidia/spark/rapids/timezone/OrcTimezonePerfSuite.scala	New optional perf benchmark gated by system property; uses per-run UUID suffix to avoid cross-run conflicts.
integration_tests/src/main/python/orc_test.py	Removes non_utc_allow_orc_scan fallback allowances and renames the non-UTC timezone integration test.

_{Reviews (6): Last reviewed commit: "Compare ORC writer/reader timezones via ..." | Re-trigger Greptile}

[greptile-apps]

Tip:

Greploops — Automatically fix all review issues by running /greploops in Claude Code. It iterates: fix, push, re-review, repeat until 5/5 confidence.

Use the Greptile plugin for Claude Code to query reviews, search comments, and manage custom context directly from your terminal.

[firestarman]

Can you address the comments from greptile first ?

[firestarman]

One low-priority nit from my pass.

[revans2]

Generally it looks good. My main comment is that it would really be nice to have the code parse/normalize the time zones once, and then pass them around instead of passing strings around. It gets really confusing to know if this string has been normalized or not, if it has then we don't need to worry about short ids, if it has not, then we do...

[revans2]

Why delete the comments that explain why we are setting the start and end timestamp to what they are? Have we tested outside of this range recently? Do we have a follow on issue to fix the range limitations?

[revans2]

Even though this section is small it is duplicated at least here and at writerTimezonesShareRules. It might be worth trying to make the conversion to a timezone a standard function. Also this is deduping time zones similarly. It might be nice to also make that generic.

[revans2]

Would it be cleaner to pass around a ZoneId instead of the string for the writer everywhere? It looks like we try to resolve it in multiple places. Might be nice to do it once.

[revans2]

Are these not rejected by java.util.TimeZone.getTimeZone? Are the different ways of parsing/normalizing them consistent?

[nvauto]

NOTE: release/26.06 has been created from main. Please retarget your PR to release/26.06 if it should be included in the release.