Align native UDF build dependencies with rapids-4-spark jar pins

[nvliyuan]

Problem

The native UDF example build needs a reliable way to compile against the same cuDF/RMM/CCCL dependency set that was used to build the libcudf.so packaged in a given rapids-4-spark jar.

This matters most for snapshot or locally built plugin jars. A jar version such as 26.06.0-SNAPSHOT does not uniquely identify the exact cuDF/RMM/CCCL commits used by the bundled native libraries. Today the native UDF build can combine dependencies from different source states:

• libcudf.so is extracted from the rapids-4-spark jar.
• cuDF headers are cloned from cudf.git.branch.
• RMM/CCCL versions are resolved by the current rapids-cmake branch.

If those inputs do not match, native UDFs can fail with undefined symbols or crash when Spark loads or executes the JNI library. This was documented as a short-term warning in #628 / #629, but the current workaround still requires users to manually understand and align RAPIDS native dependency pins.

Proposed direction

Implement a build flow similar to spark-rapids-jni so native UDF examples consume the same pinned dependency set used to build the jar's libcudf.so.

Possible approaches:

1. Publish the resolved RAPIDS native dependency pins alongside the plugin jar, for example:
   • thirdparty/cudf-pins/rapids-cmake.sha
   • thirdparty/cudf-pins/versions.json
2. Have the native UDF example build read those pins and use rapids_cpm_package_override(...) so rapids-cmake resolves the same RMM/CCCL versions as the jar build.
3. Alternatively, publish a matching cuDF/RMM/CCCL headers artifact or tarball tied to the plugin jar build, and have the native UDF build consume that instead of cloning moving branches.

Expected behavior

Given a specific rapids-4-spark jar, users should be able to rebuild native UDFs without guessing which cuDF/RMM/CCCL commits match the jar. The build should either automatically consume the jar's pinned dependency metadata or fail early with a clear message when matching metadata is unavailable.

Related context

• [BUG] Native UDF build does not guarantee ABI match between extracted libcudf.so and downloaded headers #628 documented the ABI drift failure mode.
• Document native UDF ABI alignment requirements #629 added a README warning and manual mitigation.
• This issue tracks the longer-term build-system fix so users do not have to manually align native dependency SHAs for snapshot or locally built jars.