Align native UDF builds with jar dependency pins#632
Conversation
Signed-off-by: liyuan <yuali@nvidia.com>
|
verified worked in local |
Greptile SummaryThis PR replaces the branch-name-based cuDF clone with a fully pinned dependency resolution flow: it extracts the embedded
Confidence Score: 3/5Core pin-resolution and cmake integration work end-to-end for the happy path, but three unaddressed issues from prior review threads (CPM override never applied, unconditional python3 call after unzip fallback, awk URL truncation) remain unresolved in this revision. The three issues carried over from previous review threads — the CPM override file being validated but silently ignored by rapids_cpm_init(), the python3-required validation block that breaks unzip-only hosts, and the awk -F= URL truncation — are substantive gaps in the ABI-alignment guarantee this PR is trying to establish. The new dead-code SHA-detection block and the unguarded cmake property load on a fresh skipCudfExtraction=true invocation are additional rough edges. The flow is architecturally sound and tested on the happy path, but the CPM override being a no-op means the advertised CCCL/RMM pin alignment is not actually enforced in the prebuilt build path. resolve-jni-cudf-pins.sh (python3 fallback and awk truncation), src/main/cpp/CMakeLists.txt (CPM override never passed to rapids_cpm_init, dead SHA-detection variables), pom.xml (unguarded cmake property load) Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Maven
participant extract as extract-cudf-libs.sh
participant resolve as resolve-jni-cudf-pins.sh
participant GH as raw.githubusercontent.com
participant clone as clone-cudf-repo.sh
participant cmake as CMake
Maven->>extract: mvn package -Pudf-native-examples
extract->>extract: Locate rapids-4-spark jar
extract->>resolve: resolve-jni-cudf-pins.sh JAR PINS_DIR PROPS_FILE
resolve->>resolve: Read spark-rapids-jni revision+url from jar
resolve->>resolve: Read cudf-java revision from jar
resolve->>GH: Download versions.json (JNI_REVISION/thirdparty/cudf-pins/)
resolve->>GH: Download rapids-cmake.sha
resolve->>resolve: Validate versions.json (CCCL, rmm present)
resolve->>GH: Download RAPIDS.cmake (at rapids-cmake SHA)
resolve->>resolve: Write jar-native-deps.properties
extract->>clone: clone-cudf-repo.sh CUDF_DIR CUDF_REVISION
clone->>clone: "git clone --filter=blob:none --no-checkout"
clone->>clone: git fetch --depth 1 origin CUDF_REVISION
clone->>clone: git checkout --detach FETCH_HEAD
Maven->>cmake: "cmake -DRAPIDS_CMAKE_FILE=... -DRAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE=..."
cmake->>cmake: include(RAPIDS.cmake) from pre-downloaded file
cmake->>cmake: rapids_cpm_init()
cmake->>cmake: rapids_cpm_cccl() / rapids_cpm_rmm()
cmake->>cmake: Link prebuilt libcudf.so
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Maven
participant extract as extract-cudf-libs.sh
participant resolve as resolve-jni-cudf-pins.sh
participant GH as raw.githubusercontent.com
participant clone as clone-cudf-repo.sh
participant cmake as CMake
Maven->>extract: mvn package -Pudf-native-examples
extract->>extract: Locate rapids-4-spark jar
extract->>resolve: resolve-jni-cudf-pins.sh JAR PINS_DIR PROPS_FILE
resolve->>resolve: Read spark-rapids-jni revision+url from jar
resolve->>resolve: Read cudf-java revision from jar
resolve->>GH: Download versions.json (JNI_REVISION/thirdparty/cudf-pins/)
resolve->>GH: Download rapids-cmake.sha
resolve->>resolve: Validate versions.json (CCCL, rmm present)
resolve->>GH: Download RAPIDS.cmake (at rapids-cmake SHA)
resolve->>resolve: Write jar-native-deps.properties
extract->>clone: clone-cudf-repo.sh CUDF_DIR CUDF_REVISION
clone->>clone: "git clone --filter=blob:none --no-checkout"
clone->>clone: git fetch --depth 1 origin CUDF_REVISION
clone->>clone: git checkout --detach FETCH_HEAD
Maven->>cmake: "cmake -DRAPIDS_CMAKE_FILE=... -DRAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE=..."
cmake->>cmake: include(RAPIDS.cmake) from pre-downloaded file
cmake->>cmake: rapids_cpm_init()
cmake->>cmake: rapids_cpm_cccl() / rapids_cpm_rmm()
cmake->>cmake: Link prebuilt libcudf.so
Reviews (1): Last reviewed commit: "Align native UDF builds with jar depende..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if(RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE) | ||
| if(NOT EXISTS "${RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE}") | ||
| message(FATAL_ERROR | ||
| "RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE does not exist: " | ||
| "${RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE}") | ||
| endif() | ||
| message(STATUS "Using rapids-cmake package override: ${RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE}") | ||
| endif() |
There was a problem hiding this comment.
RAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE validated but never applied
The variable is checked for existence and logged (lines 75–82), but rapids_cpm_init() is called later (line 281 in the SHOULD_USE_PREBUILT path) with no OVERRIDE argument. As a result, rapids_cpm_cccl() and rapids_cpm_rmm() resolve package versions from the defaults baked into the downloaded rapids-cmake, not from the jar-matched versions.json. The override file is effectively a no-op and the whole ABI-pin flow silently fails to apply its pins for CCCL and RMM.
| python3 - "$VERSIONS_FILE" <<'PY' | ||
| import json | ||
| import sys | ||
|
|
||
| versions_file = sys.argv[1] | ||
| with open(versions_file, encoding="utf-8") as fh: | ||
| data = json.load(fh) | ||
|
|
||
| packages = data.get("packages") | ||
| if not isinstance(packages, dict) or not packages: | ||
| raise SystemExit(f"ERROR: {versions_file} does not contain a non-empty packages map") | ||
|
|
||
| missing_metadata = [] | ||
| for name, package in sorted(packages.items()): | ||
| if "version" not in package: | ||
| missing_metadata.append(f"{name}: missing version") | ||
| has_git_source = "git_url" in package and "git_tag" in package | ||
| has_url_source = "url" in package and "url_hash" in package | ||
| if not (has_git_source or has_url_source): | ||
| missing_metadata.append(f"{name}: missing pinned git/url source") | ||
|
|
||
| if missing_metadata: | ||
| raise SystemExit("ERROR: invalid cudf-pins metadata:\n " + "\n ".join(missing_metadata)) | ||
|
|
||
| required = ["CCCL", "rmm"] | ||
| missing_required = [name for name in required if name not in packages] | ||
| if missing_required: | ||
| raise SystemExit("ERROR: cudf-pins missing required packages: " + ", ".join(missing_required)) | ||
| PY |
There was a problem hiding this comment.
python3 despite unzip fallback
read_property_from_jar (lines 40–63) handles environments where only unzip is installed. However, the versions.json validation inline script on line 125 calls python3 directly without any availability check. On a host where python3 is absent but unzip is present, property extraction succeeds, both files are downloaded, and then the build exits with an opaque python3: command not found (exit 127) rather than a meaningful diagnostic.
| break | ||
| PY | ||
| elif command -v unzip >/dev/null 2>&1; then | ||
| unzip -p "$JAR_PATH" "$entry" 2>/dev/null | awk -F= -v key="$property" '$1 == key {print $2; exit}' |
There was a problem hiding this comment.
awk fallback truncates property values containing =
The unzip fallback uses -F= and prints only $2, so a property value containing = is silently truncated. The url property is extracted this way; a query-string = in the JNI repository URL would produce a wrong RAW_BASE and silently download from an incorrect path.
2 participants
Summary
spark-rapids-jniand cuDF revisions from the selectedrapids-4-sparkjar, then fetch the matchingcudf-pinsandrapids-cmakeentrypoint.Closes #630
Test plan
bash -n examples/UDF-Examples/RAPIDS-accelerated-UDFs/resolve-jni-cudf-pins.shbash -n examples/UDF-Examples/RAPIDS-accelerated-UDFs/clone-cudf-repo.shbash -n examples/UDF-Examples/RAPIDS-accelerated-UDFs/extract-cudf-libs.shgit diff --checkexamples/UDF-Examples/RAPIDS-accelerated-UDFs.spark-yuanli, with~/work/jars/v26.08/rapids-4-spark_2.12-26.08.0-SNAPSHOT-cuda12.jar, ranmvn package -Pudf-native-examples -Drapids4spark.version=26.08.0-SNAPSHOT -DskipTestsinside the native UDF build container and producedtarget/rapids-4-spark-udf-examples_2.12-26.06.0-SNAPSHOT.jar.StringWordCountcompleted withnative udf completed rows=3.