Forward-merge branch-25.12 into branch-24.02#1440
Closed
rapids-bot[bot] wants to merge 814 commits into
Closed
Conversation
The `device_resources_snmg` utility was previously responsible for [initializing NCCL communicators during construction](https://github.com/jinsolp/raft/blob/fbdec327eca8b434fda4eb36df1b115a01dca4d3/cpp/include/raft/core/device_resources_snmg.hpp#L80). However, with the recent decoupling of multi-GPU resource management from NCCL usage, the communicators are now initialized lazily. As a result, SNMG ANN may trigger NCCL initialization during the search phase inside an OpenMP thread which can lead to the NCCL clique being invalid across threads. This PR ensures that the NCCL clique is initialized ahead of time, before entering any OpenMP thread, thereby avoiding thread-local initialization issues and ensuring global validity of the communicator group.
Forward-merge branch-25.06 into branch-25.08
This PR fixes failing CAGRA merge google tests that are causing CI to fail for all PRs and surfaced after code freeze for 25.06. --------- Co-authored-by: Artem M. Chirkin <9253178+achirkin@users.noreply.github.com> Co-authored-by: achirkin <achirkin@users.noreply.github.com> Co-authored-by: Bradley Dice <bdice@bradleydice.com> Co-authored-by: jakirkham <jakirkham@gmail.com>
Forward-merge branch-25.06 into branch-25.08
…lementation (#916) The code changes in this pull request intends to replace the C wrapper layer with the Panama API-based Java implementation. Authors: - Vivek Narang (https://github.com/narangvivek10) - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #916
Answers #894 and #895 by exposing new MG CAGRA search functions with `int64_t` neighbors output type. The PR also updates the tests and benchs to use this new search function. Additionally, the tree merge feature used to perform the KNN merge operation in-place which was an issue, this PR fixes this. Authors: - Victor Lafargue (https://github.com/viclafargue) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #975
Forward-merge branch-25.06 into branch-25.08
Fixes #978. This commit modifies the `java/build.sh` script to also copy the `pom.xml` to the `target/` directory. This will allow all artifacts for publication to Maven (i.e. all the jars and the pom) to be fetched from a single location. Authors: - MithunR (https://github.com/mythrocks) Approvers: - James Lamb (https://github.com/jameslamb) URL: #981
In #816 we started to vendor rapids_config.cmake - but this broke the symlinks that rust/java use to build. Fix the symlinks to point to the new location, and add a precommit hook to check that symlinks are valid to prevent this from happening in the future
Forward-merge branch-25.06 into branch-25.08
Update rust/cuvs-sys
Forward-merge branch-25.06 into branch-25.08
This commit does a minor rephrasing of calls to `cudaMemCpy`. Prior to this change: 1. Calls to `cudaMemcpy` were using hardcoded / undocumented magic numbers for the direction of the copy. These have been replaced an enum that mirrors what is used in the CUDA Runtime call. 2. `cudaMemcpy` would return an error value that needed to be checked / converted into a RuntimeError. This has been moved into the utility function, to reduce the noise at the call site. This change also includes some minor cleanup of unused imports in the modified files. Authors: - MithunR (https://github.com/mythrocks) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #983
Fixes #930. This commit adds a C API function to fetch the cuVS version. Authors: - MithunR (https://github.com/mythrocks) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Chris Hegarty (https://github.com/ChrisHegarty) - Corey J. Nolet (https://github.com/cjnolet) - James Lamb (https://github.com/jameslamb) URL: #935
Closes #991 Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Tarang Jain (https://github.com/tarang-jain) - Divye Gala (https://github.com/divyegala) URL: #994
…1008) cuVS 25.08 CI is blocked because the 25.06 nightlies are failing. @viclafargue is going to fix this when he's back in the office, but in the meantime we'd like to unblock CI by extending the time before the nightly checks fail. We will revert this change after nightly CI is fixed.
Depends on NVIDIA/raft#2679 with reference issue NVIDIA/raft#2681. This PR reduces the wheel size from 936.9 MB to 917.1 MB. Authors: - Divye Gala (https://github.com/divyegala) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Corey J. Nolet (https://github.com/cjnolet) - Mike Sarahan (https://github.com/msarahan) - https://github.com/jakirkham URL: #925
The verify-copyright precommit hook was excluding all files, because of an incorrect regex in the exclude leading to match all files. Fix Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - MithunR (https://github.com/mythrocks) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1017
…ifferent execution modes (#1006) This PR refactors `CagraBuildAndSearchIT`, introducing separate test for different execution modes (same thread, different thread, concurrent). This way different modes are tested every time, without the need to rely on randomization to hit a particular test path. The execution is handled by utility functions (e.g. `runConcurrently`); right now these are private to the test suite, but if we think they could be useful across other tests I can move them to a utility class. It also fixes a issue where we did not wait and/or report failures in the concurrent case (now exceptions are propagated back to the calling JUnit thread). Authors: - Lorenzo Dematté (https://github.com/ldematte) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Chris Hegarty (https://github.com/ChrisHegarty) - MithunR (https://github.com/mythrocks) URL: #1006
The build process for Java uses `jextract` to generate panama bindings. Jextract is downloaded from the JDK website. This PR excludes the downloaded Jextract archive and the Jextract files from the repository by adding them to `.gitignore` Authors: - Lorenzo Dematté (https://github.com/ldematte) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - MithunR (https://github.com/mythrocks) URL: #1005
This PR adds some lines to the Java specific readme, explaining how to run a single test/a single test suite (useful in case of debugging or error reproduction). Authors: - Lorenzo Dematté (https://github.com/ldematte) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - MithunR (https://github.com/mythrocks) - Corey J. Nolet (https://github.com/cjnolet) URL: #1004
This commit rephrases the definition of the `cuvsTieredIndexANNAlgo` enum in `tiered_index.h` to use a `typedef`.
With the old definition `enum cuvsTieredIndexANNAlgo{}`, compiling a C translation unit yields errors where `cuvsTieredIndexANNAlgo` is used without `enum`. (For instance at line#55 and line#88 of `tiered_index.h`.)
```
tiered_index.h:53:3: error: must use 'enum' tag to refer to type 'cuvsTieredIndexANNAlgo'
tiered_index.h:88:3: error: must use 'enum' tag to refer to type 'cuvsTieredIndexANNAlgo'
```
The typedef sorts this error out.
Authors:
- MithunR (https://github.com/mythrocks)
- Corey J. Nolet (https://github.com/cjnolet)
Approvers:
- Corey J. Nolet (https://github.com/cjnolet)
- Ben Frederickson (https://github.com/benfred)
URL: #996
- [x] Identify the correct dataframe columns for CPU-only algorithms
- [x] Handle `num_threads` argument in build and search params (present in HDBSCAN and DiskANN benchmarks)
* Do this in a better way by renaming params
Closes #796
Closes #1001
Authors:
- Tarang Jain (https://github.com/tarang-jain)
- Corey J. Nolet (https://github.com/cjnolet)
Approvers:
- Corey J. Nolet (https://github.com/cjnolet)
URL: #863
Authors: - Tarang Jain (https://github.com/tarang-jain) Approvers: - Divye Gala (https://github.com/divyegala) - Ben Frederickson (https://github.com/benfred) URL: #1029
Contributes to rapidsai/build-planning#105. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #998
The `cuvs-sys` crate publishing depends on its version-equivalent libcuvs conda artifact. This PR updates the job to run only after conda uploads are complete.
) xref rapidsai/build-planning#184 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - James Lamb (https://github.com/jameslamb) - Ben Frederickson (https://github.com/benfred) URL: #1046
Finishes up the shellcheck work tracked in rapidsai/build-planning#135 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - James Lamb (https://github.com/jameslamb) - Ben Frederickson (https://github.com/benfred) URL: #1019
NN descent has its distances calculated and stored in a host matrix, while the return is expected to be on device. For the slicing operation, right now we allocate a device matrix of size `nrow x node_degree`. This PR improves memory usage by doing the slicing operation on host, and copying the results into the pre-allocated device memory output. This operation is done once at the end of NN Descent, so moving the slicing device computation to be computed on host should not bring noticeable perf issues. Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Divye Gala (https://github.com/divyegala) URL: #1053
Forward-merge branch-25.10 into branch-25.12
Update for compatibility with rapidsai/rmm#2077. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Divye Gala (https://github.com/divyegala) URL: #1416
A follow-up fix for #1416 Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Divye Gala (https://github.com/divyegala) URL: #1417
Using `all_neighbors` mutual reachability feature to work towards deprecating `cuvs::neighbors::detail::reachability::mutual_reachability_graph` (related issue: #982). Also, changes `build_linkage` to use `int64_t`, because `all_neighbors::build` only supports `int64_t` types, and we will change HDBSCAN to use `int64_t` types in cuML as well. **This PR will break cuML** (which will be solved by rapidsai/cuml#7104) Waiting for #1016 to be merged. Authors: - Jinsol Park (https://github.com/jinsolp) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Tarang Jain (https://github.com/tarang-jain) URL: #1234
CMake users of an installed form of CUVS don't need cutlass, so remove it from our build and install export sets. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1411
In support of building a stable C interface for libcuvs I have moved all `C` code into a separate directory for both the headers and sources. This will make it easier to install just 'C' language headers, make it easier to apply code rules for C headers, and in the future add more CI checks for our C API. Authors: - Robert Maynard (https://github.com/robertmaynard) - MithunR (https://github.com/mythrocks) - Ben Frederickson (https://github.com/benfred) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Ben Frederickson (https://github.com/benfred) URL: #1357
…#1426) Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Divye Gala (https://github.com/divyegala) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Bradley Dice (https://github.com/bdice) URL: #1426
Fix #1419 by aligning with the format present in raft: https://github.com/rapidsai/raft/blob/branch-25.12/cpp/include/raft/core/detail/mdspan_numpy_serializer.hpp#L129-134 Authors: - Micka (https://github.com/lowener) Approvers: - Ben Frederickson (https://github.com/benfred) URL: #1429
Authors: - Paul Taylor (https://github.com/trxcllnt) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #1431
Proper support to disable OpenMP also requires that any calls to functions like `omp_get_max_threads` need to be guarded by a `_OPENMP` check. This is done by adding openmp wrapper functions into `cuvs/core` that properly handle the `_OPENMP` guards and behave as required when OpenMP is disabled. Required to fix #1322 Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Divye Gala (https://github.com/divyegala) URL: #1435
…1434) We want to remove RMM's `pinned_memory_resource` in favor of `pinned_host_memory_resource`, because the latter aligns with CCCL's MR design that we will be adopting throughout RAPIDS/RMM. This PR transitions cuVS to use `pinned_host_memory_resource`. See rapidsai/rmm#2090 for more information. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Divye Gala (https://github.com/divyegala) URL: #1434
Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Ben Frederickson (https://github.com/benfred) URL: #1413
Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Ben Frederickson (https://github.com/benfred) URL: #1436
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Contributor
Author
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
github-project-automation
Bot
moved this from Todo
to Done
in Unstructured Data Processing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to branch-25.12 that creates a PR to keep branch-24.02 up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.