java: use GCS mirror for Maven, other small changes #992
+83
−27
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| -e | ||
| -B | ||
| -Daether.connector.basic.downstreamThreads=1 | ||
| -Daether.transport.http.retryHandler.count=5 | ||
| -Daether.transport.http.retryHandler.interval=10000 | ||
| -Dmaven.wagon.http.retryHandler.count=5 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,33 +1,43 @@ | ||
| # Java KvikIO Bindings | ||
|
|
||
| ## Summary | ||
|
|
||
| These Java KvikIO bindings for GDS currently support only synchronous read and write IO operations using the underlying cuFile API. Support for batch IO and asynchronous operations are not yet supported. | ||
|
|
||
| ## Dependencies | ||
|
|
||
| The Java KvikIO bindings have been developed to work on Linux based systems and require [CUDA](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) to be installed and for [GDS](https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html) to be properly enabled. To compile the shared library it is also necessary to have a JDK installed. To run the included example, it is also necessary to install JCuda as it is used to handle memory allocations and the transfer of data between host and GPU memory. JCuda jar files supporting CUDA 12.x can be found here: | ||
|
|
||
| * [jcuda-12.0.0.jar](https://repo1.maven.org/maven2/org/jcuda/jcuda/12.0.0/jcuda-12.0.0.jar), | ||
| * [jcuda-natives-12.0.0.jar](https://repo1.maven.org/maven2/org/jcuda/jcuda-natives/12.0.0/jcuda-natives-12.0.0.jar) | ||
|
|
||
| For more information on JCuda and potentially more up to date installation instructions or jar files, see here: | ||
| [JCuda](http://javagl.de/jcuda.org/), [JCuda Usage](https://github.com/jcuda/jcuda-main/blob/master/USAGE.md), [JCuda Maven Repo](https://mvnrepository.com/artifact/org.jcuda) | ||
|
|
||
| ## Compilation and examples | ||
|
|
||
| An example for how to use the Java KvikIO bindings can be found in `src/test/java/ai/rapids/kvikio/cufile/BasicReadWriteTest.java` | ||
|
|
||
| ##### Note: This example has a dependency on JCuda so ensure that when running the example the JCuda shared library files are on the JVM library path along with the `libCuFileJNI.so` file. | ||
|
|
||
| ### Setup a test file target | ||
|
|
||
| ##### NOTE: the example as written will default to creating a temporary file in your `/tmp` directory. This directory may not be mounted in a compatible manner for use with GDS on your particular system, causing the example to run in compatibility mode. If this is the case, run the following command replacing `/mnt/nvme/` with your mount directory and update `cufile/BasicReadWriteTest.java` to point to the correct file path. | ||
|
|
||
| touch /mnt/nvme/java_test | ||
|
|
||
| ### Compile the shared library and Java files with Maven | ||
|
|
||
| ##### Note: This wil also run the example code | ||
|
|
||
| ```shell | ||
| cd kvikio/java/ | ||
| mvn clean install | ||
| ``` | ||
|
|
||
| ### Rerun example code with Maven | ||
|
|
||
| ```shell | ||
| cd kvikio/java/ | ||
| mvn test | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <!-- | ||
| SPDX-FileCopyrightText: Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
|
|
@@ -28,6 +28,60 @@ SPDX-License-Identifier: Apache-2.0 | |
| <cmake.version>4.0.3-b1</cmake.version> | ||
| </properties> | ||
|
|
||
| <repositories> | ||
| <!-- Prefer Google Cloud mirror, which has higher rate limits --> | ||
| <repository> | ||
| <id>gcs-maven-central-mirror</id> | ||
| <name>GCS Maven Central mirror</name> | ||
| <url>https://maven-central.storage-download.googleapis.com/maven2/</url> | ||
| <releases> | ||
| <enabled>true</enabled> | ||
| </releases> | ||
| <snapshots> | ||
| <enabled>false</enabled> | ||
| </snapshots> | ||
| </repository> | ||
| <!-- Fall back to Maven upstream for packages not found in GCS mirror --> | ||
| <repository> | ||
| <id>central</id> | ||
| <name>Maven Central</name> | ||
| <url>https://repo.maven.apache.org/maven2</url> | ||
| <releases> | ||
| <enabled>true</enabled> | ||
| </releases> | ||
| <snapshots> | ||
| <enabled>false</enabled> | ||
| </snapshots> | ||
| </repository> | ||
| </repositories> | ||
|
|
||
| <pluginRepositories> | ||
| <!-- Prefer Google Cloud mirror, which has higher rate limits --> | ||
| <pluginRepository> | ||
| <id>gcs-maven-central-mirror</id> | ||
| <name>GCS Maven Central mirror</name> | ||
| <url>https://maven-central.storage-download.googleapis.com/maven2/</url> | ||
| <releases> | ||
| <enabled>true</enabled> | ||
| </releases> | ||
| <snapshots> | ||
| <enabled>false</enabled> | ||
| </snapshots> | ||
| </pluginRepository> | ||
| <!-- Fall back to Maven upstream for packages not found in GCS mirror --> | ||
| <pluginRepository> | ||
| <id>central</id> | ||
| <name>Maven Plugin Repository</name> | ||
| <url>https://repo.maven.apache.org/maven2</url> | ||
| <releases> | ||
| <enabled>true</enabled> | ||
| </releases> | ||
| <snapshots> | ||
| <enabled>false</enabled> | ||
| </snapshots> | ||
| </pluginRepository> | ||
| </pluginRepositories> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.jcuda</groupId> | ||
|
|
@@ -58,16 +112,9 @@ SPDX-License-Identifier: Apache-2.0 | |
| <build> | ||
| <pluginManagement> | ||
| <plugins> | ||
| <plugin> | ||
| <artifactId>maven-clean-plugin</artifactId> | ||
| <version>3.1.0</version> | ||
|
Member
Author
There was a problem hiding this comment.
|
||
| </plugin> | ||
| <plugin> | ||
| <artifactId>maven-compiler-plugin</artifactId> | ||
|
|
@@ -88,7 +135,7 @@ SPDX-License-Identifier: Apache-2.0 | |
| <dependency> | ||
| <groupId>org.junit.jupiter</groupId> | ||
| <artifactId>junit-jupiter-engine</artifactId> | ||
| <version>${junit.version}</version> | ||
|
Member
Author
There was a problem hiding this comment. This version is stored a few lines up: Line 26 in 37790a1 Reusing that value reduces the risk of inconsistencies. |
||
| </dependency> | ||
| </dependencies> | ||
| </plugin> | ||
|
|
@@ -104,14 +151,6 @@ SPDX-License-Identifier: Apache-2.0 | |
| <artifactId>maven-deploy-plugin</artifactId> | ||
| <version>2.8.2</version> | ||
| </plugin> | ||
|
Member
Author
There was a problem hiding this comment. These don't have entries in |
||
| </plugins> | ||
| </pluginManagement> | ||
| <plugins> | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See NVIDIA/cuvs#2253 (comment) for an explanation of these flags.
I believe comments are not allowed in this file because its contents are passed literally to
mvn.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if it's worth a CI re-run, but one option (maybe) would be to document the options in a
READMEinside the.mvndirectory. But that shouldn't block this.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah fair. I went back and forth on that and settled on not doing it because I thought the git blame pointing back to this PR + the mostly informative names was probably enough to communicate the intent.
I'm gonna treat that "I don't know" and "(maybe)" as you also being unsure about how to handle this 😂
Think I'll just merge it as-is.