Skip to content

Google Cloud SDK Shading for GCS PerfIO #15130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sdrp713 wants to merge 2 commits into
base: main
Choose a base branch
from
+53 −8
Open

Google Cloud SDK Shading for GCS PerfIO #15130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ade98a3
Initial Commit
sdrp713 Jun 23, 2026
511302a
Exclude protobuf from relocation
sdrp713 Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions aggregator/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,31 @@
<shadedPattern>META-INF/versions/$1/${rapids.shade.package.path}/org/roaringbitmap</shadedPattern>
<rawString>true</rawString>
</relocation>
<!--
Relocate bundled Google libraries. Dataproc ships its own GCS
connector and Google client libraries, so the plugin-owned GCS SDK
stack must stay isolated from the cluster classpath. This also covers
com.google.flatbuffers with the same shaded path used historically.
-->
<relocation>
<pattern>com.google.flatbuffers</pattern>
<shadedPattern>${rapids.shade.package}.com.google.flatbuffers</shadedPattern>
<pattern>com.google</pattern>
<shadedPattern>${rapids.shade.package}.com.google</shadedPattern>
<excludes>
<!--
ORC and Spark expose unshaded protobuf message instances to
RAPIDS shim code, so protobuf references in RAPIDS classes must
stay in the original com.google.protobuf namespace.
-->
<exclude>com.google.protobuf.**</exclude>
</excludes>
</relocation>
Comment thread
sdrp713 marked this conversation as resolved.
<relocation>
<pattern>io.grpc</pattern>
<shadedPattern>${rapids.shade.package}.io.grpc</shadedPattern>
</relocation>
<relocation>
<pattern>io.opencensus</pattern>
<shadedPattern>${rapids.shade.package}.io.opencensus</shadedPattern>
</relocation>
<relocation>
<pattern>org.roaringbitmap</pattern>
Expand Down
26 changes: 24 additions & 2 deletions scala2.13/aggregator/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,31 @@
<shadedPattern>META-INF/versions/$1/${rapids.shade.package.path}/org/roaringbitmap</shadedPattern>
<rawString>true</rawString>
</relocation>
<!--
Relocate bundled Google libraries. Dataproc ships its own GCS
connector and Google client libraries, so the plugin-owned GCS SDK
stack must stay isolated from the cluster classpath. This also covers
com.google.flatbuffers with the same shaded path used historically.
-->
<relocation>
<pattern>com.google.flatbuffers</pattern>
<shadedPattern>${rapids.shade.package}.com.google.flatbuffers</shadedPattern>
<pattern>com.google</pattern>
<shadedPattern>${rapids.shade.package}.com.google</shadedPattern>
<excludes>
<!--
ORC and Spark expose unshaded protobuf message instances to
RAPIDS shim code, so protobuf references in RAPIDS classes must
stay in the original com.google.protobuf namespace.
-->
<exclude>com.google.protobuf.**</exclude>
</excludes>
</relocation>
Comment thread
sdrp713 marked this conversation as resolved.
<relocation>
<pattern>io.grpc</pattern>
<shadedPattern>${rapids.shade.package}.io.grpc</shadedPattern>
</relocation>
<relocation>
<pattern>io.opencensus</pattern>
<shadedPattern>${rapids.shade.package}.io.opencensus</shadedPattern>
</relocation>
<relocation>
<pattern>org.roaringbitmap</pattern>
Expand Down
9 changes: 5 additions & 4 deletions sql-plugin/src/main/java/com/nvidia/spark/rapids/fileio/RapidsInputFiles.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

package com.nvidia.spark.rapids.fileio;

import com.nvidia.spark.rapids.PerfIO;
import com.nvidia.spark.rapids.PerfIOConf;
import org.apache.spark.SparkEnv;

Expand All @@ -39,16 +40,16 @@ public static boolean isS3PerfEnabled() {
return env.conf().getBoolean(PerfIOConf.S3PERF_ENABLED().key(), false);
}
/**
* True iff {@code spark.rapids.perfio.gcs.enabled} is set to {@code true} on
* the active SparkConf. Returns false when no {@link SparkEnv} is initialized
* (e.g. before driver bring-up) so callers default to the non-PerfIO path.
* True iff PerfIO initialized GCS support on this executor. Returns false when
* no {@link SparkEnv} is initialized (e.g. before driver bring-up) so callers
* default to the non-PerfIO path.
*/
public static boolean isGCSPerfEnabled() {
SparkEnv env = SparkEnv.get();
if (env == null) {
return false;
}
return env.conf().getBoolean(PerfIOConf.GCSPERF_ENABLED().key(), false);
return PerfIO.isGCSPerfEnabled();
}

}
Loading