Skip to content

[VL] Support cudf.allowCpuFallback config for Velox cuDF backend#12451

Merged
marin-ma merged 5 commits into
apache:mainfrom
ReemaAlzaid:cudf-allow-cpu-fallback
Jul 7, 2026
Merged

[VL] Support cudf.allowCpuFallback config for Velox cuDF backend#12451
marin-ma merged 5 commits into
apache:mainfrom
ReemaAlzaid:cudf-allow-cpu-fallback

Conversation

@ReemaAlzaid

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Adds spark.gluten.sql.columnar.backend.velox.cudf.allowCpuFallback
(default true) and passes it through to Velox's CudfConfig at backend
init, letting users control CPU fallback for the cuDF/Velox backend.

How was this patch tested?

by using gluten-it, with this command:

sbin/gluten-it.sh queries-compare \
  --local --preset=velox --benchmark-type=ds --error-on-memleak \
  --off-heap-size=10g -s=1 --threads=1 --iterations=1 \
  --decimal-as-double=true \
  --extra-conf=spark.gluten.sql.columnar.backend.velox.cudf.allowCpuFallback=false \
  --extra-conf=spark.gluten.sql.columnar.cudf=true

@github-actions github-actions Bot added the VELOX label Jul 5, 2026
@ReemaAlzaid

Copy link
Copy Markdown
Contributor Author

@marin-ma

@marin-ma

marin-ma commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

@ReemaAlzaid To add a new user-exposed configuration, you also need to add an entry in VeloxConfig.scala, and run dev/gen-all-config-docs.sh to update the documentation.

@github-actions github-actions Bot added the DOCS label Jul 6, 2026
@ReemaAlzaid

Copy link
Copy Markdown
Contributor Author

Added them @marin-ma thanks

@marin-ma marin-ma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ReemaAlzaid


def cudfEnableValidation: Boolean = getConf(CUDF_ENABLE_VALIDATION)

def cudfAllowCpuFallback: Boolean = getConf(CUDF_ALLOW_CPU_FALLBACK)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not needed.

@marin-ma marin-ma merged commit bd2c1ce into apache:main Jul 7, 2026
53 of 55 checks passed
@ReemaAlzaid ReemaAlzaid deleted the cudf-allow-cpu-fallback branch July 7, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants