-
Notifications
You must be signed in to change notification settings - Fork 289
[DOC] add Databricks support matrix [skip ci] #15090
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
nvliyuan
wants to merge
1
commit into
NVIDIA:main
Choose a base branch
from
nvliyuan:docs-databricks-support-matrix
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| --- | ||
| layout: page | ||
| title: Databricks Support Matrix | ||
| nav_order: 4 | ||
| --- | ||
|
|
||
| # Databricks Support Matrix | ||
|
|
||
| This page summarizes the Databricks runtime combinations supported by the | ||
| current RAPIDS Accelerator release. Use it together with the | ||
| [Download](./download.md) page to select the correct plugin artifact before | ||
| deploying to a Databricks cluster. | ||
|
|
||
| ## Runtime Compatibility | ||
|
|
||
| The following matrix applies to RAPIDS Accelerator for Apache Spark v26.06.0. | ||
| Databricks runtime images provide the JVM and system libraries for each row; use | ||
| the runtime-provided JDK unless a Databricks support note explicitly instructs | ||
| otherwise. | ||
|
|
||
| | RAPIDS Accelerator | Databricks Runtime | Apache Spark | Scala | JDK runtime | CUDA jar variants | Minimum NVIDIA driver | Notes | | ||
| |--------------------|--------------------|--------------|-------|-------------|-------------------|-----------------------|-------| | ||
| | v26.06.0 | 13.3 ML LTS GPU | 3.4.1 | 2.12 | Databricks runtime default | CUDA 12, CUDA 13 | R525+ | Supported Databricks 13.3 runtime line. | | ||
| | v26.06.0 | 14.3 ML LTS GPU | 3.5.0 | 2.12 | Databricks runtime default | CUDA 12, CUDA 13 | R525+ | Supported Databricks 14.3 runtime line. | | ||
| | v26.06.0 | 17.3 ML LTS GPU | 4.0.0 | 2.13 | Databricks runtime default | CUDA 12, CUDA 13 | R525+ | Spark 4 / Scala 2.13 Databricks runtime line. | | ||
|
|
||
| The v26.06.0 download page publishes Scala 2.12 and Scala 2.13 artifacts for | ||
| both CUDA 12 and CUDA 13. Use the Scala artifact that matches the Databricks | ||
| runtime's Spark/Scala line. The CUDA classifier controls which RAPIDS native | ||
| libraries are bundled in the plugin jar; it does not change the Spark or Scala | ||
| compatibility of the artifact. | ||
|
|
||
| ## Delta Lake GPU Support on Databricks | ||
|
|
||
| Databricks runtimes use Databricks-specific Delta Lake implementations. The | ||
| following table summarizes Delta Lake GPU support for the Databricks runtime | ||
| lines with feature-specific coverage in the current release. `GPU` means the | ||
| operation is expected to run on the GPU when the rest of the query plan is also | ||
| GPU-compatible. `CPU fallback` means the RAPIDS Accelerator leaves that Delta | ||
| operation on the CPU for that runtime. | ||
|
|
||
| | Delta feature | DBR 14.3 | DBR 17.3 | | ||
| |---------------|----------|----------| | ||
| | Reads without deletion vectors | GPU | GPU | | ||
| | Deletion vector reads | CPU fallback | GPU only with metadata row index and RAPIDS deletion-vector predicate pushdown | | ||
| | Delta writes | GPU for append, overwrite, CTAS, and RTAS | GPU for append and overwrite. CTAS and RTAS fall back to CPU. | | ||
| | Delta writes with deletion vectors | CPU fallback | CPU fallback | | ||
| | DELETE and UPDATE | GPU for copy-on-write. Operations that write deletion vectors fall back to CPU. | Same as DBR 14.3. Liquid-clustered paths also fall back to CPU. | | ||
| | MERGE | GPU, including liquid clustering | GPU for non-liquid-clustered tables only. Liquid-clustered and persistent deletion-vector writes fall back to CPU. | | ||
| | OPTIMIZE | CPU fallback | GPU for standard deletion-vector-free, non-liquid-clustered tables only | | ||
| | Auto compaction | GPU when triggered by supported GPU writes | GPU for inline deletion-vector-free, non-liquid-clustered tables only | | ||
| | Liquid clustering | GPU support | CPU fallback | | ||
|
|
||
| ## Compatibility Caveats | ||
|
|
||
| - Databricks may patch existing runtime versions without changing the public | ||
| runtime line. Use a RAPIDS Accelerator release that explicitly lists the | ||
| Databricks runtime you plan to run. | ||
| - Runtime patch changes can surface as binary compatibility failures such as | ||
| `NoSuchMethodError` against Spark or Databricks-internal classes. If this | ||
| happens, first verify the RAPIDS Accelerator release and Databricks runtime | ||
| combination against this page and the release notes. | ||
| - Earlier v26.04.0 artifacts on Databricks 17.3 could encounter a | ||
| `NoSuchMethodError` on `CatalogTable.copy` in | ||
| `GpuCreateDataSourceTableAsSelectCommand`; use v26.04.1 or later for that | ||
| runtime line. | ||
| - Delta feature support is operation-specific. Even when a runtime is listed as | ||
| supported, individual Delta reads, writes, DML, `OPTIMIZE`, auto compaction, | ||
| deletion-vector, or liquid-clustering paths may still fall back to CPU as | ||
| shown above. | ||
| - `spark.rapids.sql.explain=NOT_ON_GPU` can be used to confirm whether a | ||
| particular query plan stayed on the GPU or fell back to CPU. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Runtime Compatibility section lists three supported runtimes (DBR 13.3, 14.3, and 17.3), but the Delta Lake GPU Support table only covers DBR 14.3 and DBR 17.3. Users deploying to DBR 13.3 — a listed and supported runtime — have no guidance on which Delta Lake operations run on the GPU vs. fall back to CPU, which is precisely the kind of ambiguity this document is intended to resolve.