[DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry - #12549
Draft
jackylee-ch wants to merge 1 commit into
Draft
[DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry#12549jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
2 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
2 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
…ve NativeConfRegistry
Which configurations get delivered from JVM to native side used to be decided by
hard-coded string lists inside GlutenConfig.getNativeSessionConf /
getNativeBackendConf: a `nativeKeys` set of 40+ raw keys, two "configs having
default values" Seqs duplicating each conf's key and default, and per-key
special cases inline in the selection methods. Adding a native conf meant
editing central lists far from the conf's definition, defaults could drift, and
backend-specific keys (Velox S3 keys) lived in common code.
This replaces those lists with a declarative mechanism where each module
declares its own native confs.
## The mechanism
NativeConfRegistry (gluten-core) tracks conf keys per NativeScope:
- RUNTIME: delivered on each native runtime creation (getNativeSessionConf)
- BACKEND: delivered once at native backend initialization
(getNativeBackendConf)
- ALL: both
Runtime and backend scopes are tracked separately, so one key can carry
different semantics per scope. Each entry supports `defaultToPass` (deliver the
default when the user did not set the conf - replaces the old default-value
Seqs) and `transform` (normalize user-set values - replaces the byte-unit and
upper-casing special cases).
Gluten's own confs declare delivery at their definition:
val COLUMNAR_MAX_BATCH_SIZE =
buildConf("spark.gluten.sql.columnar.maxBatchSize")
.passToNative() // dynamic -> RUNTIME
.intConf
.createWithDefault(4096)
val DEBUG_CUDF =
buildStaticConf("spark.gluten.sql.debug.cudf") // static -> BACKEND
.passToNative()
.passDefault() // native relies on the key: pass the default when unset
.booleanConf
.createWithDefault(false)
The scope is derived from staticness: a static conf cannot change after startup,
so delivering it once at backend init is lossless; a dynamic conf must be
re-delivered per runtime. The derivation deliberately does not emit both scopes
for dynamic confs - that would make the rule carry no information and remove the
review pressure to justify each dual-scope case. Confs consumed by both sides
keep the derived RUNTIME registration and add an explicit BACKEND one, with a
comment stating why.
Spark / Hadoop keys that have no Gluten ConfigEntry are registered explicitly
via NativeConfRegistry.register, from the conf object of the module that needs
them: common ones in GlutenConfig, Velox-only ones in VeloxConfig, and the
ClickHouse-only ones (spark.hadoop.input.*.timeout,
spark.hadoop.dfs.client.log.severity, spark.sql.orc.compression.codec - only
CHTransformerApi consumes them) in CHConfig.
## Extensibility
A conf object is a Scala object, so its registrations only happen once something
touches it. Component.confs(): Seq[ConfigRegistry] lets a component declare its
conf objects; Gluten initializes them right after component discovery, before
any component's onDriverStart / onExecutorStart. This is the only entry point
early enough for the BACKEND channel: backends are root nodes of the component
DAG, so a backend's onDriverStart - where native backend init happens - runs
before any dependent component's. Only runtime-compatible components are
visited, so an excluded component's confs never reach native side. VeloxBackend
and CHBackend declare VeloxConfig / CHConfig through the hook.
ConfigRegistry.ensureRegistered() is the supported way to force a conf object's
initialization; a reference to a constant val can be constant-folded away by the
compiler, which would silently drop a module's registrations.
ConfigRegistry.get now has a default implementation so a conf object that only
declares configurations does not have to implement an accessor it never uses.
The whole chain works from outside the org.apache.gluten package: ConfigRegistry,
ConfigEntry, NativeConfRegistry and NativeScope are public, buildConf /
buildStaticConf are protected members of ConfigRegistry. Verified by compiling a
third-party component and conf object against gluten-core.
## Staticness alignment with native consumption scope
Confs whose declared staticness did not match where native consumes them were
fixed:
- spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan: static ->
dynamic. Read per query from the runtime conf (WholeStageResultIterator,
SubstraitToVeloxPlan, VeloxRuntime), not at backend init.
- spark.gluten.memoryOverhead.size.in.bytes: dynamic -> static. Set on
SparkConf at driver start and consumed by Velox backend init only.
- spark.gluten.velox.s3UseProxyFromEnv / s3PayloadSigningPolicy: promoted from
raw string registrations to ConfigEntry in VeloxConfig.
Three confs stay non-static although native reads two of them at backend init
only: GlutenAutoAdjustStageResourceProfile rewrites
spark.gluten.numTaskSlotsPerExecutor, spark.gluten.memory.offHeap.size.in.bytes
and spark.gluten.memory.task.offHeap.size.in.bytes on SQLConf per stage, and
JVM-side readers observe the rewritten values. (Note SQLConf.setConfString does
not itself reject static keys - that guard lives in RuntimeConfig.set and SET -
but declaring these static would assert an immutability that does not hold.)
## What is removed
- The nativeKeys set, the two default-value Seqs, and all per-key special
cases in the two selection methods. Both now do: registry selection +
existing prefix rules + UGI tokens (session only).
- BackendSettingsApi.extraNativeSessionConfKeys / extraNativeBackendConfKeys.
Added shortly before this change and never overridden by any backend;
NativeConfRegistry supersedes them and, unlike a Set[String], can express
per-channel scope, defaults and normalization. An out-of-tree backend
migrates by declaring its conf object through Component.confs().
- GlutenConfigUtil.mapByteConfValue, superseded by per-entry transform.
- spark.gluten.velox.fs.s3a.retry.mode, which has had no native reader since
apache#8123 moved the S3 config path to velox's S3Config: native reads the retry
mode from spark.hadoop.fs.s3a.retry.mode.
Prefix rules (spark.gluten.sql.columnar.backend.<backend>,
spark.hadoop.fs.s3a. etc.) are kept as-is: they are pattern rules covering
open-ended key families rather than enumerable registrations.
## Behavior compatibility
The selected key/value results of both methods are identical to before,
including every default value and value normalization. Notable equivalences:
| Old code | New declaration |
|--------------------------------|------------------------------------------|
| nativeKeys filter | .passToNative() / register(key, RUNTIME) |
| session default-value Seq | .passDefault() / register(..., default) |
| mapByteConfValue special cases | register(..., transform = ...) |
| timeParserPolicy.toUpperCase | register(..., transform = ...) |
| backend default-value Seq | .passDefault() on static confs |
| backend keys filter | .passToNative() / register(key, BACKEND) |
| Velox keys in common code | declared in VeloxConfig |
One behavior note worth calling out: awsSdkLogLevel, s3UseProxyFromEnv and
s3PayloadSigningPolicy remain dynamic rather than becoming static, because
native createHiveConnectorConfig is not backend-init-only - it also runs per
write on the runtime conf map (VeloxParquetDataSourceS3::initSink,
IcebergWriter). They keep the RUNTIME registration so a session-level change
still applies to subsequent writes, and add a BACKEND registration for their
defaults.
## Design document
docs/developers/NativeConfPassing.md documents the two delivery channels, the
scope derivation and why it stops where it does, the builder API, explicit
registration, the Component.confs() extension point with a copyable example,
prefix rules and registry-vs-prefix precedence, object initialization, the
dual-scope confs with their split semantics, and the pre-existing S3 read/write
default asymmetry.
## How was this patch tested?
- NativeConfRegistrySuite (gluten-core): the registry mechanism - scope
derivation from staticness, parsed-default registration for bytes confs,
passDefault constraint checks, per-scope mixed semantics, duplicate
registration rejection, defaultToPass and transform semantics.
- NativeConfPassingSuite (gluten-substrait, new): the delivered result end to
end - what getNativeSessionConf / getNativeBackendConf actually select,
including byte-string normalization (spark.shuffle.file.buffer=32k ->
"32768"), per-channel scoping, always-present defaults and prefix rules.
This restores the byte-string coverage lost with GlutenConfigUtilSuite.
- ComponentSuite (gluten-core): Component.confs() defaults to empty and is
overridable, and initializing a component's conf objects registers their
native confs into the expected channels.
- Both generated config doc golden files verified up to date (GLUTEN_UPDATE=0
AllGlutenConfiguration / AllVeloxConfiguration).
- gluten-core, gluten-substrait, backends-velox compile clean with
-Pspark-3.5,backends-velox (scalastyle / spotless / scalafmt clean).
backends-clickhouse cannot be compiled locally due to a pre-existing
breakage unrelated to this change (src-delta33 needs Scala 2.13;
LazyAggregateExpandRule predates ColumnarShuffleExchangeExec gaining
mapperStageMode on main), so its coverage relies on the ClickHouse CI.
jackylee-ch
force-pushed
the
config-pass-to-native
branch
from
July 27, 2026 11:08
0fb8466 to
527243f
Compare
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
[DNM] Do Not Merge — for design discussion.
This PR fully replaces the hard-coded, string-based native conf key lists in
GlutenConfig.getNativeSessionConf/getNativeBackendConfwith a declarative registration mechanism, so each module declares its own native confs on demand instead of common code maintaining giant key lists.The mechanism
NativeConfRegistry(gluten-core): a central registry of conf keys to pass to native side, keyed byNativeScope:RUNTIME: dynamic, passed each time a native runtime instance is created;BACKEND: static, passed once during native backend initialization;ALL: both.Runtime and backend scopes are tracked separately, so the same key can carry different semantics per scope. Each entry supports:
defaultToPass: if defined, the key is always passed, using the default value when unset by user (replaces the old "configs having default values" lists);transform: applied to user-set values before passing (replaces the old per-key special cases such as byte-unit conversion forspark.shuffle.file.bufferand upper-casing forspark.sql.legacy.timeParserPolicy).ConfigBuilder.passToNative(): Gluten config entries declare native passing right at their definitions. No scope argument — the native scope is derived from the conf's staticness: static confs (buildStaticConf) go toBACKEND, dynamic confs (buildConf) go toRUNTIME:passDefault()marks that the entry's own default value (parsed form, e.g. abytesConfdefault"1KB"is delivered as"1024") is passed even when the conf is not set by user.Confs whose native consumption scope differs from the derived one (e.g. static velox
cachePrefetchMinPctread on runtime creation; dynamic memory-sizing confs consumed at backend init) register explicitly viaNativeConfRegistry.registerat the end of their conf object body, with comments explaining the mismatch.NativeConfRegistry.register: for non-Gluten keys (SQLConf / Spark core / Hadoop / S3 / GCS) that have no GlutenConfigEntry, and for the scope-mismatch cases above. Common registrations live inregisterNativeConfs()inGlutenConfig; velox-only keys are registered fromVeloxConfigin backends-velox instead of common code. Registrations are naturally modular: a backend's registrations only take effect when its conf object loads, so backend-specific keys never leak across deployments.What is removed
nativeKeysset (40+ hard-coded strings, including velox-specific keys living in common code) inGlutenConfig.Seqs and all per-key special-case code ingetNativeSessionConf/getNativeBackendConf. Both methods now do: registry selection + existing prefix rules + UGI tokens (session only).BackendSettingsApi.extraNativeSessionConfKeys/extraNativeBackendConfKeys(no backend ever overrode them).GlutenConfigUtil.mapByteConfValue(superseded by entry-leveltransform).Behavior compatibility
The selected key/value results of
getNativeSessionConf/getNativeBackendConfare intended to be identical to before. Notable equivalence mappings:nativeKeysfilter.passToNative()on dynamic confs /register(key, RUNTIME)Seq.passToNative().passDefault()/register(..., defaultToPass)mapByteConfValuespecial casesregister(..., transform = byte conversion)timeParserPolicy.toUpperCaseregister(..., transform = upper-case)Seq.passToNative().passDefault()on static confs /register(..., BACKEND, defaultToPass)keysfilter.passToNative()on static confs /register(key, BACKEND)VeloxConfig(backends-velox)Prefix-based rules (
spark.gluten.sql.columnar.backend.<backend>,spark.hadoop.fs.s3a.etc.) are kept as-is, since they are pattern rules rather than string key registrations.Staticness alignment with native consumption scope
Static confs are for native backend initialization (immutable afterwards); non-static confs are for native session. Confs whose declared staticness did not match their actual native consumption were fixed:
Static but consumed per native runtime -> made non-static
spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan: read by native inWholeStageResultIterator/SubstraitToVeloxPlanon every query, not at backend init, so it is session-tunable now.Non-static but consumed only at native backend init -> made static
spark.gluten.velox.awsSdkLogLevel,spark.gluten.velox.fs.s3a.retry.mode: only read when building the reused native HiveConnector at backend init; session changes never took effect.spark.gluten.memoryOverhead.size.in.bytes: only set on SparkConf at driver start (VeloxListenerApi), consumed by Velox backend init for global memory sizing.spark.gluten.velox.s3UseProxyFromEnv,spark.gluten.velox.s3PayloadSigningPolicy: promoted from raw string registrations to staticConfigEntrys inVeloxConfig.Non-static and consumed by both backend init and session -> allowed, dual-registered and documented in code
spark.gluten.sql.debugspark.gluten.sql.columnar.cudfspark.gluten.memory.task.offHeap.size.in.bytesGlutenAutoAdjustStageResourceProfilespark.gluten.memory.offHeap.size.in.bytesspark.gluten.numTaskSlotsPerExecutorDesign document
docs/developers/NativeConfPassing.mddocuments the full design: the two delivery channels (BACKEND at native backend init, RUNTIME at native runtime creation), scope derivation from conf staticness, thepassToNative()/passDefault()builder API, explicitNativeConfRegistry.registerfor non-Gluten keys, prefix rules, and the enumerated dual-scope confs with their split semantics.How was this patch tested?
NativeConfRegistrySuitecovering: scope derivation from conf staticness, parsed-default registration for bytes confs, passDefault constraint checks, per-scope mixed semantics, duplicate registration rejection, defaultToPass and transform semantics.-Pspark-3.5,backends-velox(checkstyle/scalastyle/spotless clean). backends-clickhouse Java code compiles; the delta33 Scala compile failure on this machine is pre-existing and unrelated.AI disclosure: this PR was developed with the assistance of Cursor (Fable 5), with human review of all changes.