Skip to content

[DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry - #12549

Draft
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:config-pass-to-native
Draft

[DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry#12549
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:config-pass-to-native

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

[DNM] Do Not Merge — for design discussion.

This PR fully replaces the hard-coded, string-based native conf key lists in GlutenConfig.getNativeSessionConf / getNativeBackendConf with a declarative registration mechanism, so each module declares its own native confs on demand instead of common code maintaining giant key lists.

The mechanism

  1. NativeConfRegistry (gluten-core): a central registry of conf keys to pass to native side, keyed by NativeScope:

    • RUNTIME: dynamic, passed each time a native runtime instance is created;
    • BACKEND: static, passed once during native backend initialization;
    • ALL: both.

    Runtime and backend scopes are tracked separately, so the same key can carry different semantics per scope. Each entry supports:

    • defaultToPass: if defined, the key is always passed, using the default value when unset by user (replaces the old "configs having default values" lists);
    • transform: applied to user-set values before passing (replaces the old per-key special cases such as byte-unit conversion for spark.shuffle.file.buffer and upper-casing for spark.sql.legacy.timeParserPolicy).
  2. ConfigBuilder.passToNative(): Gluten config entries declare native passing right at their definitions. No scope argument — the native scope is derived from the conf's staticness: static confs (buildStaticConf) go to BACKEND, dynamic confs (buildConf) go to RUNTIME:

val COLUMNAR_MAX_BATCH_SIZE =
  buildConf("spark.gluten.sql.columnar.maxBatchSize")
    .passToNative()   // dynamic conf -> passed on each native runtime creation
    .intConf
    .createWithDefault(4096)

val DEBUG_CUDF =
  buildStaticConf("spark.gluten.sql.debug.cudf")   // static conf -> passed at backend init
    .passToNative()
    .passDefault()    // native relies on the key: pass the default when unset
    .booleanConf
    .createWithDefault(false)

passDefault() marks that the entry's own default value (parsed form, e.g. a bytesConf default "1KB" is delivered as "1024") is passed even when the conf is not set by user.

Confs whose native consumption scope differs from the derived one (e.g. static velox cachePrefetchMinPct read on runtime creation; dynamic memory-sizing confs consumed at backend init) register explicitly via NativeConfRegistry.register at the end of their conf object body, with comments explaining the mismatch.

  1. NativeConfRegistry.register: for non-Gluten keys (SQLConf / Spark core / Hadoop / S3 / GCS) that have no Gluten ConfigEntry, and for the scope-mismatch cases above. Common registrations live in registerNativeConfs() in GlutenConfig; velox-only keys are registered from VeloxConfig in backends-velox instead of common code. Registrations are naturally modular: a backend's registrations only take effect when its conf object loads, so backend-specific keys never leak across deployments.

What is removed

  • The nativeKeys set (40+ hard-coded strings, including velox-specific keys living in common code) in GlutenConfig.
  • The two "configs having default values" Seqs and all per-key special-case code in getNativeSessionConf / getNativeBackendConf. Both methods now do: registry selection + existing prefix rules + UGI tokens (session only).
  • BackendSettingsApi.extraNativeSessionConfKeys / extraNativeBackendConfKeys (no backend ever overrode them).
  • GlutenConfigUtil.mapByteConfValue (superseded by entry-level transform).

Behavior compatibility

The selected key/value results of getNativeSessionConf / getNativeBackendConf are intended to be identical to before. Notable equivalence mappings:

Old code New declaration
nativeKeys filter .passToNative() on dynamic confs / register(key, RUNTIME)
session default-value Seq .passToNative().passDefault() / register(..., defaultToPass)
mapByteConfValue special cases register(..., transform = byte conversion)
timeParserPolicy.toUpperCase register(..., transform = upper-case)
backend default-value Seq .passToNative().passDefault() on static confs / register(..., BACKEND, defaultToPass)
backend keys filter .passToNative() on static confs / register(key, BACKEND)
velox keys in common code declared in VeloxConfig (backends-velox)

Prefix-based rules (spark.gluten.sql.columnar.backend.<backend>, spark.hadoop.fs.s3a. etc.) are kept as-is, since they are pattern rules rather than string key registrations.

Staticness alignment with native consumption scope

Static confs are for native backend initialization (immutable afterwards); non-static confs are for native session. Confs whose declared staticness did not match their actual native consumption were fixed:

  1. Static but consumed per native runtime -> made non-static

    • spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan: read by native in WholeStageResultIterator / SubstraitToVeloxPlan on every query, not at backend init, so it is session-tunable now.
  2. Non-static but consumed only at native backend init -> made static

    • spark.gluten.velox.awsSdkLogLevel, spark.gluten.velox.fs.s3a.retry.mode: only read when building the reused native HiveConnector at backend init; session changes never took effect.
    • spark.gluten.memoryOverhead.size.in.bytes: only set on SparkConf at driver start (VeloxListenerApi), consumed by Velox backend init for global memory sizing.
    • spark.gluten.velox.s3UseProxyFromEnv, spark.gluten.velox.s3PayloadSigningPolicy: promoted from raw string registrations to static ConfigEntrys in VeloxConfig.
  3. Non-static and consumed by both backend init and session -> allowed, dual-registered and documented in code

Conf Backend-init consumption Session consumption Special note
spark.gluten.sql.debug keep user glog levels per-task debug dumps log level frozen at startup; session toggle only affects per-query dumps
spark.gluten.sql.columnar.cudf one-time GPU env init per-query CPU/GPU offload decision enabling in session without startup enablement will not initialize GPU
spark.gluten.memory.task.offHeap.size.in.bytes CH external sort/agg settings Velox per-task spill limit mutated at runtime by GlutenAutoAdjustStageResourceProfile
spark.gluten.memory.offHeap.size.in.bytes CH spill thresholds (JVM-side only) must stay non-static for resource-profile adjustment; BACKEND-only native registration
spark.gluten.numTaskSlotsPerExecutor Velox io/spill thread sizing (key must always be present, default passed when unset) (JVM-side only) same as above

Design document

docs/developers/NativeConfPassing.md documents the full design: the two delivery channels (BACKEND at native backend init, RUNTIME at native runtime creation), scope derivation from conf staticness, the passToNative()/passDefault() builder API, explicit NativeConfRegistry.register for non-Gluten keys, prefix rules, and the enumerated dual-scope confs with their split semantics.

How was this patch tested?

  • New/extended NativeConfRegistrySuite covering: scope derivation from conf staticness, parsed-default registration for bytes confs, passDefault constraint checks, per-scope mixed semantics, duplicate registration rejection, defaultToPass and transform semantics.
  • Existing config suites in gluten-core / gluten-substrait pass.
  • Compiled gluten-core, gluten-substrait, backends-velox with -Pspark-3.5,backends-velox (checkstyle/scalastyle/spotless clean). backends-clickhouse Java code compiles; the delta33 Scala compile failure on this machine is pre-existing and unrelated.

AI disclosure: this PR was developed with the assistance of Cursor (Fable 5), with human review of all changes.

@github-actions github-actions Bot added the CORE works for Gluten Core label Jul 17, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions github-actions Bot added the VELOX label Jul 17, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@jackylee-ch jackylee-ch changed the title [DNM][CORE] Introduce ConfigBuilder.passToNative to declaratively register native confs [DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry Jul 17, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

2 similar comments
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions github-actions Bot added the DOCS label Jul 18, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

2 similar comments
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

…ve NativeConfRegistry

Which configurations get delivered from JVM to native side used to be decided by
hard-coded string lists inside GlutenConfig.getNativeSessionConf /
getNativeBackendConf: a `nativeKeys` set of 40+ raw keys, two "configs having
default values" Seqs duplicating each conf's key and default, and per-key
special cases inline in the selection methods. Adding a native conf meant
editing central lists far from the conf's definition, defaults could drift, and
backend-specific keys (Velox S3 keys) lived in common code.

This replaces those lists with a declarative mechanism where each module
declares its own native confs.

## The mechanism

NativeConfRegistry (gluten-core) tracks conf keys per NativeScope:

  - RUNTIME: delivered on each native runtime creation (getNativeSessionConf)
  - BACKEND: delivered once at native backend initialization
    (getNativeBackendConf)
  - ALL: both

Runtime and backend scopes are tracked separately, so one key can carry
different semantics per scope. Each entry supports `defaultToPass` (deliver the
default when the user did not set the conf - replaces the old default-value
Seqs) and `transform` (normalize user-set values - replaces the byte-unit and
upper-casing special cases).

Gluten's own confs declare delivery at their definition:

  val COLUMNAR_MAX_BATCH_SIZE =
    buildConf("spark.gluten.sql.columnar.maxBatchSize")
      .passToNative()   // dynamic -> RUNTIME
      .intConf
      .createWithDefault(4096)

  val DEBUG_CUDF =
    buildStaticConf("spark.gluten.sql.debug.cudf")   // static -> BACKEND
      .passToNative()
      .passDefault()    // native relies on the key: pass the default when unset
      .booleanConf
      .createWithDefault(false)

The scope is derived from staticness: a static conf cannot change after startup,
so delivering it once at backend init is lossless; a dynamic conf must be
re-delivered per runtime. The derivation deliberately does not emit both scopes
for dynamic confs - that would make the rule carry no information and remove the
review pressure to justify each dual-scope case. Confs consumed by both sides
keep the derived RUNTIME registration and add an explicit BACKEND one, with a
comment stating why.

Spark / Hadoop keys that have no Gluten ConfigEntry are registered explicitly
via NativeConfRegistry.register, from the conf object of the module that needs
them: common ones in GlutenConfig, Velox-only ones in VeloxConfig, and the
ClickHouse-only ones (spark.hadoop.input.*.timeout,
spark.hadoop.dfs.client.log.severity, spark.sql.orc.compression.codec - only
CHTransformerApi consumes them) in CHConfig.

## Extensibility

A conf object is a Scala object, so its registrations only happen once something
touches it. Component.confs(): Seq[ConfigRegistry] lets a component declare its
conf objects; Gluten initializes them right after component discovery, before
any component's onDriverStart / onExecutorStart. This is the only entry point
early enough for the BACKEND channel: backends are root nodes of the component
DAG, so a backend's onDriverStart - where native backend init happens - runs
before any dependent component's. Only runtime-compatible components are
visited, so an excluded component's confs never reach native side. VeloxBackend
and CHBackend declare VeloxConfig / CHConfig through the hook.

ConfigRegistry.ensureRegistered() is the supported way to force a conf object's
initialization; a reference to a constant val can be constant-folded away by the
compiler, which would silently drop a module's registrations.
ConfigRegistry.get now has a default implementation so a conf object that only
declares configurations does not have to implement an accessor it never uses.

The whole chain works from outside the org.apache.gluten package: ConfigRegistry,
ConfigEntry, NativeConfRegistry and NativeScope are public, buildConf /
buildStaticConf are protected members of ConfigRegistry. Verified by compiling a
third-party component and conf object against gluten-core.

## Staticness alignment with native consumption scope

Confs whose declared staticness did not match where native consumes them were
fixed:

  - spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan: static ->
    dynamic. Read per query from the runtime conf (WholeStageResultIterator,
    SubstraitToVeloxPlan, VeloxRuntime), not at backend init.
  - spark.gluten.memoryOverhead.size.in.bytes: dynamic -> static. Set on
    SparkConf at driver start and consumed by Velox backend init only.
  - spark.gluten.velox.s3UseProxyFromEnv / s3PayloadSigningPolicy: promoted from
    raw string registrations to ConfigEntry in VeloxConfig.

Three confs stay non-static although native reads two of them at backend init
only: GlutenAutoAdjustStageResourceProfile rewrites
spark.gluten.numTaskSlotsPerExecutor, spark.gluten.memory.offHeap.size.in.bytes
and spark.gluten.memory.task.offHeap.size.in.bytes on SQLConf per stage, and
JVM-side readers observe the rewritten values. (Note SQLConf.setConfString does
not itself reject static keys - that guard lives in RuntimeConfig.set and SET -
but declaring these static would assert an immutability that does not hold.)

## What is removed

  - The nativeKeys set, the two default-value Seqs, and all per-key special
    cases in the two selection methods. Both now do: registry selection +
    existing prefix rules + UGI tokens (session only).
  - BackendSettingsApi.extraNativeSessionConfKeys / extraNativeBackendConfKeys.
    Added shortly before this change and never overridden by any backend;
    NativeConfRegistry supersedes them and, unlike a Set[String], can express
    per-channel scope, defaults and normalization. An out-of-tree backend
    migrates by declaring its conf object through Component.confs().
  - GlutenConfigUtil.mapByteConfValue, superseded by per-entry transform.
  - spark.gluten.velox.fs.s3a.retry.mode, which has had no native reader since
    apache#8123 moved the S3 config path to velox's S3Config: native reads the retry
    mode from spark.hadoop.fs.s3a.retry.mode.

Prefix rules (spark.gluten.sql.columnar.backend.<backend>,
spark.hadoop.fs.s3a. etc.) are kept as-is: they are pattern rules covering
open-ended key families rather than enumerable registrations.

## Behavior compatibility

The selected key/value results of both methods are identical to before,
including every default value and value normalization. Notable equivalences:

  | Old code                       | New declaration                          |
  |--------------------------------|------------------------------------------|
  | nativeKeys filter              | .passToNative() / register(key, RUNTIME)  |
  | session default-value Seq      | .passDefault() / register(..., default)   |
  | mapByteConfValue special cases | register(..., transform = ...)            |
  | timeParserPolicy.toUpperCase   | register(..., transform = ...)            |
  | backend default-value Seq      | .passDefault() on static confs            |
  | backend keys filter            | .passToNative() / register(key, BACKEND)  |
  | Velox keys in common code      | declared in VeloxConfig                   |

One behavior note worth calling out: awsSdkLogLevel, s3UseProxyFromEnv and
s3PayloadSigningPolicy remain dynamic rather than becoming static, because
native createHiveConnectorConfig is not backend-init-only - it also runs per
write on the runtime conf map (VeloxParquetDataSourceS3::initSink,
IcebergWriter). They keep the RUNTIME registration so a session-level change
still applies to subsequent writes, and add a BACKEND registration for their
defaults.

## Design document

docs/developers/NativeConfPassing.md documents the two delivery channels, the
scope derivation and why it stops where it does, the builder API, explicit
registration, the Component.confs() extension point with a copyable example,
prefix rules and registry-vs-prefix precedence, object initialization, the
dual-scope confs with their split semantics, and the pre-existing S3 read/write
default asymmetry.

## How was this patch tested?

  - NativeConfRegistrySuite (gluten-core): the registry mechanism - scope
    derivation from staticness, parsed-default registration for bytes confs,
    passDefault constraint checks, per-scope mixed semantics, duplicate
    registration rejection, defaultToPass and transform semantics.
  - NativeConfPassingSuite (gluten-substrait, new): the delivered result end to
    end - what getNativeSessionConf / getNativeBackendConf actually select,
    including byte-string normalization (spark.shuffle.file.buffer=32k ->
    "32768"), per-channel scoping, always-present defaults and prefix rules.
    This restores the byte-string coverage lost with GlutenConfigUtilSuite.
  - ComponentSuite (gluten-core): Component.confs() defaults to empty and is
    overridable, and initializing a component's conf objects registers their
    native confs into the expected channels.
  - Both generated config doc golden files verified up to date (GLUTEN_UPDATE=0
    AllGlutenConfiguration / AllVeloxConfiguration).
  - gluten-core, gluten-substrait, backends-velox compile clean with
    -Pspark-3.5,backends-velox (scalastyle / spotless / scalafmt clean).
    backends-clickhouse cannot be compiled locally due to a pre-existing
    breakage unrelated to this change (src-delta33 needs Scala 2.13;
    LazyAggregateExpandRule predates ColumnarShuffleExchangeExec gaining
    mapperStageMode on main), so its coverage relies on the ClickHouse CI.
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant