Skip to content

feat(bindings/python): typed service configs via from_config#7883

Open
chitralverma wants to merge 3 commits into
apache:mainfrom
chitralverma:python-typed-service-configs
Open

feat(bindings/python): typed service configs via from_config#7883
chitralverma wants to merge 3 commits into
apache:mainfrom
chitralverma:python-typed-service-configs

Conversation

@chitralverma

@chitralverma chitralverma commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Follow-up to #7824.

Rationale for this change

#7824 replaced pyo3-stub-gen with native PyO3 introspection and, in doing so, dropped the per-service constructor typing. Operators are currently built with an opaque Operator(scheme, **kwargs) — no editor completion, type checking, or discoverability of a service's config fields. There is also no typed, per-service configuration object. this is a necessary follow-up for restoration.

What changes are included in this PR?

  • Typed config classes. One config class per service (e.g. S3Config) is generated into opendal.services, wrapping the core config struct. Types and docstrings come from PyO3 introspection over the pyclasses, so there is no hand-maintained Rust→Python type mapping. Building through Operator::from_config means nested fields (lists, maps) and all core field types are supported. Each config binds its own scheme (not a settable argument), so it cannot be paired with the wrong service.

  • from_config constructor. Operator.from_config / AsyncOperator.from_config build an operator from a typed config:

    from opendal.services import S3Config
    
    op = opendal.Operator.from_config(S3Config(bucket="my-bucket"))
  • Consistent kwargs errors. Operator(scheme, **kwargs) and from_uri still accept string options only, but a non-string value now raises a clear error naming the option and pointing to from_config (which accepts native bool / int / os.PathLike), instead of a raw PyO3 TypeError.

  • MkDocs Inclusion. A single docs/api/services.md documents all config classes using mkdocstrings directive. The stub post-processor rewrites the generated __new__ to __init__ so the typed constructor signature renders.

  • Stubs are generated with the services-all feature so the committed stubs match released wheels.

  • Tests cover the config classes, from_config, and behavioral equivalence across all three construction paths (Operator(scheme, **kwargs), from_uri, from_config), which converge on the same core deserializer and produce identical operators.

How this is better than before

  • No duplicated per-operator surface. The typed constructor overloads previously had to be generated per service on both Operator and AsyncOperator (Rust + stub). That is collapsed to one config class per service, consumed by a single from_config on each operator — the typed definition lives in one place instead of being duplicated 2×.
  • No extra toolchain. Reuses the experimental-inspect path from refactor(bindings/python): generate type stubs with pyo3 introspection #7824 — no pyo3-stub-gen dependency, stub_gen bin, or #[gen_stub*] attributes.
  • Complex types work. Configurator::from_iter + Operator::from_config parse durations, enums, and map-valued fields that the flat string-map path cannot express.
  • Stub/runtime consistency. Stubs are generated from the same feature set as released wheels and guarded by a CI drift check.

Trade-offs: larger generated services.rs; tighter coupling to the core config structs.

Alternative considered: frozen Python dataclasses generated as .py (no Rust config pyclasses) — smaller and more loosely coupled, but requires a hand-maintained type mapping and cannot carry map-valued config fields.

Are there any user-facing changes?

Additive.

  • New Operator.from_config / AsyncOperator.from_config and stubs for opendal.services.*Config classes.
  • Operator(scheme, **kwargs) and from_uri keep the same behavior; only their error message for a non-string option value is improved.

AI Usage Statement

Assisted by an AI coding agent (Claude). All changes were reviewed by the author.

@chitralverma chitralverma force-pushed the python-typed-service-configs branch from 4907c54 to 4bd82ad Compare July 8, 2026 08:38
@chitralverma chitralverma force-pushed the python-typed-service-configs branch from 4bd82ad to 35a657b Compare July 8, 2026 09:18
@chitralverma chitralverma marked this pull request as ready for review July 8, 2026 11:59
Copilot AI review requested due to automatic review settings July 8, 2026 11:59
@chitralverma chitralverma requested a review from tisonkun as a code owner July 8, 2026 11:59
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Jul 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the OpenDAL Python bindings by introducing per-service typed configuration classes and a new from_config constructor path, restoring editor discoverability/typing that was lost when stub generation moved to PyO3 introspection.

Changes:

  • Generate opendal.services.<Service>Config pyclasses (one per service) and a ServiceConfig base class, backed by core *Config structs.
  • Add Operator.from_config / AsyncOperator.from_config, plus clearer errors when Operator(..., **kwargs) receives non-string option values.
  • Update stub generation + post-processing and add tests/docs to cover the new config-based construction flow.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
dev/src/generate/python.rs Extends Python binding codegen to emit typed service config classes and helper methods (signature ordering, field typing, docs, pickling map).
dev/src/generate/python.j2 Template updates to generate ServiceConfig + per-service *Config classes and export them via the services submodule.
bindings/python/src/operator.rs Adds from_config constructors, improves **kwargs type errors, and tracks picklability for operators built from map-valued configs.
bindings/python/src/lib.rs Exposes new config_types module and re-exports the generated services submodule.
bindings/python/src/config_types.rs Introduces PyPath to accept `str
bindings/python/scripts/postprocess_stubs.py Injects PathLike import for services stubs and rewrites config __new__ stubs into __init__ for MkDocs rendering.
bindings/python/ruff.toml Adds per-file ignores for generated docstring style issues in stubs.
bindings/python/justfile Generates stubs with --features services-all so committed stubs match wheel feature sets.
bindings/python/mkdocs.yml Adds a “Services” API page to MkDocs navigation.
bindings/python/docs/api/services.md New documentation page for typed service config classes (mkdocstrings over opendal.services).
bindings/python/tests/test_from_config.py New test suite for config classes, from_config, pickle behavior, and constructor-path equivalence.
bindings/python/python/opendal/services.pyi Updated generated stubs including all *Config classes and ServiceConfig.
bindings/python/python/opendal/operator.pyi Updated stubs to include from_config on both blocking and async operators.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dev/src/generate/python.rs
Comment thread dev/src/generate/python.rs
Comment thread dev/src/generate/python.rs
Comment thread dev/src/generate/python.rs
@chitralverma

Copy link
Copy Markdown
Contributor Author

@erickguan @Xuanwo please have a look when you get a chance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants