Documentation: Plan custom expressions #15353
Open
+83
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
documentation
alamb
approved these changes
Mar 22, 2025
There was a problem hiding this comment.
Thank you so much @Jiashu-Hu -- this looks great.
I have some suggestions on how to improve this section and example, but we could also do it as a follow on PR.
It is great to see the docs being improved
| @@ -1160,6 +1160,89 @@ async fn main() -> Result<()> { | |||
| // +---+ | |||
| ``` | |||
|
|
|||
| ## Custom Expression Planning | |||
|
|
|||
| DataFusion provides native support for a limited set of SQL operators by default. For operators not natively defined, developers can extend DataFusion's functionality by implementing custom expression planning. This extensibility is a core feature of DataFusion, allowing it to be customized for particular workloads and requirements. | |||
There was a problem hiding this comment.
Here are some wordsmithing suggestions:
Suggested change
| DataFusion provides native support for a limited set of SQL operators by default. For operators not natively defined, developers can extend DataFusion's functionality by implementing custom expression planning. This extensibility is a core feature of DataFusion, allowing it to be customized for particular workloads and requirements. | |
| DataFusion provides native support for common SQL operators by default such as `+`, `-`, `||`. However it does not provide support for other operators such as `@>`. To override DataFusion's default handling or support unsupported operators, developers can extend DataFusion by implementing custom expression planning, a core feature of DataFusion |
|
|
||
| 1. Implement the `ExprPlanner` trait: This allows you to define custom logic for planning expressions that DataFusion doesn't natively recognize. The trait provides the necessary interface to translate logical expressions into physical execution plans. | ||
|
|
||
| For a detailed documentation please see: [Trait ExprPlanner](https://docs.rs/datafusion/latest/datafusion/logical_expr/planner/trait.ExprPlanner.html) |
There was a problem hiding this comment.
Suggested change
| For a detailed documentation please see: [Trait ExprPlanner](https://docs.rs/datafusion/latest/datafusion/logical_expr/planner/trait.ExprPlanner.html) | |
| For detailed documentation please see: [Trait ExprPlanner](https://docs.rs/datafusion/latest/datafusion/logical_expr/planner/trait.ExprPlanner.html) |
|
|
||
| To extend DataFusion with support for custom operators not natively available, you need to: | ||
|
|
||
| 1. Implement the `ExprPlanner` trait: This allows you to define custom logic for planning expressions that DataFusion doesn't natively recognize. The trait provides the necessary interface to translate logical expressions into physical execution plans. |
There was a problem hiding this comment.
I think ExprPlanner actually converts the SQL AST to Exprs:
Suggested change
| 1. Implement the `ExprPlanner` trait: This allows you to define custom logic for planning expressions that DataFusion doesn't natively recognize. The trait provides the necessary interface to translate logical expressions into physical execution plans. | |
| 1. Implement the `ExprPlanner` trait: This allows you to define custom logic for planning expressions that DataFusion doesn't natively recognize. The trait provides the necessary interface to translate SQL AST nodes into logical `Expr`. |
|
|
||
| 2. Register your custom planner: Integrate your implementation with DataFusion's `SessionContext` to ensure your custom planning logic is invoked during the query optimization and execution planning phase. | ||
|
|
||
| For a detailed documentation please see: [fn register_expr_planner](https://docs.rs/datafusion/latest/datafusion/execution/trait.FunctionRegistry.html#method.register_expr_planner) |
There was a problem hiding this comment.
Suggested change
| For a detailed documentation please see: [fn register_expr_planner](https://docs.rs/datafusion/latest/datafusion/execution/trait.FunctionRegistry.html#method.register_expr_planner) | |
| For a detailed documentation see: [fn register_expr_planner](https://docs.rs/datafusion/latest/datafusion/execution/trait.FunctionRegistry.html#method.register_expr_planner) |
| # // Define the custom planner | ||
| # struct MyCustomPlanner; | ||
|
|
||
| // Implement ExprPlanner for cutom operator logic |
There was a problem hiding this comment.
Suggested change
| // Implement ExprPlanner for cutom operator logic | |
| // Implement ExprPlanner to add support for the `->` custom operator |
| ctx.register_expr_planner(Arc::new(MyCustomPlanner))?; | ||
| let results = ctx.sql("select 'foo'->'bar';").await?.collect().await?; | ||
|
|
||
| pretty::print_batches(&results)?; |
There was a problem hiding this comment.
Could you please change this to use `assert_batches_eq! so the actual output is in the test and it is tested in CI
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This PR adds documentation for using the
ExprPlannerAPI to plan custom expressions, as requested in #15267. Clear documentation with an example will help users extend DataFusion to support custom operators like PostgreSQL's->.What changes are included in this PR?
## Custom Expression Planningto documentExprPlannerdoes and why it’s useful.ExprPlannerto make the->operator concatenate strings (e.g.,'foo'->'bar'outputsfoobar).ExprPlannerandFunctionRegistry) for further reading.Are these changes tested?
yes
Are there any user-facing changes?
Yes, since this is an documentation chang