Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

include some BinaryOperator from sqlparser #15327

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

waynexia
Copy link
Member

Which issue does this PR close?

Rationale for this change

Follows #15326, this patch includes Arrow, LongArrow, HashArrow, HashLongArrow, AtAt, IntegerDivide, HashMinus, AtQuestion, Question, QuestionAnd and QuestionPipe binary operators.

What changes are included in this PR?

New operators. Only XOR is actually implemented (functional) among them, by mapping to existing BitwiseXor.

Are these changes tested?

Compiler tests it

Are there any user-facing changes?

New APIs

@github-actions github-actions bot added sql SQL Planner logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates substrait Changes to the substrait crate labels Mar 20, 2025
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @waynexia -- I think the idea is great. However, I think there should be sql level tests (sqllogitests) that run these operators. I think it will make extending operators easier for other systems built on DataFusion

The tests can / should probably error with unsupported

It might also be a good idea to include some documentation in the operators themselves that DataFusion doesn't have default implementations

AtArrow | ArrowAt | Arrow | LongArrow | HashArrow | HashLongArrow | AtAt
| HashMinus | AtQuestion | Question | QuestionAnd | QuestionPipe
| IntegerDivide => {
unreachable!("These operators should be rewritten to functions")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think this code is unreachable -- I got it to panic:

DataFusion CLI v46.0.1
> select 'foo' @> 'bar';

thread 'main' panicked at datafusion/physical-expr/src/expressions/binary.rs:799:17:
internal error: entered unreachable code: These operators should be rewritten to functions
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized <@ and @> are only supported for lists 🤣

Signed-off-by: Ruihang Xia <[email protected]>
@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Mar 20, 2025
@waynexia
Copy link
Member Author

I think there should be sql level tests (sqllogitests) that run these operators

That's a good idea! I think I know them much better after writing some SQLs (though none of them can run... lol)

@waynexia
Copy link
Member Author

It might also be a good idea to include some documentation in the operators themselves that DataFusion doesn't have default implementations

Added in 5828cba 👍

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THanks @waynexia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sql SQL Planner sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants