fix: translate SQL wildcards in SIMILAR TO patterns (#22263)

[oc7o]

SIMILAR TO previously passed the pattern straight to Arrow's regex engine, so SQL wildcards were never translated and matches were unanchored:

SELECT 'abc' SIMILAR TO 'a%';  -- returned false
SELECT 'x'   SIMILAR TO '_';   -- returned false

Translate % to (?s:.*) and _ to (?s:.) (dot-all so they match newlines), then wrap the pattern in ^(?:...)$ so the regex matches the entire string. ., ^, $, and \ are escaped as SQL literals. The POSIX metacharacters that SIMILAR TO defines (| * + ? ( ) { } [ ]) pass through to the regex unchanged.

Which issue does this PR close?

• Closes PostgreSQL compatibility: SIMILAR TO should treat % as a wildcard #22263.

Rationale for this change

SIMILAR TO is a SQL standard operator with well-defined wildcard semantics (% = any sequence, _ = single character, full-string match). DataFusion's previous behavior silently produced wrong results for the most basic patterns, which is a correctness bug for anyone porting queries from Postgres or other SQL engines.

What changes are included in this PR?

• Added sql_similar_to_regex helper that translates %/_ and anchors the pattern with ^(?:...)$. It tracks bracket state so ^ inside [...] is treated as bracket negation, not as a literal.
• Added a new SqlSimilarToPattern physical expression in datafusion/physical-expr/src/expressions/similar_to_pattern.rs that applies the translation at runtime for non-literal patterns.
• similar_to() now translates literal Utf8 / LargeUtf8 / Utf8View patterns at planning time and wraps non-literal patterns in SqlSimilarToPattern for runtime translation.
• NULL patterns pass through and return NULL instead of crashing with an internal error.
• Restored a plan-time type check in datafusion/sql/src/expr/mod.rs that rejects non-string patterns with a clean plan_err!.
• Runtime type errors in SqlSimilarToPattern are reported as exec_err! rather than internal_err!.

Are these changes tested?

Yes:

• test_similar_to_sql_literal_metachars confirms that ., ^, $, and \ are treated as SQL literals, not as regex operators.
• test_similar_to_posix_metachars confirms that |, *, +, ?, (, ), {, }, [, ], [^...], and [a-z] behave as SIMILAR TO metacharacters.
• test_similar_to_wildcards_match_newlines confirms that % and _ match newlines.
• test_similar_to covers basic %/_ semantics, full-string anchoring, and case sensitivity.
• test_similar_to_dynamic_pattern covers column-based patterns.
• test_similar_to_null_pattern and test_similar_to_non_literal_pattern_errors cover the NULL pattern and non-string literal pattern paths.
• End-to-end coverage in datafusion/sqllogictest/test_files/strings.slt was updated: the pre-existing SIMILAR TO 'p[12].*' / NOT SIMILAR TO 'p[12].*' cases (which only worked because of the bug) were removed, and new cases cover literal metachars, POSIX metachars, newline matching, dynamic column patterns, dynamic function patterns, SELECT 'a' SIMILAR TO NULL, and rejection of SELECT 'a' SIMILAR TO 1.

Are there any user-facing changes?

Yes:

• SIMILAR TO now produces correct results for queries that were previously returning wrong answers.
• Queries that relied on the buggy behavior (e.g., treating ., ^, or $ as regex metacharacters) now follow standard SQL SIMILAR TO semantics.
• Queries using valid SIMILAR TO POSIX metacharacters (| * + ? ( ) { } [ ]) now work as expected.
• % and _ wildcards now work.
• Non-literal patterns (e.g., column patterns) now work instead of returning a not_impl_err!.
• SELECT ... SIMILAR TO NULL now returns NULL instead of crashing.
• SELECT ... SIMILAR TO <non-string> now returns a clean plan error instead of an internal error.

[oc7o]

@huaxingao @viirya @wesm Could one of you trigger CI for me please? Thanks!

[viirya]

@oc7o Triggered. CI is running now.

[kosiew]

@oc7o
Thanks for working on this. The direction looks good, but I think there are still a couple of correctness issues in the SIMILAR TO translation that should be addressed before this can be merged. I also have one small test coverage suggestion.

[oc7o]

@kosiew I gotta thank you for the effort you put into this. It was really with the eye for detail andI really learned a lot! 😊

I think with how it now is we're a bit closer to the standard SQL implementation. Maybe (when this PR is ready 🤞) escaping could be a good follow up topic for me. Since we currently still treat \ as a literal. So that 'a%' SIMILAR TO 'a\%' then returns true. Currently it would return false.

I'm curios for any responses!

[kosiew]

@oc7o
Thanks for the updates here. The literal-pattern handling looks much better now: the regex-only literal characters are escaped, % and _ handle newlines, and the added Rust and SLT coverage is helpful.

I do still see one regression around dynamic pattern expressions. SIMILAR TO previously allowed the right-hand side to be a normal expression, but this PR now rejects non-literal patterns during planning. Could you please address that before this lands?

[oc7o]

@kosiew

Thx again for the review. I learn so much new stuff when working on this!

I now added support for dynamic values on the RHS alongside some bug fixes like for the SIMILAR TO NULL expression.

One issue still is that datafusion-proto can't serialize SqlSimilarToPattern but I would argue that this is out of scope for this PR.

Also a decision I consciously made was disregarding strings inside Dictionary and RunEndEncoded for now.

🐙