[SPARK-51611][SQL] New iteration of single-pass Analyzer functionality

[vladimirg-db]

What changes were proposed in this pull request?

New single-pass Analyzer features:

• GROUP BY
• ORDER BY
• JOIN
• EXCEPT
• INTERSECT
• Correlated subqueries
• Other small features and bugfixes

Why are the changes needed?

To replace the fixed-point Analyzer with the single-pass one.

Does this PR introduce any user-facing change?

No, single-pass Analyzer is not yet enabled by default.

How was this patch tested?

Logical plans were compared in tests with ANALYZER_DUAL_RUN_LEGACY_AND_SINGLE_PASS_RESOLVER.

Was this patch authored or co-authored using generative AI tooling?

Yes.

[vladimirg-db]

Thanks @mihailoale-db and @mihailotim-db for this work!

[cloud-fan]

LGTM if CI is green

[vladimirg-db]

Thanks @mihailom-db for EXCEPT/INTERSECT!

[cloud-fan]

The test failures seem to be valid

[vladimirg-db]

Yeah, sorry... let me fix that real quick...

[vladimirg-db]

@cloud-fan tests passed!

[cloud-fan]

thanks, merging to master!

[iwanttobepowerful]

@vladimirg-db hi, does new Analyzer's principle similar to calcite's validator?

[vladimirg-db]

Hello @iwanttobepowerful. I never worked with Calcite.

The new Analyzer processes the logical plan in one* bottom-up traversal. For example, expression IDs get propagated bottom-up (ExpressionIdAssigner) or subqueries can find outer references from already resolved subtrees. That means that sometimes we have to build additional data-structures as we descend to leaf nodes before starting the bottom-up analysis, e.g. for CTE resolution we build a stack of lookup scopes.

• We have one pass for the main algebra (95% of the Analysis), but there is also one pre-pass to resolve the metadata (blocking calls), and a couple of rules running after the Analysis before the Optimizer to stay compatible with the old Analyzer in PlanRewriter (e.g. CleanupAliases).

[iwanttobepowerful]

Hello @iwanttobepowerful. I never worked with Calcite.

The new Analyzer processes the logical plan in one* bottom-up traversal. For example, expression IDs get propagated bottom-up (ExpressionIdAssigner) or subqueries can find outer references from already resolved subtrees. That means that sometimes we have to build additional data-structures as we descend to leaf nodes before starting the bottom-up analysis, e.g. for CTE resolution we build a stack of lookup scopes.

• We have one pass for the main algebra (95% of the Analysis), but there is also one pre-pass to resolve the metadata (blocking calls), and a couple of rules running after the Analysis before the Optimizer to stay compatible with the old Analyzer in PlanRewriter (e.g. CleanupAliases).

thanks!!! very nice