Feature Summary
Inside a control block (LoopStart/LoopEnd), the iteration state — the loop's variables such as K — reaches every operator as a state message, but no built-in operator can use it. An operator's properties are read once when its executor is initialized, before any message has arrived, and the executor is built from a frozen descriptor JSON with no substitution stage. Today the only operator that reads a state key is If; a Filter inside a loop cannot take its threshold from the loop.
Let a property of any operator inside a block refer to a loop variable as $K, with no change to the operator's source.
Proposed Solution or Design
- Compile phase (unchanged wire format). The frontend sends
$K as the property value. When the logical plan is parsed, a rewriter records each $name reference on the operator (stateReferences: JSON pointer → variable name) and, for non-string fields, replaces the value with a typed placeholder so the descriptor still deserializes. The sidecar is hidden from the property-panel schema.
- Shared registration. When a state message arrives, the worker registers the state's key–value map in the operator base (
OperatorExecutor) before calling processState — the same for every operator, UDF or built-in.
- Late binding. An executor whose descriptor carries
stateReferences (or a literal $name string) is created as a late-bound wrapper: it starts with the placeholders unbound, binds them from the registered state when the state arrives (patching the descriptor and instantiating the real executor), and reuses the bound values for every tuple of the iteration. A tuple that arrives before the state, or a name missing from the state, is an error.
The Python operator base gets the same registration hook for symmetry. Out of scope here: operators whose Scala descriptor interpolates property values into generated Python code (the sklearn operators); their values never reach a Scala executor, so Python-side binding is a follow-up. UDF-side sugar and the frontend are separate issues (#8636).
Acceptance
- A
Filter inside a loop with predicate value $i filters with the current iteration's i, verified end to end in LoopIntegrationSpec.
- Unit tests for the plan rewriter (string, integer and nested-pointer references) and for the late-bound executor (bind at the first state, reuse, error when unbound / unknown name).
- No per-operator code changes.
Affected Area
Workflow Engine (Amber)
Feature Summary
Inside a control block (LoopStart/LoopEnd), the iteration state — the loop's variables such as
K— reaches every operator as a state message, but no built-in operator can use it. An operator's properties are read once when its executor is initialized, before any message has arrived, and the executor is built from a frozen descriptor JSON with no substitution stage. Today the only operator that reads a state key isIf; aFilterinside a loop cannot take its threshold from the loop.Let a property of any operator inside a block refer to a loop variable as
$K, with no change to the operator's source.Proposed Solution or Design
$Kas the property value. When the logical plan is parsed, a rewriter records each$namereference on the operator (stateReferences: JSON pointer → variable name) and, for non-string fields, replaces the value with a typed placeholder so the descriptor still deserializes. The sidecar is hidden from the property-panel schema.OperatorExecutor) before callingprocessState— the same for every operator, UDF or built-in.stateReferences(or a literal$namestring) is created as a late-bound wrapper: it starts with the placeholders unbound, binds them from the registered state when the state arrives (patching the descriptor and instantiating the real executor), and reuses the bound values for every tuple of the iteration. A tuple that arrives before the state, or a name missing from the state, is an error.The Python operator base gets the same registration hook for symmetry. Out of scope here: operators whose Scala descriptor interpolates property values into generated Python code (the sklearn operators); their values never reach a Scala executor, so Python-side binding is a follow-up. UDF-side sugar and the frontend are separate issues (#8636).
Acceptance
Filterinside a loop with predicate value$ifilters with the current iteration'si, verified end to end inLoopIntegrationSpec.Affected Area
Workflow Engine (Amber)