feat(vector): implement exact informative semantics

docs/native-vector-core.md

@@ -64,15 +64,77 @@ TA-Lib multi-output calls are represented by generic output names such as
 `aroondown`, `fastk`, or `upperband`; tuple assignment and constant output
 subscripts compile without Signal- or strategy-specific branches.
 
+## Informative frames and resampling
+
+M20-05 pins informative alignment to Freqtrade 2026.5.1. Every source frame
+has an explicit `(pair, timeframe)` identity; a BTC informative frame cannot be
+silently substituted for the traded pair, even when its column names match.
+For slower timeframes, visibility begins at:
+
+```text
+informative open + informative merge minutes - base merge minutes
+```
+
+Consequently, a 1h candle opened at `00:00` first becomes visible to a 5m base
+frame at `00:55`, never earlier. Freqtrade obtains both merge durations with
+CCXT seconds divided by 60, so sub-minute timeframes deliberately use its
+floor-minute behavior while resampling retains their real seconds.
+`ffill=False` performs an exact-key left join,
+leaving missing matches null and preserving duplicate Cartesian rows.
+`ffill=True` carries only an already-visible historical row. Monthly `1M`
+visibility uses the next calendar month start minus the base duration.
+
+The preprocessing contract mirrors the pinned Freqtrade anchors: ordinary
+timeframes use fixed-second bins, weekly data uses Monday anchors, months use
+month starts (`nMS`), and years use year starts (`nYS`). In particular, the
+pinned multiweek branch maps `2w` to `1W-MON`. Cleanup and optional OHLCV gap
+filling remain separate from informative merge; the merge does not synthesize
+candles.
+
+Complete-frame alignment reproduces the official missing, duplicate, unsorted,
+suffix, monthly, and cross-pair cases. Streaming alignment requires ordered
+chunks and explicit identities, retains at most one historical informative row,
+and rejects future rows, timestamp regressions, schema drift, unsupported
+faster-frame joins, and column collisions at the source location instead of
+guessing. Because Freqtrade can repair the whole prefix before its first exact
+match, a bounded `ffill=True` stream requires that match in its first non-empty
+base chunk; otherwise it stops before emitting provisional rows.
+
+Latest X7 performs its merges with `ffill=False` and then calls one final
+source-order `DataFrame.ffill()`. The Rust fill primitive treats both null and
+NaN as missing, preserves the bits of retained finite values, infinities, and
+signed zero, and retains at most one value per column across chunks. It never
+adds or removes candle rows.
+
+The committed oracle was generated by the official Freqtrade helper at tag
+2026.5.1 (commit `6fa47093`). Regeneration is an evidence-authoring command and
+requires that exact checkout at
+`.nfi/roadmap-acceptance/M20-05/freqtrade-2026.5.1`:
+
+```bash
+uv run python scripts/generate_informative_fixture.py
+```
+
+It writes
+`benchmarks/reference/informative/freqtrade-2026.5.1.json`; both the Python
+compatibility layer and Rust alignment tests replay the same cases exactly.
+
 ## Current claim boundary
 
 The latest X7 indicator operation set now has exact Native kernels. This does not
-yet claim that the entire latest X7 source compiles: compilation currently stops
-at `strategy.py:3168:21` on an informative/config attribute that belongs to the
-source and multi-timeframe lowering work in M20-05. EWM remains fail-closed
-because it is not reachable in this upstream inventory. Unknown functions,
-non-SMA TA-Lib MA types, centered rolling windows, Arrow null indicator inputs,
-and unselected multi-output calls also stop rather than being guessed.
+yet claim that the entire latest X7 source compiles or executes Full Native.
+M20-05 establishes exact multi-timeframe primitives and compiler contracts,
+but their in-memory `VectorEngine` connection is deliberately owned by M21.
+M21 also owns Native
+signal assignments, tag generation, independent Python/Rust vector shadowing,
+and the in-memory simulator connection. M22 owns latest-upstream Spot/Futures
+full-state qualification, removal of Python strategy execution from the Native
+lane, and release certification. Until those proofs pass, unsupported source
+constructs remain fail-closed and the official Freqtrade fallback remains the
+execution path. EWM also remains fail-closed because it is not reachable in the
+current upstream inventory. Unknown functions, non-SMA TA-Lib MA types, centered
+rolling windows, Arrow null indicator inputs, and unselected multi-output calls
+stop rather than being guessed.
 
 The Rust boundary can validate a compiled contract directly:
 

planning/roadmap-state.json

@@ -1,8 +1,8 @@
 {
   "schema_version": "1.0.0",
   "roadmap_id": "nfi-backtest-engine-post-v1.1.0",
-  "revision": 131,
-  "updated_at": "2026-08-11T00:55:12+09:00",
+  "revision": 133,
+  "updated_at": "2026-08-11T02:31:11+09:00",
   "acceptance_commands": "planning/acceptance-commands.json",
   "active_task_id": null,
   "execution_policy": {
@@ -4767,7 +4767,7 @@
       "order": 2005,
       "milestone": "M20",
       "title": "Implement multi-timeframe and informative data semantics",
-      "status": "pending",
+      "status": "completed",
       "depends_on": [
         "M20-04"
       ],
@@ -4786,10 +4786,17 @@
         "informative merge exact fixtures",
         "cross-pair dependency tests"
       ],
-      "started_at": null,
-      "completed_at": null,
-      "commit_sha": null,
-      "evidence": [],
+      "started_at": "2026-08-11T01:26:42+09:00",
+      "completed_at": "2026-08-11T02:31:11+09:00",
+      "commit_sha": "68d82daeadcf798e8e82534d470b8709d9191bb9",
+      "evidence": [
+        ".nfi/roadmap-acceptance/M20-05/68d82daeadcf798e8e82534d470b8709d9191bb9/acceptance-report.json",
+        ".nfi/roadmap-acceptance/M20-05/68d82daeadcf798e8e82534d470b8709d9191bb9/latest-program-gap.json",
+        ".nfi/roadmap-acceptance/M20-05/68d82daeadcf798e8e82534d470b8709d9191bb9/SHA256SUMS.txt",
+        ".nfi/roadmap-acceptance/M20-05/68d82daeadcf798e8e82534d470b8709d9191bb9/x7-spot/run.json",
+        ".nfi/roadmap-acceptance/M20-05/68d82daeadcf798e8e82534d470b8709d9191bb9/x7-futures/run.json",
+        "benchmarks/reference/informative/freqtrade-2026.5.1.json"
+      ],
       "blocker": null
     },
     {
@@ -8348,6 +8355,52 @@
         "next_eligible_task": "M20-05",
         "next_task_manual_gate": false
       }
+    },
+    {
+      "sequence": 179,
+      "timestamp": "2026-08-11T01:26:42+09:00",
+      "task_id": "M20-05",
+      "event": "task_started",
+      "details": {
+        "objective": "implement exact Freqtrade multi-timeframe and informative data visibility semantics for latest NFI",
+        "latest_upstream_commit": "897a1523391b8222ee711eba9714b59a3e77265a",
+        "fabricating_missing_candles_allowed": false,
+        "forward_looking_join_allowed": false,
+        "pair_or_timeframe_hardcoding_allowed": false,
+        "cross_pair_dependencies_must_be_explicit": true
+      }
+    },
+    {
+      "sequence": 180,
+      "timestamp": "2026-08-11T02:31:11+09:00",
+      "task_id": "M20-05",
+      "event": "task_completed",
+      "details": {
+        "implementation_commit": "68d82daeadcf798e8e82534d470b8709d9191bb9",
+        "latest_upstream_commit": "897a1523391b8222ee711eba9714b59a3e77265a",
+        "latest_strategy_sha256": "e99b4f58ecf507da86d2ba94e641c2994f17a1456133c2cff26946fa6b4a1afb",
+        "freqtrade_version": "2026.5.1",
+        "informative_oracle_case_count": 20,
+        "informative_oracle_fingerprint": "0c680501981ae7eb4fff1e4494c7c0d27fee5dcfefbf5131a857c47327440939",
+        "resample_anchors_exact": true,
+        "informative_visibility_exact": true,
+        "missing_duplicate_unsorted_and_cross_pair_exact": true,
+        "nan_null_infinity_and_signed_zero_exact": true,
+        "streaming_fail_closed_and_bounded": true,
+        "final_forward_fill_bounded": true,
+        "python_test_count": 763,
+        "future_compatibility_test_count": 134,
+        "rust_workspace_test_count": 211,
+        "spot_trade_surface_and_full_state_exact": true,
+        "futures_trade_surface_and_full_state_exact": true,
+        "runtime_hardcoding_added": false,
+        "latest_x7_fully_compiled": false,
+        "latest_x7_fail_closed_source": "strategy.py:3168:21",
+        "vector_engine_multiframe_connection": false,
+        "full_native_strategy_claim": false,
+        "next_eligible_task": "M21-01",
+        "next_task_manual_gate": false
+      }
     }
   ]
 }

python/nfi_backtest_engine/indicator_program.py

@@ -47,6 +47,22 @@
 }
 _SCALAR_CALLS = {"abs", "bool", "float", "int", "max", "min"}
 _WINDOW_REDUCERS = {"max", "mean", "min", "std", "sum"}
+_INFORMATIVE_MERGE_PARAMETERS = (
+    "dataframe",
+    "informative",
+    "timeframe",
+    "timeframe_inf",
+    "ffill",
+    "append_timeframe",
+    "date_column",
+    "suffix",
+)
+_INFORMATIVE_MERGE_DEFAULTS: Mapping[str, Any] = {
+    "ffill": True,
+    "append_timeframe": True,
+    "date_column": "date",
+    "suffix": None,
+}
 
 
 class IndicatorProgramCompileError(StrategyAnalysisError):
@@ -763,7 +779,12 @@ def method_call(self, node: ast.Call, callable_name: str) -> str:
                 self.node_types[base],
                 inputs=[base],
                 parameters={"direction": "forward"},
-                lookback=self.lookback(base),
+                lookback={
+                    "kind": "recursive",
+                    "candles": None,
+                    "expression": _safe_expression(node),
+                    "causal": bool(self.lookback(base)["causal"]),
+                },
             )
         if method in {"bfill", "backfill"}:
             self.unsupported(node, "backward fill would look ahead")
@@ -816,19 +837,68 @@ def window_call(self, node: ast.Call) -> str | None:
         )
 
     def informative_merge(self, node: ast.Call) -> str:
-        if len(node.args) < 4:
-            self.unsupported(node, "informative merge signature")
-        base = self.expression(node.args[0])
-        informative = self.expression(node.args[1])
+        arguments = _bind_informative_merge_arguments(node, self)
+        base = self.expression(arguments["dataframe"])
+        informative = self.expression(arguments["informative"])
         if self.node_types[base] != "dataframe" or self.node_types[informative] != "dataframe":
             self.unsupported(node, "informative merge dataframe inputs")
-        base_timeframe = _static_value(node.args[2], self.class_constants)
-        informative_timeframe = _static_value(node.args[3], self.class_constants)
+
+        timeframe_node = arguments["timeframe"]
+        informative_timeframe_node = arguments["timeframe_inf"]
+        base_timeframe = _required_static(timeframe_node, self)
+        informative_timeframe = _required_static(informative_timeframe_node, self)
         if not isinstance(base_timeframe, str) or not isinstance(informative_timeframe, str):
             self.unsupported(node, "dynamic informative merge timeframe")
-        keyword_values = _keyword_value_map(node, self)
-        if keyword_values.get("ffill", False) is not False:
-            self.unsupported(node, "informative merge must not fill before source-ordered fill")
+
+        ffill = _static_informative_option(arguments, "ffill", self)
+        append_timeframe = _static_informative_option(arguments, "append_timeframe", self)
+        date_column = _static_informative_option(arguments, "date_column", self)
+        suffix = _static_informative_option(arguments, "suffix", self)
+        if not isinstance(ffill, bool):
+            self.unsupported(arguments.get("ffill", node), "non-boolean informative merge ffill")
+        if not isinstance(append_timeframe, bool):
+            self.unsupported(
+                arguments.get("append_timeframe", node),
+                "non-boolean informative merge append_timeframe",
+            )
+        if not isinstance(date_column, str):
+            self.unsupported(
+                arguments.get("date_column", node),
+                "non-string informative merge date_column",
+            )
+        if suffix is not None and not isinstance(suffix, str):
+            self.unsupported(arguments.get("suffix", node), "non-string informative merge suffix")
+        if suffix and append_timeframe:
+            self.unsupported(
+                arguments.get("suffix", node),
+                "informative merge suffix conflicts with append_timeframe",
+            )
+        normalized_suffix = suffix or None
+        if not append_timeframe and normalized_suffix is None:
+            self.unsupported(
+                arguments.get("append_timeframe", node),
+                "informative merge without an output suffix",
+            )
+
+        base_minutes = _freqtrade_timeframe_minutes(timeframe_node, base_timeframe, self)
+        informative_minutes = _freqtrade_timeframe_minutes(
+            informative_timeframe_node,
+            informative_timeframe,
+            self,
+        )
+        if base_minutes > informative_minutes:
+            self.unsupported(
+                informative_timeframe_node,
+                "faster informative timeframe would create rows",
+            )
+        lookback = self.merged_lookback([base, informative])
+        if ffill:
+            lookback = {
+                "kind": "recursive",
+                "candles": None,
+                "expression": _safe_expression(node),
+                "causal": bool(lookback["causal"]),
+            }
         merged = self.emit(
             node,
             "informative-merge",
@@ -837,9 +907,12 @@ def informative_merge(self, node: ast.Call) -> str:
             parameters={
                 "base_timeframe": base_timeframe,
                 "informative_timeframe": informative_timeframe,
-                "ffill": False,
+                "ffill": ffill,
+                "append_timeframe": append_timeframe,
+                "date_column": date_column,
+                "suffix": normalized_suffix,
             },
-            lookback=self.merged_lookback([base, informative]),
+            lookback=lookback,
         )
         self.informative_nodes.append(merged)
         return merged
@@ -951,6 +1024,68 @@ def _window_parameters(
     return values
 
 
+def _bind_informative_merge_arguments(
+    node: ast.Call,
+    compiler: _Compiler,
+) -> dict[str, ast.expr]:
+    if len(node.args) > len(_INFORMATIVE_MERGE_PARAMETERS):
+        compiler.unsupported(node, "informative merge signature")
+    arguments: dict[str, ast.expr] = dict(
+        zip(_INFORMATIVE_MERGE_PARAMETERS, node.args, strict=False)
+    )
+    for keyword in node.keywords:
+        if keyword.arg is None:
+            compiler.unsupported(keyword.value, "expanded informative merge keyword arguments")
+        if keyword.arg not in _INFORMATIVE_MERGE_PARAMETERS:
+            compiler.unsupported(
+                keyword.value,
+                f"unknown informative merge keyword {keyword.arg}",
+            )
+        if keyword.arg in arguments:
+            compiler.unsupported(
+                keyword.value,
+                f"duplicate informative merge argument {keyword.arg}",
+            )
+        arguments[keyword.arg] = keyword.value
+    missing = [name for name in _INFORMATIVE_MERGE_PARAMETERS[:4] if name not in arguments]
+    if missing:
+        compiler.unsupported(node, f"informative merge missing arguments: {missing}")
+    return arguments
+
+
+def _static_informative_option(
+    arguments: Mapping[str, ast.expr],
+    name: str,
+    compiler: _Compiler,
+) -> Any:
+    node = arguments.get(name)
+    if node is None:
+        return _INFORMATIVE_MERGE_DEFAULTS[name]
+    return _required_static(node, compiler)
+
+
+def _freqtrade_timeframe_minutes(
+    node: ast.expr,
+    timeframe: str,
+    compiler: _Compiler,
+) -> int:
+    """Mirror pinned CCXT parsing used by Freqtrade's timeframe helper."""
+    try:
+        amount = int(timeframe[:-1])
+        scale = {
+            "y": 31_536_000,
+            "M": 2_592_000,
+            "w": 604_800,
+            "d": 86_400,
+            "h": 3_600,
+            "m": 60,
+            "s": 1,
+        }[timeframe[-1]]
+    except (IndexError, KeyError, ValueError):
+        compiler.unsupported(node, f"invalid informative merge timeframe {timeframe!r}")
+    return amount * scale // 60
+
+
 def _literal_keyword_arguments(
     node: ast.Call,
     compiler: _Compiler,