Fresh R NNS 13.0 parity regeneration and corrective Python fixes

[OVVO-Financial]

Summary

This PR freshly regenerated tests/_r_cache.json from vendored live R NNS 13.0 after starting from an empty cache, and fixes the deterministic Python parity gaps that the fresh run revealed.

• Python implementation changes in this PR are limited to matching fresh R NNS 13.0 outputs.
• Cache-only parity was run only after fresh regeneration, as a replay check.

What was done

Made regeneration truly fresh. Added a --fresh mode to scripts/regenerate_r_cache.py that refuses to run in CI, verifies a live local R NNS 13.0 install, moves the existing tests/_r_cache.json to tests/_r_cache.json.bak, and repopulates every entry from a live R call. No existing entries were reused.

Installed live vendored R NNS 13.0. Installed via scripts/install_local_r_nns.py (R CMD INSTALL tools/NNS), never from CRAN. Verified independently:

R version 4.3.3 (2024-02-29)
Rscript (R) version 4.3.3 (2024-02-29)
packageVersion("NNS") == 13.0
source: tools/NNS (extracted package dir)

R dependencies needed to load NNS came from Ubuntu binaries; Rfast (+zigg) was built from upstream GitHub release v2.1.5.1-apollo because CRAN was unreachable. NNS itself came only from tools/NNS.

Regenerated the full cache from live R. python scripts/regenerate_r_cache.py --fresh -- -n 0 tests/parity. Result: nns_version: 13.0, schema_version: 1, 2385 entries (was 2406). The 21 dropped keys are stale — an instrumented cache-only replay confirmed no current parity test computes them. 456 of the 2384 shared keys changed value; fresh live R 13.0 values are authoritative.

Fixed Python to match fresh R 13.0 (the fresh run surfaced 216 failing tests, all from four deterministic divergences):

1. LPM.VaR/UPM.VaR integer degrees 1–4 (var.py): ported R 13.0's exact polynomial root-finding inversion (.NNS_LPM_VaR_integer) replacing the old optimize() search. Also fixed dependent regression/M.reg CIs and stack/boost prediction intervals.
2. Distance bulk-path kernel (distance.py): added nns_distance_path_single_bulk using the population sd of ranks sqrt((k²−1)/12) in the lognormal weight (the single-point kernel keeps sample sd), and used it for n.best>1 fitted values and multi-point estimates, as R does.
3. NNS.M.reg out-of-hull multi-point extrapolation (multivariate_regression.py): mirrored R 13.0's vectorized outsider path and dropped the obsolete single-outsider dims-dropping quirk.
4. NNS.ARMA numeric multi-lag seasonal weighting (arma.py): weight each seasonal factor by its position in the vector (R's seq(n,1,-i)), not its lag value. This resolved the two previously xfail-ed Sunspots ARMA and macro VAR practical examples, which now pass against live R; their xfail markers were removed.

Structural test/return updates justified by fresh R output: R 13.0's NNS.boost no longer returns n.best, so the Python return dict and stale assertions were updated, and final-estimate NaN handling now matches R.

ARMA nonseasonal nonlinear reconciliation. Live R 13.0 returns [128.5, 113.5, 155.5, 213.667]; Python returns the same. The previously reported [125.25, 107.75, 158.75, …] Python values do not reproduce with the current implementation. Smoke expectations were verified against this live run and needed no change.

Manual diagnostic scripts. No scripts/compare_nns*.py exist in this repo, so there was no $RPM-based univariate NNS.reg extraction to fix. The univariate regression-point diagnostic in test_r13_smoke.py already uses NNS.reg(..., multivariate.call = TRUE)$y.

Verification (all run after fresh regeneration)

python scripts/install_local_r_nns.py                              # NNS 13.0
python scripts/regenerate_r_cache.py --fresh -- -n 0 tests/parity  # 1794 passed, 1 xfailed (from empty)
PYNNS_R_CACHE_ONLY=1 python -m pytest -q -n 0 tests/parity         # 1794 passed, 1 xfailed
python -m pytest -q tests/parity/test_r13_smoke.py                 # 6 passed
python -m pytest -q tests/invariants                               # 317 passed
ruff check .                                                       # pass
mypy                                                               # pass
python -m build                                                    # sdist + wheel built

If --fresh were not used, the equivalent manual step is:

mv tests/_r_cache.json tests/_r_cache.pre-fresh-r13.json
python scripts/regenerate_r_cache.py -- -n 0 tests/parity

Two independent from-empty regenerations agree on 2384/2385 entries; the only difference is the stochastic bootstrap CI block of one NNS.ANOVA entry (deterministic core identical), covered by a @pytest.mark.stochastic range check. The sole remaining xfail is the balanced-Iris NNS.boost diagnostic, documented as a stochastic RNG sampling gap. No deterministic parity gap is excluded from the suite.

Full provenance: docs/r13_cache_regeneration.md.

https://claude.ai/code/session_015pEPxpJWbiHizgcFEm1AWf

---

Generated by Claude Code