Parity apparatus efficiency: shard R cache, close automation loop, dedupe CI, pin dev env

Cache sharding (schema v2): tests/_r_cache.json (15.7MB monolith, opaque hash
keys) becomes tests/_r_cache/ with one JSON shard per R function label and
keys of the form "<label>|<sha256>". Regeneration diffs now name the function
they touch, git delta-compresses the shards, and a live-R run rewrites only
the shard it changed. All 2314 entries were migrated by replaying tests/parity
against the old monolith (the sha256 payload is unchanged, so legacy keys are
the digest part of new keys); zero orphans, full suite green cache-only.
regenerate_r_cache.py, run_live_r_parity_for_changed_api.py, the sync
manifest/scripts, and docs follow the new layout.

Automation loop: the regen workflow now records r_commit/r_version in
sync/nns_source.json itself and posts a per-function changed-entry table in
the PR body, so a cache PR is self-consistent and reviewable without forensic
scripts.

CI dedupe: native-backend-ci ran the identical Python 3.11-3.14 matrix twice
per PR push (push + pull_request events). push is now main-only, with a
concurrency group cancelling superseded runs.

Dev env pinning: pre-commit hooks run ruff/mypy via `python -m` in the
project environment, so local results match CI instead of a foreign mypy
reporting spurious numpy import errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxBV5HmXHY5nCUvW9SwDvb

Author: claude

.github/workflows/inspect-r-api-update.yml

@@ -185,24 +185,120 @@ jobs:
           import os
           from pathlib import Path
 
-          path = Path('tests/_r_cache.json')
-          if not path.exists():
-              raise SystemExit('tests/_r_cache.json was not generated')
-          payload = json.loads(path.read_text(encoding='utf-8'))
-          actual = payload.get('nns_version')
+          cache_dir = Path('tests/_r_cache')
+          shards = sorted(cache_dir.glob('*.json')) if cache_dir.is_dir() else []
+          if not shards:
+              raise SystemExit('tests/_r_cache/ contains no shard files')
           expected = os.environ['EXPECTED_VERSION']
-          if actual != expected:
-              raise SystemExit(f'cache version {actual!r} != dispatched version {expected!r}')
-          entries = payload.get('entries')
-          if not isinstance(entries, dict) or not entries:
-              raise SystemExit('regenerated cache contains no entries')
-          print(f'Validated {len(entries)} cache entries for NNS {actual}.')
+          total = 0
+          for shard_path in shards:
+              shard = json.loads(shard_path.read_text(encoding='utf-8'))
+              actual = shard.get('nns_version')
+              if actual != expected:
+                  raise SystemExit(
+                      f'{shard_path.name}: cache version {actual!r} '
+                      f'!= dispatched version {expected!r}'
+                  )
+              entries = shard.get('entries')
+              if not isinstance(entries, dict) or not entries:
+                  raise SystemExit(f'{shard_path.name} contains no entries')
+              total += len(entries)
+          print(
+              f'Validated {total} cache entries across {len(shards)} '
+              f'shards for NNS {expected}.'
+          )
           PY
 
       - name: Remove transient cache files
         if: always()
         shell: bash
-        run: rm -f tests/_r_cache.json.bak tests/_r_cache.lock
+        run: rm -rf tests/_r_cache.bak && rm -f tests/_r_cache.lock
+
+      - name: Record cache provenance in sync manifest
+        if: always()
+        shell: bash
+        env:
+          R_COMMIT: ${{ steps.package.outputs.r_commit }}
+          R_VERSION: ${{ steps.package.outputs.r_version }}
+        run: |
+          python - <<'PY'
+          import json
+          import os
+          from pathlib import Path
+
+          path = Path('sync/nns_source.json')
+          manifest = json.loads(path.read_text(encoding='utf-8'))
+          manifest['r_commit'] = os.environ['R_COMMIT']
+          manifest['r_version'] = os.environ['R_VERSION']
+          path.write_text(json.dumps(manifest, indent=2) + '\n', encoding='utf-8')
+          print(f"Manifest now records {manifest['r_repo']}@{manifest['r_commit']} "
+                f"(NNS {manifest['r_version']}).")
+          PY
+
+      - name: Summarize cache changes by function
+        id: cachediff
+        if: always()
+        shell: bash
+        run: |
+          python - <<'PY'
+          import json
+          import subprocess
+          from pathlib import Path
+
+          def committed(name: str) -> dict:
+              proc = subprocess.run(
+                  ['git', 'show', f'HEAD:tests/_r_cache/{name}'],
+                  capture_output=True, text=True,
+              )
+              if proc.returncode != 0:
+                  return {}
+              return json.loads(proc.stdout).get('entries', {})
+
+          cache_dir = Path('tests/_r_cache')
+          lines = ['### Cache changes by function', '']
+          if not cache_dir.is_dir():
+              lines.append('_No cache directory was generated; see the regeneration log._')
+          else:
+              proc = subprocess.run(
+                  ['git', 'ls-tree', '--name-only', 'HEAD', 'tests/_r_cache/'],
+                  capture_output=True, text=True,
+              )
+              old_names = {Path(p).name for p in proc.stdout.split() if p.endswith('.json')}
+              new_names = {p.name for p in cache_dir.glob('*.json')}
+              rows = []
+              for name in sorted(old_names | new_names):
+                  before = committed(name) if name in old_names else {}
+                  after = (
+                      json.loads((cache_dir / name).read_text()).get('entries', {})
+                      if name in new_names else {}
+                  )
+                  added = len(set(after) - set(before))
+                  removed = len(set(before) - set(after))
+                  changed = sum(
+                      1 for k in set(before) & set(after) if before[k] != after[k]
+                  )
+                  if added or removed or changed:
+                      label = name[: -len('.json')]
+                      rows.append(
+                          f'| `{label}` | {len(before)} | {len(after)} '
+                          f'| {added} | {removed} | {changed} |'
+                      )
+              if rows:
+                  lines += [
+                      '| function | before | after | added | removed | changed |',
+                      '| --- | --- | --- | --- | --- | --- |',
+                      *rows,
+                  ]
+              else:
+                  lines.append('_No cache entries changed._')
+          Path('cache-diff.md').write_text('\n'.join(lines) + '\n', encoding='utf-8')
+          print('\n'.join(lines))
+          PY
+          {
+            echo 'summary<<CACHE_DIFF_EOF'
+            cat cache-diff.md
+            echo 'CACHE_DIFF_EOF'
+          } >> "$GITHUB_OUTPUT"
 
       - name: Verify committed-cache mode
         id: verify
@@ -246,10 +342,13 @@ jobs:
             - Live regeneration result: `${{ steps.regenerate.outcome }}`
             - Cache-only verification result: `${{ steps.verify.outcome }}`
 
-            The exact R source package was installed and the committed parity cache was regenerated. Any remaining Python/R parity mismatches are retained in the workflow diagnostics and should be repaired against this R-authored baseline.
+            The exact R source package was installed and the committed parity cache was regenerated. `sync/nns_source.json` records this R commit as the behavioral-truth provenance. Any remaining Python/R parity mismatches are retained in the workflow diagnostics and should be repaired against this R-authored baseline.
+
+            ${{ steps.cachediff.outputs.summary }}
           add-paths: |
             tests/_r.py
-            tests/_r_cache.json
+            tests/_r_cache/**
+            sync/nns_source.json
 
       - name: Report parity status
         if: always()

.github/workflows/native-backend-ci.yml

@@ -1,12 +1,20 @@
 name: Native backend CI
 
+# pull_request covers every branch with an open PR; push is limited to main so
+# a PR-branch push does not run the identical matrix twice (once per event).
 on:
   pull_request:
   push:
+    branches: [main]
 
 permissions:
   contents: read
 
+# A superseded push to the same ref cancels the in-flight run.
+concurrency:
+  group: native-backend-ci-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
 jobs:
   native-backend:
     name: Python ${{ matrix.python-version }}

.gitignore

@@ -16,7 +16,7 @@ __pycache__/
 
 # R cache lock and fresh-regeneration backup
 tests/_r_cache.lock
-tests/_r_cache.json.bak
+tests/_r_cache.bak/
 
 # Vendored R NNS local-install build artifacts
 tools/NNS/src/*.o

.pre-commit-config.yaml

@@ -0,0 +1,21 @@
+# Local hooks run `python -m ...` with language: system so they use the
+# project environment's interpreter — the same one that has numpy and the
+# package installed. A mypy/ruff binary from a different environment produces
+# spurious import-not-found noise (numpy stubs missing) that masks real errors
+# CI would catch; running through the project env keeps local == CI.
+repos:
+  - repo: local
+    hooks:
+      - id: ruff
+        name: ruff check (project env)
+        entry: python -m ruff check --force-exclude
+        language: system
+        types_or: [python, pyi]
+        require_serial: true
+      - id: mypy
+        name: mypy (project env)
+        entry: python -m mypy
+        language: system
+        types_or: [python, pyi]
+        pass_filenames: false
+        require_serial: true

docs/install.md

@@ -50,6 +50,15 @@ uv run ruff check .
 uv run mypy
 ```
 
+Optionally install the pre-commit hooks, which run `ruff` and `mypy` through
+the project environment's interpreter so local results match CI (a `mypy`
+binary from another environment lacks the project's dependencies and reports
+spurious import errors):
+
+```bash
+uv run pre-commit install
+```
+
 The default parity suite is cache-backed and does not require `Rscript`.
 `Rscript` and the R `NNS` package are needed only when regenerating parity caches
 or running live R comparison scripts.

docs/parity.md

@@ -9,8 +9,9 @@ and remains the native C++ foundation for accelerated partial-moment routines.
 Full package parity is **not** claimed. Parity is bounded by the committed tests
 and cache:
 
-- `tests/_r_cache.json` — cache-only R result fixtures (2,406 keyed entries,
-  schema version `1`, `nns_version == "13.0"`),
+- `tests/_r_cache/` — cache-only R result fixtures, sharded into one JSON
+  file per R function label (keys are `<label>|<sha256>`, schema version
+  `2`, every shard stamped with the generating `nns_version`),
 - `tests/parity/` — public behavior parity checks,
 - `tests/invariants/` — Python-native contracts and invariants, and
 - `tests/fixtures/original_tests_expected.json` — adopted original R tests
@@ -85,9 +86,10 @@ If full regeneration is slow or unstable, regenerate deterministic chunks one
 file at a time, for example
 `python scripts/regenerate_r_cache.py -- -n 0 tests/parity/test_core.py`, then
 continue through the remaining parity files. The committed result must remain a
-single valid `tests/_r_cache.json` with `nns_version == "13.0"`,
-`schema_version == 1`, and non-empty `entries`; `scripts/regenerate_r_cache.py`
-enforces those guardrails after the pytest run.
+valid `tests/_r_cache/` shard directory: every shard carries the same
+`nns_version`, `schema_version == 2`, a `label` matching its filename, and
+non-empty `entries`; `scripts/regenerate_r_cache.py` enforces those
+guardrails after the pytest run.
 
 Validate the regenerated cache offline:
 
@@ -130,7 +132,7 @@ NNS R API change
 ### Fix rules (applied by the maintainer; later by the agent)
 
 - Edit **`src/nns/**` only**. Never edit `extern/NNS-core/**`, `tools/NNS/**`,
-  or `tests/_r_cache.json` to make a check pass.
+  or `tests/_r_cache/**` to make a check pass.
 - Classify the root cause and act accordingly:
   - **Python port bug** → fix in `src/nns/**`.
   - **R changed behavior** → do not chase a cache value; cache regeneration is a

docs/plot_parity_policy.md

@@ -36,7 +36,7 @@ colors and which element they sit on*, never rendered images.
 
 - Numeric return values (scalars, vectors, matrices, nested result dicts) from
   every ported function, against committed R fixtures and the committed R cache
-  (`tests/_r_cache.json`).
+  (`tests/_r_cache/`).
 - Structural contracts (result keys, shapes, dtypes, finiteness) via the
   invariant suite.
 

pyproject.toml

@@ -42,6 +42,7 @@ dependencies = [
 dev = [
     "hypothesis",
     "mypy",
+    "pre-commit",
     "pytest",
     "pytest-benchmark",
     "pytest-cov",