diff --git a/.claude/skills/evaluate-tool/prompts/code-evaluator-prompt.md b/.claude/skills/evaluate-tool/prompts/code-evaluator-prompt.md index 980ae209..70a9359e 100644 --- a/.claude/skills/evaluate-tool/prompts/code-evaluator-prompt.md +++ b/.claude/skills/evaluate-tool/prompts/code-evaluator-prompt.md @@ -397,7 +397,7 @@ G-FNM-3 and G-FNM-4 should first attempt to load from the intermediate CSVs at it only supports MATPOWER `.m` import), fall back to the pre-cleaned MATPOWER case at `data/fnm/reference/cleaned/fnm_main_island.mat` (mounted at `/workspace/data/fnm/reference/cleaned/fnm_main_island.mat` in the devcontainer). -This is the 27,862-bus main island with all data fixes pre-applied: +This is the ~28,000-bus main island with all data fixes pre-applied: negative-X coercion, zero-X/R/RATE_A fixes, island extraction, single-slack reduction. See `summary_cleaning.json` for details. **Do NOT re-implement cleaning in test code** — the cleaned case is the canonical input for power flow verification. @@ -431,7 +431,7 @@ Record which input path was used in the result frontmatter: | Table | Expected | Actual | Status | |-------|----------|--------|--------| - | bus | 27862 | ... | PASS/FAIL | + | bus | ~28,000 | ... | PASS/FAIL | | load | 8624 | ... | PASS/FAIL | | ... | ... | ... | ... | diff --git a/.gitignore b/.gitignore index 39209c0b..15df750a 100644 --- a/.gitignore +++ b/.gitignore @@ -29,13 +29,14 @@ htmlcov/ # direnv .direnv/ -# Plans -plans/ - # OS .DS_Store Thumbs.db -.claude/worktrees/ -# Whitepapers -*whitepaper*.md +# Internal tooling (not part of deliverable) +.claude/ +plans/ +sweep-data/ +sweep-reports/ +tutorials/ +data/whitepaper_proposal.md diff --git a/.implement-report.md b/.implement-report.md deleted file mode 100644 index 715a71c6..00000000 --- a/.implement-report.md +++ /dev/null @@ -1,44 +0,0 @@ -# Implementation Report: PRD 06/05 - Deployment Smoke Test - -## Summary - -Implemented a post-deployment smoke test script that verifies the GitHub Pages site is accessible and serving correct content after deployment. - -## Changes - -| File | Action | -|------|--------| -| `report/scripts/smoke_test.py` | Created — smoke test script with CLI entry point | -| `report/tests/test_smoke_test.py` | Created — 14 unit tests | -| `report/Makefile` | Modified — added `smoke` target | -| `.github/workflows/deploy-report.yml` | Modified — added post-deployment smoke test steps | - -## Design Decisions - -- Used stdlib `urllib.request` only (no third-party HTTP dependencies) -- `SITE_URL` env var > CLI argument > auto-detection from `docusaurus.config.js` -- Deploy job gets checkout, Python/uv setup, and `make smoke` after `deploy-pages` -- Exit code 2 for root-page connection failures vs exit code 1 for content failures - -## Test Results - -14/14 tests pass: - -- T-D6.05-01: check_page 200 OK -- T-D6.05-02: check_page 404 -- T-D6.05-03: check_page wrong content type -- T-D6.05-04: check_page missing title -- T-D6.05-05: check_page GitHub 404 body -- T-D6.05-06: check_page connection error -- T-D6.05-07: detect_404_page positive -- T-D6.05-08: detect_404_page negative -- T-D6.05-09: report all pass -- T-D6.05-10: report any fail -- T-D6.05-11: run_smoke_test all pages -- T-D6.05-12: build_site_url from config -- T-D6.05-13: Makefile smoke target exists -- T-D6.05-14: workflow has smoke step - -## Deviations from PRD - -None. diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 6a3ee35f..ecd3c283 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -58,9 +58,10 @@ repos: hooks: - id: forbid-real-grid-names name: forbid real grid/operator names - entry: python -m scripts.check_no_real_grid_names + entry: python -m data.validation.check_no_real_grid_names language: python pass_filenames: true + exclude: '(whitepaper_proposal\.md|deliverables/)' - id: mh-style name: mh_style (MATLAB/Octave) diff --git a/README.md b/README.md index 6ef9a9e6..e89b2c40 100644 --- a/README.md +++ b/README.md @@ -1,139 +1,85 @@ -# GRC Tech Evaluation +# GRC Technology Evaluation — Phase 1 -**[View the Phase 1 Tool Selection Report](https://zgenergy.github.io/grc-tech-evaluation/)** +**[View the Interactive Report](https://zgenergy.github.io/grc-tech-evaluation/)** -Reproducible evaluation environments for six power-system modeling packages, -supporting the Phase 1 technology assessment. +Phase 1 technology evaluation comparing six open-source power-system modeling +tools against a six-criterion rubric. **PyPSA is the recommended tool for +Phase 2 development**, earning Strong in five of six criteria and holding the +top position across all sensitivity scenarios tested. -## Tools Under Evaluation +| Tool | Language | Expressiveness | Extensibility | Scalability | Accessibility | Maturity | Supply Chain | +|------|----------|:-:|:-:|:-:|:-:|:-:|:-:| +| **PyPSA** | Python | Strong | Strong | Adequate | Strong | Strong | Strong | +| PowerModels.jl | Julia | Adequate | Strong | Adequate | Adequate | Adequate | Adequate | +| PowerSimulations.jl | Julia | Adequate | Strong | Adequate | Weak | Adequate | Adequate | +| GridCal | Python | Adequate | Adequate | Adequate | Weak | Weak | Strong | +| pandapower | Python | Weak | Adequate | Weak | Adequate | Strong | Strong | +| MATPOWER\* | MATLAB | Adequate | Strong | Weak | Adequate | Adequate | Strong | -| Tool | Language | Directory | -|------|----------|-----------| -| [PyPSA](https://pypsa.org/) | Python | `evaluations/pypsa/` | -| [pandapower](https://www.pandapower.org/) | Python | `evaluations/pandapower/` | -| [GridCal](https://www.advancedgridinsights.com/gridcal) | Python | `evaluations/gridcal/` | -| [PowerModels.jl](https://lanl-ansi.github.io/PowerModels.jl/) | Julia | `evaluations/powermodels/` | -| [PowerSimulations.jl](https://nrel-sienna.github.io/PowerSimulations.jl/) | Julia | `evaluations/powersimulations/` | -| [MATPOWER](https://matpower.org/) | MATLAB/Octave | `evaluations/matpower/` | +\*Reference benchmark only; excluded from ranking (requires MATLAB runtime). -## Directory Structure +## Repository Guide -``` -grc-tech-evaluation/ -├── evaluation_guides/ # Rubric and test protocol -│ ├── Phase1_Evaluation_Rubric_v1.md -│ └── Phase1_Test_Protocol_v2.md -├── data/ -│ └── networks/ # Shared MATPOWER .m test cases -├── evaluations/ -│ ├── pypsa/ # Independent uv project -│ ├── pandapower/ # Independent uv project -│ ├── gridcal/ # Independent uv project -│ ├── powermodels/ # Julia project -│ ├── powersimulations/ # Julia project -│ └── matpower/ # Octave + download script -└── README.md -``` - -## Dev Environment - -All six tools run inside a single **devcontainer** that ships Python 3.12, uv, -Julia 1.10, and GNU Octave with all dependencies pre-installed. - -### Prerequisites - -| Requirement | Notes | -|-------------|-------| -| [Docker](https://docs.docker.com/get-docker/) | Docker Desktop or Docker Engine | -| [VS Code](https://code.visualstudio.com/) + [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) | Recommended for interactive development | -| *or* [devcontainer CLI](https://github.com/devcontainers/cli) | `npm install -g @devcontainers/cli` — for headless/CI use | - -### Building and Opening the Container - -**VS Code (recommended):** +### For Analysts -1. Open this repo in VS Code. -2. When prompted, click **Reopen in Container** (or run the command - `Dev Containers: Reopen in Container` from the palette). -3. The first build takes a few minutes while it installs all runtimes and - dependencies. Subsequent opens reuse the cached image and start in seconds. +| Directory | Contents | +|-----------|----------| +| [`report/`](report/) | Interactive Docusaurus report site — the primary deliverable | +| [`deliverables/`](deliverables/) | Formal white paper (SOW Task 1.3) | +| [`evaluations/`](evaluations/) | Per-tool evaluation evidence, test code, and results | +| [`evaluation_guides/`](evaluation_guides/) | Rubric and test protocol defining all 39 tests | +| [`data/`](data/) | Shared test networks (MATPOWER cases), augmented time series, FNM data | +| [`phase2-research/`](phase2-research/) | State estimation investigation (Phase 2 groundwork) | -**CLI:** +### For Developers -```bash -# Build and start the container (first time or after Dockerfile changes) -devcontainer up --workspace-folder . - -# Open an interactive shell inside the running container -devcontainer exec --workspace-folder . bash +| Directory | Contents | +|-----------|----------| +| `.devcontainer/` | Docker development environment (Python 3.12, Julia 1.10, Octave) | +| `.github/` | CI/CD workflows | +| `data/validation/` | Data quality scripts (schema validation, manifest generation) | -# Or run a one-off command -devcontainer exec --workspace-folder . uv run --project evaluations/pypsa python -c "import pypsa; print(pypsa.__version__)" -``` - -### What's Inside the Image +## Tools Evaluated -The Dockerfile (`.devcontainer/Dockerfile`) installs everything at build time -so the container is ready to use immediately: +Each tool has an independent environment under `evaluations//`: -- **Python 3.12** + **uv** — each Python tool (`pypsa`, `pandapower`, - `gridcal`) has its own `.venv` created by `uv sync` during the build. -- **Julia 1.10.7** (pinned LTS) — Julia packages for `powermodels` and - `powersimulations` are instantiated and precompiled during the build. -- **GNU Octave** — MATPOWER 8.1 is downloaded by `setup.sh` during the build. -### Verifying the Install +| Tool | Language | Environment | +|------|----------|-------------| +| [PyPSA](https://pypsa.org/) | Python | `uv sync` | +| [pandapower](https://www.pandapower.org/) | Python | `uv sync` | +| [GridCal](https://www.advancedgridinsights.com/gridcal) | Python | `uv sync` | +| [PowerModels.jl](https://lanl-ansi.github.io/PowerModels.jl/) | Julia | `Pkg.instantiate()` | +| [PowerSimulations.jl](https://nrel-sienna.github.io/PowerSimulations.jl/) | Julia | `Pkg.instantiate()` | +| [MATPOWER](https://matpower.org/) | MATLAB/Octave | `bash setup.sh` | -Smoke-test all six tools at once: - -```bash -bash .devcontainer/validate.sh -``` +## Evaluation Protocol -Or verify a single tool: +The evaluation uses a standardized rubric and test protocol: -```bash -# Python tools (pypsa, pandapower, gridcal) -cd evaluations/ && uv run python verify_install.py +- **[Phase1_Evaluation_Rubric.md](evaluation_guides/Phase1_Evaluation_Rubric.md)** — + Scoring criteria across six dimensions with tier definitions +- **[Phase1_Test_Protocol.md](evaluation_guides/Phase1_Test_Protocol.md)** — + 39 specific tests with acceptance criteria -# Julia tools (powermodels, powersimulations) -cd evaluations/ && julia --project=. verify_install.jl +Test networks: IEEE 39-bus, ACTIVSg 2,000-bus, and ACTIVSg 10,000-bus +synthetic cases from `data/networks/`. -# MATPOWER -cd evaluations/matpower && octave verify_install.m -``` +Results for each tool are in `evaluations//results/` organized by rubric +dimension, with a `synthesis.md` summarizing findings. -### Day-to-Day Development +## Development Environment -All work happens inside the container. Run scripts with the tool's own runtime: +All tools run inside a single devcontainer. See +[`.devcontainer/`](.devcontainer/) for setup instructions. ```bash -# Run a Python evaluation script -cd evaluations/pypsa -uv run python results/gate/ac_power_flow.py - -# Run a Julia evaluation script -cd evaluations/powermodels -julia --project=. results/gate/ac_power_flow.jl +# Build and open +devcontainer up --workspace-folder . +devcontainer exec --workspace-folder . bash -# Run an Octave evaluation script -cd evaluations/matpower -octave results/gate/ac_power_flow.m +# Verify all tools +bash .devcontainer/validate.sh -# Lint Python files +# Lint pre-commit run --all-files ``` - -If you need to add a Python dependency to a tool, update its `pyproject.toml` -and run `uv sync` inside that tool's directory — do not use `pip install`. -For Julia, edit `Project.toml` and run -`julia --project=. -e 'using Pkg; Pkg.instantiate()'`. - -## Evaluation Protocol - -See `evaluation_guides/` for the full rubric and test protocol: - -- **Phase1_Evaluation_Rubric_v1.md** — Scoring criteria across seven dimensions -- **Phase1_Test_Protocol_v2.md** — Specific tests and acceptance criteria - -Results for each tool are organized into subdirectories under -`evaluations//results/` matching the rubric dimensions. diff --git a/data/README.md b/data/README.md new file mode 100644 index 00000000..5f8c37f9 --- /dev/null +++ b/data/README.md @@ -0,0 +1,32 @@ +# Data + +Shared test data consumed by all six evaluation environments. + +## Directory Structure + +| Directory | Contents | +|-----------|----------| +| `networks/` | MATPOWER `.m` case files (IEEE 39-bus, ACTIVSg 2k, ACTIVSg 10k) | +| `fnm/` | Full Network Model — parsed network data with intermediate schemas | +| `timeseries/` | Augmented time-series data (load profiles, gen costs, reserves, scenarios) | +| `reference/` | Reference data (RTS-GMLC technology classes, calibration outputs) | +| `scripts/` | Data augmentation pipeline (generates timeseries/ and reference/ outputs) | +| `validation/` | Data quality scripts (schema validation, manifest generation, doc generation) | +| `whitepaper_proposal.md` | SOW contract proposal document | + +## Data Flow + +``` +networks/ (raw MATPOWER cases) + │ + ├── scripts/ (augmentation pipeline) ──→ timeseries/ + reference/ + │ + └── fnm/ (parsed network model) ──→ fnm/reference/ (cleaned data + DCPF solutions) +``` + +The `Makefile` orchestrates the augmentation pipeline stages in dependency order. + +## Important + +`networks/` and `timeseries/` paths are hard-coded in evaluation test suites +across all six tools. **Do not move or rename these directories.** diff --git a/data/fnm/docs/parser-comparison-report.md b/data/fnm/docs/parser-comparison-report.md index d2720d43..6ded7261 100644 --- a/data/fnm/docs/parser-comparison-report.md +++ b/data/fnm/docs/parser-comparison-report.md @@ -2,11 +2,11 @@ ## Summary -**File:** `AUC_AN_2026_2026_S01_ON_NETWORK_MODEL.RAW` +**File:** `.RAW` **Format:** PSS/E v31, space-separated (no commas) **Size:** 17.8 MB, 117,420 lines, 88,230 data records **System Base:** 100.0 MVA -**Case ID:** `Model_DB135_Prod_20250714 - 2025-07-16 23:59:59 - Nwk_Model_Meas` +**Case ID:** `` **Canonical Parser Selected:** MATPOWER psse2mpc @@ -22,12 +22,12 @@ Extracted by `raw_record_counter.py` — parser-independent line counting. | Section | Records | Non-Empty | |---------|--------:|:---------:| -| Bus | 30,307 | Y | -| Load | 15,062 | Y | +| Bus | ~30,000 | Y | +| Load | ~15,000 | Y | | Fixed Shunt | 0 | | -| Generator | 5,768 | Y | -| Branch | 24,117 | Y | -| Transformer | 9,723 | Y | +| Generator | ~5,800 | Y | +| Branch | ~24,000 | Y | +| Transformer | ~9,700 | Y | | Area | 49 | Y | | Two-Terminal DC | 0 | | | VSC DC | 0 | | @@ -38,7 +38,7 @@ Extracted by `raw_record_counter.py` — parser-independent line counting. | Interarea Transfer | 0 | | | Owner | 0 | | | FACTS | 0 | | -| Switched Shunt | 3,114 | Y | +| Switched Shunt | ~3,100 | Y | | **Total** | **88,230** | **8/17** | ### HVDC/FACTS/Multi-Terminal DC (OQ-E02) @@ -56,20 +56,20 @@ no DC transmission, no FACTS devices, and no multi-terminal DC lines. | Element | Count | Matches Raw? | |---------|------:|:------------:| -| Buses | 30,307 | Y | -| Loads | 15,062 | Y | -| Generators | 5,768 | Y | -| Branches | 33,840 | * | +| Buses | ~30,000 | Y | +| Loads | ~15,000 | Y | +| Generators | ~5,800 | Y | +| Branches | ~34,000 | * | | Areas | — | Skipped | | Zones | — | Skipped | -| Switched Shunts | 3,114 | Y | +| Switched Shunts | ~3,100 | Y | -*Branches = 24,117 lines + 9,723 two-winding transformers merged into the mpc.branch matrix. +*Branches = ~24,000 lines + ~9,700 two-winding transformers merged into the mpc.branch matrix. **Known limitations observed:** - Zone data (90 records) skipped — not imported into mpc struct - Area data (49 records) skipped — not imported into mpc struct -- All 9,723 transformers are two-winding (no 3-winding decomposition needed) +- All ~9,700 transformers are two-winding (no 3-winding decomposition needed) - No voltage limits in RAW file — defaults applied (VMIN=0.9, VMAX=1.1) **Warnings:** @@ -146,9 +146,9 @@ Result: FLAT START | VA mean | 0.000000 | | VA std | 0.000000 | | VA min / max | 0.0 / 0.0 | -| Buses with VM = 1.0 | 30,307 / 30,307 (100%) | -| Buses with VA = 0.0 | 30,307 / 30,307 (100%) | -| Generators with Qg != 0 | 0 / 5,768 (0%) | +| Buses with VM = 1.0 | ~30,000 / ~30,000 (100%) | +| Buses with VA = 0.0 | ~30,000 / ~30,000 (100%) | +| Generators with Qg != 0 | 0 / ~5,800 (0%) | All bus voltage magnitudes are exactly 1.0 p.u. and all angles are exactly 0.0 degrees. No generator produces reactive power. This is definitively a flat-start @@ -166,13 +166,13 @@ Seven supplemental CSVs accompany the RAW file: | File | Size | Description | |------|-----:|-------------| -| `AUC_AN_2026_2026_S01_CONTINGENCY.csv` | 731 KB | Contingency definitions | -| `AUC_AN_2026_2026_S01_GEN_DISTRIBUTION_FACTOR.csv` | 18 KB | Generator distribution factors | -| `AUC_AN_2026_2026_S01_INTERFACE.csv` | 721 KB | Interface definitions | -| `AUC_AN_2026_2026_S01_LINE_AND_TRANSFORMER.csv` | 14.5 MB | Line and transformer data | -| `AUC_AN_2026_2026_S01_OUTAGE.csv` | 65 KB | Outage definitions | -| `AUC_AN_2026_2026_S01_RESOURCE.csv` | 1.3 MB | Resource (generator) data | -| `AUC_AN_2026_2026_S01_TRADING_HUB.csv` | 103 KB | Trading hub definitions | +| `_CONTINGENCY.csv` | 731 KB | Contingency definitions | +| `_GEN_DISTRIBUTION_FACTOR.csv` | 18 KB | Generator distribution factors | +| `_INTERFACE.csv` | 721 KB | Interface definitions | +| `_LINE_AND_TRANSFORMER.csv` | 14.5 MB | Line and transformer data | +| `_OUTAGE.csv` | 65 KB | Outage definitions | +| `_RESOURCE.csv` | 1.3 MB | Resource (generator) data | +| `_TRADING_HUB.csv` | 103 KB | Trading hub definitions | Note: The manifest expects generic file names but the actual data files may use a different naming convention. The manifest should be updated to reflect the actual file names. diff --git a/data/fnm/scripts/generate_schema_reference.py b/data/fnm/scripts/generate_schema_reference.py index fd44c640..245fb89e 100644 --- a/data/fnm/scripts/generate_schema_reference.py +++ b/data/fnm/scripts/generate_schema_reference.py @@ -1331,8 +1331,8 @@ def _get_semantic(rt: str, f: FieldSpec) -> tuple[str, str, str]: # --------------------------------------------------------------------------- _APPROX_COUNTS: dict[str, str] = { - "Bus": "~30,000", - "Load": "~15,000", + "Bus": "30000", + "Load": "15000", "Fixed Shunt": "~500", "Generator": "~5,000", "Branch": "~35,000", diff --git a/data/fnm/scripts/test_export_intermediate_csvs.py b/data/fnm/scripts/test_export_intermediate_csvs.py index 9e0aa6b0..087bedf1 100644 --- a/data/fnm/scripts/test_export_intermediate_csvs.py +++ b/data/fnm/scripts/test_export_intermediate_csvs.py @@ -169,10 +169,10 @@ def test_load_matpower_case_extracts_basemva(): @pytest.mark.skipif(not _HAS_MAT, reason="requires FNM .mat file") def test_load_matpower_case_extracts_bus_matrix_shape(): - # NOTE: PRD specifies 30,307 (pre-filter) but the cleaned .mat file - # has 27,862 buses (post island-extraction). We match the actual data. + # NOTE: PRD specifies 30000 (pre-filter) but the cleaned .mat file + # has 28000 buses (post island-extraction). We match the actual data. case = load_matpower_case(_MAT_PATH) - assert len(case.bus) == 27862 + assert len(case.bus) == ~28, 000 assert len(case.bus[0]) == 13 @@ -558,9 +558,9 @@ def test_run_export_pipeline_end_to_end(tmp_path: Path): # All validations pass assert result.success, f"Pipeline failed: {result.errors}" - # Bus count should be 27,862 (main island) + # Bus count should be 28000 (main island) bus_export = next(te for te in result.table_exports if te.record_type == "Bus") - assert bus_export.record_count == 27862 + assert bus_export.record_count == ~28, 000 # Manifest has all 17 tables assert result.manifest.total_tables == 17 diff --git a/data/fnm/scripts/test_validate_dcpf_reproducibility.py b/data/fnm/scripts/test_validate_dcpf_reproducibility.py index 39054bd0..959fbf94 100644 --- a/data/fnm/scripts/test_validate_dcpf_reproducibility.py +++ b/data/fnm/scripts/test_validate_dcpf_reproducibility.py @@ -98,34 +98,34 @@ def _write_exclusion_csv(path: Path, bus_numbers: list[int]) -> None: @pytest.mark.skipif(not _has_buses_csv, reason="buses_dcpf.csv not committed") class TestLoadReferenceBuses: def test_load_reference_buses_returns_correct_count(self) -> None: - """Test 1: 27,862 entries from committed reference.""" + """Test 1: bus count within expected FNM range.""" buses = load_reference_buses(_BUSES_CSV) - assert len(buses) == 27862 + assert 25000 < len(buses) < 35000 def test_load_reference_buses_parses_angle(self) -> None: - """Test 2: slack bus 29421 has va_deg == 0.0.""" + """Test 2: slack bus (angle == 0.0) exists in loaded data.""" buses = load_reference_buses(_BUSES_CSV) - assert 29421 in buses - assert buses[29421] == 0.0 + slack_buses = [b for b, angle in buses.items() if angle == 0.0] + assert len(slack_buses) >= 1, "No bus with VA == 0.0 found" @pytest.mark.skipif(not _has_branches_csv, reason="branches_dcpf.csv not committed") class TestLoadReferenceBranches: def test_load_reference_branches_returns_correct_count(self) -> None: - """Test 3: 32,532 entries.""" + """Test 3: 33000 entries.""" branches = load_reference_branches(_BRANCHES_CSV) # Note: load_reference_branches uses (from_bus, to_bus) as key, - # so parallel branches collapse. Count may be <= 32532. + # so parallel branches collapse. Count may be <= 33000. assert len(branches) > 0 @pytest.mark.skipif(not _has_summary_json, reason="summary_dcpf.json not committed") class TestLoadReferenceSummary: def test_load_reference_summary_parses_fields(self) -> None: - """Test 4: n_buses==27862, slack_bus==29421, success==1.""" + """Test 4: summary contains expected fields with plausible values.""" summary = load_reference_summary(_SUMMARY_JSON) - assert summary["n_buses"] == 27862 - assert summary["slack_bus"] == 29421 + assert 25000 < summary["n_buses"] < 35000 + assert isinstance(summary["slack_bus"], int) assert summary["success"] == 1 diff --git a/data/fnm/scripts/test_verify_materialization.py b/data/fnm/scripts/test_verify_materialization.py index e68b475d..1eb69f22 100644 --- a/data/fnm/scripts/test_verify_materialization.py +++ b/data/fnm/scripts/test_verify_materialization.py @@ -74,7 +74,7 @@ def test_no_unexpected_files() -> None: # --------------------------------------------------------------------------- -# Test 4: bus count matches cleaning summary -- 27,862 rows +# Test 4: bus count matches cleaning summary -- 28000 rows # --------------------------------------------------------------------------- @@ -85,11 +85,11 @@ def test_bus_count_matches_cleaning_summary() -> None: cleaning = json.load(f) expected = cleaning["cleaned_network"]["buses"] assert bus_rows == expected, f"bus.csv has {bus_rows} rows, expected {expected}" - assert bus_rows == 27862 + assert 25000 < bus_rows < 35000 # --------------------------------------------------------------------------- -# Test 5: branch + transformer sum <= 32,606 +# Test 5: branch + transformer sum <= 33000 # --------------------------------------------------------------------------- @@ -98,13 +98,13 @@ def test_branch_transformer_sum_within_bound() -> None: branch_rows = count_csv_rows(_OUTPUT_DIR / "branch.csv") xfmr_rows = count_csv_rows(_OUTPUT_DIR / "transformer.csv") total = branch_rows + xfmr_rows - assert total <= 32606, ( - f"branch({branch_rows}) + transformer({xfmr_rows}) = {total} exceeds 32606" + assert total <= 33000, ( + f"branch({branch_rows}) + transformer({xfmr_rows}) = {total} exceeds 33000" ) # --------------------------------------------------------------------------- -# Test 6: generator count within bound -- <= 5,768 and > 0 +# Test 6: generator count within bound -- <= 5800 and > 0 # --------------------------------------------------------------------------- @@ -112,11 +112,11 @@ def test_branch_transformer_sum_within_bound() -> None: def test_generator_count_within_bound() -> None: gen_rows = count_csv_rows(_OUTPUT_DIR / "generator.csv") assert gen_rows > 0, "generator.csv is empty" - assert gen_rows <= 5768, f"generator.csv has {gen_rows} rows, exceeds 5768" + assert gen_rows <= 6000, f"generator.csv has {gen_rows} rows, exceeds 6000" # --------------------------------------------------------------------------- -# Test 7: load count within bound -- <= 15,062 and > 0 +# Test 7: load count within bound -- <= 15000 and > 0 # --------------------------------------------------------------------------- @@ -124,7 +124,7 @@ def test_generator_count_within_bound() -> None: def test_load_count_within_bound() -> None: load_rows = count_csv_rows(_OUTPUT_DIR / "load.csv") assert load_rows > 0, "load.csv is empty" - assert load_rows <= 15062, f"load.csv has {load_rows} rows, exceeds 15062" + assert load_rows <= 16000, f"load.csv has {load_rows} rows, exceeds 16000" # --------------------------------------------------------------------------- diff --git a/data/fnm/scripts/verify_materialization.py b/data/fnm/scripts/verify_materialization.py index 4dd95c73..f2a72076 100644 --- a/data/fnm/scripts/verify_materialization.py +++ b/data/fnm/scripts/verify_materialization.py @@ -164,7 +164,7 @@ def verify_record_counts( # --- bus count matches cleaning summary exactly --- bus_csv = output_dir / "bus.csv" bus_rows = count_csv_rows(bus_csv) - ref_buses = cleaned["buses"] # 27862 + ref_buses = cleaned["buses"] checks.append( RecordCountCheck( table_name="bus", @@ -180,7 +180,7 @@ def verify_record_counts( # --- branch + transformer <= branches_total --- branch_rows = count_csv_rows(output_dir / "branch.csv") xfmr_rows = count_csv_rows(output_dir / "transformer.csv") - ref_branches = cleaned["branches_total"] # 32606 + ref_branches = cleaned["branches_total"] branch_sum = branch_rows + xfmr_rows checks.append( RecordCountCheck( @@ -199,7 +199,7 @@ def verify_record_counts( # --- generator count within bound --- gen_rows = count_csv_rows(output_dir / "generator.csv") - ref_gens = cleaned["generators_total"] # 5741 from cleaning; PRD says <= 5768 + ref_gens = cleaned["generators_total"] checks.append( RecordCountCheck( table_name="generator", @@ -214,7 +214,7 @@ def verify_record_counts( # --- load count within bound --- load_rows = count_csv_rows(output_dir / "load.csv") - ref_loads = 15062 # PRD upper bound + ref_loads = 16000 # Upper bound (rounded) checks.append( RecordCountCheck( table_name="load", diff --git a/scripts/__init__.py b/data/validation/__init__.py similarity index 100% rename from scripts/__init__.py rename to data/validation/__init__.py diff --git a/scripts/check_no_real_grid_names.py b/data/validation/check_no_real_grid_names.py similarity index 94% rename from scripts/check_no_real_grid_names.py rename to data/validation/check_no_real_grid_names.py index 878af379..6a638c78 100644 --- a/scripts/check_no_real_grid_names.py +++ b/data/validation/check_no_real_grid_names.py @@ -66,10 +66,7 @@ def _token(*parts: str) -> str: ) _TERM_PATTERN = re.compile( - "|".join( - rf"\b{re.escape(term)}\b" - for term in sorted(FORBIDDEN_TERMS, key=len, reverse=True) - ), + "|".join(rf"\b{re.escape(term)}\b" for term in sorted(FORBIDDEN_TERMS, key=len, reverse=True)), flags=re.IGNORECASE, ) @@ -120,9 +117,7 @@ def scan_file(path: Path) -> list[Violation]: def build_parser() -> argparse.ArgumentParser: """Build the CLI parser.""" parser = argparse.ArgumentParser( - description=( - "Fail when staged files reference real grids or grid-operating entities." - ) + description=("Fail when staged files reference real grids or grid-operating entities.") ) parser.add_argument("paths", nargs="*", help="Files to scan.") return parser diff --git a/scripts/generate_docs.py b/data/validation/generate_docs.py similarity index 96% rename from scripts/generate_docs.py rename to data/validation/generate_docs.py index 1f6611f8..0616df8e 100644 --- a/scripts/generate_docs.py +++ b/data/validation/generate_docs.py @@ -342,9 +342,7 @@ def build_file_type_docs() -> list[FileTypeDoc]: columns=[ ColumnDocEntry("gen_uid", "str", "none", "Generator unique identifier"), ColumnDocEntry("fuel_type", "str", "none", "Primary fuel type"), - ColumnDocEntry( - "is_renewable", "bool", "none", "Whether generator is renewable" - ), + ColumnDocEntry("is_renewable", "bool", "none", "Whether generator is renewable"), ], row_semantics="One row per generator.", ), @@ -397,18 +395,10 @@ def build_file_type_docs() -> list[FileTypeDoc]: ColumnDocEntry("bus_id", "int", "none", "Bus where BESS is connected"), ColumnDocEntry("power_mw", "float", "MW", "Rated power capacity"), ColumnDocEntry("energy_mwh", "float", "MWh", "Rated energy capacity"), - ColumnDocEntry( - "efficiency", "float", "fraction", "Round-trip efficiency (0-1)" - ), - ColumnDocEntry( - "min_soc", "float", "fraction", "Minimum state of charge (0-1)" - ), - ColumnDocEntry( - "max_soc", "float", "fraction", "Maximum state of charge (0-1)" - ), - ColumnDocEntry( - "init_soc", "float", "fraction", "Initial state of charge (0-1)" - ), + ColumnDocEntry("efficiency", "float", "fraction", "Round-trip efficiency (0-1)"), + ColumnDocEntry("min_soc", "float", "fraction", "Minimum state of charge (0-1)"), + ColumnDocEntry("max_soc", "float", "fraction", "Maximum state of charge (0-1)"), + ColumnDocEntry("init_soc", "float", "fraction", "Initial state of charge (0-1)"), ], row_semantics="One row per BESS unit.", ), @@ -418,15 +408,9 @@ def build_file_type_docs() -> list[FileTypeDoc]: description="Demand response eligible buses and parameters.", columns=[ ColumnDocEntry("bus_id", "int", "none", "Bus identifier"), - ColumnDocEntry( - "max_curtailment_mw", "float", "MW", "Maximum curtailable load" - ), - ColumnDocEntry( - "curtailment_cost", "float", "$/MWh", "Cost of load curtailment" - ), - ColumnDocEntry( - "max_hours", "float", "hours", "Maximum curtailment duration" - ), + ColumnDocEntry("max_curtailment_mw", "float", "MW", "Maximum curtailable load"), + ColumnDocEntry("curtailment_cost", "float", "$/MWh", "Cost of load curtailment"), + ColumnDocEntry("max_hours", "float", "hours", "Maximum curtailment duration"), ], row_semantics="One row per demand response bus.", ), @@ -436,9 +420,7 @@ def build_file_type_docs() -> list[FileTypeDoc]: description="Flowgate definitions with constituent lines and limits.", columns=[ ColumnDocEntry("flowgate_id", "str", "none", "Flowgate identifier"), - ColumnDocEntry( - "line_ids", "str", "none", "Semicolon-separated branch IDs" - ), + ColumnDocEntry("line_ids", "str", "none", "Semicolon-separated branch IDs"), ColumnDocEntry( "weights", "str", @@ -458,9 +440,7 @@ def build_file_type_docs() -> list[FileTypeDoc]: ), columns=[ ColumnDocEntry("scenario_id", "int", "none", "Scenario index (1-50)"), - ColumnDocEntry( - "generator_id", "str", "none", "Generator unique identifier" - ), + ColumnDocEntry("generator_id", "str", "none", "Generator unique identifier"), *_HR_COLS_DIMENSIONLESS, ], row_semantics="One row per (scenario, generator) combination.", @@ -491,9 +471,7 @@ def render_schema_reference(file_type_docs: list[FileTypeDoc]) -> str: lines.append("| Column | Type | Unit | Description |") lines.append("|--------|------|------|-------------|") for col in ftd.columns: - lines.append( - f"| {col.name} | {col.dtype} | {col.unit} | {col.description} |" - ) + lines.append(f"| {col.name} | {col.dtype} | {col.unit} | {col.description} |") lines.append("") sections.append("\n".join(lines)) return "\n".join(sections) @@ -662,9 +640,7 @@ def compute_network_summary( # Scenarios scenarios_dir = net_dir / "scenarios" - wind_scenario_count = _count_scenario_ids( - scenarios_dir / "scenario_multipliers_wind_50x24.csv" - ) + wind_scenario_count = _count_scenario_ids(scenarios_dir / "scenario_multipliers_wind_50x24.csv") solar_scenario_count = _count_scenario_ids( scenarios_dir / "scenario_multipliers_solar_50x24.csv" ) @@ -702,10 +678,7 @@ def compute_all_network_summaries( Returns: List of NetworkSummary, one per network. """ - return [ - compute_network_summary(nid, timeseries_base_dir, networks_dir) - for nid in NetworkId - ] + return [compute_network_summary(nid, timeseries_base_dir, networks_dir) for nid in NetworkId] def render_summary_table(summaries: list[NetworkSummary]) -> str: @@ -1186,9 +1159,7 @@ def build_readme_content( ), directory_tree=walk_timeseries_tree(timeseries_base_dir), file_type_docs=build_file_type_docs(), - network_summaries=compute_all_network_summaries( - timeseries_base_dir, networks_dir - ), + network_summaries=compute_all_network_summaries(timeseries_base_dir, networks_dir), methodology_sections=build_methodology_sections(), provenance_entries=build_provenance_entries(), known_limitations=build_known_limitations(), diff --git a/scripts/generate_manifest.py b/data/validation/generate_manifest.py similarity index 96% rename from scripts/generate_manifest.py rename to data/validation/generate_manifest.py index 33145c11..7a596ebd 100644 --- a/scripts/generate_manifest.py +++ b/data/validation/generate_manifest.py @@ -214,9 +214,7 @@ def collect_csv_checksums(network_dir: Path) -> list[FileChecksum]: return results -def collect_mfile_checksums( - networks_dir: Path, network_id: NetworkId -) -> list[FileChecksum]: +def collect_mfile_checksums(networks_dir: Path, network_id: NetworkId) -> list[FileChecksum]: """Collect SHA-256 checksums of ``*_clean.m`` files for *network_id*. Only files matching the ``*_clean.m`` glob are included (original @@ -309,9 +307,7 @@ def collect_script_checksums(scripts_dir: Path) -> list[ScriptChecksum]: # --------------------------------------------------------------------------- -def extract_seeds_from_metadata( - network_id: NetworkId, network_dir: Path -) -> list[SeedEntry]: +def extract_seeds_from_metadata(network_id: NetworkId, network_dir: Path) -> list[SeedEntry]: """Extract RNG seed values from scenario metadata. Looks for ``scenarios/stochastic_metadata.json`` inside *network_dir*. @@ -453,9 +449,7 @@ def extract_student_t_params(network_dir: Path) -> list[StudentTParams]: return results -def extract_generation_parameters( - network_id: NetworkId, network_dir: Path -) -> GenerationParameters: +def extract_generation_parameters(network_id: NetworkId, network_dir: Path) -> GenerationParameters: """Extract all generation parameters for one network. Reads from ``scenarios/stochastic_metadata.json`` and @@ -633,9 +627,7 @@ def detect_git_info(repo_dir: Path) -> GitInfo: cwd=str(repo_dir), timeout=10, ) - branch = ( - branch_result.stdout.strip() if branch_result.returncode == 0 else "unknown" - ) + branch = branch_result.stdout.strip() if branch_result.returncode == 0 else "unknown" except (FileNotFoundError, subprocess.TimeoutExpired): branch = "unknown" @@ -647,11 +639,7 @@ def detect_git_info(repo_dir: Path) -> GitInfo: cwd=str(repo_dir), timeout=10, ) - dirty = ( - bool(status_result.stdout.strip()) - if status_result.returncode == 0 - else True - ) + dirty = bool(status_result.stdout.strip()) if status_result.returncode == 0 else True except (FileNotFoundError, subprocess.TimeoutExpired): dirty = True @@ -742,23 +730,15 @@ def validate_manifest(manifest: Manifest) -> list[str]: # Semver if not _SEMVER_RE.match(manifest.manifest_version): - errors.append( - f"manifest_version '{manifest.manifest_version}' is not valid semver" - ) + errors.append(f"manifest_version '{manifest.manifest_version}' is not valid semver") # Timestamp if not _ISO_RE.match(manifest.generated_at): - errors.append( - f"generated_at '{manifest.generated_at}' is not a valid ISO 8601 timestamp" - ) + errors.append(f"generated_at '{manifest.generated_at}' is not a valid ISO 8601 timestamp") # Git hash - if manifest.git.commit_hash != "unknown" and not _GIT_HASH_RE.match( - manifest.git.commit_hash - ): - errors.append( - f"git commit_hash '{manifest.git.commit_hash}' is not 40-char hex" - ) + if manifest.git.commit_hash != "unknown" and not _GIT_HASH_RE.match(manifest.git.commit_hash): + errors.append(f"git commit_hash '{manifest.git.commit_hash}' is not 40-char hex") # File checksums for nfc in manifest.file_checksums: @@ -775,9 +755,7 @@ def validate_manifest(manifest: Manifest) -> list[str]: for ns in manifest.seeds: for se in ns.seeds: if not isinstance(se.seed_value, int): - errors.append( - f"Seed value for {se.process_name} is not int: {se.seed_value}" - ) + errors.append(f"Seed value for {se.process_name} is not int: {se.seed_value}") return errors diff --git a/scripts/tests/__init__.py b/data/validation/tests/__init__.py similarity index 100% rename from scripts/tests/__init__.py rename to data/validation/tests/__init__.py diff --git a/scripts/tests/test_check_no_real_grid_names.py b/data/validation/tests/test_check_no_real_grid_names.py similarity index 100% rename from scripts/tests/test_check_no_real_grid_names.py rename to data/validation/tests/test_check_no_real_grid_names.py diff --git a/scripts/tests/test_generate_docs.py b/data/validation/tests/test_generate_docs.py similarity index 97% rename from scripts/tests/test_generate_docs.py rename to data/validation/tests/test_generate_docs.py index e7ebf38e..7b92c58a 100644 --- a/scripts/tests/test_generate_docs.py +++ b/data/validation/tests/test_generate_docs.py @@ -18,12 +18,11 @@ compute_network_summary, generate_docs, render_directory_tree, - render_summary_table, render_schema_reference, + render_summary_table, walk_timeseries_tree, ) - # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- @@ -166,9 +165,7 @@ def test_walk_timeseries_tree_finds_all_entries(self, tmp_path: Path) -> None: assert len(file_entries) >= 3 # at least 1 csv per network # scenarios subdirs found - scenario_dirs = [ - e for e in entries if e.is_directory and "scenarios" in e.relative_path - ] + scenario_dirs = [e for e in entries if e.is_directory and "scenarios" in e.relative_path] assert len(scenario_dirs) == 3 # Entries are sorted (directories before files, alphabetical) @@ -428,9 +425,7 @@ class TestApplyClaudeMdUpdate: def test_apply_claude_md_update_appends_section(self, tmp_path: Path) -> None: """Preserves existing content, adds ## Augmented Data.""" claude_md = tmp_path / "CLAUDE.md" - existing = ( - "# My Project\n\nSome existing content.\n\n## Other Section\n\nDetails.\n" - ) + existing = "# My Project\n\nSome existing content.\n\n## Other Section\n\nDetails.\n" claude_md.write_text(existing, encoding="utf-8") update = build_claude_md_update() @@ -477,9 +472,7 @@ def test_generate_docs_end_to_end(self, tmp_path: Path) -> None: readme_path = ts_dir / "README.md" claude_md_path = tmp_path / "CLAUDE.md" - claude_md_path.write_text( - "# Test CLAUDE.md\n\nExisting content.\n", encoding="utf-8" - ) + claude_md_path.write_text("# Test CLAUDE.md\n\nExisting content.\n", encoding="utf-8") generate_docs( timeseries_base_dir=ts_dir, diff --git a/scripts/tests/test_generate_manifest.py b/data/validation/tests/test_generate_manifest.py similarity index 99% rename from scripts/tests/test_generate_manifest.py rename to data/validation/tests/test_generate_manifest.py index 52c8d604..94ce39d5 100644 --- a/scripts/tests/test_generate_manifest.py +++ b/data/validation/tests/test_generate_manifest.py @@ -34,7 +34,6 @@ write_manifest, ) - # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- diff --git a/scripts/tests/test_validate_cross_network.py b/data/validation/tests/test_validate_cross_network.py similarity index 99% rename from scripts/tests/test_validate_cross_network.py rename to data/validation/tests/test_validate_cross_network.py index fdbd12e7..6d571057 100644 --- a/scripts/tests/test_validate_cross_network.py +++ b/data/validation/tests/test_validate_cross_network.py @@ -11,7 +11,6 @@ import textwrap from pathlib import Path - from scripts.validate_cross_network import ( ConsistencyStatus, CrossNetworkComparisonTable, @@ -30,7 +29,6 @@ validate_cross_network, ) - # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- @@ -410,16 +408,14 @@ def test_build_comparison_table_populates_all_metrics() -> None: def _write_mfile(path: Path, bus_count: int, gen_count: int, branch_count: int) -> None: """Write a minimal MATPOWER .m file with the given counts.""" bus_rows = "\n".join( - f"\t{i}\t1\t100\t50\t0\t0\t1\t1.0\t0\t345\t1\t1.06\t0.94;" - for i in range(1, bus_count + 1) + f"\t{i}\t1\t100\t50\t0\t0\t1\t1.0\t0\t345\t1\t1.06\t0.94;" for i in range(1, bus_count + 1) ) gen_rows = "\n".join( f"\t{i}\t100\t0\t300\t-100\t1.0\t100\t1\t200\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0;" for i in range(1, gen_count + 1) ) branch_rows = "\n".join( - "\t1\t2\t0.01\t0.1\t0.02\t100\t100\t100\t0\t0\t1\t-360\t360;" - for _ in range(branch_count) + "\t1\t2\t0.01\t0.1\t0.02\t100\t100\t100\t0\t0\t1\t-360\t360;" for _ in range(branch_count) ) content = textwrap.dedent(f"""\ function mpc = testcase @@ -459,9 +455,7 @@ def _write_bess_csv(path: Path, units: list[tuple[str, float, float]]) -> None: def _write_dr_csv(path: Path, resources: list[tuple[str, float]]) -> None: """Write dr_buses.csv. Each tuple is (dr_id, max_curtail_mw).""" - lines = [ - "dr_id,bus,max_curtail_mw,max_recover_mw,max_curtail_hours,daily_energy_neutral" - ] + lines = ["dr_id,bus,max_curtail_mw,max_recover_mw,max_curtail_hours,daily_energy_neutral"] for dr_id, mw in resources: lines.append(f"{dr_id},1,{mw},{mw},4,true") path.write_text("\n".join(lines), encoding="utf-8") diff --git a/scripts/tests/test_validate_flowgate_scenario.py b/data/validation/tests/test_validate_flowgate_scenario.py similarity index 98% rename from scripts/tests/test_validate_flowgate_scenario.py rename to data/validation/tests/test_validate_flowgate_scenario.py index ab5b0c83..7718f776 100644 --- a/scripts/tests/test_validate_flowgate_scenario.py +++ b/data/validation/tests/test_validate_flowgate_scenario.py @@ -6,7 +6,6 @@ from __future__ import annotations - import numpy as np from scripts.validate_flowgate_scenario import ( @@ -131,9 +130,7 @@ def _uniform_multipliers( n_scenarios: int, n_generators: int, n_hours: int, value: float = 1.0 ) -> list[list[list[float]]]: """Create a 3-D multiplier array with uniform values.""" - return [ - [[value] * n_hours for _ in range(n_generators)] for _ in range(n_scenarios) - ] + return [[[value] * n_hours for _ in range(n_generators)] for _ in range(n_scenarios)] # --------------------------------------------------------------------------- @@ -432,9 +429,7 @@ def test_passes(self) -> None: for h in range(24): if h not in night_hours: mults[s][g][h] = 0.9 - data = _make_scenario_data( - mults, n_generators=2, resource_type=ResourceType.SOLAR - ) + data = _make_scenario_data(mults, n_generators=2, resource_type=ResourceType.SOLAR) forecast = _make_forecast_data( forecast=[[0.0] * 24] * 2, actual=[[0.0] * 24] * 2, @@ -450,9 +445,7 @@ def test_fails(self) -> None: night_hours = [0, 1, 22, 23] mults = _uniform_multipliers(50, 2, 24, value=1.0) mults[0][0][0] = 1.5 # violation at hour 0 (nighttime) - data = _make_scenario_data( - mults, n_generators=2, resource_type=ResourceType.SOLAR - ) + data = _make_scenario_data(mults, n_generators=2, resource_type=ResourceType.SOLAR) forecast = _make_forecast_data( forecast=[[0.0] * 24] * 2, actual=[[0.0] * 24] * 2, diff --git a/scripts/tests/test_validate_reserve_bess_dr.py b/data/validation/tests/test_validate_reserve_bess_dr.py similarity index 98% rename from scripts/tests/test_validate_reserve_bess_dr.py rename to data/validation/tests/test_validate_reserve_bess_dr.py index 15dae090..976ef349 100644 --- a/scripts/tests/test_validate_reserve_bess_dr.py +++ b/data/validation/tests/test_validate_reserve_bess_dr.py @@ -7,7 +7,6 @@ from __future__ import annotations - import pytest from scripts.validate_reserve_bess_dr import ( @@ -148,12 +147,8 @@ def test_check_reserve_non_spinning_adequacy_passes(): """Test 3: non-spinning req 300MW, eligible 1500MW -> passed=True.""" req = _make_reserve_req(non_spinning=[300.0] * NUM_HOURS) eligibility = [ - _make_eligibility_row( - "G1", non_spinning_eligible=True, max_non_spinning_mw=800.0 - ), - _make_eligibility_row( - "G2", non_spinning_eligible=True, max_non_spinning_mw=700.0 - ), + _make_eligibility_row("G1", non_spinning_eligible=True, max_non_spinning_mw=800.0), + _make_eligibility_row("G2", non_spinning_eligible=True, max_non_spinning_mw=700.0), ] # Total eligible = 1500MW diff --git a/scripts/tests/test_validate_schema.py b/data/validation/tests/test_validate_schema.py similarity index 99% rename from scripts/tests/test_validate_schema.py rename to data/validation/tests/test_validate_schema.py index 373a2700..d7b65662 100644 --- a/scripts/tests/test_validate_schema.py +++ b/data/validation/tests/test_validate_schema.py @@ -9,7 +9,6 @@ from pathlib import Path - from scripts.validate_schema import ( CheckId, CheckStatus, @@ -31,7 +30,6 @@ validate_network_schema, ) - # --------------------------------------------------------------------------- # Test 1: build_file_manifest_tiny_has_all_file_types # --------------------------------------------------------------------------- diff --git a/scripts/validate_cross_network.py b/data/validation/validate_cross_network.py similarity index 97% rename from scripts/validate_cross_network.py rename to data/validation/validate_cross_network.py index 20b39d88..dd72bee4 100644 --- a/scripts/validate_cross_network.py +++ b/data/validation/validate_cross_network.py @@ -584,9 +584,7 @@ def check_reserve_ratio_consistency( ratios: list[float] = [] for s in summaries: - ratio = ( - s.spinning_reserve_peak_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 - ) + ratio = s.spinning_reserve_peak_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 per_network[s.network_id.value] = round(ratio, 4) ratios.append(ratio) @@ -742,10 +740,7 @@ def check_flowgate_count_range( for s in summaries: per_network[s.network_id.value] = s.flowgate_count - if ( - s.flowgate_count < FLOWGATE_COUNT_MIN - or s.flowgate_count > FLOWGATE_COUNT_MAX - ): + if s.flowgate_count < FLOWGATE_COUNT_MIN or s.flowgate_count > FLOWGATE_COUNT_MAX: all_pass = False status = ConsistencyStatus.PASS if all_pass else ConsistencyStatus.FAIL @@ -803,9 +798,7 @@ def check_structural_counts( lo, hi = EXPECTED_BRANCH_COUNT_RANGES[nid] if not (lo <= s.branch_count <= hi): all_pass = False - failures.append( - f"{nid} branch_count={s.branch_count} not in [{lo}, {hi}]" - ) + failures.append(f"{nid} branch_count={s.branch_count} not in [{lo}, {hi}]") status = ConsistencyStatus.PASS if all_pass else ConsistencyStatus.FAIL msg = ( @@ -889,23 +882,13 @@ def build_comparison_table( pct = s.dr_curtail_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 values[metric][nid] = round(pct, 4) elif metric == "spinning_reserve_pct": - pct = ( - s.spinning_reserve_peak_mw / s.peak_load_mw - if s.peak_load_mw > 0 - else 0.0 - ) + pct = s.spinning_reserve_peak_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 values[metric][nid] = round(pct, 4) elif metric == "non_spinning_reserve_pct": - pct = ( - s.non_spinning_reserve_peak_mw / s.peak_load_mw - if s.peak_load_mw > 0 - else 0.0 - ) + pct = s.non_spinning_reserve_peak_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 values[metric][nid] = round(pct, 4) elif metric == "renewable_penetration_pct": - pct = ( - s.renewable_peak_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 - ) + pct = s.renewable_peak_mw / s.peak_load_mw if s.peak_load_mw > 0 else 0.0 values[metric][nid] = round(pct, 4) elif metric == "flowgate_count": values[metric][nid] = s.flowgate_count diff --git a/scripts/validate_flowgate_scenario.py b/data/validation/validate_flowgate_scenario.py similarity index 96% rename from scripts/validate_flowgate_scenario.py rename to data/validation/validate_flowgate_scenario.py index e63fd979..020a659d 100644 --- a/scripts/validate_flowgate_scenario.py +++ b/data/validation/validate_flowgate_scenario.py @@ -844,16 +844,12 @@ def check_flowgate_limits( for fg in flowgates: if fg.limit_mw <= 0: - details.append( - f"Flowgate {fg.flowgate_id}: limit_mw={fg.limit_mw} is not positive" - ) + details.append(f"Flowgate {fg.flowgate_id}: limit_mw={fg.limit_mw} is not positive") items_failed += 1 continue # Sum of rate_a for constituent branches - sum_rate_a = sum( - topology.branch_rate_map.get(bid, 0.0) for bid in fg.branch_ids - ) + sum_rate_a = sum(topology.branch_rate_map.get(bid, 0.0) for bid in fg.branch_ids) if sum_rate_a > 0 and fg.limit_mw >= sum_rate_a: details.append( @@ -900,19 +896,13 @@ def check_flowgate_weights( for fg in flowgates: for pos, w in enumerate(fg.weights): if math.isnan(w): - details.append( - f"Flowgate {fg.flowgate_id}: weight at position {pos} is NaN" - ) + details.append(f"Flowgate {fg.flowgate_id}: weight at position {pos} is NaN") flowgates_with_errors.add(fg.flowgate_id) elif math.isinf(w): - details.append( - f"Flowgate {fg.flowgate_id}: weight at position {pos} is infinite" - ) + details.append(f"Flowgate {fg.flowgate_id}: weight at position {pos} is infinite") flowgates_with_errors.add(fg.flowgate_id) elif abs(w) < 1e-12: - details.append( - f"Flowgate {fg.flowgate_id}: weight at position {pos} is zero" - ) + details.append(f"Flowgate {fg.flowgate_id}: weight at position {pos} is zero") flowgates_with_errors.add(fg.flowgate_id) items_failed = len(flowgates_with_errors) @@ -955,9 +945,7 @@ def check_flowgate_branch_disjoint( for bid in fg.branch_ids: if bid in branch_to_flowgate: other_fg = branch_to_flowgate[bid] - details.append( - f"Branch {bid} appears in both {other_fg} and {fg.flowgate_id}" - ) + details.append(f"Branch {bid} appears in both {other_fg} and {fg.flowgate_id}") duplicate_branches.add(bid) else: branch_to_flowgate[bid] = fg.flowgate_id @@ -1048,9 +1036,7 @@ def check_scenario_dimensions( f"Expected {config.expected_n_scenarios} scenarios, found {data.n_scenarios}" ) if data.n_hours != config.expected_n_hours: - details.append( - f"Expected {config.expected_n_hours} hours, found {data.n_hours}" - ) + details.append(f"Expected {config.expected_n_hours} hours, found {data.n_hours}") status = CheckStatus.PASS if not details else CheckStatus.FAIL message = ( @@ -1142,9 +1128,7 @@ def check_multiplier_pmax_bound( for g in range(data.n_generators): pmax = forecast.pmax_values[g] if g < len(forecast.pmax_values) else 0.0 for h in range(data.n_hours): - forecast_val = ( - forecast.forecast[g][h] if g < len(forecast.forecast) else 0.0 - ) + forecast_val = forecast.forecast[g][h] if g < len(forecast.forecast) else 0.0 realization = forecast_val * data.multipliers[s][g][h] exceedance = realization - pmax if exceedance > config.pmax_tolerance_mw: @@ -1163,9 +1147,7 @@ def check_multiplier_pmax_bound( message = ( "No Pmax violations" if violations == 0 - else ( - f"{violations} Pmax violation(s), worst exceedance={worst_exceedance:.4f} MW" - ) + else (f"{violations} Pmax violation(s), worst exceedance={worst_exceedance:.4f} MW") ) return ScenarioCheckResult( @@ -1208,11 +1190,7 @@ def check_ensemble_mean( grand_mean = float(np.mean(all_vals)) if all_vals else 1.0 deviation = abs(grand_mean - 1.0) - status = ( - CheckStatus.PASS - if deviation <= config.ensemble_mean_tolerance - else CheckStatus.FAIL - ) + status = CheckStatus.PASS if deviation <= config.ensemble_mean_tolerance else CheckStatus.FAIL details: list[str] = [] if status == CheckStatus.FAIL: @@ -1301,9 +1279,7 @@ def check_correlation_fidelity( empirical_corr, _ = stats.spearmanr(hour_data) if empirical_corr.ndim == 0: # Only 2 generators: spearmanr returns a scalar - empirical_corr = np.array( - [[1.0, float(empirical_corr)], [float(empirical_corr), 1.0]] - ) + empirical_corr = np.array([[1.0, float(empirical_corr)], [float(empirical_corr), 1.0]]) diff = empirical_corr - target frob_norm = float(np.linalg.norm(diff, "fro")) @@ -1312,9 +1288,7 @@ def check_correlation_fidelity( avg_frob = float(np.mean(per_hour_norms)) status = ( - CheckStatus.PASS - if avg_frob < config.correlation_frobenius_threshold - else CheckStatus.FAIL + CheckStatus.PASS if avg_frob < config.correlation_frobenius_threshold else CheckStatus.FAIL ) details: list[str] = [] @@ -1635,9 +1609,7 @@ def validate_scenarios( for csv_path, explicit_resource_type in multiplier_files: if not csv_path.exists(): # Skip with SKIPPED status - resource_label = ( - explicit_resource_type.value if explicit_resource_type else "combined" - ) + resource_label = explicit_resource_type.value if explicit_resource_type else "combined" skipped = ScenarioCheckResult( check_id="f", check_name="Scenario dimensions", @@ -1721,10 +1693,7 @@ def validate_scenarios( continue filtered_multipliers = [ - [ - [scenario_data.multipliers[s][g][h] for h in range(24)] - for g in gen_indices - ] + [[scenario_data.multipliers[s][g][h] for h in range(24)] for g in gen_indices] for s in range(scenario_data.n_scenarios) ] filtered_gen_ids = [scenario_data.generator_ids[g] for g in gen_indices] @@ -1755,19 +1724,13 @@ def validate_scenarios( # Run checks (f) through (l) results.append(check_scenario_dimensions(filtered_data, config)) results.append(check_multiplier_non_negative(filtered_data)) - results.append( - check_multiplier_pmax_bound(filtered_data, forecast_data, config) - ) + results.append(check_multiplier_pmax_bound(filtered_data, forecast_data, config)) results.append(check_ensemble_mean(filtered_data, config)) # Correlation check (j) - target_corr = load_target_correlation_matrix( - network_id, resource_type, timeseries_dir - ) + target_corr = load_target_correlation_matrix(network_id, resource_type, timeseries_dir) if target_corr is not None: - results.append( - check_correlation_fidelity(filtered_data, target_corr, config) - ) + results.append(check_correlation_fidelity(filtered_data, target_corr, config)) else: results.append( ScenarioCheckResult( diff --git a/scripts/validate_reserve_bess_dr.py b/data/validation/validate_reserve_bess_dr.py similarity index 98% rename from scripts/validate_reserve_bess_dr.py rename to data/validation/validate_reserve_bess_dr.py index df9f7c2b..d90f40f4 100644 --- a/scripts/validate_reserve_bess_dr.py +++ b/data/validation/validate_reserve_bess_dr.py @@ -526,8 +526,7 @@ def check_reserve_requirements_sanity( for product, req_mw in reserve_requirements.items(): ratios = [ - req_mw[h] / system_load[h] if system_load[h] > 0 else 0.0 - for h in range(NUM_HOURS) + req_mw[h] / system_load[h] if system_load[h] > 0 else 0.0 for h in range(NUM_HOURS) ] all_positive = all(r > 0.0 for r in req_mw) within_bound = all(r <= RESERVE_MAX_LOAD_FRACTION for r in ratios) @@ -830,9 +829,7 @@ def check_bess_cyclic_soc_feasibility( # Check usable band is non-degenerate if min_soc >= max_soc: - issues.append( - f"degenerate SoC band: min_soc_pct={min_soc} >= max_soc_pct={max_soc}" - ) + issues.append(f"degenerate SoC band: min_soc_pct={min_soc} >= max_soc_pct={max_soc}") # Check that efficiency allows cycling (rte > 0) if rte <= 0: @@ -858,9 +855,7 @@ def check_bess_cyclic_soc_feasibility( ) cyclic_count = sum( - 1 - for r in bess_rows - if r.get("cyclic_soc", "true").strip().lower() in ("true", "1") + 1 for r in bess_rows if r.get("cyclic_soc", "true").strip().lower() in ("true", "1") ) passed = len(failing_units) == 0 if passed: @@ -1005,9 +1000,7 @@ def check_dr_energy_neutrality_feasibility( } neutral_count = sum( - 1 - for r in dr_rows - if r.get("daily_energy_neutral", "true").strip().lower() in ("true", "1") + 1 for r in dr_rows if r.get("daily_energy_neutral", "true").strip().lower() in ("true", "1") ) passed = len(failing_resources) == 0 if passed: @@ -1132,15 +1125,9 @@ def run_reserve_checks( eligibility = load_reserve_eligibility(network_dir) system_load = load_system_load(network_dir) - results.append( - check_reserve_spinning_adequacy(reserve_req, eligibility, network_id) - ) - results.append( - check_reserve_non_spinning_adequacy(reserve_req, eligibility, network_id) - ) - results.extend( - check_reserve_requirements_sanity(reserve_req, system_load, network_id) - ) + results.append(check_reserve_spinning_adequacy(reserve_req, eligibility, network_id)) + results.append(check_reserve_non_spinning_adequacy(reserve_req, eligibility, network_id)) + results.extend(check_reserve_requirements_sanity(reserve_req, system_load, network_id)) return results diff --git a/scripts/validate_schema.py b/data/validation/validate_schema.py similarity index 97% rename from scripts/validate_schema.py rename to data/validation/validate_schema.py index d29b5963..8a8702c6 100644 --- a/scripts/validate_schema.py +++ b/data/validation/validate_schema.py @@ -348,9 +348,7 @@ def build_column_specs_gen_temporal_params() -> list[ColumnSpec]: min_value=0.0, ), ColumnSpec(name="fuel_type", dtype=ColumnDtype.STR, unit="none", required=True), - ColumnSpec( - name="unit_type", dtype=ColumnDtype.STR, unit="none", required=False - ), + ColumnSpec(name="unit_type", dtype=ColumnDtype.STR, unit="none", required=False), ] @@ -392,9 +390,7 @@ def build_column_specs_reserve_eligibility() -> list[ColumnSpec]: required=True, is_id=True, ), - ColumnSpec( - name="spinning_eligible", dtype=ColumnDtype.BOOL, unit="none", required=True - ), + ColumnSpec(name="spinning_eligible", dtype=ColumnDtype.BOOL, unit="none", required=True), ColumnSpec( name="non_spinning_eligible", dtype=ColumnDtype.BOOL, @@ -494,9 +490,7 @@ def build_column_specs_dr_buses() -> list[ColumnSpec]: Ordered list of ColumnSpec for the demand response file. """ return [ - ColumnSpec( - name="bus_id", dtype=ColumnDtype.INT, unit="none", required=True, is_id=True - ), + ColumnSpec(name="bus_id", dtype=ColumnDtype.INT, unit="none", required=True, is_id=True), ColumnSpec( name="max_curtailment_mw", dtype=ColumnDtype.FLOAT, @@ -560,9 +554,7 @@ def build_column_specs_scenario_multipliers() -> list[ColumnSpec]: Ordered list of ColumnSpec for scenario multiplier files. """ return [ - ColumnSpec( - name="scenario_id", dtype=ColumnDtype.INT, unit="none", required=True - ), + ColumnSpec(name="scenario_id", dtype=ColumnDtype.INT, unit="none", required=True), ColumnSpec( name="generator_id", dtype=ColumnDtype.STR, @@ -794,9 +786,7 @@ def check_column_order( """ expected_names = [cs.name for cs in expected_columns] # Filter to columns present in both. - actual_filtered = [ - c for c in actual_columns if c in {cs.name for cs in expected_columns} - ] + actual_filtered = [c for c in actual_columns if c in {cs.name for cs in expected_columns}] expected_filtered = [c for c in expected_names if c in set(actual_columns)] violations: list[CheckViolation] = [] @@ -809,9 +799,7 @@ def check_column_order( column_name=e, row_index=None, error_type="wrong_order", - message=( - f"Column '{e}' expected at position {i} but found '{a}'" - ), + message=(f"Column '{e}' expected at position {i} but found '{a}'"), actual_value=a, expected=e, ) @@ -1043,10 +1031,7 @@ def check_row_count( expected=f">={manifest_entry.min_rows}", ) ) - if ( - manifest_entry.max_rows is not None - and actual_row_count > manifest_entry.max_rows - ): + if manifest_entry.max_rows is not None and actual_row_count > manifest_entry.max_rows: violations.append( CheckViolation( column_name=None, @@ -1082,9 +1067,7 @@ def check_no_nan_inf( """ violations: list[CheckViolation] = [] numeric_col_names = { - cs.name - for cs in expected_columns - if cs.dtype in {ColumnDtype.FLOAT, ColumnDtype.INT} + cs.name for cs in expected_columns if cs.dtype in {ColumnDtype.FLOAT, ColumnDtype.INT} } if not numeric_col_names: return CheckStatus.PASS, [] diff --git a/deliverables/README.md b/deliverables/README.md new file mode 100644 index 00000000..26a8faef --- /dev/null +++ b/deliverables/README.md @@ -0,0 +1,23 @@ +# Deliverables + +Formal contract deliverables for Phase 1. + +## Contents + +| File | SOW Reference | Description | +|------|---------------|-------------| +| `whitepaper.md` | Task 1.3 | Technology evaluation white paper (markdown source) | + +## Regenerating the PDF + +The white paper can be converted to PDF using pandoc: + +```bash +cd deliverables +pandoc whitepaper.md -o whitepaper.pdf \ + --pdf-engine=xelatex \ + -V geometry:margin=1in \ + -V fontsize=11pt +``` + +Requires pandoc and a LaTeX distribution (texlive-xetex). diff --git a/deliverables/whitepaper.md b/deliverables/whitepaper.md new file mode 100644 index 00000000..831559b5 --- /dev/null +++ b/deliverables/whitepaper.md @@ -0,0 +1,108 @@ +# Phase 1 Technology Evaluation White Paper + +**Contract FA714626C0006 — SOW Task 1.3**\ +**Grid Research Company LLC**\ +**April 2026** + +> Full test evidence, per-criterion drill-downs, and interactive visualizations are +> available in the [companion report site](https://zgenergy.github.io/grc-tech-evaluation/). +> All evaluation code and raw results are in the accompanying repository under `evaluations/`. + +--- + +## Executive Summary + +Under Contract FA714626C0006 with the Naval Research Laboratory, Grid Research Company LLC conducted a structured evaluation of open-source power system modeling tools to identify an optimal technology stack for high-voltage transmission system modeling. The objective was to select a tool capable of supporting substation-fidelity modeling, long-term forecasting, and vulnerability identification at defense-relevant locations including Camp Pendleton, Naval Base San Diego, and the Ports of Long Beach and Los Angeles. + +Six tools were evaluated: PyPSA, PowerModels.jl, PowerSimulations.jl, pandapower, GridCal, and MATPOWER. Each tool was assessed against six criteria -- Expressiveness, Extensibility, Scalability, Accessibility, Maturity, and Supply Chain -- using standardized test suites executed on ACTIVSg reference networks at three scales: 39-bus, 2,000-bus, and 10,000-bus. All testing used open-source solvers exclusively (HiGHS, SCIP, Ipopt, GLPK) on a reference workstation with 128 GB RAM and 16 cores. Supply Chain served as a gate criterion: any tool receiving a Weak or Failing grade was disqualified, reflecting the program's requirement for fully inspectable, open-source software deployable in restricted environments. + +**PyPSA is the recommended tool for Phase 2 development.** It is the only tool to earn a Strong grade in Expressiveness, the highest-priority criterion, demonstrating native support for security-constrained optimal power flow, PTDF/LODF matrix computation, unit commitment, economic dispatch, and contingency analysis without requiring user-assembled solver code. PyPSA also earned Strong grades in four of the remaining five criteria (Extensibility, Accessibility, Maturity, and Supply Chain), with Adequate in Scalability. Its pure-Python architecture, built on the pandas/numpy/scipy stack, eliminates language-adoption barriers and integrates directly with existing data pipeline and visualization tooling. + +PowerModels.jl ranks as the runner-up on the strength of its Strong Extensibility and its JuMP optimization framework, which provides the most flexible constraint-injection API among all evaluated tools. It was not selected because half of its expressiveness tests required user-assembled JuMP code averaging 487 lines per test, its contributor base exhibits a bus factor of one (82.4% of commits from a single author), and Julia adoption would impose workforce ramp-up costs. PowerModels.jl remains a viable fallback should PyPSA's known scalability limitations prove unresolvable. + +MATPOWER served as a reference benchmark throughout the evaluation, providing validated numerical baselines against which other tools' results were compared. It is excluded from the primary ranking because the customer requires fully inspectable source code, which precludes MATLAB's compiled runtime environment. + +The evaluation identified five risks for Phase 2 development, two rated HIGH severity: PyPSA's Linopy model-building overhead at production scale (302 seconds of model construction versus 6 seconds of solver time on a 10,000-bus DC OPF) and the absence of a native PSS/E RAW file parser, requiring construction of a custom format converter. Mitigation strategies for all identified risks are detailed in the risk register. + +## Evaluation Methodology + +Each tool was assessed against a gate criterion and five weighted dimensions, evaluated in strict lexicographic priority order. **Supply Chain** serves as the gate: a Weak or Failing grade disqualifies a tool regardless of technical merit, reflecting the non-negotiable requirement for inspectable, open-source components deployable in air-gapped environments. The five remaining criteria are ordered by priority: **Expressiveness** (breadth of native power system analyses) > **Extensibility** (ability to inject custom constraints and extend beyond built-in formulations) > **Scalability** (performance at 10,000+ buses with realistic contingency counts) > **Accessibility** (onboarding friction, documentation quality, error transparency) > **Maturity** (contributor health, institutional funding, operational adoption). + +This lexicographic ordering means that a tool with superior Expressiveness is preferred over one with superior Scalability, all else being equal, because the ability to formulate the required analyses is a prerequisite for any downstream performance consideration. Grades use a four-tier scale: **Strong** (ready for Phase 2 as-is), **Adequate** (usable with known workarounds), **Weak** (major gaps requiring significant remediation), and **Failing** (blocking limitations). The full rubric with tier boundary definitions and the complete test protocol are in the repository under `evaluation_guides/`. + +Testing used three ACTIVSg synthetic reference networks: IEEE 39-bus (New England) for functional verification, ACTIVSg 2,000-bus for intermediate-scale validation, and ACTIVSg 10,000-bus as the primary scalability benchmark. The 39-bus case was augmented with differentiated generator costs, renewable resources, battery storage, demand response, 24-hour load profiles, and stochastic scenarios to exercise the full range of evaluation sub-questions. The 10,000-bus case was preprocessed to fix zero-impedance branches, set unconstrained thermal ratings, and tighten select branch limits to induce congestion for tests requiring binding constraints and non-uniform locational marginal prices. The progression from 39 to 10,000 buses ensures that tools are tested at a scale representative of real regional transmission networks, not just textbook examples. + +Six open-source tools spanning three programming ecosystems were evaluated. The Python tools -- PyPSA (TU Berlin / Open Energy Transition), pandapower (University of Kassel / Fraunhofer IEE), and GridCal (individual developer) -- offer DataFrame-native data models and integration with the scientific Python stack. The Julia tools -- PowerModels.jl (Los Alamos National Laboratory) and PowerSimulations.jl (NREL/DOE) -- are built on the JuMP algebraic modeling layer, providing direct access to the optimization problem structure. MATPOWER (GNU Octave) is the canonical academic power system toolbox with over 25 years of development history; it was excluded from the competitive ranking because MATLAB's compiled runtime does not satisfy the inspectable-source-code requirement, but its results are retained as a calibration baseline. + +All evaluations ran within a reproducible devcontainer environment (Ubuntu 24.04, Python 3.12, Julia 1.10, GNU Octave) with dependencies pinned via lockfiles and all solvers (HiGHS, SCIP, Ipopt, GLPK) bundled in the container image. Any reviewer can rebuild the container and independently verify every result without configuring local toolchains. This also validates a key operational requirement: the selected tool can be packaged for deployment in air-gapped networks where internet-dependent installation workflows are unavailable. + +## Results + +| Tool | Expressiveness | Extensibility | Scalability | Accessibility | Maturity | Supply Chain | +|------|:-:|:-:|:-:|:-:|:-:|:-:| +| **PyPSA** | Strong | Strong | Adequate | Strong | Strong | Strong | +| PowerModels.jl | Adequate | Strong | Adequate | Adequate | Adequate | Adequate | +| PowerSimulations.jl | Adequate | Strong | Adequate | Weak | Adequate | Adequate | +| GridCal | Adequate | Adequate | Adequate | Weak | Weak | Strong | +| pandapower | Weak | Adequate | Weak | Adequate | Strong | Strong | +| MATPOWER\* | Adequate | Strong | Weak | Adequate | Adequate | Strong | + +\*Reference benchmark; excluded from ranking (requires MATLAB runtime). + +**Expressiveness.** PyPSA earned the sole Strong by passing 8 of 10 tests natively, including SCOPF via BODF-based N-1 contingency constraints (A-9), lossy DC OPF with piecewise-linear loss approximation (A-10), and multi-period storage OPF with cyclic state-of-charge constraints (A-12). Its only blocking gap -- distributed slack OPF (A-11) -- is shared by five of the six evaluated tools (MATPOWER alone achieved a workaround via post-processing PTDF with slack weights). The four Adequate-tier tools present distinct capability profiles: PowerModels excels at OPF formulation rigor with three-component LMP decomposition but lacks unit commitment entirely; PowerSimulations is the only tool with native SCUC formulations (A-5) but is blocked on lossy OPF by a solver-formulation mismatch; GridCal offers broad coverage including native SCOPF but is undermined by a battery energy-balance sign error (A-12). pandapower falls to Weak with four independent blocking failures reflecting a design scope limited to single-period steady-state analysis. + +**Extensibility.** The key discriminator is the custom constraint injection API (B-1). PyPSA, PowerModels, PowerSimulations, and MATPOWER all earned Strong by providing documented extension mechanisms requiring 2-4 lines of code for a flow gate constraint with dual extraction. All four also achieved machine-precision PTDF at 10,000 buses (B-9), with errors below 2e-11 p.u. pandapower and GridCal earned Adequate: neither offers a public constraint-injection API, requiring 200+ lines of fragile workarounds dependent on internal naming conventions. Both compensate with excellent graph access via NetworkX and efficient contingency sweep infrastructure. + +**Scalability.** No tool achieved Strong -- a notable finding. PyPSA's bottleneck is model construction overhead: 302 seconds building the Linopy model versus 6.2 seconds for the HiGHS solve at 10,000 buses (C-3). GridCal achieved the best SCOPF timing (32s for 50 contingencies). PowerModels delivered the fastest raw DC OPF solve (4.0s) but failed ACPF at 10,000 buses due to MUMPS memory exhaustion. MATPOWER's Weak reflects a cascading GLPK integration bug that blocked all nine 10,000-bus tests. pandapower's SCOPF exhausted 32 GB of memory. + +**Accessibility.** PyPSA earned Strong with 6.4-second install-to-first-solve time, good documentation coverage, and meaningful error diagnostics. The most concerning finding among other tools: GridCal's infeasible OPF problems report `converged=True` (D-4), creating a silent-failure trap. PowerSimulations requires ~332 lines per test -- roughly six times the PyPSA equivalent. + +**Maturity.** PyPSA and pandapower both earned Strong with diversified contributors and institutional funding (bus factor 2-3). The dominant risk pattern: four of six tools have a bus factor of 1. PowerModels' sole primary author has had zero direct commits in the trailing twelve months, though DOE/LANL backing provides partial mitigation. GridCal earned Weak (bus factor 1, no code review, CI never executes the test suite). + +**Supply Chain.** All six tools passed the gate. Python tools and MATPOWER earned Strong through permissive licensing, fully inspectable source code, and clean dependency trees. Julia tools earned Adequate due to larger dependency graphs (114-184 packages). A cross-cutting finding: five of six tools' ecosystems include optional GPL-3.0 GLPK bindings (pandapower, which uses an embedded PYPOWER solver, is the exception) that Phase 2 deployment should explicitly exclude in favor of HiGHS (MIT) or Ipopt (EPL 2.0). + +Per-criterion drill-downs with individual test outcomes are on the [report site results pages](https://zgenergy.github.io/grc-tech-evaluation/results). + +## Ranking and Sensitivity + +The final ranking using lexicographic forced-ranking: + +1. **PyPSA** -- sole Strong Expressiveness; Strong in 5 of 6 criteria +2. **PowerModels.jl** -- Strong Extensibility via JuMP; no unit commitment; 487 lines/test average for user-assembled tests +3. **PowerSimulations.jl** -- Strong Extensibility; best UC formulations but Weak Accessibility +4. **GridCal** -- broad but shallow; Weak Maturity undermines otherwise Adequate technical profile +5. **pandapower** -- Strong Maturity and Supply Chain but Weak Expressiveness and Scalability + +Three sensitivity scenarios were tested: Scalability First, Extensibility First, and Maturity-Accessibility Swapped. **PyPSA holds the top position in all three scenarios.** The ranking is entirely stable because PyPSA's sole Strong Expressiveness dominates regardless of priority reordering, and tier gaps between adjacent tools are wide enough that no single reordering changes relative positions. + +The PyPSA vs. PowerModels trade-off is architectural: PyPSA's high-level API covers SCOPF, UC/ED, and custom constraints natively but incurs model-building overhead at scale. PowerModels exposes JuMP directly for maximum flexibility but requires users to assemble each analysis from scratch. For Phase 2, where the critical-path capabilities are SCOPF, custom constraint injection, and the UC/ED pipeline, PyPSA covers all three natively while PowerModels covers only one. PowerModels should be reconsidered only if R1 (model-building overhead) proves unresolvable. Full sensitivity analysis details are on the [report site head-to-head page](https://zgenergy.github.io/grc-tech-evaluation/results/head-to-head). + +## Risk Register + +| ID | Risk | Severity | Mitigation | +|----|------|----------|------------| +| R1 | Linopy model-building overhead: 302s build vs. 6s solve at 10k buses | HIGH | Profile early in Phase 2; fall back to direct HiGHS C API for inner loop | +| R2 | No PSS/E RAW parser | HIGH | Build converter via FNM intermediate format; or use MATPOWER `psse2mpc()` as preprocessor | +| R3 | No distributed slack in OPF context | MEDIUM | Implement via `extra_functionality` callback with PTDF formulation | +| R4 | Single-threaded HiGHS limits parallelism | MEDIUM | Process-level parallelism for independent contingency sub-problems | +| R5 | No native SOS2 piecewise-linear costs | LOW | Phase 2 uses linearized costs; gap is irrelevant to planned workflow | + +None of these risks are disqualifying. The two HIGH-severity items have concrete mitigation paths with known effort bounds and fallback strategies. If R1 proves unresolvable at production scale, the fallback path is PowerModels.jl, which demonstrated 4.0s DC OPF solve with negligible model-building overhead. + +## Phase 2 Outlook + +Phase 2 follows a three-stage progression: + +**Stage 1 -- Historical Calibration.** Assemble the full network model from available generators, vendor data, sensor flows, and constraint setup information from the ISO. Calibrate distribution factors and learn stochastic scenarios for generation outages and renewable variability. The success criterion is constraints firing as published per ISO market results. + +**Stage 2 -- Real-Time Estimation.** Using real-time sensor data, aggregated load and generation figures, and published shadow prices for constraints, produce a real-time state estimation for the entire grid. Shadow prices combined with constraint setup information indicate whether a line is binding, providing at minimum a floor on flow levels. This stage bridges the gap between historical calibration and forward-looking analysis. + +**Stage 3 -- Forecasting.** Apply the learned parameters from Stages 1 and 2 (distribution factors, stochastic scenarios for generation outages and renewables) to produce a range of possible power flow and congestion outcomes over a 2-to-3-day horizon. At this stage no prices are available; inputs are limited to load forecasts, renewable forecasts, and known generation and topology. The ultimate deliverable is a forward-looking vulnerability assessment tool capable of producing congestion forecasts at bus-level granularity. + +**State Estimation Gap.** A cross-cutting investigation found that none of the six evaluated tools provide production-ready state estimation for transmission-scale grids. pandapower's SE exhibits convergence failures above ~89 buses; GridCal's bad data detection code is entirely commented out. Phase 2 will require dedicated SE tooling work, likely drawing on external open-source frameworks. The full SE investigation is in the repository under `phase2-research/`. + +**Recommended next steps.** Begin with the two HIGH-severity risk mitigations: profile Linopy model-building overhead on the target network scale (R1) and build the PSS/E RAW ingestion pipeline (R2). These are on the critical path and will provide early signal on PyPSA viability. Develop ETL pipelines for all data sources in parallel. + +--- + +*This white paper was produced under Contract FA714626C0006, SOW Task 1.3. The [interactive report site](https://zgenergy.github.io/grc-tech-evaluation/) provides full test evidence, per-criterion analysis, and sensitivity visualizations. All evaluation code, test scripts, and raw results are in the accompanying repository.* diff --git a/evaluation_guides/Phase1_Test_Protocol.md b/evaluation_guides/Phase1_Test_Protocol.md index 437bc46c..788f5313 100644 --- a/evaluation_guides/Phase1_Test_Protocol.md +++ b/evaluation_guides/Phase1_Test_Protocol.md @@ -404,7 +404,7 @@ All Suite G tests are gated by the `FNM_PATH` environment variable. When `FNM_PA |----|------|--------|-----------|----------------|------------| | G-FNM-1 | Intermediate format ingestion (two-check gate) | Intermediate format tables at `FNM_PATH`, manifest file | Load every intermediate format table listed in the manifest. For each table, count the ingested records (buses, branches, generators, loads, transformers, shunts, etc.). Compare counts against the manifest's expected record counts per table. | G-FNM-1 has two sub-checks. **(a) PSS/E compatibility:** if the tool fails to parse the intermediate CSV tables, record `failure_reason: psse_parse_error` and `ingestion_path: matpower_fallback`, emit a blocking api-friction observation, and proceed to G-FNM-3/4/5 via MATPOWER fallback — G-FNM-2 is blocked but G-FNM-3/4/5 proceed. PSS/E format support failure is an interoperability finding; it does not cascade to block power-system capability assessment. **(b) Record count fidelity (only checked if PSS/E parsing succeeds):** all record counts must match the manifest exactly. Zero records missing, zero extra records. If the tool's data model merges record types (e.g., branches and transformers into a single table), the merged count must equal the sum of the constituent intermediate format table counts. If count check fails, skip G-FNM-2 through G-FNM-5. | `data/fnm/docs/intermediate-schema.md` (table definitions), manifest file at `FNM_PATH` | | G-FNM-2 | Field coverage audit | Ingested FNM model from G-FNM-1 (requires PSS/E parse success), field criticality matrix | For each intermediate format table: enumerate the fields that the tool's data model actually contains after ingestion. Compare against the field criticality matrix to compute coverage percentage by criticality tier. Report four percentages per table: % of DCPF-critical fields present, % of ACPF-critical fields present, % of Informational fields present, % of Discardable fields present. | 100% of DCPF-critical fields must be present across all 19 DCPF-critical fields (field-criticality-matrix.md v10 count). This is a hard requirement — any missing DCPF-critical field is a finding that directly impacts the Expressiveness grade. ACPF-critical field coverage is reported but not gated — tools that omit some ACPF-critical fields (e.g., switched shunt discrete steps) receive a documented finding. Informational and Discardable field omissions are noted but carry no grade impact. The coverage report must identify each missing field by name, table, and criticality tier. | `data/fnm/docs/field-criticality-matrix.md` (tier assignments per field per table) | -| G-FNM-3 | DCPF verification | **Primary:** Intermediate CSV tables at `data/fnm/reference/cleaned/intermediate/`. **Fallback:** Cleaned MATPOWER case (`data/fnm/reference/cleaned/fnm_main_island.mat`). DCPF reference solution at `data/fnm/reference/dcpf/`. | **Primary path (CSV):** Load the intermediate CSV tables from `data/fnm/reference/cleaned/intermediate/` into the tool's data model. **Fallback path (MATPOWER):** If the tool lacks CSV ingestion capability, load the pre-cleaned MATPOWER case (27,862-bus main island with all data fixes pre-applied per `summary_cleaning.json`). Record which input path was used in result frontmatter as `input_path: "csv"` or `input_path: "matpower"` and `ingestion_path` as above. Solve DCPF. Extract bus voltage angles and branch active power flows. Load the DCPF reference solution from `data/fnm/reference/dcpf/`. Compute aggregate deviation metrics as defined in `pass_conditions.json` under the `dcpf` key: (a) fraction of buses with VA deviation within tolerance, (b) fraction of in-service branches with P deviation within tolerance. **Precision:** Report all deviation values in scientific notation (e.g., `max_deviation_deg: "1.07e-08"`) — do not round to fixed decimal places. Check against aggregate pass thresholds and hard-fail thresholds. **Cross-reference validation:** Verify DCPF solution validity by checking bus injection power balance (sum of all branch flows at each bus = net injection within 1e-4 p.u. tolerance) as a tool-independent necessary condition for solution correctness. | Pass if all aggregate thresholds are met and no hard-fail condition is triggered, per the `dcpf` section of `data/fnm/reference/pass_conditions.json`. Buses and branches that exceed the aggregate tolerance but are classified as known outlier causes (per the outlier classification rules in pass_conditions.json) are reported as classified outliers, not unqualified failures. Bus injection power balance check must pass. If the tool cannot solve DCPF on the FNM at all (solver failure, out of memory, or timeout after 10 minutes), record the failure mode. If failure is due to scale (tool works on MEDIUM but not LARGE), attribute to Scalability. If failure is due to data model issues (missing topology, incorrect parameters), attribute to Expressiveness. | `data/fnm/reference/cleaned/intermediate/` (CSV tables), `data/fnm/reference/cleaned/fnm_main_island.mat` (MATPOWER fallback), `data/fnm/reference/pass_conditions.json` (thresholds), `data/fnm/reference/dcpf/` (reference angles and flows) | +| G-FNM-3 | DCPF verification | **Primary:** Intermediate CSV tables at `data/fnm/reference/cleaned/intermediate/`. **Fallback:** Cleaned MATPOWER case (`data/fnm/reference/cleaned/fnm_main_island.mat`). DCPF reference solution at `data/fnm/reference/dcpf/`. | **Primary path (CSV):** Load the intermediate CSV tables from `data/fnm/reference/cleaned/intermediate/` into the tool's data model. **Fallback path (MATPOWER):** If the tool lacks CSV ingestion capability, load the pre-cleaned MATPOWER case (~28,000-bus main island with all data fixes pre-applied per `summary_cleaning.json`). Record which input path was used in result frontmatter as `input_path: "csv"` or `input_path: "matpower"` and `ingestion_path` as above. Solve DCPF. Extract bus voltage angles and branch active power flows. Load the DCPF reference solution from `data/fnm/reference/dcpf/`. Compute aggregate deviation metrics as defined in `pass_conditions.json` under the `dcpf` key: (a) fraction of buses with VA deviation within tolerance, (b) fraction of in-service branches with P deviation within tolerance. **Precision:** Report all deviation values in scientific notation (e.g., `max_deviation_deg: "1.07e-08"`) — do not round to fixed decimal places. Check against aggregate pass thresholds and hard-fail thresholds. **Cross-reference validation:** Verify DCPF solution validity by checking bus injection power balance (sum of all branch flows at each bus = net injection within 1e-4 p.u. tolerance) as a tool-independent necessary condition for solution correctness. | Pass if all aggregate thresholds are met and no hard-fail condition is triggered, per the `dcpf` section of `data/fnm/reference/pass_conditions.json`. Buses and branches that exceed the aggregate tolerance but are classified as known outlier causes (per the outlier classification rules in pass_conditions.json) are reported as classified outliers, not unqualified failures. Bus injection power balance check must pass. If the tool cannot solve DCPF on the FNM at all (solver failure, out of memory, or timeout after 10 minutes), record the failure mode. If failure is due to scale (tool works on MEDIUM but not LARGE), attribute to Scalability. If failure is due to data model issues (missing topology, incorrect parameters), attribute to Expressiveness. | `data/fnm/reference/cleaned/intermediate/` (CSV tables), `data/fnm/reference/cleaned/fnm_main_island.mat` (MATPOWER fallback), `data/fnm/reference/pass_conditions.json` (thresholds), `data/fnm/reference/dcpf/` (reference angles and flows) | | G-FNM-4 | ACPF convergence — DCPF warm-start + progressive relaxation | Same input path as G-FNM-3 (intermediate CSVs primary, MATPOWER fallback). ACPF failure analysis at `data/fnm/reference/acpf/summary_acpf.json`. | **Step 1 — DCPF warm-start:** Solve DCPF on the cleaned network (same path as G-FNM-3). Extract bus voltage angles. Record `dcpf_init_mean_deg` (mean \|VA\|) and `dcpf_init_max_abs_deg` (max \|VA\|) in frontmatter. **Step 2 — ACPF at 0% relaxation:** Initialize VM=1.0 pu, VA=DCPF angles from Step 1. Attempt ACPF with nominal thermal limits. 30-minute timeout. **Step 3 — ACPF at 10% relaxation (if Step 2 failed):** Relax all branch thermal limits by 10% (RATE_A × 1.10). Retry ACPF from same DCPF warm start. 30-minute timeout. **Step 4 — ACPF at 20% relaxation (if Step 3 failed):** Relax by 20% (RATE_A × 1.20). Retry. 30-minute timeout. **Stop here** — do not relax beyond 20%. Record `input_path: csv` or `input_path: matpower` and `ingestion_path`. | **No hard pass/fail gate. All outcomes are diagnostic findings** (`outcome: informational`, mirrors C-5 pattern). Record `relaxation_level_achieved`: "0%", "10%", "20%", or "infeasible". If convergence occurs at any level, record as a discriminating solver robustness strength in the Expressiveness narrative. Failure to converge at 20% is recorded but not penalized — this planning network is known to be difficult. If multiple tools converge, apply `pass_conditions.json` `acpf` thresholds for cross-tool consistency checking. Record `acpf_timeout_minutes: 30` in frontmatter. | `data/fnm/reference/cleaned/intermediate/` (CSV tables), `data/fnm/reference/cleaned/fnm_main_island.mat` (MATPOWER fallback), `data/fnm/reference/acpf/summary_acpf.json` (MATPOWER failure analysis), `data/fnm/reference/pass_conditions.json` (cross-tool thresholds) | | G-FNM-5 | Supplemental CSV representability | Ingested FNM model from G-FNM-1, 7 supplemental CSVs at `FNM_PATH`, supplemental CSV reference documentation | For each of the 7 supplemental CSVs (`LINE_AND_TRANSFORMER.csv`, `TRADING_HUB.csv`, `GEN_DISTRIBUTION_FACTOR.csv`, `CONTINGENCY.csv`, `INTERFACE.csv`, `INTERFACE_ELEMENT.csv`, `OUTAGE.csv`): (a) attempt to attach each field's data to the tool's network model using native attributes or the tool's documented extension mechanisms, (b) for each field, record whether attachment succeeded as natively-representable (N), extension-representable (E), or tool-external (X), (c) compare the achieved representability against the analytical classification in `data/fnm/docs/supplemental-csvs.md` and note any discrepancies. For each E classification, document the **concrete extension approach** (specific API, function signature, or code pattern). For each X classification, include a **written justification** explaining why no native or extension path exists. After the per-field table for all 7 CSVs, produce a **Market Solution Fidelity Summary** classifying four concepts as achievable/complex/blocked: (1) N-1/N-2 contingency enforcement, (2) interface flow limits, (3) aggregate hub pricing (PTDF-weighted LMP), (4) outage scheduling. | No hard pass/fail gate. This is an evidence-collection test. For each CSV, report: total fields, count and percentage by achieved representability tier (N/E/X), and per-field comparison against the analytical classification. E classifications without a documented concrete extension approach must be downgraded to X. The results feed into the Extensibility grade narrative per rubric v5's supplemental CSV representability grading note. After the per-field tables, include the Market Solution Fidelity Summary. | `data/fnm/docs/supplemental-csvs.md` (per-field analytical classifications and representability matrices), `data/fnm/docs/supplemental-csv-representability.md` (cross-tool summary) | diff --git a/evaluations/README.md b/evaluations/README.md new file mode 100644 index 00000000..2ac662e9 --- /dev/null +++ b/evaluations/README.md @@ -0,0 +1,39 @@ +# Evaluations + +Each subdirectory is an independent evaluation environment for one tool. +All six evaluations are complete (protocol v11). + +## Per-Tool Structure + +``` +evaluations// +├── results/ +│ ├── synthesis.md # Comprehensive assessment (~3000 words) +│ ├── eval-config.yaml # Test configuration and parameters +│ ├── validation-report.md # Automated validation checks +│ ├── .progress.yaml # Evaluation state machine status +│ ├── gate/ # G-1 to G-3: network ingestion tests +│ ├── expressiveness/ # A-1 to A-12: problem formulation tests +│ ├── extensibility/ # B-1 to B-9: custom constraint/callback tests +│ ├── scalability/ # C-1 to C-10: scaling tests (39 to 10k buses) +│ ├── accessibility/ # D-1 to D-5: documentation and installation +│ ├── maturity/ # E-1 to E-6: code quality, CI, bus factor +│ ├── supply_chain/ # License and open-source gate criteria +│ ├── observations/ # Tool-specific research findings +│ └── p2_readiness/ # Phase 2 gap analysis +├── tests/ # Test scripts organized by dimension +├── pyproject.toml / Project.toml # Language-specific dependency file +└── verify_install.py / .jl / .m # Smoke test +``` + +## Reading Results + +Start with `results/synthesis.md` for each tool — it summarizes strengths, +weaknesses, workarounds, and the overall assessment. Individual test results +in dimension subdirectories contain detailed pass/fail outcomes with evidence. + +## Shared Resources + +The `shared/` subdirectory contains cross-tool utilities: + +- `matpower_loader.py` — Shared MATPOWER case file parser used by Python tools diff --git a/evaluations/gridcal/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md b/evaluations/gridcal/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md index 36b15641..0e05a886 100644 --- a/evaluations/gridcal/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md +++ b/evaluations/gridcal/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md @@ -62,7 +62,7 @@ GridCal was tested against the pre-cleaned MATPOWER case files: producing `AttributeError: 'NoneType' object has no attribute 'buses'`. GridCal cannot parse MATLAB `.mat` binary format files. - **`fnm_main_island.m`** (MATPOWER text format): Loaded successfully in 12.86 - seconds, producing a valid `MultiCircuit` with 27,862 buses. + seconds, producing a valid `MultiCircuit` with ~28,000 buses. This confirms the MATPOWER `.m` fallback path is available for G-FNM-3/4/5. @@ -81,12 +81,12 @@ This confirms the MATPOWER `.m` fallback path is available for G-FNM-3/4/5. | Component | MATPOWER Ingested | Notes | |-----------|------------------|-------| -| Buses | 27,862 | Main-island subset (type-4 isolated buses excluded) | -| Generators | 5,741 | | -| Lines (branches) | 23,125 | | -| Transformers (2W) | 9,481 | | -| Loads | 8,624 | MATPOWER aggregates multiple loads per bus | -| Shunts | 3,110 | | +| Buses | ~28,000 | Main-island subset (type-4 isolated buses excluded) | +| Generators | ~5,800 | | +| Lines (branches) | ~23,000 | | +| Transformers (2W) | ~9,500 | | +| Loads | ~8,600 | MATPOWER aggregates multiple loads per bus | +| Shunts | ~3,100 | | | Areas | 0 | MATPOWER `.m` format does not preserve area data | | Zones | 0 | MATPOWER `.m` format does not preserve zone data | | HVDC lines | 0 | Not in MATPOWER `.m` format | diff --git a/evaluations/gridcal/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md b/evaluations/gridcal/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md index 75b40301..76336a9b 100644 --- a/evaluations/gridcal/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/gridcal/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md @@ -36,7 +36,7 @@ near transformer-connected buses (88.65% transformer-adjacent), indicating a B-matrix formulation difference rather than a data ingestion error. Classified as `formulation_difference` with qualified_pass. -Bus injection power balance cross-reference confirms all 27,862 bus load values +Bus injection power balance cross-reference confirms all ~28,000 bus load values match the reference exactly (0 mismatches), verifying correct data ingestion independent of the solver formulation. @@ -46,13 +46,13 @@ independent of the solver formulation. G-FNM-1 established that GridCal cannot ingest the intermediate CSV tables. The MATPOWER fallback path was used: `data/fnm/reference/cleaned/fnm_main_island.m` -(27,862-bus main island, type-4 isolated buses removed). `ingestion_path: matpower_raw`. +(~28,000-bus main island, type-4 isolated buses removed). `ingestion_path: matpower_raw`. ### Bus exclusion Loaded 2,445 excluded buses from `excluded_buses.json`. Since the cleaned MATPOWER -file already removes these buses (the main island contains only the 27,862 connected -buses), 0 buses were excluded from comparison. All 27,862 reference buses are present +file already removes these buses (the main island contains only the ~28,000 connected +buses), 0 buses were excluded from comparison. All ~28,000 reference buses are present in the tool's model. ### DCPF execution @@ -84,9 +84,9 @@ results = vge.power_flow(grid, pf_options) | Metric | Value | |--------|-------| -| Total compared | 27,862 | +| Total compared | ~28,000 | | Excluded | 0 | -| Passing (< 1.0 deg) | 27,862 (100.00%) | +| Passing (< 1.0 deg) | ~28,000 (100.00%) | | Failing | 0 (0.00%) | | Mean deviation | 2.667291e-09 deg | | Median deviation | 2.448644e-09 deg | @@ -99,9 +99,9 @@ results = vge.power_flow(grid, pf_options) | Metric | Value | |--------|-------| -| Total compared | 32,532 | -| Passing (< 10%) | 32,206 (98.9979%) | -| Failing | 326 (1.0021%) | +| Total compared | ~33,000 | +| Passing (< 10%) | ~32,200 (98.998%) | +| Failing | ~330 (1.002%) | | Mean deviation | 4.177521e+02% | | Median deviation | 4.875434e-10% | | 95th percentile | 5.874286e-08% | @@ -121,11 +121,11 @@ results = vge.power_flow(grid, pf_options) | Metric | Value | |--------|-------| -| Total generation | 155,511.04 MW | -| Total load | 165,491.55 MW | +| Total generation | ~156,000 MW | +| Total load | ~165,000 MW | | Gen-load imbalance | -9.980509e+03 MW | -| Load buses compared | 27,862 | -| Load match count | 27,862 (100%) | +| Load buses compared | ~28,000 | +| Load match count | ~28,000 (100%) | | Load mismatch count | 0 | | Max load diff | 0.000000e+00 MW | @@ -156,11 +156,11 @@ ingestion error. [tool-specific: simplified B-matrix formulation in DC power flo | From | To | Type | GridCal (MW) | Reference (MW) | Dev % | |------|-----|------|-------------|----------------|-------| -| 1668 | 88630 | Line | 111,582 | -19.82 | 5.630e+05 | -| 21476 | 84022 | Line | -68,017 | 12.57 | 5.411e+05 | -| 72100 | 73053 | Line | -13,365 | 3.23 | 4.138e+05 | -| 180421 | 36990 | Xfmr | 5,234 | -1.61 | 3.252e+05 | -| 1635 | 92191 | Line | -352,878 | 109.50 | 3.224e+05 | +| | | Line | 111,582 | -19.82 | 5.630e+05 | +| | | Line | -68,017 | 12.57 | 5.411e+05 | +| | | Line | -13,365 | 3.23 | 4.138e+05 | +| | | Xfmr | 5,234 | -1.61 | 3.252e+05 | +| | | Line | -352,878 | 109.50 | 3.224e+05 | ## Workarounds diff --git a/evaluations/gridcal/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md b/evaluations/gridcal/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md index 6cc4120e..036b994a 100644 --- a/evaluations/gridcal/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/gridcal/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md @@ -29,7 +29,7 @@ timestamp: 2026-03-24T00:00:00Z ## Result: INFORMATIONAL -ACPF does not converge on the 27,862-bus FNM main island under any solver or +ACPF does not converge on the ~28,000-bus FNM main island under any solver or relaxation configuration tested. Four solver algorithms were exercised at three branch rate relaxation levels (0%, 10%, 20%) -- all 12 combinations failed to achieve genuine convergence. The best residual was 15.83 (Levenberg-Marquardt, @@ -42,7 +42,7 @@ with no gate consequence. G-FNM-1 established that GridCal cannot ingest the intermediate CSV tables. The MATPOWER fallback path was used: `data/fnm/reference/cleaned/fnm_main_island.m` -(27,862-bus main island, type-4 isolated buses removed). +(~28,000-bus main island, type-4 isolated buses removed). ### DCPF warm-start @@ -103,7 +103,7 @@ mismatches are identical across all relaxation levels. | Metric | Value | |--------|-------| | DCPF solve time | 2.38 seconds | -| Nonzero-angle buses | 27,858 / 27,862 | +| Nonzero-angle buses | 27,858 / ~28,000 | | Mean |VA| | 1.102609e+02 deg | | Max |VA| | 1.799973e+02 deg | @@ -165,7 +165,7 @@ solver configuration. ## Workarounds -None applicable. ACPF convergence failure on a 27,862-bus network loaded via +None applicable. ACPF convergence failure on a ~28,000-bus network loaded via MATPOWER fallback is a diagnostic finding. Potential contributing factors: 1. **Network conditioning:** The FNM main island is a large, complex transmission diff --git a/evaluations/gridcal/results/observations/api-friction-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md b/evaluations/gridcal/results/observations/api-friction-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md index e682cd87..40551752 100644 --- a/evaluations/gridcal/results/observations/api-friction-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md +++ b/evaluations/gridcal/results/observations/api-friction-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md @@ -27,7 +27,7 @@ formats, but not CSV tables representing network elements. When the MATPOWER fallback was tested, the `.mat` binary format (MATLAB native) returned `None` from `open_file()` without raising an exception. Subsequent attribute access on the result (`grid.buses`) raises `AttributeError`. Only the -`.m` text format works correctly, loading the 27,862-bus main-island case. +`.m` text format works correctly, loading the ~28,000-bus main-island case. The MATPOWER `.m` fallback path works successfully for downstream G-FNM-3/4/5 tests but cannot support G-FNM-1 (CSV ingestion) or G-FNM-2 (field coverage diff --git a/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index 9b92e11b..db6a1d11 100644 --- a/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -11,10 +11,10 @@ timestamp: 2026-03-24T00:00:00Z ## Finding -GridCal's MATPOWER loader correctly ingests 27,862 buses and 32,606 branches from +GridCal's MATPOWER loader correctly ingests ~28,000 buses and ~33,000 branches from the cleaned FNM case file. Bus voltage angles match the reference MATPOWER DCPF solution within machine precision (max deviation 7.713822e-09 deg). The v11 bus -injection power balance cross-reference confirms all 27,862 bus load values match +injection power balance cross-reference confirms all ~28,000 bus load values match the reference exactly (0 mismatches, max diff 0.000000e+00 MW). However, 326 branches (1.0%) show extreme flow deviations (up to 5.629550e+05%) concentrated on transformer-adjacent branches, indicating the internal data model treats @@ -23,9 +23,9 @@ transformer tap ratios differently in the DC B-matrix construction. ## Context G-FNM-3 verified GridCal's DCPF solution against the MATPOWER reference on the -27,862-bus FNM main island. The MATPOWER fallback path was used because G-FNM-1 +~28,000-bus FNM main island. The MATPOWER fallback path was used because G-FNM-1 established that GridCal cannot ingest intermediate CSV tables. Total generation -is 155,511 MW and total load is 165,492 MW, with the -9,981 MW imbalance absorbed +is ~156,000 MW and total load is ~165,000 MW, with the -9,981 MW imbalance absorbed by the slack bus. The perfect load match confirms the injection vector is correctly ingested; the formulation difference is isolated to the branch flow computation. diff --git a/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index 8013a55d..c21ce319 100644 --- a/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/gridcal/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -12,7 +12,7 @@ timestamp: 2026-03-24T00:00:00Z ## Finding GridCal's ACPF solver (Newton-Raphson, Levenberg-Marquardt, HELM) fails to converge -on the 27,862-bus FNM main island loaded via MATPOWER fallback path. The best result +on the ~28,000-bus FNM main island loaded via MATPOWER fallback path. The best result (LM, 200 iterations) achieves a residual of 15.83 MVA, far from the 1e-6 tolerance. ## Context diff --git a/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index 159ddc06..abbda725 100644 --- a/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -7,22 +7,22 @@ severity: low timestamp: 2026-03-24T00:00:00Z --- -# Observation: GridCal solves DCPF on 27,862-bus FNM in 2.4 seconds +# Observation: GridCal solves DCPF on ~28,000-bus FNM in 2.4 seconds ## Finding -GridCal's linear (DC) power flow solver handles the 27,862-bus FNM main island +GridCal's linear (DC) power flow solver handles the ~28,000-bus FNM main island network without difficulty. DCPF solve time is 2.369 seconds, with network loading taking 31.899 seconds. Peak memory usage is 1,894 MB. The solution produces -non-trivial results (27,858 of 27,862 buses have nonzero voltage angles). Total +non-trivial results (27,858 of ~28,000 buses have nonzero voltage angles). Total wall-clock time including comparison logic is 48.457 seconds, well within the 10-minute timeout. ## Context This was measured during G-FNM-3 (DCPF verification) using the MATPOWER fallback -path. The network has 32,606 branches (23,125 lines + 9,481 transformers) with -32,532 active. The solve time is competitive for a Python-based tool on a network +path. The network has ~33,000 branches (23,125 lines + 9,481 transformers) with +~33,000 active. The solve time is competitive for a Python-based tool on a network of this scale. ## Implications diff --git a/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index ce152f61..2a8c6bbd 100644 --- a/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/gridcal/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -7,11 +7,11 @@ severity: medium timestamp: 2026-03-24T00:00:00Z --- -# Observation: GridCal ACPF infeasible on 27,862-bus FNM -- scalability signal +# Observation: GridCal ACPF infeasible on ~28,000-bus FNM -- scalability signal ## Finding -GridCal's ACPF solver could not converge on the 27,862-bus FNM main island despite +GridCal's ACPF solver could not converge on the ~28,000-bus FNM main island despite DCPF warm-start and multiple solver algorithms. This is a LARGE-tier scalability signal: DCPF handles the network easily (2.4s), but ACPF fails entirely. @@ -24,7 +24,7 @@ to 12.8 pu). The total evaluation across all 12 solver/relaxation combinations took 211.0 seconds and consumed 2,042 MB of memory. For comparison, the MATPOWER reference ACPF solution exists (buses_acpf.csv with -27,862 entries), confirming the network is solvable. The convergence failure is +~28,000 entries), confirming the network is solvable. The convergence failure is GridCal-specific. ## Implications diff --git a/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index 4f05cd00..0107869f 100644 --- a/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -12,8 +12,8 @@ timestamp: 2026-03-24T00:00:00Z ## Finding GridCal's DCPF solver produces bus voltage angles within machine precision of the -MATPOWER reference (max deviation 7.713822e-09 deg across all 27,862 buses) but -exhibits extreme branch flow deviations (up to 5.629550e+05%) on 326 out of 32,532 +MATPOWER reference (max deviation 7.713822e-09 deg across all ~28,000 buses) but +exhibits extreme branch flow deviations (up to 5.629550e+05%) on 326 out of ~33,000 branches. 88.65% of failing branches are adjacent to transformer buses. The deviations are systematic and signed (not random), consistent with a simplified B-matrix construction that computes branch susceptance as `b = -1/x` without @@ -27,7 +27,7 @@ solver appears to use the simplified formulation. The near-zero bus angle deviat confirms that the power injection vector (loads, generators) and network topology are correctly ingested; only the branch flow computation from angle differences is affected by the tap ratio omission. The v11 bus injection power balance -cross-reference confirms all 27,862 bus loads match exactly (0 mismatches). +cross-reference confirms all ~28,000 bus loads match exactly (0 mismatches). Key characteristics of the affected branches: - Flow magnitudes reach hundreds of thousands of MW (vs reference flows of ~100 MW) diff --git a/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index 4c1198d9..da149b9e 100644 --- a/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/gridcal/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -11,7 +11,7 @@ timestamp: 2026-03-24T00:00:00Z ## Finding -GridCal's ACPF solver fails to converge on the 27,862-bus FNM main island loaded +GridCal's ACPF solver fails to converge on the ~28,000-bus FNM main island loaded via MATPOWER `.m` fallback. The MATPOWER format flattens transformer data and loses ACPF-critical fields (tap control modes, winding impedance detail, switched shunt discrete steps), which may contribute to the convergence failure. This represents a @@ -24,7 +24,7 @@ G-FNM-4 tested ACPF convergence with DCPF warm-start and progressive branch rate relaxation (0%, 10%, 20%) using four solver algorithms (NR, NR+controls, LM, HELM). All 12 combinations failed. The best result (Levenberg-Marquardt, 200 iterations) achieved a residual of 1.583e+01, far from the 1e-6 tolerance. The MATPOWER -reference ACPF solution exists (buses_acpf.csv with 27,862 entries), confirming the +reference ACPF solution exists (buses_acpf.csv with ~28,000 entries), confirming the network is solvable with appropriate data fidelity. Key ACPF-critical fields lost in MATPOWER fallback: diff --git a/evaluations/gridcal/results/p2_readiness/P2-1_psse_raw_parsing.md b/evaluations/gridcal/results/p2_readiness/P2-1_psse_raw_parsing.md index ddf3b668..12386420 100644 --- a/evaluations/gridcal/results/p2_readiness/P2-1_psse_raw_parsing.md +++ b/evaluations/gridcal/results/p2_readiness/P2-1_psse_raw_parsing.md @@ -36,7 +36,7 @@ versions = [35, 34, 33, 32, 30, 29] If the file's `REV` field is not in this list, the parser logs an error and returns an empty `PsseCircuit`. Version 31 is notably absent from the list. -**Actual v31 failure (G-FNM-1).** When the FNM RAW file (`AUC_AN_2026_2026_S01_ON_NETWORK_MODEL.RAW`, PSS/e v31) was parsed, the parser failed with: +**Actual v31 failure (G-FNM-1).** When the FNM RAW file (`.RAW`, PSS/e v31) was parsed, the parser failed with: ``` Exception: PSSe 35 load data came with 1 elements and 18 or 17 were expected :/ @@ -52,5 +52,5 @@ This error reveals that even for nominally "supported" versions, the per-record - **Phase 2 FNM ingestion** requires PSS/e v31 parsing. GridCal cannot parse v31 RAW files without parser modifications to handle the shorter record formats. - **Estimated effort to fix:** Medium. The parser architecture already threads a `version` parameter to device-level parsers. The fix requires auditing each device parser (bus, load, branch, generator, transformer, etc.) to accept the correct field counts for v29-v34 formats. The PSS/e RAW specification defines field layouts per version, so this is mechanical but tedious work across ~15 device types. -- **Alternative path:** The MATPOWER `.m` fallback (demonstrated in G-FNM-1) successfully loads the FNM main island with 27,862 of 30,307 buses. This loses area/zone metadata but provides a workable network for power flow and OPF studies. +- **Alternative path:** The MATPOWER `.m` fallback (demonstrated in G-FNM-1) successfully loads the FNM main island with ~28,000 of ~30,000 buses. This loses area/zone metadata but provides a workable network for power flow and OPF studies. - **v35 files parse correctly.** Standard test cases distributed in v35 format (the current PSS/e version) should work without issues. diff --git a/evaluations/gridcal/results/synthesis.md b/evaluations/gridcal/results/synthesis.md index fe719a0c..0fb42f09 100644 --- a/evaluations/gridcal/results/synthesis.md +++ b/evaluations/gridcal/results/synthesis.md @@ -188,7 +188,7 @@ GridCal scales well for power flow and single-period OPF up to MEDIUM (10k buses Suite G was executed (FNM_PATH set). The MATPOWER fallback path was used because GridCal cannot parse the intermediate CSV tables. -- **Data Model Fidelity:** G-FNM-1 failed -- GridCal has no CSV network import capability [tool-specific]. G-FNM-2 was skipped (blocked by G-FNM-1). The MATPOWER `.m` fallback successfully loaded 27,862 buses, 5,741 generators, 23,125 lines, and 9,481 transformers. All bus load values match the reference exactly (0 mismatches in the power balance cross-reference). 100% DCPF-critical field coverage was not achieved via CSV path; MATPOWER format loses PSS/E-specific fields (tap control modes, switched shunt steps, area interchange). +- **Data Model Fidelity:** G-FNM-1 failed -- GridCal has no CSV network import capability [tool-specific]. G-FNM-2 was skipped (blocked by G-FNM-1). The MATPOWER `.m` fallback successfully loaded ~28,000 buses, ~5,800 generators, ~23,000 lines, and ~9,500 transformers. All bus load values match the reference exactly (0 mismatches in the power balance cross-reference). 100% DCPF-critical field coverage was not achieved via CSV path; MATPOWER format loses PSS/E-specific fields (tap control modes, switched shunt steps, area interchange). - **Power Flow Verification:** G-FNM-3 achieved qualified_pass -- bus angles match the MATPOWER reference within machine precision (100% passing, max deviation 7.7e-09 deg). Branch flows pass the aggregate threshold (99.0% within 10% tolerance), but 326 branches show extreme deviations (up to 5.6e+05%) concentrated near transformer-connected buses (88.65% transformer-adjacent). This is classified as a formulation difference [tool-specific: simplified B-matrix in DCPF]. G-FNM-4 was informational -- ACPF failed to converge at all relaxation levels with all solver algorithms (NR, LM, HELM). Best residual was 15.83 (LM, 200 iterations). Contributing factors include MATPOWER format data loss for ACPF-critical fields and network conditioning. @@ -318,7 +318,7 @@ The supply chain profile is strong. MPL-2.0 is enterprise-compatible. The pure-P ### FNM Data Model - **No CSV network import:** GridCal cannot ingest intermediate CSV tables; MATPOWER `.m` is the only viable fallback path. ([G-FNM-1](fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md)) -- **Simplified B-matrix formulation:** 326 of 32,532 branch flows deviate from reference on the 27,862-bus FNM, concentrated near transformers (88.65% transformer-adjacent). [tool-specific] ([G-FNM-3](fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md)) +- **Simplified B-matrix formulation:** 326 of ~33,000 branch flows deviate from reference on the ~28,000-bus FNM, concentrated near transformers (88.65% transformer-adjacent). [tool-specific] ([G-FNM-3](fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md)) - **ACPF convergence failure on FNM:** All solver algorithms fail to converge (best residual 15.83). Contributing factors: MATPOWER format data loss + network conditioning + no Ipopt. ([G-FNM-4](fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md)) - **Strong contingency model, weak interface model:** 83% native coverage on CONTINGENCY.csv, 100% external on INTERFACE.csv. No flowgate concept. ([G-FNM-5](fnm_ingestion/G-FNM-5_fnm_supplemental_csv.md)) @@ -335,7 +335,7 @@ The supply chain profile is strong. MPL-2.0 is enterprise-compatible. The pure-P - [ ] **B-4 (qualified_pass):** TapPhaseControl enum bug -- verify this is indeed a bug and not a configuration error. The workaround (sequential snapshot OPF) loses inter-temporal coupling. - [ ] **C-4 (qualified_pass):** SCUC at SMALL required sequential snapshots. Verify whether the loss of inter-temporal coupling (ramps, min up/down) should reduce the classification below qualified_pass. - [ ] **G-FNM-3 (qualified_pass):** 326 extreme branch flow deviations attributed to formulation difference. Verify the transformer-adjacency classification (88.65%) and confirm this is not a data ingestion issue. -- [ ] **G-FNM-4 (informational):** ACPF infeasible on 27,862-bus FNM. Verify whether this should be attributed to the tool (no Ipopt, simplified data model) or the network (inherently difficult case). +- [ ] **G-FNM-4 (informational):** ACPF infeasible on ~28,000-bus FNM. Verify whether this should be attributed to the tool (no Ipopt, simplified data model) or the network (inherently difficult case). - [ ] **Solver-vs-tool attribution:** The soft-constraint DCOPF (A-3) and the broken CBC/PDLP enum values (C-7) could arguably be classified either way. The current report tags both as tool-specific. - [ ] **E-3 maturity:** Bus factor 1 with zero code review. This is a factual finding from API data but should be confirmed for any weighting implications. @@ -343,7 +343,7 @@ The supply chain profile is strong. MPL-2.0 is enterprise-compatible. The pure-P ## 6. Methodology Notes -- **Scale cap:** None applied. Tests ran at MEDIUM (10,000 buses) for scalability, LARGE (27,862 buses) for FNM ingestion. +- **Scale cap:** None applied. Tests ran at MEDIUM (10,000 buses) for scalability, LARGE (~28,000 buses) for FNM ingestion. - **FNM status:** Suite G executed (FNM_PATH set). MATPOWER `.m` fallback used because GridCal cannot parse intermediate CSV tables. - **Tests skipped:** G-FNM-2 (blocked by G-FNM-1 failure). C-10 not executed (cascaded from A-11). - **Solver versions:** HiGHS (bundled via highspy >= 1.8.0), SCIP (via PuLP SCIP_CMD), PuLP 3.3.0. diff --git a/evaluations/gridcal/tests/fnm_ingestion/gfnm3_output.json b/evaluations/gridcal/tests/fnm_ingestion/gfnm3_output.json new file mode 100644 index 00000000..f4a81a51 --- /dev/null +++ b/evaluations/gridcal/tests/fnm_ingestion/gfnm3_output.json @@ -0,0 +1,252 @@ +{ + "status": "qualified_pass", + "wall_clock_seconds": 46.859, + "details": { + "pass_conditions": { + "va_tolerance_deg": 1.0, + "min_bus_passing_fraction": 0.95, + "p_tolerance_pct": 10.0, + "p_base_floor_mw": 1.0, + "min_branch_passing_fraction": 0.9, + "formulation_difference_max_abs_deg": null + }, + "veragrid_version": "unknown", + "input_path": "matpower", + "matpower_file": "/workspace/data/fnm/reference/cleaned/fnm_main_island.m", + "load_time_seconds": 31.704, + "bus_count": 28000, + "sbase": 100.0, + "solve_time_seconds": 2.412, + "nonzero_angle_buses": 27858, + "ref_bus_count": 28000, + "ref_branch_count": 33000, + "bus_angle": { + "total_compared": 28000, + "passing": 28000, + "failing": 0, + "pass_fraction": 1.0, + "mean_diff_deg": 0.0, + "median_diff_deg": 0.0, + "p95_diff_deg": 0.0, + "p99_diff_deg": 0.0, + "max_diff_deg": 0.0, + "meets_threshold": true + }, + "branch_flow": { + "total_compared": 33000, + "matched": 33000, + "passing": 32200, + "failing": 326, + "pass_fraction": 0.989979, + "mean_dev_pct": 417.7521, + "median_dev_pct": 0.0, + "p95_dev_pct": 0.0, + "p99_dev_pct": 186.8568, + "max_dev_pct": 562955.0, + "meets_threshold": true + }, + "hard_fail": { + "bus_fail_fraction_exceeded": false, + "bus_fail_fraction": 0.0, + "branch_fail_fraction_exceeded": false, + "branch_fail_fraction": 0.010021, + "extreme_branch_dev_exceeded": "True", + "max_branch_dev_pct": 562955.0, + "any_triggered": "True" + }, + "formulation_difference": { + "failing_branches": 326, + "transformer_adjacent_count": 289, + "transformer_adjacent_fraction": 0.8865, + "threshold_fraction": 0.8, + "max_abs_threshold": null, + "qualified": true + }, + "top_failing_branches": [ + { + "from_bus": 10001, + "to_bus": 10002, + "gc_pf_mw": 111582.05, + "ref_pf_mw": -19.82, + "dev_pct": 562955.0, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10003, + "to_bus": 10004, + "gc_pf_mw": -68016.72, + "ref_pf_mw": 12.57, + "dev_pct": 541074.5, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10005, + "to_bus": 10006, + "gc_pf_mw": -13365.25, + "ref_pf_mw": 3.23, + "dev_pct": 413787.2, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10007, + "to_bus": 10008, + "gc_pf_mw": 5234.33, + "ref_pf_mw": -1.61, + "dev_pct": 325193.3, + "branch_type": "Transformer2W", + "transformer_adjacent": true + }, + { + "from_bus": 10009, + "to_bus": 10010, + "gc_pf_mw": -352878.44, + "ref_pf_mw": 109.5, + "dev_pct": 322365.0, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10009, + "to_bus": 10010, + "gc_pf_mw": -347029.63, + "ref_pf_mw": 107.68, + "dev_pct": 322365.0, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10011, + "to_bus": 10012, + "gc_pf_mw": 225916.37, + "ref_pf_mw": -97.49, + "dev_pct": 231837.8, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10013, + "to_bus": 10014, + "gc_pf_mw": -288086.29, + "ref_pf_mw": 133.22, + "dev_pct": 2100090.3, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10015, + "to_bus": 10013, + "gc_pf_mw": 69387.55, + "ref_pf_mw": -32.24, + "dev_pct": 215320.3, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10016, + "to_bus": 10017, + "gc_pf_mw": 115022.09, + "ref_pf_mw": -54.56, + "dev_pct": 210909.7, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10018, + "to_bus": 10028, + "gc_pf_mw": 27058.38, + "ref_pf_mw": -13.81, + "dev_pct": 196049.6, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10010, + "to_bus": 10019, + "gc_pf_mw": 81663.71, + "ref_pf_mw": -42.21, + "dev_pct": 193565.6, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10020, + "to_bus": 10021, + "gc_pf_mw": 25674.28, + "ref_pf_mw": -13.31, + "dev_pct": 192936.7, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10022, + "to_bus": 10023, + "gc_pf_mw": 97969.75, + "ref_pf_mw": -51.86, + "dev_pct": 189022.8, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10024, + "to_bus": 10019, + "gc_pf_mw": 89708.03, + "ref_pf_mw": -51.76, + "dev_pct": 173417.7, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10024, + "to_bus": 10019, + "gc_pf_mw": 80157.8, + "ref_pf_mw": -46.25, + "dev_pct": 173417.7, + "branch_type": "Line", + "transformer_adjacent": false + }, + { + "from_bus": 10025, + "to_bus": 10026, + "gc_pf_mw": -2190.26, + "ref_pf_mw": 1.29, + "dev_pct": 169550.4, + "branch_type": "Transformer2W", + "transformer_adjacent": true + }, + { + "from_bus": 10027, + "to_bus": 10028, + "gc_pf_mw": 15506.36, + "ref_pf_mw": -9.21, + "dev_pct": 168441.6, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10029, + "to_bus": 10021, + "gc_pf_mw": 51470.41, + "ref_pf_mw": -31.11, + "dev_pct": 165557.2, + "branch_type": "Line", + "transformer_adjacent": true + }, + { + "from_bus": 10021, + "to_bus": 10030, + "gc_pf_mw": -59519.43, + "ref_pf_mw": 36.83, + "dev_pct": 161710.2, + "branch_type": "Line", + "transformer_adjacent": true + } + ], + "qualification_reason": "Hard fail triggered by extreme_branch_flow_deviation (max=562955.0%), but 289/326 (88.7%) failing branches are transformer-adjacent, indicating a B-matrix formulation difference (simplified vs full treatment of transformer tap ratios), not a data ingestion error.", + "peak_memory_mb": 1892.8 + }, + "errors": [], + "workarounds": [] +} diff --git a/evaluations/matpower/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md b/evaluations/matpower/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md index 15fdb480..bf3e1f1d 100644 --- a/evaluations/matpower/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/matpower/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md @@ -53,18 +53,18 @@ disconnected islands). | Metric | Value | |--------|-------| -| Buses | 27,862 | -| Branches | 32,606 | -| Generators | 5,741 | +| Buses | ~28,000 | +| Branches | ~33,000 | +| Generators | ~5,700 | | baseMVA | 100.0 | | Excluded buses | 2,445 | -| Non-excluded buses | 27,862 | +| Non-excluded buses | ~28,000 | ### Bus Angle Comparison | Metric | Value | |--------|-------| -| Passing fraction | 1.0000 (27,862 / 27,862) | +| Passing fraction | 1.0000 (~28,000 / ~28,000) | | Threshold | >= 0.95 required | | Max VA deviation | 4.999976e-09 deg | | Mean VA deviation | 2.493020e-09 deg | @@ -84,7 +84,7 @@ disconnected islands). | Metric | Value | |--------|-------| -| Passing fraction | 1.0000 (32,532 / 32,532) | +| Passing fraction | 1.0000 (~33,000 / ~33,000) | | Threshold | >= 0.90 required | | Max branch deviation | 4.999400e-07 pct | | Mean branch deviation | 1.107121e-08 pct | @@ -94,10 +94,10 @@ disconnected islands). | Metric | Value | |--------|-------| -| Non-excluded buses checked | 27,862 | +| Non-excluded buses checked | ~28,000 | | Max |mismatch| | 6.184564e-09 MW | | Mean |mismatch| | 5.618584e-11 MW | -| Buses balanced (< 0.1 MW) | 27,862 / 27,862 (1.0000) | +| Buses balanced (< 0.1 MW) | ~28,000 / ~28,000 (1.0000) | | Power balance | PASS | ### Pass/Fail Gates diff --git a/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index ca4d2df9..f330f178 100644 --- a/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -7,11 +7,11 @@ severity: low timestamp: "2026-03-24T18:00:00Z" --- -# Observation: MATPOWER DCPF scales efficiently to 27,862-bus FNM +# Observation: MATPOWER DCPF scales efficiently to ~28,000-bus FNM ## Finding -MATPOWER solves DCPF on the 27,862-bus FNM main island network in 0.217 +MATPOWER solves DCPF on the ~28,000-bus FNM main island network in 0.217 seconds (Octave, single-threaded), with peak RSS of 1.9 MB. This is the fastest DCPF solve time observed across all evaluated tools on this network. @@ -24,7 +24,7 @@ deviation 5.0e-7 pct). Both solutions were generated by the same MATPOWER version from the same `.mat` file, confirming deterministic reproduction. The v11 bus injection power balance check confirmed max mismatch of -6.2e-9 MW across all 27,862 non-excluded buses. +6.2e-9 MW across all ~28,000 non-excluded buses. ## Implications diff --git a/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index c4fdf61e..8638a3fe 100644 --- a/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/matpower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -11,7 +11,7 @@ timestamp: "2026-03-24T18:00:00Z" ## Finding -MATPOWER's built-in Newton-Raphson solver fails to converge on the 27,862-bus +MATPOWER's built-in Newton-Raphson solver fails to converge on the ~28,000-bus FNM main island network at 0%, 10%, and 20% thermal limit relaxation. The failure mode is a singular Jacobian matrix (rcond ~ 1.9e-17) persisting across all 100 NR iterations at each relaxation level. diff --git a/evaluations/matpower/results/scalability/C-2_acpf_scale_SMALL.md b/evaluations/matpower/results/scalability/C-2_acpf_scale_SMALL.md index 7275e681..aa5680e6 100644 --- a/evaluations/matpower/results/scalability/C-2_acpf_scale_SMALL.md +++ b/evaluations/matpower/results/scalability/C-2_acpf_scale_SMALL.md @@ -46,7 +46,7 @@ set to 1e-8. No DC warm start or tolerance relaxation was needed. | Total load | 67,109.21 MW | | Total P losses | 1,631.66 MW (2.37%) | | Total Q losses | 10,367.86 MVAr | -| VM range | [0.9723, 1.0400] pu | +| VM range | [0.~9,700, 1.0400] pu | | VA range | [-73.95, 0.00] deg | | VM differs from flat start | 95.1% of buses | diff --git a/evaluations/matpower/results/scalability/C-5_ac_feasibility_relaxation_SMALL.md b/evaluations/matpower/results/scalability/C-5_ac_feasibility_relaxation_SMALL.md index d7f210e2..a6b274ac 100644 --- a/evaluations/matpower/results/scalability/C-5_ac_feasibility_relaxation_SMALL.md +++ b/evaluations/matpower/results/scalability/C-5_ac_feasibility_relaxation_SMALL.md @@ -45,9 +45,9 @@ start or tolerance relaxation was needed at any level. | Relaxation | VM Range (pu) | V Over | V Under | Thermal Violations | |------------|---------------|--------|---------|-------------------| -| 0% | [0.9723, 1.0400] | 0 | 0 | 0 / 3206 | -| 10% | [0.9723, 1.0400] | 0 | 0 | 0 / 3206 | -| 20% | [0.9723, 1.0400] | 0 | 0 | 0 / 3206 | +| 0% | [0.~9,700, 1.0400] | 0 | 0 | 0 / 3206 | +| 10% | [0.~9,700, 1.0400] | 0 | 0 | 0 / 3206 | +| 20% | [0.~9,700, 1.0400] | 0 | 0 | 0 / 3206 | The AC solution is well within the original voltage and thermal limits at all relaxation levels. The solution is identical across all three levels because the base case has no diff --git a/evaluations/matpower/results/synthesis.md b/evaluations/matpower/results/synthesis.md index cb27d261..d9399f40 100644 --- a/evaluations/matpower/results/synthesis.md +++ b/evaluations/matpower/results/synthesis.md @@ -135,7 +135,7 @@ MATPOWER's extensibility is strong. The combination of native PTDF/LODF computat - ACPF on SMALL converges from flat start in 5 NR iterations, 0.165s ([C-2 SMALL](scalability/C-2_acpf_scale_SMALL.md)) - DC OPF on SMALL via MIPS in 0.507s with all LMPs resolved ([C-3 SMALL](scalability/C-3_dcopf_scale_SMALL.md)) - AC feasibility with progressive relaxation: SMALL converges at all levels with no violations ([C-5 SMALL](scalability/C-5_ac_feasibility_relaxation_SMALL.md)) -- DCPF on FNM (27,862-bus LARGE) in 0.217s via G-FNM-3 ([G-FNM-3](fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md)) +- DCPF on FNM (~28,000-bus LARGE) in 0.217s via G-FNM-3 ([G-FNM-3](fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md)) #### Weaknesses @@ -168,7 +168,7 @@ None applicable -- the SCUC failure is a solver integration bug, not addressable #### Findings Summary -MATPOWER demonstrates strong scalability for power flow and DC OPF at SMALL scale (2000-bus): DCPF in 0.1s, ACPF in 0.17s, DC OPF in 0.5s. The FNM DCPF result (27,862-bus in 0.217s) provides additional evidence of scalability for linear solves. However, the Suite C SMALL gate failed because C-4 SCUC failed -- this is a cascaded failure from the A-5 GLPK integration bug (blocked_by: A-5), not a scalability limitation. MOST assembled and GLPK solved the 162K-variable SCUC problem in 1.1s, but the solution could not be extracted. All 9 MEDIUM-tier tests are skipped as a consequence. 1 independent failure (C-4 SMALL, but blocked_by A-5 -- effectively 0 independent scalability failures) + 9 blocked skips. +MATPOWER demonstrates strong scalability for power flow and DC OPF at SMALL scale (2000-bus): DCPF in 0.1s, ACPF in 0.17s, DC OPF in 0.5s. The FNM DCPF result (~28,000-bus in 0.217s) provides additional evidence of scalability for linear solves. However, the Suite C SMALL gate failed because C-4 SCUC failed -- this is a cascaded failure from the A-5 GLPK integration bug (blocked_by: A-5), not a scalability limitation. MOST assembled and GLPK solved the 162K-variable SCUC problem in 1.1s, but the solution could not be extracted. All 9 MEDIUM-tier tests are skipped as a consequence. 1 independent failure (C-4 SMALL, but blocked_by A-5 -- effectively 0 independent scalability failures) + 9 blocked skips. **Scale cap: SMALL** -- applied due to C-4 SCUC failure triggering the Suite C SMALL gate. Evidence from G-FNM-3 (DCPF on 28k-bus FNM) suggests DCPF/ACPF would scale to MEDIUM/LARGE, but this cannot be verified within the gated protocol. @@ -274,7 +274,7 @@ MATPOWER achieves a perfect 9/9 on supply chain tests. The self-contained distri ### Power Flow Verification -**G-FNM-3: PASS** -- DCPF on the 27,862-bus FNM main island (via MATPOWER fallback `.mat` path) in 0.217s. All deviations at float64 machine noise (max VA deviation 5.0e-9 deg, max branch flow deviation 5.0e-7%). Bus injection power balance confirmed to machine precision (max mismatch 6.2e-9 MW). MATPOWER is the reference DCPF implementation for this network ([G-FNM-3](fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md)). +**G-FNM-3: PASS** -- DCPF on the ~28,000-bus FNM main island (via MATPOWER fallback `.mat` path) in 0.217s. All deviations at float64 machine noise (max VA deviation 5.0e-9 deg, max branch flow deviation 5.0e-7%). Bus injection power balance confirmed to machine precision (max mismatch 6.2e-9 MW). MATPOWER is the reference DCPF implementation for this network ([G-FNM-3](fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md)). **G-FNM-4: INFORMATIONAL** -- ACPF fails to converge at all relaxation levels (0%, 10%, 20%) with singular Jacobian (rcond ~1.9e-17). DCPF angles reach 537 degrees absolute maximum, indicating structurally ill-conditioned network. The ACPF reference data contains non-physical values (VM up to 379,646 pu), confirming this is a network characteristic rather than a tool limitation [solver-specific: Newton-Raphson on structurally ill-conditioned network] ([G-FNM-4](fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md)). diff --git a/evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md b/evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md index aa23ed2f..fc5270f3 100644 --- a/evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md +++ b/evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md @@ -38,7 +38,7 @@ difference. ## Approach -1. Loaded the pre-cleaned MATPOWER case (`fnm_main_island.m`, 27,862-bus main +1. Loaded the pre-cleaned MATPOWER case (`fnm_main_island.m`, ~28,000-bus main island) using `matpowercaseframes.CaseFrames` + `from_ppc` (MATPOWER fallback since pandapower has no native CSV import). 2. Applied zero RATE_A workaround (set to 9999 before `from_ppc`). @@ -55,7 +55,7 @@ difference. | Metric | Value | |--------|-------| -| Total non-excluded buses | 27,862 | +| Total non-excluded buses | ~28,000 | | Passing (VA dev < 1.0 deg) | 27,761 (99.64%) | | Failing | 101 (0.36%) | | Threshold | >= 95% | @@ -81,8 +81,8 @@ difference. | Metric | Value | |--------|-------| -| Total in-service branches | 32,532 | -| Matched to tool | 32,532 (100%) | +| Total in-service branches | ~33,000 | +| Matched to tool | ~33,000 (100%) | | Passing (dev < 10%) | 32,424 (99.67%) | | Failing | 108 (0.33%) | | Threshold | >= 90% | @@ -101,7 +101,7 @@ difference. | Metric | Value | |--------|-------| -| Buses checked | 27,862 | +| Buses checked | ~28,000 | | Max mismatch (p.u.) | 8.602342e-11 | | Max mismatch (MW) | 8.602342e-09 | | Tolerance (p.u.) | 1.000000e-04 | diff --git a/evaluations/pandapower/results/fnm_ingestion/G-FNM-4_acpf_convergence.md b/evaluations/pandapower/results/fnm_ingestion/G-FNM-4_acpf_convergence.md index fa23ab0b..508ff04f 100644 --- a/evaluations/pandapower/results/fnm_ingestion/G-FNM-4_acpf_convergence.md +++ b/evaluations/pandapower/results/fnm_ingestion/G-FNM-4_acpf_convergence.md @@ -32,12 +32,12 @@ timestamp: 2026-03-24T12:00:00Z ACPF did not converge at any relaxation level (0%, 10%, 20%). The `relaxation_level_achieved` is **infeasible**. pandapower's Newton-Raphson -solver reaches 100 iterations without convergence on this 27,862-bus FNM +solver reaches 100 iterations without convergence on this ~28,000-bus FNM main island network at all three relaxation levels. ## Approach -1. Loaded the pre-cleaned MATPOWER case (`fnm_main_island.m`, 27,862-bus +1. Loaded the pre-cleaned MATPOWER case (`fnm_main_island.m`, ~28,000-bus main island) using `matpowercaseframes.CaseFrames` + `from_ppc` (same MATPOWER fallback path as G-FNM-1/G-FNM-3). 2. Solved DCPF via `pandapower.rundcpp(net)` -- converges successfully. diff --git a/evaluations/pandapower/results/observations/arch-quality-extensibility-B-5_interoperability.md b/evaluations/pandapower/results/observations/arch-quality-extensibility-B-5_interoperability.md new file mode 100644 index 00000000..5c632bc0 --- /dev/null +++ b/evaluations/pandapower/results/observations/arch-quality-extensibility-B-5_interoperability.md @@ -0,0 +1,22 @@ +--- +tag: arch-quality +source_dimension: extensibility +source_test: B-5 +tool: pandapower +severity: low +timestamp: "2026-03-24T00:00:00Z" +--- + +# Observation: DataFrame-native results enable zero-friction data export + +## Finding + +pandapower stores all power flow and OPF results as pandas DataFrames (`net.res_bus`, `net.res_line`, `net.res_gen`, etc.), making data export to CSV, Parquet, or any pandas-supported format a single `.to_csv()` call per result table. No custom serialization, format conversion, or intermediate data structures are needed. + +## Context + +During B-5 (interoperability), exporting full DCPF results for all buses, lines, generators, and transformers required exactly 4 lines of code -- one `.to_csv()` call per table. CSV roundtrip verification confirmed lossless serialization. This is a direct consequence of pandapower's architectural decision to use pandas DataFrames as its primary data model. + +## Implications + +This positive architectural finding is relevant for the Accessibility dimension: any user familiar with pandas can immediately work with pandapower results without learning a custom data format or API. It also benefits integration workflows -- results feed directly into downstream analysis (plotting, statistics, comparison) without transformation. This is one of the strongest interoperability designs among the evaluated tools. diff --git a/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_intermediate_ingestion.md b/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_intermediate_ingestion.md index bb8d37cd..d776f7db 100644 --- a/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_intermediate_ingestion.md +++ b/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_intermediate_ingestion.md @@ -12,16 +12,16 @@ timestamp: 2026-03-14T03:00:00Z ## Finding pandapower's PPC import path aggregates multiple loads per bus into a single load -element (8,576 vs 15,062 expected) and classifies branches by voltage level rather -than tap ratio, splitting the 33,840 MATPOWER branch records into 24,165 lines + -2,393 trafos + 7,282 impedances instead of the intermediate format's 24,117 -branches + 9,723 transformers. Additionally, 55 extra sgen elements are created +element (8,576 vs ~15,000 expected) and classifies branches by voltage level rather +than tap ratio, splitting the ~34,000 MATPOWER branch records into 24,165 lines + +2,393 trafos + 7,282 impedances instead of the intermediate format's ~24,000 +branches + ~9,700 transformers. Additionally, 55 extra sgen elements are created from buses with negative active power demand. ## Context -During G-FNM-1 ingestion testing, pandapower loaded the 30,307-bus FNM network -via `from_ppc()`. The merged branch total (33,840) and bus count (30,307) match +During G-FNM-1 ingestion testing, pandapower loaded the ~30,000-bus FNM network +via `from_ppc()`. The merged branch total (~34,000) and bus count (~30,000) match exactly. However, the per-table record counts diverge from the intermediate manifest because pandapower uses a fundamentally different element classification scheme. The load aggregation means that per-load attributes (individual load ID, diff --git a/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-2_field_coverage_audit.md b/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-2_field_coverage_audit.md index ff86e329..9a13528d 100644 --- a/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-2_field_coverage_audit.md +++ b/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-2_field_coverage_audit.md @@ -20,7 +20,7 @@ line shunt conductances. ## Context G-FNM-2 audited pandapower's data model against the field-criticality-matrix (v10) -after importing the 30,307-bus FNM via `scipy.io.loadmat` + `from_ppc`. The PPC +after importing the ~30,000-bus FNM via `scipy.io.loadmat` + `from_ppc`. The PPC format is a lossy intermediate: it flattens transformer I/O codes into impedance values, aggregates per-bus shunts, and drops area interchange parameters entirely. diff --git a/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md index 4c76898f..5d9a9745 100644 --- a/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/pandapower/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -18,7 +18,7 @@ extreme branch flow deviation (596.6% > 50% threshold). ## Context -G-FNM-3 loaded the pre-cleaned 27,862-bus FNM main island via `matpowercaseframes.CaseFrames` +G-FNM-3 loaded the pre-cleaned ~28,000-bus FNM main island via `matpowercaseframes.CaseFrames` + `from_ppc` (MATPOWER fallback path, since pandapower lacks native CSV ingestion). The DCPF converges and the aggregate metrics are strong (99.64% buses pass, 99.67% branches pass), but a localized cluster of outliers in the 69-138 kV sub-network causes the worst-case branch diff --git a/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md index 12552145..a900c43a 100644 --- a/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -7,18 +7,18 @@ severity: low timestamp: 2026-03-14T04:00:00Z --- -# Observation: pandapower DCPF solves 27,862-bus FNM in 0.40 seconds +# Observation: pandapower DCPF solves ~28,000-bus FNM in 0.40 seconds ## Finding -pandapower's `rundcpp()` solved the 27,862-bus FNM main island DCPF in 0.40 seconds +pandapower's `rundcpp()` solved the ~28,000-bus FNM main island DCPF in 0.40 seconds wall-clock time. Network loading via `matpowercaseframes` + `from_ppc` took an additional 0.18 seconds. Total ingestion-to-solution time was under 2 seconds. ## Context -G-FNM-3 loaded the pre-cleaned FNM main island (27,862 buses, 32,606 branches, -5,741 generators) and ran DCPF. Despite the test failing due to a hard-fail +G-FNM-3 loaded the pre-cleaned FNM main island (~28,000 buses, ~33,000 branches, +~5,700 generators) and ran DCPF. Despite the test failing due to a hard-fail condition on localized branch flow outliers, the solver itself performed well: the DCPF converged immediately and produced results matching the reference solution on 99.64% of buses and 99.67% of branches. diff --git a/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md b/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md index eb54b120..21bdf515 100644 --- a/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md +++ b/evaluations/pandapower/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md @@ -7,11 +7,11 @@ severity: medium timestamp: 2026-03-24T12:00:00Z --- -# Observation: pandapower ACPF non-convergence on 27,862-bus FNM +# Observation: pandapower ACPF non-convergence on ~28,000-bus FNM ## Finding -pandapower's internal Newton-Raphson solver fails to converge on the 27,862-bus FNM main +pandapower's internal Newton-Raphson solver fails to converge on the ~28,000-bus FNM main island at all three progressive relaxation levels (0%, 10%, 20%). The network is numerically ill-conditioned for ACPF, with DCPF angles reaching 536.9 degrees maximum absolute value. diff --git a/evaluations/pandapower/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/pandapower/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md index 168ae79b..b9afd2d8 100644 --- a/evaluations/pandapower/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/pandapower/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -20,7 +20,7 @@ data_ingestion_error per the formulation difference classification protocol. ## Context G-FNM-3 compared pandapower's DCPF solution against the MATPOWER reference on -27,862 buses. Aggregate performance is strong (99.64% bus pass rate, 99.67% branch +~28,000 buses. Aggregate performance is strong (99.64% bus pass rate, 99.67% branch pass rate), but the hard-fail condition is triggered by a maximum branch flow deviation of 596.6% on a single branch (14102->48022). The affected buses form a radial cluster with zero load and zero generation, where small impedance handling diff --git a/evaluations/pandapower/results/synthesis.md b/evaluations/pandapower/results/synthesis.md index fb817600..7a21dd54 100644 --- a/evaluations/pandapower/results/synthesis.md +++ b/evaluations/pandapower/results/synthesis.md @@ -256,7 +256,7 @@ pandapower has an exemplary supply chain profile. The BSD license, fully inspect #### Data Model Fidelity -**G-FNM-1 (Intermediate Ingestion):** FAIL. pandapower has no PSS/E parser of any kind. The tool cannot ingest the 17-table intermediate CSV format. All FNM testing proceeds via MATPOWER fallback path (`fnm_main_island.m` via `matpowercaseframes` + `from_ppc`). The MATPOWER path aggregates per-load records (8,576 vs 15,062 expected) and reclassifies branches by voltage level rather than PSS/E record type. ([G-FNM-1](fnm_ingestion/G-FNM-1_intermediate_ingestion.md)) +**G-FNM-1 (Intermediate Ingestion):** FAIL. pandapower has no PSS/E parser of any kind. The tool cannot ingest the 17-table intermediate CSV format. All FNM testing proceeds via MATPOWER fallback path (`fnm_main_island.m` via `matpowercaseframes` + `from_ppc`). The MATPOWER path aggregates per-load records (8,576 vs ~15,000 expected) and reclassifies branches by voltage level rather than PSS/E record type. ([G-FNM-1](fnm_ingestion/G-FNM-1_intermediate_ingestion.md)) **G-FNM-2 (Field Coverage):** SKIP, blocked by G-FNM-1. Prior v10 assessment via MATPOWER path found 100% DCPF-critical coverage (19/19), 55.8% ACPF-critical coverage (29/52), and 27.6% informational coverage (24/87). The 55.8% ACPF-critical gap includes area interchange controls, switched shunt parameters, and transformer control modes. ([G-FNM-2](fnm_ingestion/G-FNM-2_field_coverage_audit.md)) @@ -266,9 +266,9 @@ pandapower has an exemplary supply chain profile. The BSD license, fully inspect **G-FNM-3 (DCPF Verification):** FAIL (hard-fail triggered). Aggregate performance is strong: 99.64% of buses pass VA tolerance, 99.67% of branches pass flow tolerance, and bus injection power balance passes at machine precision (max mismatch 8.6e-11 p.u.). However, a localized cluster of ~101 subtransmission buses produces systematic 14-21 degree angle deviations, causing one branch to exceed the 50% flow deviation hard-fail ceiling (596.6%). The outlier cluster has zero transformer adjacency, ruling out formulation difference classification. This is attributed to localized impedance handling differences in the MATPOWER PPC import path [tool-specific]. ([G-FNM-3](fnm_ingestion/G-FNM-3_dcpf_verification.md)) -**G-FNM-4 (ACPF Convergence):** INFORMATIONAL -- infeasible at all relaxation levels (0%, 10%, 20%). pandapower's internal Newton-Raphson solver fails to converge on the 27,862-bus FNM at 100 iterations per attempt. Contributing factors: PPC import flattens transformer AC data (tap control modes, switched shunt steps), the ~101 outlier buses create Jacobian ill-conditioning, and pandapower uses its own NR implementation (not Ipopt) which may have different convergence properties on ill-conditioned large-scale networks. ([G-FNM-4](fnm_ingestion/G-FNM-4_acpf_convergence.md)) +**G-FNM-4 (ACPF Convergence):** INFORMATIONAL -- infeasible at all relaxation levels (0%, 10%, 20%). pandapower's internal Newton-Raphson solver fails to converge on the ~28,000-bus FNM at 100 iterations per attempt. Contributing factors: PPC import flattens transformer AC data (tap control modes, switched shunt steps), the ~101 outlier buses create Jacobian ill-conditioning, and pandapower uses its own NR implementation (not Ipopt) which may have different convergence properties on ill-conditioned large-scale networks. ([G-FNM-4](fnm_ingestion/G-FNM-4_acpf_convergence.md)) -*Scalability implication:* DCPF solves the 27,862-bus FNM in 2.9 seconds, demonstrating adequate LARGE-tier DCPF scalability. ACPF cannot be assessed at LARGE tier due to convergence failure, which is primarily an ingestion fidelity issue. +*Scalability implication:* DCPF solves the ~28,000-bus FNM in 2.9 seconds, demonstrating adequate LARGE-tier DCPF scalability. ACPF cannot be assessed at LARGE tier due to convergence failure, which is primarily an ingestion fidelity issue. #### Supplemental Data Representability @@ -319,7 +319,7 @@ pandapower has an exemplary supply chain profile. The BSD license, fully inspect ### FNM Data Model -- **Load aggregation in PPC import:** Multiple PSS/E loads per bus aggregated into single pandapower load element, losing per-load granularity (8,576 vs 15,062 expected). (Source: [G-FNM-1 observation](observations/fnm-data-model-fnm_ingestion-G-FNM-1_intermediate_ingestion.md)) +- **Load aggregation in PPC import:** Multiple PSS/E loads per bus aggregated into single pandapower load element, losing per-load granularity (8,576 vs ~15,000 expected). (Source: [G-FNM-1 observation](observations/fnm-data-model-fnm_ingestion-G-FNM-1_intermediate_ingestion.md)) - **ACPF-critical field coverage limited:** 55.8% via MATPOWER path (missing area interchange, switched shunt parameters, transformer control modes). (Source: [G-FNM-2 observation](observations/fnm-data-model-fnm_ingestion-G-FNM-2_field_coverage_audit.md)) - **No interface/flowgate or contingency definition models:** 43% of supplemental CSV fields are tool-external. Interface flow limits cannot be enforced in OPF. (Source: [G-FNM-5 observation](observations/fnm-data-model-fnm_ingestion-G-FNM-5_supplemental_csv_representability.md)) - **Thermal rating unit difference:** `line.max_i_ka` uses current (kA) not power (MVA), requiring voltage-dependent conversion at ingestion. (Source: [G-FNM-5 observation](observations/formulation-difference-fnm_ingestion-G-FNM-5_supplemental_csv_representability.md)) diff --git a/evaluations/pandapower/tests/fnm_ingestion/_output.json b/evaluations/pandapower/tests/fnm_ingestion/_output.json new file mode 100644 index 00000000..7978277f --- /dev/null +++ b/evaluations/pandapower/tests/fnm_ingestion/_output.json @@ -0,0 +1,65 @@ +{ + "status": "pass", + "wall_clock_seconds": 8.843149347929284, + "details": { + "load_time_seconds": 2.84173441096209, + "baseMVA": { + "expected": 100.0, + "actual": 100.0 + }, + "table_results": { + "bus": { + "expected": 30000, + "actual": 30000, + "match": true + }, + "generator": { + "expected": 5800, + "actual_gen": 4668, + "actual_sgen": 1151, + "actual_ext_grid": 4, + "actual_total": 5823, + "match": false, + "note": "pandapower splits generators into gen (4668), sgen (1151), ext_grid (4). Total 5823 vs expected 5800. Difference of 55 due to 56 buses with negative Pd creating extra sgen elements." + }, + "branch_merged": { + "expected_branch": 24000, + "expected_transformer": 9700, + "expected_merged": 34000, + "actual_line": 24165, + "actual_trafo": 2393, + "actual_impedance": 7282, + "actual_merged": 34000, + "merged_match": true, + "note": "pandapower classifies branches by voltage level difference (line=same kV, trafo=different kV) rather than tap ratio (branch=tap==0, transformer=tap!=0). Merged total matches. pandapower: line=24165, trafo=2393, impedance=7282. Intermediate: branch=24000, transformer=9700." + }, + "load": { + "expected": 15000, + "actual": 8576, + "match": false, + "note": "PPC import aggregates multiple loads per bus into a single load. Actual 8576 vs expected 15000." + }, + "switched_shunt": { + "expected": 3100, + "actual": 3110, + "match": false, + "note": "Shunts from bus Bs column: 3110 vs expected 3100. Difference of 4." + }, + "area": { + "expected": 49, + "actual": 74, + "note": "pandapower has no separate area table; area info embedded in bus" + }, + "zone": { + "expected": 90, + "note": "pandapower has no separate zone table; zone info embedded in bus" + } + }, + "zero_rate_a_branches_fixed": 28 + }, + "errors": [], + "workarounds": [ + "from_mpc fails due to missing 'version' field in .mat struct. Used scipy.io.loadmat + from_ppc instead (stable workaround).", + "from_ppc bug: variable 'sn' reuse causes IndexError when branches have zero RATE_A. Pre-set zero RATE_A to 9999 before conversion (stable workaround, deterministic pre-processing)." + ] +} diff --git a/evaluations/powermodels/results/fnm_ingestion/G-FNM-1_intermediate_ingestion.md b/evaluations/powermodels/results/fnm_ingestion/G-FNM-1_intermediate_ingestion.md index e4d434b2..17c467f0 100644 --- a/evaluations/powermodels/results/fnm_ingestion/G-FNM-1_intermediate_ingestion.md +++ b/evaluations/powermodels/results/fnm_ingestion/G-FNM-1_intermediate_ingestion.md @@ -34,7 +34,7 @@ PowerModels.jl cannot ingest the PSS/E intermediate CSV format. It supports only MATPOWER `.m`, PSS/E `.raw` (v33 spec), and PowerModels JSON formats. The intermediate CSV tables are tabular extracts from PSS/E, not a native format PowerModels can parse. -The MATPOWER `.m` fallback file loads successfully (3.05s, 27,862 buses), confirming the +The MATPOWER `.m` fallback file loads successfully (3.05s, ~28,000 buses), confirming the fallback ingestion path works for downstream G-FNM-3/4/5 tests. ## Approach @@ -71,22 +71,22 @@ to verify the fallback path for downstream tests. This does NOT change G-FNM-1 s | File | `data/fnm/reference/cleaned/fnm_main_island.m` | | Load time | 3.05 s | | baseMVA | 100 | -| Slack bus | 29421 (bus_type=3) | +| Slack bus | (bus_type=3) | | Tap=0 branches | 0 (correctly mapped to 1.0 by MATPOWER converter) | | Tap=1.0 branches | 30,248 | -| Buses | 27,862 | -| Branches | 32,606 | -| Generators | 5,741 | +| Buses | ~28,000 | +| Branches | ~33,000 | +| Generators | ~5,700 | | Loads | 8,624 | ### Record Count Comparison (MATPOWER fallback vs manifest) | Table | Manifest Expected | MATPOWER Actual | Delta | % Diff | |-------|------------------:|----------------:|------:|-------:| -| bus | 30,307 | 27,862 | -2,445 | -8.1% | -| load | 15,062 | 8,624 | -6,438 | -42.7% | -| generator | 5,768 | 5,741 | -27 | -0.5% | -| branch+transformer | 33,840 | 32,606 | -1,234 | -3.6% | +| bus | ~30,000 | ~28,000 | -2,445 | -8.1% | +| load | ~15,000 | 8,624 | -6,438 | -42.7% | +| generator | ~5,800 | ~5,700 | -27 | -0.5% | +| branch+transformer | ~34,000 | ~33,000 | -1,234 | -3.6% | Count mismatches are attributable to the MATPOWER fallback being a pre-cleaned main-island subset, not a PowerModels ingestion error. Isolated buses (IDE=4), de-energized equipment, @@ -97,7 +97,7 @@ and off-island fragments were removed during the external cleaning process. | Check | Result | Detail | |-------|--------|--------| | baseMVA | 100 | Correct (matches manifest sbase) | -| Slack bus present | Yes | Bus 29421 (bus_type=3) | +| Slack bus present | Yes | Bus (bus_type=3) | | Tap ratio preservation | OK | 0 branches with tap=0; 30,248 with tap=1.0 | ## Workarounds diff --git a/evaluations/powermodels/results/fnm_ingestion/G-FNM-3_dcpf_verification.md b/evaluations/powermodels/results/fnm_ingestion/G-FNM-3_dcpf_verification.md index 0221025f..bf0de882 100644 --- a/evaluations/powermodels/results/fnm_ingestion/G-FNM-3_dcpf_verification.md +++ b/evaluations/powermodels/results/fnm_ingestion/G-FNM-3_dcpf_verification.md @@ -26,7 +26,7 @@ timestamp: "2026-03-24T12:00:00Z" ## Result: FAIL PowerModels' `solve_dc_pf` with `DCPPowerModel` produces DCPF results with systematic -deviations from the MATPOWER reference on the 27,862-bus FNM main island. Bus angle pass +deviations from the MATPOWER reference on the ~28,000-bus FNM main island. Bus angle pass rate is 2.43% (need >=95%) and branch flow pass rate is 78.88% (need >=90%). All three hard-fail conditions are triggered. The deviations are caused by a documented formulation difference: `DCPPowerModel` uses a simplified B-matrix (`b = -1/x`) that ignores @@ -36,7 +36,7 @@ taps via `makeBdc()`. [tool-specific: DCPPowerModel formulation choice] ## Approach 1. Loaded the cleaned FNM case (`data/fnm/reference/cleaned/fnm_main_island.m`) via - `PowerModels.parse_file` (27,862 buses, 32,606 branches, 5,741 generators, baseMVA=100). + `PowerModels.parse_file` (~28,000 buses, ~33,000 branches, ~5,700 generators, baseMVA=100). 2. Applied zero-reactance preprocessing (0 fixes needed). No rate fixes needed. 3. Solved DCPF using `PowerModels.solve_dc_pf(data, HiGHS.Optimizer)` which internally uses `DCPPowerModel`. Per the task specification, `solve_dc_pf` is used rather than @@ -60,7 +60,7 @@ taps via `makeBdc()`. [tool-specific: DCPPowerModel formulation choice] | Simplex iterations | 12,904 | | Solve time | 8.20 s | | HiGHS wall time | 6.79 s | -| Nonzero VA buses | 27,858 / 27,862 | +| Nonzero VA buses | 27,858 / ~28,000 | ### Power Balance Check @@ -82,7 +82,7 @@ internal solution is self-consistent (OPTIMAL termination, zero objective). | Metric | Value | Pass Condition | |--------|-------|----------------| -| Non-excluded buses | 27,862 | -- | +| Non-excluded buses | ~28,000 | -- | | Passing (\|dev\| < 1.0 deg) | 678 (2.43%) | >= 95% | | Failing | 27,184 | -- | | Mean deviation | 5.098394e+00 deg | -- | @@ -94,7 +94,7 @@ internal solution is self-consistent (OPTIMAL termination, zero objective). | Metric | Value | Pass Condition | |--------|-------|----------------| -| In-service branches | 32,532 | -- | +| In-service branches | ~33,000 | -- | | Passing (dev < 10%) | 25,660 (78.88%) | >= 90% | | Failing | 6,872 | -- | | Mean deviation | 3.129475e+01% | -- | diff --git a/evaluations/powermodels/results/fnm_ingestion/G-FNM-4_acpf_convergence.md b/evaluations/powermodels/results/fnm_ingestion/G-FNM-4_acpf_convergence.md index 0e95f66a..38977924 100644 --- a/evaluations/powermodels/results/fnm_ingestion/G-FNM-4_acpf_convergence.md +++ b/evaluations/powermodels/results/fnm_ingestion/G-FNM-4_acpf_convergence.md @@ -25,7 +25,7 @@ timestamp: "2026-03-24T12:30:00Z" ## Result: INFORMATIONAL -ACPF does not converge on the 27,862-bus FNM case at any relaxation level (0%, 10%, 20%). +ACPF does not converge on the ~28,000-bus FNM case at any relaxation level (0%, 10%, 20%). Ipopt diverges rapidly: primal infeasibility grows from 171 to 4.17e6 by iteration 14, with MUMPS requiring repeated memory reallocation. The solver hangs during MUMPS workspace expansion after iteration 14. This outcome is consistent with the ACPF reference data, @@ -89,7 +89,7 @@ failure because: | Solver | HiGHS 1.13.1 | | Termination | OPTIMAL | | Solve time | 8.75 s | -| Nonzero VA buses | 27,858 / 27,862 | +| Nonzero VA buses | 27,858 / ~28,000 | | dcpf_init_mean_deg | 214.4886 | | dcpf_init_max_abs_deg | 554.7958 | diff --git a/evaluations/powermodels/results/observations/api-friction-fnm_ingestion-G-FNM-1_intermediate_ingestion.md b/evaluations/powermodels/results/observations/api-friction-fnm_ingestion-G-FNM-1_intermediate_ingestion.md index 88fcf9f4..47145049 100644 --- a/evaluations/powermodels/results/observations/api-friction-fnm_ingestion-G-FNM-1_intermediate_ingestion.md +++ b/evaluations/powermodels/results/observations/api-friction-fnm_ingestion-G-FNM-1_intermediate_ingestion.md @@ -23,7 +23,7 @@ from PSS/E v31 records). PowerModels' `parse_file()` dispatches on file extensio does not recognize `.csv`. Additionally, its PSS/E RAW parser fails on the FNM's v31 header format due to a single-line Case Identification parsing limitation. -The MATPOWER fallback path works (3.05s load time, 27,862 buses), but the fallback file +The MATPOWER fallback path works (3.05s load time, ~28,000 buses), but the fallback file is a pre-cleaned main-island subset with fewer records than the raw source (bus count deficit: -8.1%, load count deficit: -42.7%). diff --git a/evaluations/powermodels/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_matpower_fallback_count_discrepancy.md b/evaluations/powermodels/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_matpower_fallback_count_discrepancy.md index cf9131a6..4410c741 100644 --- a/evaluations/powermodels/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_matpower_fallback_count_discrepancy.md +++ b/evaluations/powermodels/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_matpower_fallback_count_discrepancy.md @@ -22,10 +22,10 @@ G-FNM-1 record count comparison (manifest vs. PowerModels actual): | Table | Manifest | Actual | Delta | % Diff | |-------|----------|--------|-------|--------| -| bus | 30,307 | 27,862 | -2,445 | -8.1% | -| load | 15,062 | 8,624 | -6,438 | -42.7% | -| generator | 5,768 | 5,741 | -27 | -0.5% | -| branch+transformer | 33,840 | 32,606 | -1,234 | -3.6% | +| bus | ~30,000 | ~28,000 | -2,445 | -8.1% | +| load | ~15,000 | 8,624 | -6,438 | -42.7% | +| generator | ~5,800 | ~5,700 | -27 | -0.5% | +| branch+transformer | ~34,000 | ~33,000 | -1,234 | -3.6% | The manifest counts raw PSS/E v31 records including isolated buses (IDE=4), de-energized equipment, and off-island network fragments. The `fnm_main_island.m` fallback is a @@ -39,7 +39,7 @@ are proportionally smaller and consistent with removing isolated/off-island elem This discrepancy does not indicate a PowerModels data-model defect. It reflects that the two inputs (raw PSS/E manifest vs. cleaned MATPOWER fallback) have different scope. -The post-ingestion fidelity checks (baseMVA=100, slack bus 29421, tap ratio preservation) +The post-ingestion fidelity checks (baseMVA=100, slack bus , tap ratio preservation) all pass on the loaded data, confirming PowerModels correctly parses what it receives. A valid full-fidelity comparison would require either PowerModels successfully parsing the raw PSS/E file (currently blocked) or a manifest derived from the same cleaned scope. diff --git a/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_matpower_load_performance.md b/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_matpower_load_performance.md index 2499799c..14e2f31b 100644 --- a/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_matpower_load_performance.md +++ b/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_matpower_load_performance.md @@ -11,14 +11,14 @@ timestamp: "2026-03-13T00:00:00Z" ## Finding -PowerModels loaded the 27,862-bus MATPOWER fallback file in 2.97 seconds. This confirms +PowerModels loaded the ~28,000-bus MATPOWER fallback file in 2.97 seconds. This confirms that PowerModels' MATPOWER parser handles LARGE-scale networks without difficulty, though -the actual FNM (30,307 buses) could not be tested due to PSS/E parser incompatibility. +the actual FNM (~30,000 buses) could not be tested due to PSS/E parser incompatibility. ## Context -The MATPOWER fallback (`fnm_main_island.m`) contains 27,862 buses, 32,606 branches, -5,741 generators, and 8,624 loads. PowerModels parsed this file without errors (warnings +The MATPOWER fallback (`fnm_main_island.m`) contains ~28,000 buses, ~33,000 branches, +~5,700 generators, and 8,624 loads. PowerModels parsed this file without errors (warnings about angle limits and branch orientation reversals were handled automatically). The 2.97-second load time is well within acceptable bounds for a network of this scale. diff --git a/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-2_field_coverage_audit.md b/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-2_field_coverage_audit.md index 0d3afab8..efbd96cb 100644 --- a/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-2_field_coverage_audit.md +++ b/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-2_field_coverage_audit.md @@ -7,11 +7,11 @@ severity: medium timestamp: "2026-03-13T23:15:00Z" --- -# Observation: MATPOWER fallback carries 27,862 buses and 32,606 branches successfully +# Observation: MATPOWER fallback carries ~28,000 buses and ~33,000 branches successfully ## Finding -PowerModels successfully loaded and indexed the 27,862-bus, 32,606-branch MATPOWER fallback +PowerModels successfully loaded and indexed the ~28,000-bus, ~33,000-branch MATPOWER fallback of the regional FNM in approximately 3 seconds. The data model correctly represents all core network elements (buses, loads, generators, branches including transformer-as-branch, and fixed shunts) at this scale. diff --git a/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md b/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md index 015236ee..cb0d3238 100644 --- a/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md +++ b/evaluations/powermodels/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md @@ -7,11 +7,11 @@ severity: medium timestamp: "2026-03-24T12:30:00Z" --- -# Observation: ACPF Diverges on 27,862-Bus FNM with ~7 GB Memory Consumption +# Observation: ACPF Diverges on ~28,000-Bus FNM with ~7 GB Memory Consumption ## Finding -PowerModels' `solve_ac_pf` with Ipopt/MUMPS diverges on the FNM 27,862-bus case, +PowerModels' `solve_ac_pf` with Ipopt/MUMPS diverges on the FNM ~28,000-bus case, consuming approximately 7 GB of memory. Ipopt's MUMPS linear solver required 4 memory reallocation attempts (icntl[13] from 1000 to 16000) before continuing to diverge. The 67,206-variable NLP with 380,065 Jacobian nonzeros is tractable in terms of diff --git a/evaluations/powermodels/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/powermodels/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md index d2145644..c6572468 100644 --- a/evaluations/powermodels/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/powermodels/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -12,7 +12,7 @@ timestamp: "2026-03-24T12:00:00Z" ## Finding PowerModels' `solve_dc_pf` (which hardcodes `DCPPowerModel`) uses a simplified B-matrix -that ignores transformer tap ratios. On the 27,862-bus FNM network with 12,501 +that ignores transformer tap ratios. On the ~28,000-bus FNM network with 12,501 transformer-connected buses, this produces systematic angle deviations of 5.1 degrees mean (62.2 degrees max) compared to MATPOWER's full B-matrix reference, causing 97.6% of buses to fail the 1.0-degree tolerance. diff --git a/evaluations/powermodels/results/observations/solver-issues-scalability-C5_acpf_medium_ipopt_divergence.md b/evaluations/powermodels/results/observations/solver-issues-scalability-C5_acpf_medium_ipopt_divergence.md index bc92d983..d7282b3e 100644 --- a/evaluations/powermodels/results/observations/solver-issues-scalability-C5_acpf_medium_ipopt_divergence.md +++ b/evaluations/powermodels/results/observations/solver-issues-scalability-C5_acpf_medium_ipopt_divergence.md @@ -15,7 +15,7 @@ PowerModels' `solve_ac_pf` formulation contains 0 inequality constraints, making ## Context -C-5 tests progressive AC feasibility relaxation (0%, 10%, 20% thermal limit relaxation). The protocol assumes that thermal limits may cause ACPF convergence difficulties and that relaxing them could enable convergence. However, PowerModels' `solve_ac_pf` is a pure power balance feasibility problem with no thermal limit constraints. Ipopt reports 23,392 equality constraints and 0 inequality constraints. The same finding was observed in G-FNM-4 on the 27,862-bus FNM case. +C-5 tests progressive AC feasibility relaxation (0%, 10%, 20% thermal limit relaxation). The protocol assumes that thermal limits may cause ACPF convergence difficulties and that relaxing them could enable convergence. However, PowerModels' `solve_ac_pf` is a pure power balance feasibility problem with no thermal limit constraints. Ipopt reports 23,392 equality constraints and 0 inequality constraints. The same finding was observed in G-FNM-4 on the ~28,000-bus FNM case. ## Implications diff --git a/evaluations/powermodels/results/observations/workaround-needed-fnm_ingestion-G-FNM-1_matpower_fallback_required.md b/evaluations/powermodels/results/observations/workaround-needed-fnm_ingestion-G-FNM-1_matpower_fallback_required.md index b462d562..7fc0fa25 100644 --- a/evaluations/powermodels/results/observations/workaround-needed-fnm_ingestion-G-FNM-1_matpower_fallback_required.md +++ b/evaluations/powermodels/results/observations/workaround-needed-fnm_ingestion-G-FNM-1_matpower_fallback_required.md @@ -21,7 +21,7 @@ than the raw source. G-FNM-1 tested three ingestion paths: 1. Intermediate CSV: No CSV parser in PowerModels (not attempted) 2. PSS/E v31 RAW: Parser fails on Case Identification header format -3. MATPOWER `.m` fallback: Loads successfully (2.97 s, 27,862 buses) +3. MATPOWER `.m` fallback: Loads successfully (2.97 s, ~28,000 buses) The workaround classification is `blocking` because the external conversion step (MATPOWER/Octave `psse2mpc()` or equivalent) is outside PowerModels' control and diff --git a/evaluations/powermodels/results/p2_readiness/P2-1_psse_raw_parsing.md b/evaluations/powermodels/results/p2_readiness/P2-1_psse_raw_parsing.md index ddb2c8e0..fe006af3 100644 --- a/evaluations/powermodels/results/p2_readiness/P2-1_psse_raw_parsing.md +++ b/evaluations/powermodels/results/p2_readiness/P2-1_psse_raw_parsing.md @@ -25,7 +25,7 @@ unresolved as of March 2026). ### FNM RAW File Parse Attempt -**File:** `/data/fnm-source/AUC_AN_2026_2026_S01_ON_NETWORK_MODEL.RAW` +**File:** `` #### FNM RAW version detected from file header: diff --git a/evaluations/powermodels/results/synthesis.md b/evaluations/powermodels/results/synthesis.md index f3253b3c..396a6a1b 100644 --- a/evaluations/powermodels/results/synthesis.md +++ b/evaluations/powermodels/results/synthesis.md @@ -290,7 +290,7 @@ Suite G executed (FNM_PATH set). FNM ingestion used the MATPOWER fallback path b ### Data Model Fidelity -**G-FNM-1 (fail):** PowerModels has no CSV parser. The PSS/E RAW parser fails on v31 headers. MATPOWER fallback loaded 27,862 buses / 32,606 branches / 5,741 generators / 8,624 loads (vs manifest: 30,307 / 33,840 / 5,768 / 15,062). Count deltas are attributable to the pre-cleaned main-island subset, not ingestion errors. [tool-specific: no CSV ingestion path] ([G-FNM-1](fnm_ingestion/G-FNM-1_intermediate_ingestion.md)) +**G-FNM-1 (fail):** PowerModels has no CSV parser. The PSS/E RAW parser fails on v31 headers. MATPOWER fallback loaded ~28,000 buses / ~33,000 branches / ~5,700 generators / 8,624 loads (vs manifest: ~30,000 / ~34,000 / ~5,800 / ~15,000). Count deltas are attributable to the pre-cleaned main-island subset, not ingestion errors. [tool-specific: no CSV ingestion path] ([G-FNM-1](fnm_ingestion/G-FNM-1_intermediate_ingestion.md)) **G-FNM-2 (skip):** Field coverage audit skipped because G-FNM-1 failed (no CSV ingestion). Cannot assess field-level coverage against the intermediate schema. ([G-FNM-2](fnm_ingestion/G-FNM-2_field_coverage_audit.md)) diff --git a/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.jl b/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.jl index 9c4d2efa..507e6896 100644 --- a/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.jl +++ b/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.jl @@ -2,7 +2,7 @@ Test G-FNM-3: DCPF Verification on Cleaned FNM Case Dimension: fnm_ingestion -Network: LARGE (FNM 27,862-bus main island) +Network: LARGE (FNM 28000-bus main island) Pass condition: Per pass_conditions.json dcpf section: - Bus angle: >=95% of non-excluded buses within 1.0 deg tolerance - Branch flow: >=90% of in-service branches within 10% tolerance (floor 1.0 MW) diff --git a/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.jl b/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.jl index 106ed586..fd4788bc 100644 --- a/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.jl +++ b/evaluations/powermodels/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.jl @@ -2,7 +2,7 @@ Test G-FNM-4: ACPF Convergence on Cleaned FNM Case Dimension: fnm_ingestion -Network: LARGE (FNM 27,862-bus main island) +Network: LARGE (FNM 28000-bus main island) Pass condition: Informational (all outcomes recorded, no gate consequence) Tool: PowerModels.jl Solver: Ipopt diff --git a/evaluations/powersimulations/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md b/evaluations/powersimulations/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md index e28424d7..9948b0b6 100644 --- a/evaluations/powersimulations/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md +++ b/evaluations/powersimulations/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md @@ -41,7 +41,7 @@ parsing, which is also known to fail on this FNM's Case Identification header fo **(b) Record count fidelity:** Not evaluated because sub-check (a) failed. After confirming PSS/E path failure, loaded the pre-cleaned MATPOWER fallback -(`fnm_main_island.m`, 27,862-bus main island) to verify the MATPOWER ingestion path +(`fnm_main_island.m`, ~28,000-bus main island) to verify the MATPOWER ingestion path works for downstream G-FNM-3/4/5 tests. ## Output @@ -78,20 +78,20 @@ Not evaluated because PSS/E-derived format parsing did not succeed. ### MATPOWER fallback -- verified The MATPOWER fallback file loaded successfully in 12.03 seconds. This is the -pre-cleaned main island (27,862 buses), not the full 30,307-bus FNM. +pre-cleaned main island (~28,000 buses), not the full ~30,000-bus FNM. | Component Type | MATPOWER Count | Manifest (full FNM) | Notes | |---|---|---|---| -| ACBus | 27,862 | 30,307 | Main island only (2,445 isolated buses removed) | -| Generator (all) | 5,741 | 5,768 | 27 generators on removed islands | -| Line | 23,058 | 24,117 | Lines on removed islands excluded | +| ACBus | ~28,000 | ~30,000 | Main island only (2,445 isolated buses removed) | +| Generator (all) | ~5,700 | ~5,800 | 27 generators on removed islands | +| Line | 23,058 | ~24,000 | Lines on removed islands excluded | | Transformer2W | 7,190 | -- | MATPOWER merges branch types | | TapTransformer | 2,358 | -- | MATPOWER merges branch types | | PhaseShiftingTransformer | 0 | -- | None in this network | -| Total branches | 32,606 | 33,840 (merged) | Consistent with island removal | -| ElectricLoad | 11,734 | 15,062 | Loads mapped to PowerLoad + StandardLoad | +| Total branches | ~33,000 | ~34,000 (merged) | Consistent with island removal | +| ElectricLoad | ~12,000 | ~15,000 | Loads mapped to PowerLoad + StandardLoad | | PowerLoad | 8,624 | -- | Subset of ElectricLoad | -| FixedAdmittance | 3,110 | 3,114 | Switched shunts mapped to fixed admittance | +| FixedAdmittance | 3,110 | ~3,100 | Switched shunts mapped to fixed admittance | | Area | 44 | 49 | 5 areas only on removed islands | | LoadZone | 74 | 90 | Zones on removed islands excluded | @@ -100,10 +100,10 @@ pre-cleaned main island (27,862 buses), not the full 30,307-bus FNM. | Check | Result | Detail | |-------|--------|--------| | baseMVA | 100.0 | Correct (matches manifest sbase) | -| Slack bus present | Yes | Bus 29421 (bus_type=REF) | +| Slack bus present | Yes | Bus (bus_type=REF) | | Tap ratio preservation | OK | 0 branches with tap=0 | -| Bus count (internal) | 27,862 | Consistent with cleaned .m file | -| Branch count (internal) | 32,606 | Consistent with cleaned .m file | +| Bus count (internal) | ~28,000 | Consistent with cleaned .m file | +| Branch count (internal) | ~33,000 | Consistent with cleaned .m file | **Component type mapping notes:** - PowerSystems.jl splits MATPOWER `branch` rows into `Line`, `Transformer2W`, and @@ -124,7 +124,7 @@ pre-cleaned main island (27,862 buses), not the full 30,307-bus FNM. custom CSV-to-PowerSystems.jl converter, (2) upstream patch to the PTI parser's fixed-width tokenizer, or (3) external pre-conversion to MATPOWER format. - **Grade impact:** G-FNM-1 fails. G-FNM-2 (field coverage) is blocked. G-FNM-3/4/5 - proceed via MATPOWER fallback path with reduced bus count (27,862 vs 30,307). + proceed via MATPOWER fallback path with reduced bus count (~28,000 vs ~30,000). ## Timing @@ -148,5 +148,5 @@ Key findings: # MATPOWER fallback succeeds sys = PowerSystems.System("/workspace/data/fnm/reference/cleaned/fnm_main_island.m"; runchecks=false) -# => 27,862 buses, 5,741 generators, 32,606 branches +# => ~28,000 buses, ~5,700 generators, ~33,000 branches ``` diff --git a/evaluations/powersimulations/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md b/evaluations/powersimulations/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md index e6fa0cf9..d394622b 100644 --- a/evaluations/powersimulations/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/powersimulations/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md @@ -36,7 +36,7 @@ that incorporates taps. ## Approach -1. Loaded pre-cleaned MATPOWER case `fnm_main_island.m` (27,862-bus main island) via +1. Loaded pre-cleaned MATPOWER case `fnm_main_island.m` (~28,000-bus main island) via `PowerSystems.System(path; runchecks=false)`. This is the MATPOWER fallback path because G-FNM-1 failed (PowerSystems.jl has no PSS/E CSV parser). 2. Solved DCPF using `PowerFlows.solve_powerflow(PowerFlows.DCPowerFlow(), sys)`. @@ -66,7 +66,7 @@ applying `rad2deg` to angles and multiplying flows by `baseMVA`. | Metric | Value | |--------|-------| -| Non-excluded buses compared | 27,862 | +| Non-excluded buses compared | ~28,000 | | Passing (< 1.0 deg) | 3,668 (13.16%) | | Failing | 24,194 (86.84%) | | Mean VA deviation | 2.658984e+00 deg | @@ -80,7 +80,7 @@ applying `rad2deg` to angles and multiplying flows by `baseMVA`. | Metric | Value | |--------|-------| -| In-service branches (rows) | 32,532 | +| In-service branches (rows) | ~33,000 | | Unique (from,to) pairs | 30,912 | | Matched | 30,912 (100%) | | Passing (< 10%) | 29,835 (96.52%) | @@ -121,7 +121,7 @@ Evidence: off-nominal taps. The extreme 700.4% deviation at (14333, 13343) is on a small-flow branch (ref ~3.22 MW) where the angle difference produces a disproportionate percentage error. -- The non-trivial solution check confirms the DCPF solve was valid: 27,858 of 27,862 +- The non-trivial solution check confirms the DCPF solve was valid: 27,858 of ~28,000 buses have nonzero angles. This is classified as `formulation-difference` [tool-specific: PowerFlows.jl simplified diff --git a/evaluations/powersimulations/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md b/evaluations/powersimulations/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md index 17dfe1de..315e8709 100644 --- a/evaluations/powersimulations/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/powersimulations/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md @@ -32,12 +32,12 @@ acpf_timeout_minutes: 30 ## Result: INFORMATIONAL The Newton-Raphson AC power flow solver in PowerFlows.jl failed to converge at all -three relaxation levels (0%, 10%, 20%) on the 27,862-bus FNM main island network. +three relaxation levels (0%, 10%, 20%) on the ~28,000-bus FNM main island network. Relaxation level achieved: **infeasible**. ## Approach -1. **DCPF warm-start:** Loaded `fnm_main_island.m` (MATPOWER fallback, 27,862 buses) via +1. **DCPF warm-start:** Loaded `fnm_main_island.m` (MATPOWER fallback, ~28,000 buses) via `PowerSystems.System(path; runchecks=false)`. Solved DCPF via `PowerFlows.solve_powerflow(DCPowerFlow(), sys)`. Extracted bus voltage angles from the DCPF solution. @@ -58,7 +58,7 @@ Relaxation level achieved: **infeasible**. | DCPF solve time | 10.51s | | Mean \|angle\| (non-zero buses) | 2.120131e+02 deg | | Max \|angle\| | 5.402463e+02 deg | -| Non-zero angles | 27,858 / 27,862 | +| Non-zero angles | 27,858 / ~28,000 | The large angle magnitudes (mean 212 deg) are expected for a large interconnected network where the slack bus reference sets the zero point and other buses span @@ -105,9 +105,9 @@ at the `@error` log level. ## Analysis -The ACPF non-convergence on this 27,862-bus network is attributable to several factors: +The ACPF non-convergence on this ~28,000-bus network is attributable to several factors: -1. **Network scale:** 27,862 buses is at the upper end of what open-source NR solvers +1. **Network scale:** ~28,000 buses is at the upper end of what open-source NR solvers reliably handle without specialized initialization and tuning. Commercial tools (PSS/E, PowerWorld) use multi-level initialization strategies, optimal multiplier selection, and bus-ordering heuristics not available in PowerFlows.jl. @@ -163,7 +163,7 @@ These are left as future diagnostic paths since G-FNM-4 has no hard gate consequ ### `fnm-scale` -- ACPF non-convergence on 28K-bus network -PowerFlows.jl's Newton-Raphson ACPF solver cannot converge on the 27,862-bus FNM +PowerFlows.jl's Newton-Raphson ACPF solver cannot converge on the ~28,000-bus FNM main island network even with DCPF warm-start and 20% branch rating relaxation. This is consistent with the expected behavior for large-scale AC power flow in open-source tools without specialized initialization heuristics. [tool-specific] diff --git a/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md b/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md index 91b12cc0..0ba368ad 100644 --- a/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md +++ b/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md @@ -21,8 +21,8 @@ information and fuel type classification. G-FNM-1 sub-check (a) failed because PowerSystems.jl has no parser for PSS/E-derived intermediate CSV tables and its PSS/E RAW v31 parser fails on the Case Identification -header. The MATPOWER fallback loaded 27,862 buses (vs 30,307 in the full FNM) with -32,606 branches. The MATPOWER format merges branches and transformers into a single +header. The MATPOWER fallback loaded ~28,000 buses (vs ~30,000 in the full FNM) with +~33,000 branches. The MATPOWER format merges branches and transformers into a single table, though PowerSystems.jl re-separates them based on tap ratio heuristics. ## Implications diff --git a/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index 41e780dd..fe260dc2 100644 --- a/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -19,7 +19,7 @@ analysis using the DC path. ## Context -Discovered during G-FNM-3 DCPF verification on the 27,862-bus FNM main island. Without +Discovered during G-FNM-3 DCPF verification on the ~28,000-bus FNM main island. Without the correction, branch flows would be 100x too small (per-unit vs MW) and angles would be in radians instead of degrees. The initial comparison showed nonsensical deviations until the unit mismatch was identified empirically by comparing magnitudes against the diff --git a/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index 9260c798..cda80ea0 100644 --- a/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/powersimulations/results/observations/fnm-data-model-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -20,7 +20,7 @@ quality to `binary_convergence_api` (tier 3 of 4). ## Context -Discovered during G-FNM-4 ACPF convergence testing on the 27,862-bus FNM main island. +Discovered during G-FNM-4 ACPF convergence testing on the ~28,000-bus FNM main island. All three relaxation levels (0%, 10%, 20%) produced non-convergence with no diagnostic detail beyond the boolean flag. The internal `_run_powerflow_method` function computes iteration count and logs it on success (as confirmed in G-FNM-1 observations and diff --git a/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md b/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md index d20dfe12..5bb6df66 100644 --- a/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md +++ b/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-1_fnm_ingestion_gate.md @@ -7,26 +7,26 @@ severity: low timestamp: "2026-03-24T00:00:00Z" --- -# Observation: MATPOWER fallback loads 27,862-bus main island in 12s +# Observation: MATPOWER fallback loads ~28,000-bus main island in 12s ## Finding -PowerSystems.jl loaded the 27,862-bus MATPOWER fallback file in 12.03 seconds with +PowerSystems.jl loaded the ~28,000-bus MATPOWER fallback file in 12.03 seconds with 847 MB peak RSS. This demonstrates the tool can handle LARGE-tier networks via MATPOWER ingestion, though 2,445 isolated buses (8.1% of total) are excluded from the cleaned fallback file. ## Context -The full FNM has 30,307 buses, 33,840 branches (24,117 lines + 9,723 transformers), -5,768 generators, and 15,062 loads. The MATPOWER fallback contains only the main -connected island: 27,862 buses, 32,606 branches, 5,741 generators, 11,734 loads. +The full FNM has ~30,000 buses, ~34,000 branches (~24,000 lines + ~9,700 transformers), +~5,800 generators, and ~15,000 loads. The MATPOWER fallback contains only the main +connected island: ~28,000 buses, ~33,000 branches, ~5,700 generators, ~12,000 loads. Load time of 12s includes Julia's PowerModels-derived MATPOWER parser and System construction overhead. ## Implications -For scalability assessment, the 12-second load time for a 27,862-bus network is +For scalability assessment, the 12-second load time for a ~28,000-bus network is acceptable for G-FNM-3/4/5 downstream tests. The missing 2,445 buses are isolated (IDE=4) and disconnected island fragments that do not participate in power flow, so their absence does not affect DCPF/ACPF verification accuracy. diff --git a/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index f5d728ac..88ca8b87 100644 --- a/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -7,13 +7,13 @@ severity: low timestamp: "2026-03-24T18:30:00Z" --- -# Observation: PowerFlows.jl DCPF successfully handles 27,862-bus network +# Observation: PowerFlows.jl DCPF successfully handles ~28,000-bus network ## Finding -PowerFlows.jl DCPowerFlow solve completed in 10.70 seconds on the 27,862-bus FNM main -island network with 32,532 branches. Peak memory was 1,139.8 MB. The solve produced a -non-trivial solution (27,858 of 27,862 buses with nonzero angles). Network loading via +PowerFlows.jl DCPowerFlow solve completed in 10.70 seconds on the ~28,000-bus FNM main +island network with ~33,000 branches. Peak memory was 1,139.8 MB. The solve produced a +non-trivial solution (27,858 of ~28,000 buses with nonzero angles). Network loading via PowerSystems.System took 38.11 seconds (includes JIT compilation overhead on cold start). ## Context diff --git a/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index 06ab5f61..34f685c7 100644 --- a/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/powersimulations/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -11,7 +11,7 @@ timestamp: "2026-03-24T22:00:00Z" ## Finding -PowerFlows.jl's Newton-Raphson ACPF solver cannot converge on the 27,862-bus FNM +PowerFlows.jl's Newton-Raphson ACPF solver cannot converge on the ~28,000-bus FNM main island network with DCPF warm-start initialization and progressive branch rating relaxation (0%, 10%, 20%). The solver ran 100 iterations at each level without achieving convergence. Total ACPF wall-clock time across all three attempts was diff --git a/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md b/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md index aa3ae373..413e86d9 100644 --- a/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md +++ b/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_fnm_dcpf_verification.md @@ -13,7 +13,7 @@ timestamp: "2026-03-24T18:30:00Z" PowerFlows.jl v0.9.0 DCPowerFlow uses a simplified B-matrix (`b = -1/x`) via PowerNetworkMatrices.jl that ignores transformer tap ratios and phase shift angles. On -the 27,862-bus FNM with ~2,340 off-nominal tap transformers, this produces systematic +the ~28,000-bus FNM with ~2,340 off-nominal tap transformers, this produces systematic angle deviations (mean 2.66 degrees, max 35.88 degrees) compared to MATPOWER's full B-matrix reference. Branch flows are less affected (96.52% within 10% tolerance) because flow errors depend on relative angle differences across each branch, which partially diff --git a/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md b/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md index 6da33cc4..b2810cde 100644 --- a/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md +++ b/evaluations/powersimulations/results/observations/formulation-difference-fnm_ingestion-G-FNM-4_fnm_acpf_convergence.md @@ -15,7 +15,7 @@ The DCPF warm-start angles used for ACPF initialization in G-FNM-4 were computed using PowerFlows.jl's simplified B-matrix (b = -1/x, ignoring transformer tap ratios). As documented in the G-FNM-3 formulation-difference observation, this produces mean angle deviations of approximately 2.7 degrees and maximum deviations of ~36 degrees -versus a full B-matrix reference on the 27,862-bus FNM network with 2,340 off-nominal- +versus a full B-matrix reference on the ~28,000-bus FNM network with 2,340 off-nominal- tap transformers. ## Context diff --git a/evaluations/powersimulations/results/synthesis.md b/evaluations/powersimulations/results/synthesis.md index 7ecfe12d..f7d09ba7 100644 --- a/evaluations/powersimulations/results/synthesis.md +++ b/evaluations/powersimulations/results/synthesis.md @@ -240,7 +240,7 @@ Grade: **B+**. The core license is clean, all compiled components are source-ava ### Data Model Fidelity -G-FNM-1 **failed**: PowerSystems.jl v4.6.2 cannot parse PSS/E RAW v31 files -- the PTI parser lacks fixed-width column support for pre-v33 formats. The parser fails at line 1 of the CASE IDENTIFICATION section. G-FNM-2 is **blocked** by G-FNM-1 (skip). The MATPOWER fallback path loaded the 27,862-bus main island successfully with correct component type differentiation (Line vs Transformer2W vs TapTransformer). 100% DCPF-critical coverage was **not assessed** via the native PSS/E path. +G-FNM-1 **failed**: PowerSystems.jl v4.6.2 cannot parse PSS/E RAW v31 files -- the PTI parser lacks fixed-width column support for pre-v33 formats. The parser fails at line 1 of the CASE IDENTIFICATION section. G-FNM-2 is **blocked** by G-FNM-1 (skip). The MATPOWER fallback path loaded the ~28,000-bus main island successfully with correct component type differentiation (Line vs Transformer2W vs TapTransformer). 100% DCPF-critical coverage was **not assessed** via the native PSS/E path. **Impact on Expressiveness:** The PSS/E v31 parse failure is additive negative evidence. It weakens the tool's data ingestion capability but does not change the Expressiveness grade boundary (the A-10/A-11 failures are more determinative). @@ -248,7 +248,7 @@ G-FNM-1 **failed**: PowerSystems.jl v4.6.2 cannot parse PSS/E RAW v31 files -- t G-FNM-3 **failed**: Bus angle gate fails (13.2% passing vs 95% required). Branch flow gate passes (96.5% passing vs 90% required). The failure is attributable to a formulation difference: PowerFlows.jl uses a simplified B-matrix (`b = -1/x`) that ignores transformer tap ratios, while the MATPOWER reference uses the full B-matrix. With 2,340 off-nominal tap transformers in the network, this produces systematic angle deviations (mean 2.66 deg). This is a `formulation-difference`, not a tool bug, but it represents a fidelity gap on real-world networks. -G-FNM-4 is **informational**: ACPF failed to converge at all three relaxation levels (0%, 10%, 20%) on the 27,862-bus FNM. This is consistent with expected behavior for large-scale NR without specialized initialization heuristics. The solver does not expose convergence residual, limiting root-cause analysis. +G-FNM-4 is **informational**: ACPF failed to converge at all three relaxation levels (0%, 10%, 20%) on the ~28,000-bus FNM. This is consistent with expected behavior for large-scale NR without specialized initialization heuristics. The solver does not expose convergence residual, limiting root-cause analysis. **Impact on Expressiveness:** The simplified B-matrix formulation difference weakens DCPF fidelity on networks with off-nominal tap transformers. This is moderate negative evidence. diff --git a/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_1_fnm_ingestion_gate.jl b/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_1_fnm_ingestion_gate.jl index 94411047..974329ae 100644 --- a/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_1_fnm_ingestion_gate.jl +++ b/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_1_fnm_ingestion_gate.jl @@ -30,7 +30,7 @@ end function run_test(; intermediate_dir::String="/workspace/data/fnm/intermediate", - raw_file::String="/data/fnm-source/AUC_AN_2026_2026_S01_ON_NETWORK_MODEL.RAW", + raw_file::String="", matpower_fallback::String="/workspace/data/fnm/reference/cleaned/fnm_main_island.m", ) results = Dict( @@ -45,17 +45,18 @@ function run_test(; logger = ConsoleLogger(stderr, Logging.Error) global_logger(logger) - # --- Expected counts from intermediate_manifest.json --- - manifest = Dict( - "bus" => 30307, - "load" => 15062, - "fixed_shunt" => 0, - "generator" => 5768, - "branch" => 24117, - "transformer" => 9723, - "area" => 49, - "zone" => 90, - "switched_shunt" => 3114, + # --- Expected count ranges from intermediate_manifest.json --- + # Exact counts redacted (NDA); range checks verify order-of-magnitude fidelity. + manifest_ranges = Dict( + "bus" => (25000, 35000), + "load" => (12000, 18000), + "fixed_shunt" => (0, 10), + "generator" => (4500, 7000), + "branch" => (20000, 28000), + "transformer" => (8000, 12000), + "area" => (30, 70), + "zone" => (60, 120), + "switched_shunt" => (2500, 4000), ) t0 = time() @@ -206,7 +207,7 @@ function run_test(; "counts" => matpower_counts, "base_power_mva" => base_power, "slack_bus_numbers" => slack_numbers, - "note" => "Pre-cleaned main island (27,862 buses), not full 30,307-bus FNM.", + "note" => "Pre-cleaned main island (28000 buses), not full 30000-bus FNM.", ) results["details"]["peak_rss_mb"] = peak_rss_mb() diff --git a/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_3_fnm_dcpf_verification.jl b/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_3_fnm_dcpf_verification.jl index 7ba4b7e1..982c2d2c 100644 --- a/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_3_fnm_dcpf_verification.jl +++ b/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_3_fnm_dcpf_verification.jl @@ -2,7 +2,7 @@ Test G-FNM-3: DCPF Verification Against Reference Solution Dimension: fnm_ingestion -Network: LARGE (FNM main island via MATPOWER fallback, 27,862 buses) +Network: LARGE (FNM main island via MATPOWER fallback, 28000 buses) Pass condition: All aggregate thresholds met per pass_conditions.json dcpf section. - Bus angle: >=95% of non-excluded buses within 1.0 deg - Branch flow: >=90% of in-service branches within 10% (floor 1.0 MW) diff --git a/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_4_fnm_acpf_convergence.jl b/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_4_fnm_acpf_convergence.jl index 26eb96b5..91c290f8 100644 --- a/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_4_fnm_acpf_convergence.jl +++ b/evaluations/powersimulations/tests/fnm_ingestion/test_g_fnm_4_fnm_acpf_convergence.jl @@ -2,7 +2,7 @@ Test G-FNM-4: ACPF Convergence (DCPF warm-start + progressive relaxation) Dimension: fnm_ingestion -Network: LARGE (FNM main island via MATPOWER fallback, 27,862 buses) +Network: LARGE (FNM main island via MATPOWER fallback, 28000 buses) Pass condition: No hard gate. All outcomes are diagnostic. Record relaxation_level_achieved: 0%, 10%, 20%, or infeasible. Tool: PowerSimulations.jl v0.30.2 (PowerSystems.jl v4.6.2, PowerFlows.jl v0.9.0) diff --git a/evaluations/pypsa/results/extensibility/b5_v_ang_export.csv b/evaluations/pypsa/results/extensibility/b5_v_ang_export.csv new file mode 100644 index 00000000..93a88bd0 --- /dev/null +++ b/evaluations/pypsa/results/extensibility/b5_v_ang_export.csv @@ -0,0 +1,2 @@ +snapshot,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 +now,-0.2147517671298287,-0.1414483855736976,-0.19179632777049,-0.20332290202898223,-0.18057912211390953,-0.1671955249999999,-0.2084555303500416,-0.21833073302506245,-0.22912863033680075,-0.12480407237777105,-0.13946295655784968,-0.14064547902777105,-0.13809518819769243,-0.1687252507262636,-0.1763352451050041,-0.14955174036584098,-0.1696629901340803,-0.18611918402331565,-0.05985174036584094,-0.08501190036584091,-0.1043570115196871,-0.019128403827379403,-0.02308592998122556,-0.14688941075045636,-0.11893497815558088,-0.1364468084340495,-0.17403708877750115,-0.06754370363404953,-0.01448755843404953,-0.0950671355736976,0.0,0.014295927622228949,0.036174339634159086,0.007251059634159104,0.0761453461726206,0.12923407001877446,0.009477021844419092,0.11822944156595042,-0.23494019873331473 diff --git a/evaluations/pypsa/results/fnm_ingestion/G-FNM-1_intermediate_format_ingestion.md b/evaluations/pypsa/results/fnm_ingestion/G-FNM-1_intermediate_format_ingestion.md index e1a69a5d..d7fe34d5 100644 --- a/evaluations/pypsa/results/fnm_ingestion/G-FNM-1_intermediate_format_ingestion.md +++ b/evaluations/pypsa/results/fnm_ingestion/G-FNM-1_intermediate_format_ingestion.md @@ -68,14 +68,14 @@ For reference, the intermediate manifest expects: | Table | Expected Records | |-------|-----------------| -| bus | 30,307 | -| load | 15,062 | -| generator | 5,768 | -| branch | 24,117 | -| transformer | 9,723 | +| bus | ~30,000 | +| load | ~15,000 | +| generator | ~5,800 | +| branch | ~24,000 | +| transformer | ~9,700 | | area | 49 | | zone | 90 | -| switched_shunt | 3,114 | +| switched_shunt | ~3,100 | ## Approach diff --git a/evaluations/pypsa/results/fnm_ingestion/G-FNM-3_dcpf_verification.md b/evaluations/pypsa/results/fnm_ingestion/G-FNM-3_dcpf_verification.md index f22c72f0..5af1072a 100644 --- a/evaluations/pypsa/results/fnm_ingestion/G-FNM-3_dcpf_verification.md +++ b/evaluations/pypsa/results/fnm_ingestion/G-FNM-3_dcpf_verification.md @@ -28,11 +28,11 @@ timestamp: 2026-03-24T00:00:00Z 100% of buses and 100% of branches pass all tolerance thresholds. All deviations are at float64 machine-precision level. No hard-fail conditions triggered. Bus -injection power balance check passes on all 27,862 buses. +injection power balance check passes on all ~28,000 buses. ## Approach -Loaded the pre-cleaned 27,862-bus FNM main island from +Loaded the pre-cleaned ~28,000-bus FNM main island from `data/fnm/reference/cleaned/fnm_main_island.m` via the shared `matpower_loader.load_pypsa()` utility (MATPOWER fallback, since G-FNM-1 failed with `psse_parse_error`). The loader applies three correctness patches: @@ -53,9 +53,9 @@ balance verification using post-solve generator dispatch values. | Metric | Value | |--------|-------| -| Non-excluded buses compared | 27,862 | +| Non-excluded buses compared | ~28,000 | | Tolerance | 1.0 deg | -| Buses passing | 27,862 (100.0%) | +| Buses passing | ~28,000 (100.0%) | | Required passing fraction | 95% | | Mean deviation | 3.316294e-09 deg | | Max deviation | 1.073352e-08 deg | @@ -75,9 +75,9 @@ balance verification using post-solve generator dispatch values. | Metric | Value | |--------|-------| -| Branches compared | 32,532 | +| Branches compared | ~33,000 | | Tolerance | 10% (floor 1 MW) | -| Branches passing | 32,532 (100.0%) | +| Branches passing | ~33,000 (100.0%) | | Required passing fraction | 90% | | Mean deviation | 1.175350e-08 % | | Max deviation (pct) | 5.757744e-07 % | @@ -95,7 +95,7 @@ balance verification using post-solve generator dispatch values. ### Transformer Tap Analysis -- Total transformers: 9,481 +- Total transformers: ~9,500 - Tap = 1.0: 7,123 - Tap != 1.0: 2,358 - Tap range: [0.7894, 1.4165] @@ -108,9 +108,9 @@ numerical noise only). | Metric | Value | |--------|-------| -| Buses checked | 27,862 | +| Buses checked | ~28,000 | | Tolerance | 1.000000e-03 MW | -| Buses balanced | 27,862 (100.0%) | +| Buses balanced | ~28,000 (100.0%) | | Buses imbalanced | 0 | | Max mismatch | 1.317039e-07 MW | | Mean mismatch | 6.748134e-11 MW | @@ -123,10 +123,10 @@ absorption of the 9,981 MW generation-load deficit). | Metric | PyPSA (pre-solve) | Reference | |--------|-------------------|-----------| -| Total generation | 155,511 MW | 165,492 MW | -| Total load | 165,492 MW | 165,492 MW | +| Total generation | ~156,000 MW | ~165,000 MW | +| Total load | ~165,000 MW | ~165,000 MW | -The 9,981 MW generation-load imbalance (6.0%) is absorbed at the slack bus (bus 29421). +The 9,981 MW generation-load imbalance (6.0%) is absorbed at the slack bus (bus ). ### Hard-Fail Conditions diff --git a/evaluations/pypsa/results/fnm_ingestion/G-FNM-4_acpf_convergence.md b/evaluations/pypsa/results/fnm_ingestion/G-FNM-4_acpf_convergence.md index af584862..41e26a63 100644 --- a/evaluations/pypsa/results/fnm_ingestion/G-FNM-4_acpf_convergence.md +++ b/evaluations/pypsa/results/fnm_ingestion/G-FNM-4_acpf_convergence.md @@ -30,7 +30,7 @@ timestamp: 2026-03-24T00:00:00Z ## Result: INFORMATIONAL PyPSA's Newton-Raphson ACPF solver did not converge at any relaxation level -(0%, 10%, 20%) on the 27,862-bus FNM main island. This is consistent with +(0%, 10%, 20%) on the ~28,000-bus FNM main island. This is consistent with MATPOWER 8.1's failure on the same network. The FNM is a planning model with no solved voltage profile. @@ -116,10 +116,10 @@ on ill-conditioned admittance matrix] | Metric | Value | |--------|-------| -| Buses | 27,862 | +| Buses | ~28,000 | | Lines | 23,125 (69 inactive) | | Transformers | 9,481 (5 inactive) | -| Generators | 5,741 | +| Generators | ~5,700 | | Loads | 8,624 | | baseMVA | 100.0 | diff --git a/evaluations/pypsa/results/latency_bench_report.md b/evaluations/pypsa/results/latency_bench_report.md new file mode 100644 index 00000000..41955c5d --- /dev/null +++ b/evaluations/pypsa/results/latency_bench_report.md @@ -0,0 +1,297 @@ +# PyPSA/Linopy Latency Benchmark Report + +**Purpose**: Determine whether PyPSA/Linopy is fast enough for a web-app showing +interactive contingency sweeps on target ISO-scale (~10k bus) networks. + +**Environment**: devcontainer (Python 3.12, PyPSA 1.1.2, HiGHS solver, single thread) + +--- + +## Executive Summary + +BODF pre-compute + numpy matrix multiply is the viable architecture for interactive +contingency analysis. For the target workflow — user clicks a bus, system sweeps +N-m contingencies out to h hops — latency depends heavily on m (outage depth) and +the combinatorial expansion of scoped branches: + +- **N-1 + N-2 at h<=3**: sub-200 ms on typical buses at all network sizes, including 10k +- **N-3**: viable at h<=2 on typical buses; hits seconds at hub nodes or h>=3 +- **Single N-1 contingency**: ~3-9 us — effectively instant + +The re-solve path (rebuilding and solving the LP per contingency) is 4-5 orders of +magnitude slower and unsuitable for interactive use at any scale above 39 buses. + +PyPSA is sufficient for this architecture. Julia is not needed for the contingency +sweep itself, though it may offer faster initial BODF pre-computation if startup +latency matters. + +--- + +## B1: Baseline Solve Time Decomposition (10k bus) + +| Metric | DCPF (`n.lpf()`) | DCOPF (`n.optimize()`) | +|--------|:-----------------:|:----------------------:| +| Median | 19.98s | 208.3s | +| Min | 19.85s | 206.3s | +| Max | 20.24s | 258.9s | +| Peak Memory | 2,099 MB | 4,403 MB | + +**Finding**: Neither DCPF nor DCOPF is interactive on a 10k-bus network. +DCOPF is ~10x slower than DCPF due to Linopy model construction + LP solve overhead. + +--- + +## B2: Linopy Build vs Solve Isolation + +The key question: is the bottleneck in Linopy symbolic construction or the solver? + +| Network | Buses | `create_model()` | `solve_model()` | Build % | +|---------|------:|------------------:|-----------------:|--------:| +| case39 | 39 | 0.76s | 0.25s | **75%** | +| case2000 | 2,000 | 4.58s | 1.33s | **77%** | +| case10000 | 10,000 | 21.84s | 187.58s | **10%** | + +**Finding**: At small/medium scale, Linopy model construction dominates (75-77%). +At 10k scale, the solver itself dominates (90%). The `io_api="direct"` option +(skipping LP file I/O) provides negligible improvement — the solver computation +itself is the bottleneck at scale. + +**Implication**: Neither skipping model construction nor switching solvers will make +DCOPF interactive at 10k scale. The architecture must avoid per-interaction LP solves. + +--- + +## B3: Incremental Re-solve (2000 bus, 5 outages) + +Three strategies for handling line outages in DCOPF: + +| Strategy | Per-Outage Time | Speedup vs A | +|----------|----------------:|-------------:| +| A: Full rebuild (`n.optimize()` from scratch) | 5.73s | 1.0x | +| B: Re-solve only (`solve_model()` reuse) | 1.28s | **4.5x** | +| C: Warm-start (`warm_start=True`) | 5.80s | 1.0x | + +**Finding**: Reusing the Linopy model and calling `solve_model()` without rebuilding +saves ~78% of per-outage time. However, 1.28s per outage is still too slow for +interactive use on larger networks. Warm-start provides no measurable benefit — +HiGHS appears to ignore the `warm_start` option for LP problems, or the basis +from the previous solve is not being passed through Linopy's abstraction layer. + +**Note**: Strategy B does not actually update constraint bounds (Linopy constraints +are immutable), so it measures the lower bound of re-solve time. True incremental +re-solve with modified constraints would require either rebuilding the model or +direct solver API access. + +--- + +## B4: Scaling Curve (Bus Count vs Solve Time) + +| Network | Buses | DCPF | DCOPF | Interactive? | +|---------|------:|-----:|------:|:------------:| +| case39 | 39 | 0.18s | 1.01s | DCPF only | +| case2000 | 2,000 | 3.95s | 5.89s | No | +| case10000 | 10,000 | 20.09s | 206.6s | No | + +**Clustering not available**: PyPSA's spatial clustering (`busmap_by_kmeans`, +`busmap_by_greedy_modularity`) requires networks built with PyPSA's native +conventions. The MATPOWER→PyPSA bridge places Pd/Qd on the bus frame and sets +all coordinates to (0,0), which breaks the clustering aggregation (`consense` +fails on non-uniform bus attributes). Intermediate sizes would require either +native PyPSA network construction or manual bus-merging logic. + +**Extrapolation**: Based on the scaling trend, DCPF crosses the 1s threshold +at approximately 100-200 buses. DCOPF would require fewer than 39 buses. +This confirms that LP-based approaches cannot serve interactive contingency +analysis at target ISO scale. + +--- + +## B5: Contingency Sweep Throughput (BODF) + +### BODF Pre-computation + +| Network | Buses | Branches | BODF Time | Memory | +|---------|------:|---------:|----------:|-------:| +| case39 | 39 | 46 | 0.13s | 0.2 MB | +| case2000 | 2,000 | 3,206 | 1.32s | 249 MB | +| case10000 | 10,000 | 12,706 | 17.1s | 5,936 MB | + +### All-N-1 Vectorized Analysis + +| Network | Branches | All-N-1 Time | Single Contingency | +|---------|------:|---------:|---------:| +| case39 | 46 | 10 us | < 1 us | +| case2000 | 3,206 | 19.4 ms | **2.9 us** | +| case10000 | 12,706 | 235.8 ms | **8.8 us** | + +### Comparison: BODF vs Re-solve + +On the 39-bus network (the only scale where re-solve is tractable): + +| Method | Total N-1 Time | Speedup | +|--------|---------------:|--------:| +| Re-solve (n.lpf per contingency) | 1.99s (54 ms/each) | 1x | +| BODF vectorized | 10 us (all at once) | **200,669x** | + +### Violation Detection + +Checking all branches for thermal violations across all N-1 contingencies on the +2000-bus network takes **14 ms** — fully interactive. + +### N-2 Composition (Bonus) + +First-order superposition for double outages on the 2000-bus network: +- 100 random N-2 pairs: **6.5 us per pair** +- This extends to full N-2 screening at interactive speeds + +--- + +## B6: Interactive N-m Sweep from Focal Bus + +This benchmark simulates the exact user workflow: click a bus on the map, BFS out +to h hops to find scoped branches, enumerate all N-1/N-2/N-3 contingency +combinations, compute post-contingency flows via BODF, and detect violations. + +Two bus types are tested: the **highest-degree hub** (worst case — degree 17-20) +and a **median-degree bus** (typical case — degree 2-3). + +### Combinatorial Expansion + +The number of scoped branches and resulting combinations drives everything: + +| Network | Bus Type | h=1 | h=2 | h=3 | h=4 | +|---------|----------|----:|----:|----:|----:| +| 10k | hub (deg 20) | 44 branches | 100 | 170 | 251 | +| 10k | typical (deg 2) | 7 branches | 15 | 27 | 47 | +| 2k | hub (deg 17) | 30 branches | 57 | 116 | 250 | +| 2k | typical (deg 3) | 4 branches | 7 | 10 | 17 | + +N-m combinations from k scoped branches: N-1 = k, N-2 = k(k-1)/2, N-3 = k(k-1)(k-2)/6. +At k=100 (10k hub, h=2): N-1 = 100, N-2 = 4,950, **N-3 = 161,700**. + +### Latency Decomposition + +Total = BFS scope + N-1 sweep + N-2 sweep + N-3 sweep. The BFS graph traversal +on NetworkX is a **constant overhead per network size** regardless of h or branch +count — this dominates at small scopes: + +| Network | BFS Scope (constant) | +|---------|---------------------:| +| 39-bus | ~2 ms | +| 2,000-bus | ~24 ms | +| 10,000-bus | ~90 ms | + +This is an implementation artifact (Python NetworkX), not fundamental — a compiled +graph library or pre-computed adjacency would reduce it to microseconds. + +### Typical Bus (degree 2-3): Total User-Perceived Latency + +| Network | h | Branches | Scope | N-1 | N-2 | N-3 | **Total** | +|---------|--:|---------:|------:|----:|----:|----:|----------:| +| 39-bus | 2 | 7 | 2 ms | 0.1 ms | 0.0 ms | 0.0 ms | **2 ms** | +| 39-bus | 4 | 18 | 2 ms | 0.1 ms | 0.1 ms | 0.5 ms | **3 ms** | +| 2,000-bus | 2 | 7 | 24 ms | 0.1 ms | 0.2 ms | 0.3 ms | **24 ms** | +| 2,000-bus | 4 | 17 | 23 ms | 0.2 ms | 1.1 ms | 12 ms | **36 ms** | +| 10,000-bus | 2 | 15 | 88 ms | 0.4 ms | 4 ms | 39 ms | **133 ms** | +| 10,000-bus | 3 | 27 | 91 ms | 0.7 ms | 25 ms | 261 ms | **377 ms** | +| 10,000-bus | 4 | 47 | 89 ms | 1.2 ms | 72 ms | 1,403 ms | **1.6s** | + +### Hub Bus (worst case): Total User-Perceived Latency + +| Network | h | Branches | Scope | N-1 | N-2 | N-3 | **Total** | +|---------|--:|---------:|------:|----:|----:|----:|----------:| +| 39-bus | 4 | 29 | 2 ms | 0.1 ms | 0.1 ms | 2.1 ms | **5 ms** | +| 2,000-bus | 1 | 30 | 24 ms | 0.3 ms | 5 ms | 106 ms | **136 ms** | +| 2,000-bus | 2 | 57 | 25 ms | 0.5 ms | 33 ms | 762 ms | **820 ms** | +| 2,000-bus | 3 | 116 | 28 ms | 1.0 ms | 134 ms | 1,240 ms | **1.4s** | +| 10,000-bus | 1 | 44 | 88 ms | 1.1 ms | 64 ms | 1,149 ms | **1.3s** | +| 10,000-bus | 2 | 100 | 91 ms | 4.5 ms | 323 ms | 4,383 ms | **4.8s** | +| 10,000-bus | 3 | 170 | 88 ms | 6.8 ms | 934 ms | skipped | **1.0s*** | +| 10,000-bus | 4 | 251 | 92 ms | 10.5 ms | 2,055 ms | skipped | **2.2s*** | + +*N-3 skipped (>500k combinations); total reflects scope+N-1+N-2 only. + +### What This Means + +**BFS scope is the floor**: ~90 ms on 10k due to Python NetworkX graph traversal. +This is constant regardless of h or m. A compiled adjacency lookup or pre-computed +hop table would eliminate this. + +**N-1 is always instant**: <11 ms even at 10k-bus/h=4 with 251 branches. + +**N-2 stays interactive for typical buses**: <100 ms compute up to h=4 on 10k. At +hub nodes it crosses 1s around h=3 on 10k (14k combinations × 12k branches each). + +**N-3 is the bottleneck**: O(k^3) combinatorics dominate. At 50+ scoped branches +it hits seconds; at 100+ it's multi-second. This is the cost of computing flows +for every triple, not the per-combination BODF math (which is ~10-20 us). + +### Strategies for N-3 at Scale + +1. **Progressive rendering**: Show N-1+N-2 results instantly (<200 ms), compute + N-3 in background, stream results as they arrive. +2. **Flow-based pruning**: Pre-filter scoped branches to exclude zero/low-flow + lines (reduces k by 20-40% based on B5 pruning ratios, cubic reduction in combos). +3. **Severity screening**: Run N-2 first, only expand to N-3 around branches that + showed N-2 violations (targeted rather than exhaustive). +4. **Hop budget**: Cap at h=2 for N-3 on 10k networks (keeps k<30 on typical buses). + +--- + +## Architecture Recommendation + +### For Interactive Contingency Web-App + +``` +Startup (one-time, per network load): + 1. Load network → PyPSA ~3-20s + 2. Run base DCPF (n.lpf) ~0.2-20s + 3. Compute BODF matrix ~0.1-17s + 4. Store BODF + base flows in memory ~250 MB - 6 GB + +Per user interaction (click a bus on the map): + 5. BFS to h hops → find scoped branches ~2-90 ms (NetworkX; reducible) + 6. Vectorized N-1 sweep (all at once) 0.1-11 ms + 7. Vectorized N-2 sweep (all combos) 0.1-72 ms (typical bus) + 8. Violation detection <1 ms per sweep + 9. Return N-1+N-2 results to UI <200 ms total (typical) + 10. (Background) N-3 sweep if requested 0.3-1.4s (typical, h<=4) +``` + +### Key Numbers + +| Metric | 2000-bus | 10k-bus | +|--------|:--------:|:-------:| +| Startup latency | ~5s | ~55s | +| Memory footprint | 250 MB | 6 GB | +| BFS scope overhead (constant) | **24 ms** | **90 ms** | +| N-1+N-2 compute at h=3, typical bus | **1 ms** | **25 ms** | +| N-1+N-2 compute at h=3, hub bus | **135 ms** | **941 ms** | +| Full N-1+N-2+N-3 compute at h=3, typical bus | **2 ms** | **286 ms** | +| Full N-1+N-2+N-3 compute at h=3, hub bus | **1.4s** | N-3 too large | + +### PyPSA vs Julia Decision + +- **BODF computation**: PyPSA computes BODF in 17s on 10k. PowerModels.jl DCPF + solves in 0.23s but does not expose BODF natively. Building BODF from repeated + DCPF solves in Julia would be slower than PyPSA's direct matrix factorization. +- **Per-click latency**: Both reduce to numpy/BLAS operations — language is irrelevant. +- **Startup latency**: Julia's JIT compilation adds 5-15s startup tax on top of + solve time. PyPSA's 55s startup on 10k is dominated by DCPF (20s) + BODF (17s), + not Python overhead. +- **Recommendation**: **PyPSA is sufficient**. The Python/numpy stack handles the + interactive path (BODF multiply) at microsecond latency. Julia adds complexity + without meaningful latency improvement for this architecture. + +--- + +## Raw Data + +| File | Contents | +|------|----------| +| `tests/latency_bench/bench_interactive_latency.py` | B1-B5 benchmark script | +| `tests/latency_bench/bench_results.json` | B1-B5 JSON results | +| `tests/latency_bench/bench_interactive_sweep.py` | B6 interactive sweep script | +| `tests/latency_bench/sweep_results.json` | B6 JSON results | + +All paths relative to `evaluations/pypsa/`. diff --git a/evaluations/pypsa/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/pypsa/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md index 4af0e7cf..464aa57c 100644 --- a/evaluations/pypsa/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/pypsa/results/observations/fnm-data-model-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -14,7 +14,7 @@ timestamp: 2026-03-24T00:00:00Z PyPSA's `import_from_pypower_ppc` imports the MATPOWER `BR_STATUS` column as a custom `status` attribute on the Lines and Transformers DataFrames but does NOT map it to PyPSA's `active` flag. All branches are treated as active regardless of their MATPOWER status. -On the 27,862-bus FNM, this means 74 inactive branches (69 lines, 5 transformers) +On the ~28,000-bus FNM, this means 74 inactive branches (69 lines, 5 transformers) participate in the DCPF B-matrix when they should be excluded. ## Context diff --git a/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md index 2ff5501a..32f90aae 100644 --- a/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -7,19 +7,19 @@ severity: low timestamp: 2026-03-24T00:00:00Z --- -# Observation: PyPSA DCPF scales to 27,862-bus FNM with 40s solve time and 16 GB memory +# Observation: PyPSA DCPF scales to ~28,000-bus FNM with 40s solve time and 16 GB memory ## Finding -PyPSA's linear power flow (`n.lpf()`) solves the 27,862-bus FNM main island (32,532 -active branches, 9,481 transformers, 5,741 generators, 8,624 loads) in 40.1 seconds +PyPSA's linear power flow (`n.lpf()`) solves the ~28,000-bus FNM main island (~33,000 +active branches, 9,481 transformers, ~5,700 generators, 8,624 loads) in 40.1 seconds wall-clock with 16,289 MB peak memory (single-threaded). The solve completes well within the 10-minute timeout and produces results matching the MATPOWER reference at float64 machine precision. ## Context -The 27,862-bus network is the largest case in the evaluation suite. The high memory usage +The ~28,000-bus network is the largest case in the evaluation suite. The high memory usage (16 GB) is driven by PyPSA's internal data structures (pandas DataFrames for all components plus the sparse B-matrix construction). The solve time is dominated by the sparse linear system factorization, which scales well for DC power flow. @@ -29,5 +29,5 @@ sparse linear system factorization, which scales well for DC power flow. PyPSA handles the LARGE network tier for DCPF without issues. The 16 GB peak memory footprint may be relevant for scalability assessment -- it suggests significant overhead per component in PyPSA's data model compared to sparse-matrix-only approaches. For -reference, the cleaned MATPOWER .m file represents ~27,862 buses in a compact matrix +reference, the cleaned MATPOWER .m file represents ~~28,000 buses in a compact matrix format that would require far less memory for the raw data alone. diff --git a/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md b/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md index bf4a61d0..e319c8d0 100644 --- a/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md +++ b/evaluations/pypsa/results/observations/fnm-scale-fnm_ingestion-G-FNM-4_acpf_convergence.md @@ -7,7 +7,7 @@ severity: medium timestamp: 2026-03-24T00:00:00Z --- -# Observation: PyPSA ACPF hits SuperLU factorization failure on 27,862-bus FNM at all relaxation levels +# Observation: PyPSA ACPF hits SuperLU factorization failure on ~28,000-bus FNM at all relaxation levels ## Finding diff --git a/evaluations/pypsa/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md b/evaluations/pypsa/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md index c8e0abc6..1cab3ff5 100644 --- a/evaluations/pypsa/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md +++ b/evaluations/pypsa/results/observations/formulation-difference-fnm_ingestion-G-FNM-3_dcpf_verification.md @@ -12,7 +12,7 @@ timestamp: 2026-03-24T00:00:00Z ## Finding PyPSA's DCPF solution (`n.lpf()`) produces deviations at float64 machine-precision level -from the MATPOWER reference on the 27,862-bus FNM main island when loaded via the shared +from the MATPOWER reference on the ~28,000-bus FNM main island when loaded via the shared `matpower_loader.load_pypsa()`. Max bus angle deviation: 1.073352e-08 degrees. Max branch flow deviation: 5.757744e-07 %. 100% of buses and 100% of branches pass all thresholds. Both tools use the same B-matrix formulation. @@ -25,7 +25,7 @@ no systematic formulation difference is observed. PyPSA's `calculate_B_H` and MA `makeBdc.m` produce equivalent B-matrix constructions when the branch status patch is applied. -Bus injection power balance was verified on all 27,862 non-excluded buses using post-solve +Bus injection power balance was verified on all ~28,000 non-excluded buses using post-solve generator dispatch values. Maximum bus mismatch: 1.317039e-07 MW. All buses balanced within 1.000000e-03 MW tolerance. diff --git a/evaluations/pypsa/results/synthesis.md b/evaluations/pypsa/results/synthesis.md index 111b9167..be3c2a7f 100644 --- a/evaluations/pypsa/results/synthesis.md +++ b/evaluations/pypsa/results/synthesis.md @@ -296,9 +296,9 @@ The supply chain gate passes. The single GPL-2.0 dependency (Levenshtein) is a c ### Power Flow Verification -**G-FNM-3 (DCPF Verification): PASS** -- 100% of buses (27,862) and 100% of branches (32,532) pass all tolerance thresholds via MATPOWER fallback path. Max bus angle deviation: 1.07e-08 degrees. Max branch flow deviation: 5.76e-07%. Bus injection power balance verified on all buses (max mismatch: 1.32e-07 MW). The shared `matpower_loader.load_pypsa()` addresses the `import_from_pypower_ppc` branch status bug that caused the original v10 G-FNM-3 failure. Solve time: 40.1s, peak memory: 16,289 MB. ([G-FNM-3](fnm_ingestion/G-FNM-3_dcpf_verification.md)) +**G-FNM-3 (DCPF Verification): PASS** -- 100% of buses (~28,000) and 100% of branches (~33,000) pass all tolerance thresholds via MATPOWER fallback path. Max bus angle deviation: 1.07e-08 degrees. Max branch flow deviation: 5.76e-07%. Bus injection power balance verified on all buses (max mismatch: 1.32e-07 MW). The shared `matpower_loader.load_pypsa()` addresses the `import_from_pypower_ppc` branch status bug that caused the original v10 G-FNM-3 failure. Solve time: 40.1s, peak memory: 16,289 MB. ([G-FNM-3](fnm_ingestion/G-FNM-3_dcpf_verification.md)) -**G-FNM-4 (ACPF Convergence): INFORMATIONAL** -- PyPSA's Newton-Raphson ACPF did not converge at any relaxation level (0%, 10%, 20%) on the 27,862-bus FNM. SuperLU factorization failure at all levels. Consistent with MATPOWER 8.1's failure on the same network. The FNM planning model lacks a feasible AC operating point at full load. PyPSA's ACPF solver offers fewer recovery options than MATPOWER (no continuation PF, no fast-decoupled variants). [solver-specific: SuperLU factorization on ill-conditioned admittance matrix] ([G-FNM-4](fnm_ingestion/G-FNM-4_acpf_convergence.md)) +**G-FNM-4 (ACPF Convergence): INFORMATIONAL** -- PyPSA's Newton-Raphson ACPF did not converge at any relaxation level (0%, 10%, 20%) on the ~28,000-bus FNM. SuperLU factorization failure at all levels. Consistent with MATPOWER 8.1's failure on the same network. The FNM planning model lacks a feasible AC operating point at full load. PyPSA's ACPF solver offers fewer recovery options than MATPOWER (no continuation PF, no fast-decoupled variants). [solver-specific: SuperLU factorization on ill-conditioned admittance matrix] ([G-FNM-4](fnm_ingestion/G-FNM-4_acpf_convergence.md)) ### Supplemental Data Representability @@ -306,9 +306,9 @@ The supply chain gate passes. The single GPL-2.0 dependency (Levenshtein) is a c ### FNM Evidence Integration -- **Expressiveness:** The G-FNM-1 failure (no PSS/E parsing) is a format gap, not an expressiveness limitation. G-FNM-3's machine-precision DCPF match confirms PyPSA's formulation correctness on large networks (27,862 buses). G-FNM-4's ACPF non-convergence is consistent with MATPOWER and reflects network characteristics, not a tool deficiency. +- **Expressiveness:** The G-FNM-1 failure (no PSS/E parsing) is a format gap, not an expressiveness limitation. G-FNM-3's machine-precision DCPF match confirms PyPSA's formulation correctness on large networks (~28,000 buses). G-FNM-4's ACPF non-convergence is consistent with MATPOWER and reflects network characteristics, not a tool deficiency. - **Extensibility:** G-FNM-5's 23% extension-representable rate demonstrates that PyPSA's DataFrame-centric architecture enables supplemental data storage, though semantic interpretation requires custom code. -- **Scalability:** G-FNM-3 demonstrates DCPF scaling to LARGE (27,862 buses) in 40s with 16 GB memory. Memory overhead is notable (16 GB for a sparse linear solve on ~28k buses). +- **Scalability:** G-FNM-3 demonstrates DCPF scaling to LARGE (~28,000 buses) in 40s with 16 GB memory. Memory overhead is notable (16 GB for a sparse linear solve on ~28k buses). --- @@ -357,7 +357,7 @@ The supply chain gate passes. The single GPL-2.0 dependency (Levenshtein) is a c - `import_from_pypower_ppc` ignores MATPOWER branch status -- shared loader patches this deterministically - No formulation difference from MATPOWER for DCPF when branch status is correctly handled -- PyPSA DCPF matches MATPOWER at float64 machine precision on the 27,862-bus FNM +- PyPSA DCPF matches MATPOWER at float64 machine precision on the ~28,000-bus FNM - 57% in-model supplemental CSV representability (34% N + 23% E) via DataFrame custom columns --- diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_1_ingestion.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_1_ingestion.py index deab80af..8d40a22d 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_1_ingestion.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_1_ingestion.py @@ -5,10 +5,10 @@ intermediate manifest. Pass condition: All non-empty table counts match the manifest expectations. - - Buses: 30307 total in RAW, minus 2370 type-4 (isolated) = 27937 importable - - Generators: 5768 - - Branches + Transformers: 33840 merged in MATPOWER; PyPSA splits by tap ratio - - Loads: 15062 (PyPSA aggregates loads per bus from PPC, so expect bus count with nonzero load) + - Buses: 30000 total in RAW, minus type-4 (isolated) = importable subset + - Generators: 5800 + - Branches + Transformers: 34000 merged in MATPOWER; PyPSA splits by tap ratio + - Loads: 15000 (PyPSA aggregates loads per bus from PPC, so expect bus count with nonzero load) Tool: PyPSA API: import_from_pypower_ppc(ppc) @@ -159,7 +159,7 @@ def run() -> dict: exp_gen = expected["generator"]["expected_record_count"] exp_branch = expected["branch"]["expected_record_count"] exp_xfmr = expected["transformer"]["expected_record_count"] - exp_branch_total = exp_branch + exp_xfmr # 33840 merged in MATPOWER + exp_branch_total = exp_branch + exp_xfmr # merged in MATPOWER exp_load = expected["load"]["expected_record_count"] exp_switched_shunt = expected["switched_shunt"]["expected_record_count"] @@ -174,7 +174,7 @@ def run() -> dict: "expected": exp_bus_after_filter, "actual": n_buses, "match": n_buses == exp_bus_after_filter, - "note": f"30307 total - {type4_count} type-4 = {exp_bus_after_filter}", + "note": f"{exp_bus} total - {type4_count} type-4 = {exp_bus_after_filter}", } # Generator count: gens on non-type-4 buses diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf.py index 8bc95898..26b99955 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf.py @@ -1,7 +1,7 @@ """G-FNM-3: DCPF verification against MATPOWER reference solution. Dimension: fnm_ingestion (Suite G) -Network: LARGE — FNM main island (27,862 buses, 32,532 active branches) +Network: LARGE — FNM main island (28000 buses, 33000 active branches) Pass condition: - >=99% of buses within 0.1 degree voltage angle tolerance - >=99% of in-service branches within 1 MW absolute (or 1% relative) tolerance diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.py index 8bcdd55e..edfd0157 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_3_dcpf_verification.py @@ -2,7 +2,7 @@ Test G-FNM-3: DCPF verification against reference solution on LARGE Dimension: fnm_ingestion -Network: LARGE (FNM Annual S01, 27862-bus main island) +Network: LARGE (FNM Annual S01, 28000-bus main island) Pass condition: Pass if all aggregate thresholds are met and no hard-fail condition is triggered, per the dcpf section of data/fnm/reference/pass_conditions.json. Bus injection power balance check must pass. diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf.py index 966fde00..26c2089b 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf.py @@ -1,7 +1,7 @@ """G-FNM-4: ACPF convergence capability on FNM main island. Dimension: fnm_ingestion (Suite G) -Network: LARGE — FNM main island (27,862 buses, 32,532 active branches) +Network: LARGE — FNM main island (28000 buses, 33000 active branches) Pass condition: Informational — convergence is a positive finding, not a requirement. MATPOWER 8.1 cannot solve this case (voltage collapse at ~30% load). If PyPSA converges, that is a solver robustness strength. diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.py index 08552e76..b662f527 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_4_acpf_convergence.py @@ -2,7 +2,7 @@ Test G-FNM-4: ACPF convergence -- DCPF warm-start + progressive relaxation on LARGE Dimension: fnm_ingestion -Network: LARGE (FNM Annual S01, 27862-bus main island) +Network: LARGE (FNM Annual S01, 28000-bus main island) Pass condition: No hard pass/fail gate. All outcomes are diagnostic findings. Record relaxation_level_achieved: 0%, 10%, 20%, or infeasible. If convergence occurs at any level, record as a discriminating solver robustness strength. @@ -249,7 +249,7 @@ def run() -> dict: if relaxation_achieved != "infeasible": results["details"]["convergence_finding"] = ( f"POSITIVE: PyPSA converged at {relaxation_achieved} relaxation " - f"on the 27,862-bus FNM main island with DC warm start, " + f"on the 28000-bus FNM main island with DC warm start, " f"where MATPOWER 8.1 failed at ~30% load via continuation power flow." ) else: diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv.py index e99824a4..127c4779 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv.py @@ -19,7 +19,7 @@ from pathlib import Path FNM_PATH = Path("/data/fnm-source") -PREFIX = "AUC_AN_2026_2026_S01_" +PREFIX = "_" # Empirical field classification for each supplemental CSV # Based on actual CSV column names (from FNM data) and PyPSA data model diff --git a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv_representability.py b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv_representability.py index 65db022b..0df66023 100644 --- a/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv_representability.py +++ b/evaluations/pypsa/tests/fnm_ingestion/test_g_fnm_5_supplemental_csv_representability.py @@ -17,7 +17,7 @@ from pathlib import Path FNM_PATH = Path("/data/fnm-source") -PREFIX = "AUC_AN_2026_2026_S01_" +PREFIX = "_" # ── Per-field representability classifications ────────────────────────── # Each entry: (tier, extension_approach_or_justification) diff --git a/evaluations/pypsa/tests/latency_bench/__init__.py b/evaluations/pypsa/tests/latency_bench/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/evaluations/pypsa/tests/latency_bench/bench_interactive_latency.py b/evaluations/pypsa/tests/latency_bench/bench_interactive_latency.py new file mode 100644 index 00000000..26956af4 --- /dev/null +++ b/evaluations/pypsa/tests/latency_bench/bench_interactive_latency.py @@ -0,0 +1,882 @@ +""" +PyPSA/Linopy Latency Benchmarking for Interactive Contingency Sweeps + +Five benchmark sections: + B1: Baseline solve time decomposition (DCPF vs DCOPF, 10k bus) + B2: Linopy build vs solve isolation (create_model / solve_model) + B3: Incremental re-solve strategies (full rebuild vs model mod vs warm-start) + B4: Scaling curve (bus count -> solve time, via clustering) + B5: Contingency sweep throughput (BODF matrix vs re-solve) + +Usage: + cd evaluations/pypsa + uv run python tests/latency_bench/bench_interactive_latency.py +""" + +from __future__ import annotations + +import gc +import json +import time +import traceback +import tracemalloc +from pathlib import Path + +import numpy as np + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent.parent +NETWORKS = { + "case39": str(REPO_ROOT / "data" / "networks" / "case39.m"), + "case2000": str(REPO_ROOT / "data" / "networks" / "case_ACTIVSg2000.m"), + "case10000": str(REPO_ROOT / "data" / "networks" / "case_ACTIVSg10k.m"), +} + +SOLVER_NAME = "highs" +SOLVER_OPTIONS = { + "threads": 1, + "presolve": "on", + "output_flag": False, + "log_to_console": False, +} + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def load_network(network_file: str, for_opf: bool = False): + """Load MATPOWER .m file into PyPSA Network. + + Args: + network_file: Path to .m file. + for_opf: If True, relax zero-rated lines to 99999 MVA and assign costs. + """ + import pypsa + from matpowercaseframes import CaseFrames + + cf = CaseFrames(network_file) + ppc = { + "version": "2", + "baseMVA": float(cf.baseMVA), + "bus": cf.bus.values, + "gen": cf.gen.values, + "branch": cf.branch.values, + } + n = pypsa.Network() + n.import_from_pypower_ppc(ppc, overwrite_zero_s_nom=1.0) + + if for_opf: + # Zero-rated = "no thermal limit" in MATPOWER, not "blocked" + n.lines.loc[n.lines.s_nom == 1.0, "s_nom"] = 99999.0 + # Assign marginal costs for merit-order dispatch + gen_names = sorted(n.generators.index) + costs = np.linspace(10, 100, len(gen_names)) + for gen_name, cost in zip(gen_names, costs): + n.generators.at[gen_name, "marginal_cost"] = float(cost) + + return n + + +def timed(func, *args, **kwargs): + """Run func, return (result, elapsed_seconds, peak_memory_mb).""" + gc.collect() + tracemalloc.start() + t0 = time.perf_counter() + result = func(*args, **kwargs) + elapsed = time.perf_counter() - t0 + _, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + return result, elapsed, peak / (1024 * 1024) + + +# --------------------------------------------------------------------------- +# B1: Baseline Solve Time Decomposition +# --------------------------------------------------------------------------- + + +def run_b1() -> dict: + """DCPF and DCOPF on 10k-bus, 5 runs each. Reports median/min/max + peak memory.""" + print("\n" + "=" * 60) + print("B1: Baseline Solve Time Decomposition (10k bus)") + print("=" * 60) + + result: dict = {"status": "fail", "details": {}, "errors": []} + N_RUNS = 3 + + try: + n_base = load_network(NETWORKS["case10000"]) + n_opf_base = load_network(NETWORKS["case10000"], for_opf=True) + result["details"]["n_buses"] = len(n_base.buses) + result["details"]["n_lines"] = len(n_base.lines) + print(f"Network: {len(n_base.buses)} buses, {len(n_base.lines)} lines") + + # DCPF (n.lpf) + lpf_times, lpf_mems = [], [] + for i in range(N_RUNS): + n = n_base.copy() + _, elapsed, peak_mb = timed(n.lpf) + lpf_times.append(elapsed) + lpf_mems.append(peak_mb) + print(f" DCPF run {i + 1}: {elapsed:.3f}s, {peak_mb:.1f} MB") + + result["details"]["dcpf"] = { + "run_times": lpf_times, + "median_s": float(np.median(lpf_times)), + "min_s": float(min(lpf_times)), + "max_s": float(max(lpf_times)), + "peak_memory_mb": float(np.median(lpf_mems)), + } + print(f" DCPF median: {np.median(lpf_times):.3f}s") + + # DCOPF (n.optimize) + opf_times, opf_mems = [], [] + for i in range(N_RUNS): + n = n_opf_base.copy() + _, elapsed, peak_mb = timed( + n.optimize, solver_name=SOLVER_NAME, solver_options=SOLVER_OPTIONS + ) + opf_times.append(elapsed) + opf_mems.append(peak_mb) + print(f" DCOPF run {i + 1}: {elapsed:.3f}s, {peak_mb:.1f} MB") + + result["details"]["dcopf"] = { + "run_times": opf_times, + "median_s": float(np.median(opf_times)), + "min_s": float(min(opf_times)), + "max_s": float(max(opf_times)), + "peak_memory_mb": float(np.median(opf_mems)), + } + print(f" DCOPF median: {np.median(opf_times):.3f}s") + result["status"] = "pass" + + except Exception as e: + result["errors"].append(f"{type(e).__name__}: {e}") + result["details"]["traceback"] = traceback.format_exc() + print(f"ERROR: {e}") + + return result + + +# --------------------------------------------------------------------------- +# B2: Linopy Build vs Solve Isolation +# --------------------------------------------------------------------------- + + +def run_b2() -> dict: + """Decompose n.optimize() into model build vs solver execution. + + Tests create_model() + solve_model() on 39, 2000, 10000 bus networks. + Also tests io_api="direct" to skip LP file I/O. + """ + print("\n" + "=" * 60) + print("B2: Linopy Build vs Solve Isolation") + print("=" * 60) + + result: dict = {"status": "fail", "details": {}, "errors": []} + + try: + for label, path in NETWORKS.items(): + # Use fewer runs for 10k to keep total time manageable (~200s per DCOPF) + n_runs = 1 if label == "case10000" else 3 + print(f"\n--- {label} ({n_runs} runs) ---") + n_base = load_network(path, for_opf=True) + n_buses = len(n_base.buses) + case_result: dict = {"n_buses": n_buses, "build": {}, "solve": {}, "solve_direct": {}} + + # Model build timing + build_times, build_mems = [], [] + for i in range(n_runs): + n = n_base.copy() + _, elapsed, peak_mb = timed(n.optimize.create_model) + build_times.append(elapsed) + build_mems.append(peak_mb) + print(f" create_model run {i + 1}: {elapsed:.3f}s, {peak_mb:.1f} MB") + if i == n_runs - 1: + # Keep last model for solve timing + _ = n + + case_result["build"] = { + "run_times": build_times, + "median_s": float(np.median(build_times)), + "min_s": float(min(build_times)), + "max_s": float(max(build_times)), + "peak_memory_mb": float(np.median(build_mems)), + } + + # Solve timing (default io_api) + solve_times, solve_mems = [], [] + for i in range(n_runs): + n = n_base.copy() + n.optimize.create_model() + _, elapsed, peak_mb = timed( + n.optimize.solve_model, + solver_name=SOLVER_NAME, + solver_options=SOLVER_OPTIONS, + ) + solve_times.append(elapsed) + solve_mems.append(peak_mb) + print(f" solve_model run {i + 1}: {elapsed:.3f}s, {peak_mb:.1f} MB") + + case_result["solve"] = { + "run_times": solve_times, + "median_s": float(np.median(solve_times)), + "min_s": float(min(solve_times)), + "max_s": float(max(solve_times)), + "peak_memory_mb": float(np.median(solve_mems)), + } + + # Solve timing with io_api="direct" (skip LP file I/O) + solve_direct_times, solve_direct_mems = [], [] + for i in range(n_runs): + n = n_base.copy() + n.optimize.create_model() + try: + _, elapsed, peak_mb = timed( + n.optimize.solve_model, + solver_name=SOLVER_NAME, + solver_options=SOLVER_OPTIONS, + io_api="direct", + ) + solve_direct_times.append(elapsed) + solve_direct_mems.append(peak_mb) + print(f" solve_model(direct) run {i + 1}: {elapsed:.3f}s, {peak_mb:.1f} MB") + except Exception as e: + print(f" solve_model(direct) run {i + 1}: FAILED — {e}") + case_result["solve_direct"]["error"] = str(e) + break + + if solve_direct_times: + case_result["solve_direct"] = { + "run_times": solve_direct_times, + "median_s": float(np.median(solve_direct_times)), + "min_s": float(min(solve_direct_times)), + "max_s": float(max(solve_direct_times)), + "peak_memory_mb": float(np.median(solve_direct_mems)), + } + + result["details"][label] = case_result + build_med = case_result["build"]["median_s"] + solve_med = case_result["solve"]["median_s"] + print( + f" Summary: build={build_med:.3f}s, solve={solve_med:.3f}s, " + f"build_pct={build_med / (build_med + solve_med) * 100:.0f}%" + ) + + result["status"] = "pass" + + except Exception as e: + result["errors"].append(f"{type(e).__name__}: {e}") + result["details"]["traceback"] = traceback.format_exc() + print(f"ERROR: {e}") + + return result + + +# --------------------------------------------------------------------------- +# B3: Incremental Re-solve +# --------------------------------------------------------------------------- + + +def run_b3() -> dict: + """After initial DCOPF, knock out lines and re-solve. Compare strategies. + + Strategy A: Full rebuild (n.optimize from scratch) + Strategy B: Model modification (modify constraint bounds, re-solve without rebuild) + Strategy C: Warm-start (pass HiGHS basis if supported) + + Uses 2000-bus network. + """ + print("\n" + "=" * 60) + print("B3: Incremental Re-solve (2000 bus)") + print("=" * 60) + + result: dict = {"status": "fail", "details": {}, "errors": []} + N_RUNS = 3 + N_OUTAGES = 5 + + try: + n_base = load_network(NETWORKS["case2000"], for_opf=True) + print(f"Network: {len(n_base.buses)} buses, {len(n_base.lines)} lines") + + # Initial solve to find high-flow lines + n_init = n_base.copy() + n_init.optimize(solver_name=SOLVER_NAME, solver_options=SOLVER_OPTIONS) + flows = n_init.lines_t.p0.iloc[0].abs() + top_lines = flows.nlargest(N_OUTAGES).index.tolist() + print(f"Top-flow lines to outage: {top_lines}") + result["details"]["outage_lines"] = top_lines + + # Strategy A: Full rebuild per outage + print("\n--- Strategy A: Full rebuild ---") + strat_a_times = [] + for run_i in range(N_RUNS): + run_times = [] + for line_name in top_lines: + n = n_base.copy() + n.lines.at[line_name, "s_nom"] = 0.0001 # effectively disable + _, elapsed, _ = timed( + n.optimize, solver_name=SOLVER_NAME, solver_options=SOLVER_OPTIONS + ) + run_times.append(elapsed) + total = sum(run_times) + strat_a_times.append(total) + print( + f" Run {run_i + 1}: {total:.3f}s ({N_OUTAGES} outages, " + f"avg={total / N_OUTAGES:.3f}s)" + ) + + result["details"]["strategy_a_full_rebuild"] = { + "total_times": strat_a_times, + "median_total_s": float(np.median(strat_a_times)), + "median_per_outage_s": float(np.median(strat_a_times)) / N_OUTAGES, + } + + # Strategy B: Model modification (modify bounds, re-solve without rebuild) + print("\n--- Strategy B: Model modification (re-solve without rebuild) ---") + strat_b_times = [] + strat_b_errors = [] + for run_i in range(N_RUNS): + n = n_base.copy() + n.optimize.create_model() + # Initial solve + n.optimize.solve_model(solver_name=SOLVER_NAME, solver_options=SOLVER_OPTIONS) + + run_times = [] + for line_name in top_lines: + try: + # Modify the line capacity in the model constraints + # Set the line's s_nom to near-zero to simulate outage + n.lines.at[line_name, "s_nom"] = 0.0001 + + # Update the model constraint bounds directly via linopy + model = n.model + if hasattr(model, "constraints") and "Line-s_nom" in model.constraints: + # Try to modify the upper bound constraint + pass # Linopy constraints are immutable — fall through + + # Rebuild model but skip full optimize() overhead + _, elapsed, _ = timed( + n.optimize.solve_model, + solver_name=SOLVER_NAME, + solver_options=SOLVER_OPTIONS, + ) + run_times.append(elapsed) + except Exception as e: + strat_b_errors.append(str(e)) + print(f" Strategy B error: {e}") + break + + if run_times: + total = sum(run_times) + strat_b_times.append(total) + print( + f" Run {run_i + 1}: {total:.3f}s ({len(run_times)} outages, " + f"avg={total / len(run_times):.3f}s)" + ) + + if strat_b_times: + result["details"]["strategy_b_model_mod"] = { + "total_times": strat_b_times, + "median_total_s": float(np.median(strat_b_times)), + "median_per_outage_s": float(np.median(strat_b_times)) / N_OUTAGES, + "note": "Re-solve only (no model rebuild), but constraint bounds unchanged — " + "Linopy constraints are immutable so this measures solve_model() reuse", + } + if strat_b_errors: + result["details"]["strategy_b_errors"] = strat_b_errors + + # Strategy C: Warm-start attempt + print("\n--- Strategy C: Warm-start (HiGHS basis reuse) ---") + strat_c_times = [] + strat_c_note = "" + try: + for run_i in range(N_RUNS): + n = n_base.copy() + # First solve + n.optimize(solver_name=SOLVER_NAME, solver_options=SOLVER_OPTIONS) + + run_times = [] + for line_name in top_lines: + n_mod = n_base.copy() + n_mod.lines.at[line_name, "s_nom"] = 0.0001 + # Attempt warm-start via solver options + warm_opts = {**SOLVER_OPTIONS, "warm_start": True} + _, elapsed, _ = timed( + n_mod.optimize, + solver_name=SOLVER_NAME, + solver_options=warm_opts, + ) + run_times.append(elapsed) + + total = sum(run_times) + strat_c_times.append(total) + print( + f" Run {run_i + 1}: {total:.3f}s ({N_OUTAGES} outages, " + f"avg={total / N_OUTAGES:.3f}s)" + ) + except Exception as e: + strat_c_note = f"Warm-start not supported: {e}" + print(f" {strat_c_note}") + + if strat_c_times: + result["details"]["strategy_c_warm_start"] = { + "total_times": strat_c_times, + "median_total_s": float(np.median(strat_c_times)), + "median_per_outage_s": float(np.median(strat_c_times)) / N_OUTAGES, + } + if strat_c_note: + result["details"]["strategy_c_note"] = strat_c_note + + result["status"] = "pass" + + except Exception as e: + result["errors"].append(f"{type(e).__name__}: {e}") + result["details"]["traceback"] = traceback.format_exc() + print(f"ERROR: {e}") + + return result + + +# --------------------------------------------------------------------------- +# B4: Scaling Curve (Bus Count -> Solve Time) +# --------------------------------------------------------------------------- + + +def run_b4() -> dict: + """Find the bus count where DC-OPF < 1s for interactive use. + + Uses raw networks (39, 2000, 10000) plus PyPSA clustering to reduce + 10k -> 1000, 500, 200, 100 buses. + """ + print("\n" + "=" * 60) + print("B4: Scaling Curve (bus count vs solve time)") + print("=" * 60) + + result: dict = {"status": "fail", "details": {}, "errors": []} + N_RUNS = 3 + + try: + # Raw networks first + raw_results = {} + for label, path in NETWORKS.items(): + print(f"\n--- {label} (raw) ---") + n_base = load_network(path, for_opf=True) + n_buses = len(n_base.buses) + + # DCPF timing + lpf_times = [] + for i in range(N_RUNS): + n = n_base.copy() + _, elapsed, _ = timed(n.lpf) + lpf_times.append(elapsed) + + # DCOPF timing + opf_times = [] + for i in range(N_RUNS): + n = n_base.copy() + _, elapsed, _ = timed( + n.optimize, solver_name=SOLVER_NAME, solver_options=SOLVER_OPTIONS + ) + opf_times.append(elapsed) + + raw_results[label] = { + "n_buses": n_buses, + "dcpf_median_s": float(np.median(lpf_times)), + "dcopf_median_s": float(np.median(opf_times)), + "dcpf_times": lpf_times, + "dcopf_times": opf_times, + } + print( + f" {n_buses} buses: DCPF={np.median(lpf_times):.3f}s, " + f"DCOPF={np.median(opf_times):.3f}s" + ) + + result["details"]["raw_networks"] = raw_results + + # Clustering: MATPOWER->PyPSA bridge puts Pd/Qd on bus frame and sets + # all coordinates to (0,0). PyPSA's clustering requires `consense` on every + # bus attribute, which fails when buses in the same cluster have different + # Pd/v_mag_pu_set values. This is a known limitation — clustering works with + # natively-built PyPSA networks, not MATPOWER imports. + print("\n--- Clustering 10k-bus network ---") + print(" SKIPPED: PyPSA clustering incompatible with MATPOWER-imported networks") + print(" (consense fails on non-uniform bus attributes Pd, v_mag_pu_set)") + cluster_results: dict = { + "note": "Clustering not possible with MATPOWER-imported networks — " + "PyPSA's get_clustering_from_busmap requires consense on all " + "bus attributes, but MATPOWER import places heterogeneous Pd/Qd " + "and v_mag_pu_set on the bus frame." + } + + result["details"]["clustered_networks"] = cluster_results + + # Build scaling table + print("\n--- Scaling Table ---") + table = [] + for label, data in raw_results.items(): + table.append( + { + "source": label, + "n_buses": data["n_buses"], + "dcpf_s": data["dcpf_median_s"], + "dcopf_s": data["dcopf_median_s"], + } + ) + + table.sort(key=lambda r: r["n_buses"]) + result["details"]["scaling_table"] = table + + print(f" {'Source':<20} {'Buses':>7} {'DCPF':>10} {'DCOPF':>10}") + print(f" {'-' * 50}") + for row in table: + dcpf_flag = " *" if row["dcpf_s"] < 1.0 else "" + opf_flag = " *" if row["dcopf_s"] < 1.0 else "" + print( + f" {row['source']:<20} {row['n_buses']:>7} " + f"{row['dcpf_s']:>9.3f}s{dcpf_flag} {row['dcopf_s']:>9.3f}s{opf_flag}" + ) + print(" (* = sub-1s, interactive candidate)") + + result["status"] = "pass" + + except Exception as e: + result["errors"].append(f"{type(e).__name__}: {e}") + result["details"]["traceback"] = traceback.format_exc() + print(f"ERROR: {e}") + + return result + + +# --------------------------------------------------------------------------- +# B5: Contingency Sweep Throughput +# --------------------------------------------------------------------------- + + +def run_b5() -> dict: + """N-1 analysis: BODF matrix approach vs re-solve. + + 1. BODF pre-computation timing (39, 2000, 10k if memory allows) + 2. Vectorized all-N-1 via numpy broadcast + 3. Single-contingency latency (simulates one user click) + 4. Re-solve baseline (n.lpf per contingency on 39 and clustered 100) + 5. Violation detection timing + 6. N-2 composition (Woodbury formula, bonus) + """ + print("\n" + "=" * 60) + print("B5: Contingency Sweep Throughput") + print("=" * 60) + + result: dict = {"status": "fail", "details": {}, "errors": []} + + try: + # --- BODF pre-computation across network sizes --- + bodf_results = {} + for label, path in NETWORKS.items(): + print(f"\n--- {label}: BODF pre-computation ---") + try: + n = load_network(path) + n.lpf() + n.determine_network_topology() + + _, bodf_elapsed, bodf_mem = timed(_compute_bodf, n) + sn = list(n.sub_networks.obj)[0] + bodf_shape = sn.BODF.shape + + bodf_results[label] = { + "n_buses": len(n.buses), + "bodf_shape": list(bodf_shape), + "compute_seconds": bodf_elapsed, + "peak_memory_mb": bodf_mem, + } + print( + f" BODF shape: {bodf_shape}, time: {bodf_elapsed:.3f}s, mem: {bodf_mem:.1f} MB" + ) + except MemoryError: + bodf_results[label] = {"error": "MemoryError — network too large for BODF"} + print(f" MemoryError on {label}") + except Exception as e: + bodf_results[label] = {"error": str(e)} + print(f" Error: {e}") + + result["details"]["bodf_precompute"] = bodf_results + + # --- Vectorized N-1 on 2000-bus --- + print("\n--- Vectorized all-N-1 (2000-bus) ---") + n = load_network(NETWORKS["case2000"]) + n.lpf() + n.determine_network_topology() + _compute_bodf(n) + + sn = list(n.sub_networks.obj)[0] + sn_branches = sn.branches() + p0_sn = _build_p0_vector(n, sn_branches) + BODF = sn.BODF + n_branches = BODF.shape[0] + + # All-N-1 via broadcast: post_flows[i, k] = p0[i] + BODF[i, k] * p0[k] + gc.collect() + t0 = time.perf_counter() + all_post_flows = p0_sn[:, np.newaxis] + BODF * p0_sn[np.newaxis, :] + vectorized_elapsed = time.perf_counter() - t0 + print( + f" All-N-1 vectorized ({n_branches}x{n_branches}): {vectorized_elapsed * 1000:.2f} ms" + ) + + result["details"]["vectorized_n1_2000"] = { + "n_branches": n_branches, + "elapsed_ms": vectorized_elapsed * 1000, + "matrix_shape": list(all_post_flows.shape), + } + + # Single-contingency latency (simulate user click) + print("\n--- Single-contingency latency (2000-bus) ---") + single_times = [] + for k in range(min(100, n_branches)): + t0 = time.perf_counter() + _ = p0_sn + BODF[:, k] * p0_sn[k] + single_times.append(time.perf_counter() - t0) + + result["details"]["single_contingency_2000"] = { + "n_samples": len(single_times), + "median_us": float(np.median(single_times)) * 1e6, + "mean_us": float(np.mean(single_times)) * 1e6, + "max_us": float(max(single_times)) * 1e6, + } + print( + f" Single contingency: median={np.median(single_times) * 1e6:.1f} us, " + f"mean={np.mean(single_times) * 1e6:.1f} us" + ) + + # Violation detection timing + print("\n--- Violation detection (2000-bus) ---") + s_nom_sn = _build_s_nom_vector(n, sn_branches) + t0 = time.perf_counter() + violations = np.abs(all_post_flows) > s_nom_sn[:, np.newaxis] + n_violations = int(violations.sum()) + violation_elapsed = time.perf_counter() - t0 + print( + f" Violation check: {violation_elapsed * 1000:.2f} ms, " + f"{n_violations} total violations across all N-1" + ) + + result["details"]["violation_detection_2000"] = { + "elapsed_ms": violation_elapsed * 1000, + "total_violations": n_violations, + "contingencies_with_violations": int((violations.sum(axis=0) > 0).sum()), + } + + # Re-solve baseline on 39-bus (lpf per contingency) + print("\n--- Re-solve baseline (39-bus, n.lpf per contingency) ---") + n39 = load_network(NETWORKS["case39"]) + n39.lpf() + n39_lines = n39.lines.index.tolist() + resolv_times = [] + for line_name in n39_lines: + n_c = n39.copy() + n_c.lines.at[line_name, "s_nom"] = 0.0001 + t0 = time.perf_counter() + n_c.lpf() + resolv_times.append(time.perf_counter() - t0) + + result["details"]["resolv_baseline_39"] = { + "n_contingencies": len(n39_lines), + "total_s": sum(resolv_times), + "per_contingency_ms": float(np.median(resolv_times)) * 1000, + } + print( + f" {len(n39_lines)} contingencies: total={sum(resolv_times):.3f}s, " + f"per={np.median(resolv_times) * 1000:.2f} ms" + ) + + # BODF on 39-bus for comparison + print("\n--- BODF N-1 on 39-bus ---") + n39b = load_network(NETWORKS["case39"]) + n39b.lpf() + n39b.determine_network_topology() + _compute_bodf(n39b) + sn39 = list(n39b.sub_networks.obj)[0] + sn39_branches = sn39.branches() + p0_39 = _build_p0_vector(n39b, sn39_branches) + BODF_39 = sn39.BODF + t0 = time.perf_counter() + _all_post_39 = p0_39[:, np.newaxis] + BODF_39 * p0_39[np.newaxis, :] + bodf_39_elapsed = time.perf_counter() - t0 + print(f" BODF all-N-1: {bodf_39_elapsed * 1e6:.1f} us") + + result["details"]["bodf_n1_39"] = { + "n_branches": BODF_39.shape[0], + "elapsed_us": bodf_39_elapsed * 1e6, + "speedup_vs_resolv": sum(resolv_times) / max(bodf_39_elapsed, 1e-9), + } + + # --- BODF on 10k if memory allows --- + print("\n--- BODF vectorized N-1 on 10k-bus (if memory allows) ---") + try: + n10k = load_network(NETWORKS["case10000"]) + n10k.lpf() + n10k.determine_network_topology() + + _, bodf_10k_elapsed, bodf_10k_mem = timed(_compute_bodf, n10k) + sn10k = list(n10k.sub_networks.obj)[0] + sn10k_branches = sn10k.branches() + p0_10k = _build_p0_vector(n10k, sn10k_branches) + BODF_10k = sn10k.BODF + + t0 = time.perf_counter() + _all_post_10k = p0_10k[:, np.newaxis] + BODF_10k * p0_10k[np.newaxis, :] + vec_10k_elapsed = time.perf_counter() - t0 + + # Single contingency + single_10k_times = [] + for k in range(min(100, BODF_10k.shape[0])): + t0 = time.perf_counter() + _ = p0_10k + BODF_10k[:, k] * p0_10k[k] + single_10k_times.append(time.perf_counter() - t0) + + result["details"]["vectorized_n1_10k"] = { + "n_branches": BODF_10k.shape[0], + "bodf_compute_s": bodf_10k_elapsed, + "bodf_memory_mb": bodf_10k_mem, + "all_n1_ms": vec_10k_elapsed * 1000, + "single_contingency_median_us": float(np.median(single_10k_times)) * 1e6, + } + print(f" BODF 10k: compute={bodf_10k_elapsed:.1f}s, mem={bodf_10k_mem:.0f} MB") + print(f" All-N-1: {vec_10k_elapsed * 1000:.1f} ms") + print(f" Single contingency: {np.median(single_10k_times) * 1e6:.1f} us") + + except MemoryError: + result["details"]["vectorized_n1_10k"] = {"error": "MemoryError"} + print(" MemoryError — 10k BODF too large") + except Exception as e: + result["details"]["vectorized_n1_10k"] = {"error": str(e)} + print(f" Error: {e}") + + # --- N-2 composition (bonus) --- + print("\n--- N-2 Woodbury composition (2000-bus, bonus) ---") + try: + # For double outage of lines k1, k2: + # Use superposition: delta_p = BODF[:, k1] * p0[k1] + BODF[:, k2] * p0[k2] + # (first-order approximation; exact Woodbury requires matrix inverse update) + n_pairs = min(100, n_branches * (n_branches - 1) // 2) + rng = np.random.default_rng(42) + pairs = set() + while len(pairs) < n_pairs: + k1, k2 = sorted(rng.choice(n_branches, 2, replace=False)) + pairs.add((k1, k2)) + + t0 = time.perf_counter() + for k1, k2 in pairs: + _ = p0_sn + BODF[:, k1] * p0_sn[k1] + BODF[:, k2] * p0_sn[k2] + n2_elapsed = time.perf_counter() - t0 + + result["details"]["n2_woodbury_2000"] = { + "n_pairs": n_pairs, + "total_ms": n2_elapsed * 1000, + "per_pair_us": n2_elapsed / n_pairs * 1e6, + "note": "First-order superposition (not exact Woodbury inverse update)", + } + print( + f" {n_pairs} N-2 pairs: total={n2_elapsed * 1000:.1f} ms, " + f"per pair={n2_elapsed / n_pairs * 1e6:.1f} us" + ) + except Exception as e: + result["details"]["n2_woodbury_2000"] = {"error": str(e)} + print(f" N-2 error: {e}") + + result["status"] = "pass" + + except Exception as e: + result["errors"].append(f"{type(e).__name__}: {e}") + result["details"]["traceback"] = traceback.format_exc() + print(f"ERROR: {e}") + + return result + + +def _compute_bodf(n): + """Compute PTDF and BODF for all sub-networks.""" + for sn in n.sub_networks.obj: + sn.calculate_PTDF() + sn.calculate_BODF() + + +def _build_p0_vector(n, sn_branches) -> np.ndarray: + """Build base-case power flow vector for sub-network branches.""" + p0 = [] + for comp, bname in sn_branches.index: + if comp == "Line" and bname in n.lines_t.p0.columns: + p0.append(float(n.lines_t.p0.iloc[0][bname])) + elif ( + comp == "Transformer" + and len(n.transformers_t.p0) > 0 + and bname in n.transformers_t.p0.columns + ): + p0.append(float(n.transformers_t.p0.iloc[0][bname])) + else: + p0.append(0.0) + return np.array(p0) + + +def _build_s_nom_vector(n, sn_branches) -> np.ndarray: + """Build s_nom vector for sub-network branches.""" + s_nom = [] + for comp, bname in sn_branches.index: + if comp == "Line" and bname in n.lines.index: + s_nom.append(float(n.lines.at[bname, "s_nom"])) + elif comp == "Transformer" and bname in n.transformers.index: + s_nom.append(float(n.transformers.at[bname, "s_nom"])) + else: + s_nom.append(1e9) + return np.array(s_nom) + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + + +def main(): + """Run all benchmarks and dump JSON results.""" + print("PyPSA/Linopy Interactive Latency Benchmarks") + print("=" * 60) + + all_results = {} + + for name, func in [ + ("b1_baseline_decomposition", run_b1), + ("b2_linopy_build_vs_solve", run_b2), + ("b3_incremental_resolv", run_b3), + ("b4_scaling_curve", run_b4), + ("b5_contingency_throughput", run_b5), + ]: + print(f"\n{'#' * 60}") + print(f"# Running {name}") + print(f"{'#' * 60}") + t0 = time.perf_counter() + try: + all_results[name] = func() + except Exception as e: + all_results[name] = {"status": "error", "error": str(e)} + all_results[name]["wall_clock_seconds"] = time.perf_counter() - t0 + + # Summary + print("\n" + "=" * 60) + print("SUMMARY") + print("=" * 60) + for name, res in all_results.items(): + status = res.get("status", "unknown") + wall = res.get("wall_clock_seconds", 0) + print(f" {name}: {status} ({wall:.1f}s)") + + # Write JSON + output_path = Path(__file__).parent / "bench_results.json" + with open(output_path, "w") as f: + json.dump(all_results, f, indent=2, default=str) + print(f"\nResults written to {output_path}") + + return all_results + + +if __name__ == "__main__": + main() diff --git a/evaluations/pypsa/tests/latency_bench/bench_interactive_sweep.py b/evaluations/pypsa/tests/latency_bench/bench_interactive_sweep.py new file mode 100644 index 00000000..fe1eb7f0 --- /dev/null +++ b/evaluations/pypsa/tests/latency_bench/bench_interactive_sweep.py @@ -0,0 +1,356 @@ +""" +Interactive contingency sweep latency benchmark. + +Simulates the exact user workflow: + 1. User clicks a bus on the map + 2. BFS out to h hops → find scoped branches + 3. Enumerate N-m contingencies (m=1,2,3) over scoped branches + 4. Compute all post-contingency flows via BODF + 5. Detect violations + +Measures wall-clock latency for steps 2-5 at 39, 2000, 10000 buses +and h = 1, 2, 3, 4 hops. +""" + +from __future__ import annotations + +import gc +import itertools +import json +import time +from pathlib import Path + +import networkx as nx +import numpy as np + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent.parent +NETWORKS = { + "case39": str(REPO_ROOT / "data" / "networks" / "case39.m"), + "case2000": str(REPO_ROOT / "data" / "networks" / "case_ACTIVSg2000.m"), + "case10000": str(REPO_ROOT / "data" / "networks" / "case_ACTIVSg10k.m"), +} + + +def load_and_prepare(network_file: str): + """Load network, run DCPF, compute BODF. Returns prepared context.""" + import pypsa + from matpowercaseframes import CaseFrames + + cf = CaseFrames(network_file) + ppc = { + "version": "2", + "baseMVA": float(cf.baseMVA), + "bus": cf.bus.values, + "gen": cf.gen.values, + "branch": cf.branch.values, + } + n = pypsa.Network() + n.import_from_pypower_ppc(ppc, overwrite_zero_s_nom=1.0) + + # DCPF + n.lpf() + + # Topology + BODF + n.determine_network_topology() + for sn in n.sub_networks.obj: + sn.calculate_PTDF() + sn.calculate_BODF() + + # Get main sub-network + main_sn = max(n.sub_networks.obj, key=lambda sn: len(sn.branches())) + sn_branches = main_sn.branches() + BODF = main_sn.BODF + + # Build p0 and s_nom vectors + p0 = [] + s_nom = [] + for comp, bname in sn_branches.index: + if comp == "Line" and bname in n.lines_t.p0.columns: + p0.append(float(n.lines_t.p0.iloc[0][bname])) + elif ( + comp == "Transformer" + and len(n.transformers_t.p0) > 0 + and bname in n.transformers_t.p0.columns + ): + p0.append(float(n.transformers_t.p0.iloc[0][bname])) + else: + p0.append(0.0) + + if comp == "Line" and bname in n.lines.index: + s_nom.append(float(n.lines.at[bname, "s_nom"])) + elif comp == "Transformer" and bname in n.transformers.index: + s_nom.append(float(n.transformers.at[bname, "s_nom"])) + else: + s_nom.append(1e9) + + p0 = np.array(p0) + s_nom = np.array(s_nom) + + # Build branch index lookup: branch_name -> position in BODF matrix + branch_idx = {} + for i, (comp, bname) in enumerate(sn_branches.index): + branch_idx[(comp, bname)] = i + + return n, BODF, p0, s_nom, branch_idx + + +def bfs_scoped_branches(n, focal_bus: str, h: int, branch_idx: dict) -> list[int]: + """BFS from focal_bus out to h hops. Return BODF column indices of scoped branches.""" + G = n.graph() + if focal_bus not in G: + return [] + + distance = nx.single_source_shortest_path_length(G, focal_bus, cutoff=h) + buses_in_scope = set(distance.keys()) + + scoped = [] + for line_name in n.lines.index: + key = ("Line", line_name) + if key not in branch_idx: + continue + bus0 = n.lines.at[line_name, "bus0"] + bus1 = n.lines.at[line_name, "bus1"] + if bus0 in buses_in_scope or bus1 in buses_in_scope: + scoped.append(branch_idx[key]) + + return scoped + + +def sweep_n1( + BODF: np.ndarray, p0: np.ndarray, s_nom: np.ndarray, scoped_indices: list[int] +) -> tuple[np.ndarray, int]: + """N-1 sweep via BODF. Returns (max_loading_per_contingency, n_violations).""" + k_indices = np.array(scoped_indices) + # Vectorized: post_flows[:, j] = p0 + BODF[:, k_j] * p0[k_j] + bodf_cols = BODF[:, k_indices] # (n_branches, n_contingencies) + p0_outaged = p0[k_indices] # (n_contingencies,) + delta = bodf_cols * p0_outaged[np.newaxis, :] # broadcast + post_flows = p0[:, np.newaxis] + delta # (n_branches, n_contingencies) + violations = np.abs(post_flows) > s_nom[:, np.newaxis] + return post_flows, int(violations.sum()) + + +def sweep_n2( + BODF: np.ndarray, p0: np.ndarray, s_nom: np.ndarray, scoped_indices: list[int] +) -> tuple[int, int]: + """N-2 sweep via BODF superposition. Returns (n_combinations, n_violations).""" + pairs = list(itertools.combinations(scoped_indices, 2)) + n_pairs = len(pairs) + + if n_pairs == 0: + return 0, 0 + + # Vectorized: for each pair (k1, k2), delta = BODF[:,k1]*p0[k1] + BODF[:,k2]*p0[k2] + k1s = np.array([p[0] for p in pairs]) + k2s = np.array([p[1] for p in pairs]) + delta = BODF[:, k1s] * p0[k1s][np.newaxis, :] + BODF[:, k2s] * p0[k2s][np.newaxis, :] + post_flows = p0[:, np.newaxis] + delta + violations = np.abs(post_flows) > s_nom[:, np.newaxis] + return n_pairs, int(violations.sum()) + + +def sweep_n3( + BODF: np.ndarray, + p0: np.ndarray, + s_nom: np.ndarray, + scoped_indices: list[int], + max_combos: int = 50000, +) -> tuple[int, int]: + """N-3 sweep via BODF superposition. Caps at max_combos to avoid OOM.""" + triples = list(itertools.islice(itertools.combinations(scoped_indices, 3), max_combos)) + n_triples = len(triples) + + if n_triples == 0: + return 0, 0 + + k1s = np.array([t[0] for t in triples]) + k2s = np.array([t[1] for t in triples]) + k3s = np.array([t[2] for t in triples]) + delta = ( + BODF[:, k1s] * p0[k1s][np.newaxis, :] + + BODF[:, k2s] * p0[k2s][np.newaxis, :] + + BODF[:, k3s] * p0[k3s][np.newaxis, :] + ) + post_flows = p0[:, np.newaxis] + delta + violations = np.abs(post_flows) > s_nom[:, np.newaxis] + return n_triples, int(violations.sum()) + + +def run_sweep(label: str, network_file: str, hop_range: list[int]) -> dict: + """Run the full interactive sweep benchmark for one network.""" + print(f"\n{'=' * 60}") + print(f"Network: {label}") + print(f"{'=' * 60}") + + # Startup (one-time cost) + t0 = time.perf_counter() + n, BODF, p0, s_nom, branch_idx = load_and_prepare(network_file) + startup = time.perf_counter() - t0 + n_buses = len(n.buses) + n_branches = BODF.shape[0] + print(f"Startup: {startup:.2f}s ({n_buses} buses, {n_branches} branches)") + print(f"BODF shape: {BODF.shape}, memory: {BODF.nbytes / 1024 / 1024:.0f} MB") + + # Pick focal bus: highest degree (worst case for scoping) + G = n.graph() + degrees = dict(G.degree()) + focal_bus = max(degrees, key=lambda b: degrees[b]) + focal_degree = degrees[focal_bus] + + # Also pick a median-degree bus for typical case + sorted_buses = sorted(degrees.items(), key=lambda x: x[1]) + median_bus = sorted_buses[len(sorted_buses) // 2][0] + median_degree = degrees[median_bus] + + print(f"Focal bus (max degree): {focal_bus} (degree={focal_degree})") + print(f"Median bus: {median_bus} (degree={median_degree})") + + result = { + "n_buses": n_buses, + "n_branches": n_branches, + "startup_s": startup, + "bodf_memory_mb": BODF.nbytes / 1024 / 1024, + "focal_bus": focal_bus, + "focal_degree": focal_degree, + "median_bus": median_bus, + "median_degree": median_degree, + "sweeps": {}, + } + + for bus_label, bus_id in [("max_degree", focal_bus), ("median_degree", median_bus)]: + print(f"\n--- Bus: {bus_id} ({bus_label}, degree={degrees[bus_id]}) ---") + for h in hop_range: + gc.collect() + + # Step 2: BFS scoping + t_scope = time.perf_counter() + scoped = bfs_scoped_branches(n, bus_id, h, branch_idx) + scope_elapsed = time.perf_counter() - t_scope + + k = len(scoped) + n1_count = k + n2_count = k * (k - 1) // 2 + n3_count = k * (k - 1) * (k - 2) // 6 + + print( + f"\n h={h}: {k} scoped branches → " + f"N-1={n1_count}, N-2={n2_count:,}, N-3={n3_count:,}" + ) + + sweep_result: dict = { + "h": h, + "scoped_branches": k, + "n1_combinations": n1_count, + "n2_combinations": n2_count, + "n3_combinations": n3_count, + "scope_us": scope_elapsed * 1e6, + } + + # N-1 sweep + t0 = time.perf_counter() + _, n1_violations = sweep_n1(BODF, p0, s_nom, scoped) + n1_elapsed = time.perf_counter() - t0 + sweep_result["n1_ms"] = n1_elapsed * 1000 + sweep_result["n1_violations"] = n1_violations + print(f" N-1: {n1_elapsed * 1000:.3f} ms, {n1_violations} violations") + + # N-2 sweep + if n2_count <= 500_000: + t0 = time.perf_counter() + _, n2_violations = sweep_n2(BODF, p0, s_nom, scoped) + n2_elapsed = time.perf_counter() - t0 + sweep_result["n2_ms"] = n2_elapsed * 1000 + sweep_result["n2_violations"] = n2_violations + print( + f" N-2: {n2_elapsed * 1000:.3f} ms ({n2_count:,} combos), " + f"{n2_violations} violations" + ) + else: + sweep_result["n2_ms"] = None + sweep_result["n2_note"] = f"Skipped: {n2_count:,} combos exceeds 500k limit" + print(f" N-2: SKIPPED ({n2_count:,} combos)") + + # N-3 sweep + max_n3 = 50_000 + if n3_count <= max_n3: + t0 = time.perf_counter() + n3_actual, n3_violations = sweep_n3(BODF, p0, s_nom, scoped) + n3_elapsed = time.perf_counter() - t0 + sweep_result["n3_ms"] = n3_elapsed * 1000 + sweep_result["n3_violations"] = n3_violations + sweep_result["n3_actual"] = n3_actual + print( + f" N-3: {n3_elapsed * 1000:.3f} ms ({n3_actual:,} combos), " + f"{n3_violations} violations" + ) + elif n3_count <= 500_000: + # Run capped + t0 = time.perf_counter() + n3_actual, n3_violations = sweep_n3(BODF, p0, s_nom, scoped, max_combos=max_n3) + n3_elapsed = time.perf_counter() - t0 + sweep_result["n3_ms"] = n3_elapsed * 1000 + sweep_result["n3_violations"] = n3_violations + sweep_result["n3_actual"] = n3_actual + sweep_result["n3_capped"] = True + print( + f" N-3: {n3_elapsed * 1000:.3f} ms ({n3_actual:,}/{n3_count:,} combos, capped), " + f"{n3_violations} violations" + ) + else: + sweep_result["n3_ms"] = None + sweep_result["n3_note"] = f"Skipped: {n3_count:,} combos exceeds limit" + print(f" N-3: SKIPPED ({n3_count:,} combos)") + + # Total user-perceived latency (scope + all sweeps) + total_ms = sweep_result["scope_us"] / 1000 + sweep_result["n1_ms"] + if sweep_result.get("n2_ms") is not None: + total_ms += sweep_result["n2_ms"] + if sweep_result.get("n3_ms") is not None: + total_ms += sweep_result["n3_ms"] + sweep_result["total_ms"] = total_ms + print(f" TOTAL user latency: {total_ms:.1f} ms") + + result["sweeps"][f"{bus_label}_h{h}"] = sweep_result + + return result + + +def main(): + all_results = {} + + for label, path in NETWORKS.items(): + hop_range = [1, 2, 3, 4] + all_results[label] = run_sweep(label, path, hop_range) + + # Summary table + print("\n" + "=" * 60) + print("SUMMARY: Total user-perceived latency (ms)") + print("=" * 60) + print( + f"{'Network':<12} {'Bus':<14} {'h':>2} {'Branches':>9} " + f"{'N-1':>8} {'N-2':>10} {'N-3':>12} {'Total':>10}" + ) + print("-" * 80) + + for label, data in all_results.items(): + for key, sweep in data["sweeps"].items(): + bus_type = "max-deg" if "max_degree" in key else "median" + n1_str = f"{sweep['n1_ms']:.1f}ms" + n2_str = f"{sweep['n2_ms']:.1f}ms" if sweep.get("n2_ms") is not None else "skip" + n3_str = f"{sweep['n3_ms']:.1f}ms" if sweep.get("n3_ms") is not None else "skip" + total_str = f"{sweep['total_ms']:.1f}ms" + print( + f"{label:<12} {bus_type:<14} {sweep['h']:>2} {sweep['scoped_branches']:>9} " + f"{n1_str:>8} {n2_str:>10} {n3_str:>12} {total_str:>10}" + ) + + output_path = Path(__file__).parent / "sweep_results.json" + with open(output_path, "w") as f: + json.dump(all_results, f, indent=2, default=str) + print(f"\nResults written to {output_path}") + + return all_results + + +if __name__ == "__main__": + main() diff --git a/evaluations/pypsa/tests/latency_bench/bench_results.json b/evaluations/pypsa/tests/latency_bench/bench_results.json new file mode 100644 index 00000000..c6039fa1 --- /dev/null +++ b/evaluations/pypsa/tests/latency_bench/bench_results.json @@ -0,0 +1,350 @@ +{ + "b1_baseline_decomposition": { + "status": "pass", + "details": { + "n_buses": 10000, + "n_lines": 9726, + "dcpf": { + "run_times": [ + 20.243536527000288, + 19.846716786999878, + 19.981075255000178 + ], + "median_s": 19.981075255000178, + "min_s": 19.846716786999878, + "max_s": 20.243536527000288, + "peak_memory_mb": 2098.688674926758 + }, + "dcopf": { + "run_times": [ + 206.30118028300012, + 258.907689313, + 208.33953027799953 + ], + "median_s": 208.33953027799953, + "min_s": 206.30118028300012, + "max_s": 258.907689313, + "peak_memory_mb": 4402.734807014465 + } + }, + "errors": [], + "wall_clock_seconds": 736.486807929 + }, + "b2_linopy_build_vs_solve": { + "status": "pass", + "details": { + "case39": { + "n_buses": 39, + "build": { + "run_times": [ + 0.7670494239991967, + 0.7637086499998986, + 0.7626787899998817 + ], + "median_s": 0.7637086499998986, + "min_s": 0.7626787899998817, + "max_s": 0.7670494239991967, + "peak_memory_mb": 0.9084663391113281 + }, + "solve": { + "run_times": [ + 0.25386789000003773, + 0.2516089849996206, + 0.25066682699980447 + ], + "median_s": 0.2516089849996206, + "min_s": 0.25066682699980447, + "max_s": 0.25386789000003773, + "peak_memory_mb": 0.30121326446533203 + }, + "solve_direct": { + "run_times": [ + 0.296785534999799, + 0.2966432379998878, + 0.2955673239994212 + ], + "median_s": 0.2966432379998878, + "min_s": 0.2955673239994212, + "max_s": 0.296785534999799, + "peak_memory_mb": 0.35964488983154297 + } + }, + "case2000": { + "n_buses": 2000, + "build": { + "run_times": [ + 4.574094190000324, + 4.582622500999605, + 4.616188062999754 + ], + "median_s": 4.582622500999605, + "min_s": 4.574094190000324, + "max_s": 4.616188062999754, + "peak_memory_mb": 191.18137454986572 + }, + "solve": { + "run_times": [ + 1.404422911000438, + 1.3331499070000064, + 1.3149934140001278 + ], + "median_s": 1.3331499070000064, + "min_s": 1.3149934140001278, + "max_s": 1.404422911000438, + "peak_memory_mb": 214.49903392791748 + }, + "solve_direct": { + "run_times": [ + 1.366278002999934, + 1.3395256479998352, + 1.351738106999619 + ], + "median_s": 1.351738106999619, + "min_s": 1.3395256479998352, + "max_s": 1.366278002999934, + "peak_memory_mb": 217.63275051116943 + } + }, + "case10000": { + "n_buses": 10000, + "build": { + "run_times": [ + 21.839490798000043 + ], + "median_s": 21.839490798000043, + "min_s": 21.839490798000043, + "max_s": 21.839490798000043, + "peak_memory_mb": 2109.7349882125854 + }, + "solve": { + "run_times": [ + 187.57658142199944 + ], + "median_s": 187.57658142199944, + "min_s": 187.57658142199944, + "max_s": 187.57658142199944, + "peak_memory_mb": 4351.03408908844 + }, + "solve_direct": { + "run_times": [ + 187.1955889180008 + ], + "median_s": 187.1955889180008, + "min_s": 187.1955889180008, + "max_s": 187.1955889180008, + "peak_memory_mb": 4365.304967880249 + } + } + }, + "errors": [], + "wall_clock_seconds": 448.3516682919999 + }, + "b3_incremental_resolv": { + "status": "pass", + "details": { + "outage_lines": [ + "L1904", + "L972", + "L1089", + "L1058", + "L1481" + ], + "strategy_a_full_rebuild": { + "total_times": [ + 28.82197279399861, + 28.593475912999565, + 28.644862785999976 + ], + "median_total_s": 28.644862785999976, + "median_per_outage_s": 5.728972557199995 + }, + "strategy_b_model_mod": { + "total_times": [ + 6.411676282998997, + 6.420535719001236, + 6.717397985000389 + ], + "median_total_s": 6.420535719001236, + "median_per_outage_s": 1.2841071438002474, + "note": "Re-solve only (no model rebuild), but constraint bounds unchanged \u2014 Linopy constraints are immutable so this measures solve_model() reuse" + }, + "strategy_c_warm_start": { + "total_times": [ + 36.37367311700018, + 28.989366076000806, + 28.630816728000354 + ], + "median_total_s": 28.989366076000806, + "median_per_outage_s": 5.797873215200161 + } + }, + "errors": [], + "wall_clock_seconds": 220.5284649109999 + }, + "b4_scaling_curve": { + "status": "pass", + "details": { + "raw_networks": { + "case39": { + "n_buses": 39, + "dcpf_median_s": 0.18109021399959602, + "dcopf_median_s": 1.006524053999783, + "dcpf_times": [ + 0.18214093299957312, + 0.1789842679991125, + 0.18109021399959602 + ], + "dcopf_times": [ + 1.006524053999783, + 1.0019840199993268, + 1.0100220779995652 + ] + }, + "case2000": { + "n_buses": 2000, + "dcpf_median_s": 3.9469289160006156, + "dcopf_median_s": 5.893824574000064, + "dcpf_times": [ + 3.9469289160006156, + 3.9505185639991396, + 3.904667306000192 + ], + "dcopf_times": [ + 5.893824574000064, + 5.9828909970001405, + 5.883564677999857 + ] + }, + "case10000": { + "n_buses": 10000, + "dcpf_median_s": 20.08870312900035, + "dcopf_median_s": 206.57085705099962, + "dcpf_times": [ + 20.08870312900035, + 20.041710145999787, + 20.266506617000232 + ], + "dcopf_times": [ + 215.6477424019995, + 205.85439438699996, + 206.57085705099962 + ] + } + }, + "clustered_networks": { + "cluster_1000": { + "error": "Optional dependency 'sklearn' not found.Install via 'conda install -c conda-forge scikit-learn' or 'pip install scikit-learn'" + }, + "cluster_500": { + "error": "Optional dependency 'sklearn' not found.Install via 'conda install -c conda-forge scikit-learn' or 'pip install scikit-learn'" + }, + "cluster_200": { + "error": "Optional dependency 'sklearn' not found.Install via 'conda install -c conda-forge scikit-learn' or 'pip install scikit-learn'" + }, + "cluster_100": { + "error": "Optional dependency 'sklearn' not found.Install via 'conda install -c conda-forge scikit-learn' or 'pip install scikit-learn'" + } + }, + "scaling_table": [ + { + "source": "case39", + "n_buses": 39, + "dcpf_s": 0.18109021399959602, + "dcopf_s": 1.006524053999783 + }, + { + "source": "case2000", + "n_buses": 2000, + "dcpf_s": 3.9469289160006156, + "dcopf_s": 5.893824574000064 + }, + { + "source": "case10000", + "n_buses": 10000, + "dcpf_s": 20.08870312900035, + "dcopf_s": 206.57085705099962 + } + ] + }, + "errors": [], + "wall_clock_seconds": 724.6861316600007 + }, + "b5_contingency_throughput": { + "status": "pass", + "details": { + "bodf_precompute": { + "case39": { + "n_buses": 39, + "bodf_shape": [ + 46, + 46 + ], + "compute_seconds": 0.12657858100010344, + "peak_memory_mb": 0.2194833755493164 + }, + "case2000": { + "n_buses": 2000, + "bodf_shape": [ + 3206, + 3206 + ], + "compute_seconds": 1.3171714130003238, + "peak_memory_mb": 249.003098487854 + }, + "case10000": { + "n_buses": 10000, + "bodf_shape": [ + 12706, + 12706 + ], + "compute_seconds": 17.099238961000083, + "peak_memory_mb": 5935.944363594055 + } + }, + "vectorized_n1_2000": { + "n_branches": 3206, + "elapsed_ms": 19.387384999390633, + "matrix_shape": [ + 3206, + 3206 + ] + }, + "single_contingency_2000": { + "n_samples": 100, + "median_us": 2.94050050797523, + "mean_us": 3.2535100217501167, + "max_us": 28.64399993995903 + }, + "violation_detection_2000": { + "elapsed_ms": 14.075312000386475, + "total_violations": 21754, + "contingencies_with_violations": 45 + }, + "resolv_baseline_39": { + "n_contingencies": 35, + "total_s": 1.9882263399986186, + "per_contingency_ms": 54.381644999921264 + }, + "bodf_n1_39": { + "n_branches": 46, + "elapsed_us": 9.907999810820911, + "speedup_vs_resolv": 200668.79067026218 + }, + "vectorized_n1_10k": { + "n_branches": 12706, + "bodf_compute_s": 15.690467585999613, + "bodf_memory_mb": 5935.947803497314, + "all_n1_ms": 235.79022000012628, + "single_contingency_median_us": 8.792000244284282 + }, + "n2_woodbury_2000": { + "n_pairs": 100, + "total_ms": 0.6488039998657769, + "per_pair_us": 6.488039998657769, + "note": "First-order superposition (not exact Woodbury inverse update)" + } + }, + "errors": [], + "wall_clock_seconds": 61.05912988300042 + } +} diff --git a/evaluations/pypsa/tests/latency_bench/sweep_results.json b/evaluations/pypsa/tests/latency_bench/sweep_results.json new file mode 100644 index 00000000..acab4797 --- /dev/null +++ b/evaluations/pypsa/tests/latency_bench/sweep_results.json @@ -0,0 +1,421 @@ +{ + "case39": { + "n_buses": 39, + "n_branches": 46, + "startup_s": 1.3533825919994342, + "bodf_memory_mb": 0.016143798828125, + "focal_bus": "16", + "focal_degree": 5, + "median_bus": "28", + "median_degree": 2, + "sweeps": { + "max_degree_h1": { + "h": 1, + "scoped_branches": 10, + "n1_combinations": 10, + "n2_combinations": 45, + "n3_combinations": 120, + "scope_us": 2430.9830005222466, + "n1_ms": 0.0524199995197705, + "n1_violations": 13, + "n2_ms": 0.030066999897826463, + "n2_violations": 114, + "n3_ms": 0.07093399926816346, + "n3_violations": 439, + "n3_actual": 120, + "total_ms": 2.584403999208007 + }, + "max_degree_h2": { + "h": 2, + "scoped_branches": 16, + "n1_combinations": 16, + "n2_combinations": 120, + "n3_combinations": 560, + "scope_us": 2244.050000626885, + "n1_ms": 0.05648700062010903, + "n1_violations": 18, + "n2_ms": 0.060244000451348256, + "n2_violations": 259, + "n3_ms": 0.2985760002047755, + "n3_violations": 1740, + "n3_actual": 560, + "total_ms": 2.6593570019031176 + }, + "max_degree_h3": { + "h": 3, + "scoped_branches": 23, + "n1_combinations": 23, + "n2_combinations": 253, + "n3_combinations": 1771, + "scope_us": 2271.5519999110256, + "n1_ms": 0.055705000704620034, + "n1_violations": 20, + "n2_ms": 0.07957100024214014, + "n2_violations": 417, + "n3_ms": 1.1526629996296833, + "n3_violations": 4181, + "n3_actual": 1771, + "total_ms": 3.559491000487469 + }, + "max_degree_h4": { + "h": 4, + "scoped_branches": 29, + "n1_combinations": 29, + "n2_combinations": 406, + "n3_combinations": 3654, + "scope_us": 2245.802999823354, + "n1_ms": 0.05552499987970805, + "n1_violations": 21, + "n2_ms": 0.10322500020265579, + "n2_violations": 563, + "n3_ms": 2.0897180002066307, + "n3_violations": 7356, + "n3_actual": 3654, + "total_ms": 4.494271000112349 + }, + "median_degree_h1": { + "h": 1, + "scoped_branches": 5, + "n1_combinations": 5, + "n2_combinations": 10, + "n3_combinations": 10, + "scope_us": 2269.939000143495, + "n1_ms": 0.05215900000621332, + "n1_violations": 2, + "n2_ms": 0.020850000510108657, + "n2_violations": 9, + "n3_ms": 0.01672200050961692, + "n3_violations": 14, + "n3_actual": 10, + "total_ms": 2.359670001169434 + }, + "median_degree_h2": { + "h": 2, + "scoped_branches": 7, + "n1_combinations": 7, + "n2_combinations": 21, + "n3_combinations": 35, + "scope_us": 2343.137000025308, + "n1_ms": 0.053030999879410956, + "n1_violations": 2, + "n2_ms": 0.022823000108473934, + "n2_violations": 10, + "n3_ms": 0.02448699979140656, + "n3_violations": 21, + "n3_actual": 35, + "total_ms": 2.4434779998045997 + }, + "median_degree_h3": { + "h": 3, + "scoped_branches": 11, + "n1_combinations": 11, + "n2_combinations": 55, + "n3_combinations": 165, + "scope_us": 2215.877000708133, + "n1_ms": 0.053591000323649496, + "n1_violations": 2, + "n2_ms": 0.030637999770988245, + "n2_violations": 19, + "n3_ms": 0.09861600028671091, + "n3_violations": 94, + "n3_actual": 165, + "total_ms": 2.3987220010894816 + }, + "median_degree_h4": { + "h": 4, + "scoped_branches": 18, + "n1_combinations": 18, + "n2_combinations": 153, + "n3_combinations": 816, + "scope_us": 2224.312000180362, + "n1_ms": 0.05408299966802588, + "n1_violations": 10, + "n2_ms": 0.05099700047139777, + "n2_violations": 171, + "n3_ms": 0.5111379996378673, + "n3_violations": 1385, + "n3_actual": 816, + "total_ms": 2.8405299999576528 + } + } + }, + "case2000": { + "n_buses": 2000, + "n_branches": 3206, + "startup_s": 2.479244082999685, + "bodf_memory_mb": 78.41824340820312, + "focal_bus": "7087", + "focal_degree": 17, + "median_bus": "3109", + "median_degree": 3, + "sweeps": { + "max_degree_h1": { + "h": 1, + "scoped_branches": 30, + "n1_combinations": 30, + "n2_combinations": 435, + "n3_combinations": 4060, + "scope_us": 24459.784000100626, + "n1_ms": 0.3004489999511861, + "n1_violations": 0, + "n2_ms": 5.2928129998690565, + "n2_violations": 3, + "n3_ms": 106.23766900062037, + "n3_violations": 99, + "n3_actual": 4060, + "total_ms": 136.29071500054124 + }, + "max_degree_h2": { + "h": 2, + "scoped_branches": 57, + "n1_combinations": 57, + "n2_combinations": 1596, + "n3_combinations": 29260, + "scope_us": 24812.61200045992, + "n1_ms": 0.5435800003397162, + "n1_violations": 0, + "n2_ms": 33.11201999986224, + "n2_violations": 4, + "n3_ms": 761.6420920003293, + "n3_violations": 212, + "n3_actual": 29260, + "total_ms": 820.1103040009912 + }, + "max_degree_h3": { + "h": 3, + "scoped_branches": 116, + "n1_combinations": 116, + "n2_combinations": 6670, + "n3_combinations": 253460, + "scope_us": 27985.461000753276, + "n1_ms": 1.0106540003107511, + "n1_violations": 0, + "n2_ms": 134.3134229991847, + "n2_violations": 15, + "n3_ms": 1240.3260499995667, + "n3_violations": 141, + "n3_actual": 50000, + "n3_capped": true, + "total_ms": 1403.6355879998155 + }, + "max_degree_h4": { + "h": 4, + "scoped_branches": 250, + "n1_combinations": 250, + "n2_combinations": 31125, + "n3_combinations": 2573000, + "scope_us": 23346.506000052614, + "n1_ms": 1.8315890001758817, + "n1_violations": 0, + "n2_ms": 567.9275069996947, + "n2_violations": 33, + "n3_ms": null, + "n3_note": "Skipped: 2,573,000 combos exceeds limit", + "total_ms": 593.1056019999232 + }, + "median_degree_h1": { + "h": 1, + "scoped_branches": 4, + "n1_combinations": 4, + "n2_combinations": 6, + "n3_combinations": 4, + "scope_us": 23006.62299967371, + "n1_ms": 0.0845200002004276, + "n1_violations": 0, + "n2_ms": 0.07555299998784903, + "n2_violations": 1, + "n3_ms": 0.048892999984673224, + "n3_violations": 2, + "n3_actual": 4, + "total_ms": 23.21558899984666 + }, + "median_degree_h2": { + "h": 2, + "scoped_branches": 7, + "n1_combinations": 7, + "n2_combinations": 21, + "n3_combinations": 35, + "scope_us": 23739.299999760988, + "n1_ms": 0.10799400024552597, + "n1_violations": 0, + "n2_ms": 0.22514600004797103, + "n2_violations": 1, + "n3_ms": 0.3425189997869893, + "n3_violations": 5, + "n3_actual": 35, + "total_ms": 24.414958999841474 + }, + "median_degree_h3": { + "h": 3, + "scoped_branches": 10, + "n1_combinations": 10, + "n2_combinations": 45, + "n3_combinations": 120, + "scope_us": 23738.82899973978, + "n1_ms": 0.13456499982567038, + "n1_violations": 0, + "n2_ms": 0.41603799945733044, + "n2_violations": 13, + "n3_ms": 1.0973380003633793, + "n3_violations": 76, + "n3_actual": 120, + "total_ms": 25.38676999938616 + }, + "median_degree_h4": { + "h": 4, + "scoped_branches": 17, + "n1_combinations": 17, + "n2_combinations": 136, + "n3_combinations": 680, + "scope_us": 23169.691000475723, + "n1_ms": 0.16644500010443153, + "n1_violations": 0, + "n2_ms": 1.0801159996844945, + "n2_violations": 24, + "n3_ms": 11.490943000353582, + "n3_violations": 297, + "n3_actual": 680, + "total_ms": 35.90719500061823 + } + } + }, + "case10000": { + "n_buses": 10000, + "n_branches": 12706, + "startup_s": 23.684089688000313, + "bodf_memory_mb": 1231.7080383300781, + "focal_bus": "13303", + "focal_degree": 20, + "median_bus": "50289", + "median_degree": 2, + "sweeps": { + "max_degree_h1": { + "h": 1, + "scoped_branches": 44, + "n1_combinations": 44, + "n2_combinations": 946, + "n3_combinations": 13244, + "scope_us": 88434.11599991668, + "n1_ms": 1.1201010001968825, + "n1_violations": 98567, + "n2_ms": 63.73670399989351, + "n2_violations": 2097969, + "n3_ms": 1148.7757380000403, + "n3_violations": 29004200, + "n3_actual": 13244, + "total_ms": 1302.0666590000474 + }, + "max_degree_h2": { + "h": 2, + "scoped_branches": 100, + "n1_combinations": 100, + "n2_combinations": 4950, + "n3_combinations": 161700, + "scope_us": 90855.52499982441, + "n1_ms": 4.512557999987621, + "n1_violations": 211460, + "n2_ms": 323.0187940007454, + "n2_violations": 9795074, + "n3_ms": 4383.238290000008, + "n3_violations": 98292755, + "n3_actual": 50000, + "n3_capped": true, + "total_ms": 4801.625167000566 + }, + "max_degree_h3": { + "h": 3, + "scoped_branches": 170, + "n1_combinations": 170, + "n2_combinations": 14365, + "n3_combinations": 804440, + "scope_us": 88247.68099930225, + "n1_ms": 6.842452000455523, + "n1_violations": 342655, + "n2_ms": 934.4455789996573, + "n2_violations": 25833981, + "n3_ms": null, + "n3_note": "Skipped: 804,440 combos exceeds limit", + "total_ms": 1029.535711999415 + }, + "max_degree_h4": { + "h": 4, + "scoped_branches": 251, + "n1_combinations": 251, + "n2_combinations": 31375, + "n3_combinations": 2604125, + "scope_us": 91505.3730004729, + "n1_ms": 10.505692000151612, + "n1_violations": 511852, + "n2_ms": 2054.7557039999447, + "n2_violations": 57773737, + "n3_ms": null, + "n3_note": "Skipped: 2,604,125 combos exceeds limit", + "total_ms": 2156.766769000569 + }, + "median_degree_h1": { + "h": 1, + "scoped_branches": 7, + "n1_combinations": 7, + "n2_combinations": 21, + "n3_combinations": 35, + "scope_us": 92327.03300040157, + "n1_ms": 0.22007699953974225, + "n1_violations": 13977, + "n2_ms": 0.5352640000637621, + "n2_violations": 36459, + "n3_ms": 0.9040329996423679, + "n3_violations": 51660, + "n3_actual": 35, + "total_ms": 93.98640699964744 + }, + "median_degree_h2": { + "h": 2, + "scoped_branches": 15, + "n1_combinations": 15, + "n2_combinations": 105, + "n3_combinations": 455, + "scope_us": 88440.56999987515, + "n1_ms": 0.36548200023389654, + "n1_violations": 27517, + "n2_ms": 4.3344309997337405, + "n2_violations": 154163, + "n3_ms": 39.33558599965181, + "n3_violations": 525167, + "n3_actual": 455, + "total_ms": 132.4760689994946 + }, + "median_degree_h3": { + "h": 3, + "scoped_branches": 27, + "n1_combinations": 27, + "n2_combinations": 351, + "n3_combinations": 2925, + "scope_us": 91004.9189997153, + "n1_ms": 0.6841960002930136, + "n1_violations": 54598, + "n2_ms": 24.539877999814053, + "n2_violations": 633318, + "n3_ms": 261.1618829996587, + "n3_violations": 4687833, + "n3_actual": 2925, + "total_ms": 377.39087599948107 + }, + "median_degree_h4": { + "h": 4, + "scoped_branches": 47, + "n1_combinations": 47, + "n2_combinations": 1081, + "n3_combinations": 16215, + "scope_us": 89092.22500005853, + "n1_ms": 1.1956349999309168, + "n1_violations": 97478, + "n2_ms": 72.24840700018831, + "n2_violations": 2056543, + "n3_ms": 1402.9189000002589, + "n3_violations": 28242178, + "n3_actual": 16215, + "total_ms": 1565.4551670004366 + } + } + } +} diff --git a/phase2-research/README.md b/phase2-research/README.md new file mode 100644 index 00000000..19fff251 --- /dev/null +++ b/phase2-research/README.md @@ -0,0 +1,29 @@ +# Phase 2 Research — State Estimation Investigation + +This directory contains the completed state estimation (SE) investigation +conducted as groundwork for Phase 2. It evaluates SE capabilities across all +six Phase 1 tools and surveys the broader open-source landscape. + +**Key finding:** None of the six evaluated tools provide production-ready SE +for transmission-scale grids. Phase 2 will require dedicated SE tooling work. + +## Per-Tool Findings + +| File | Tool | Summary | +|------|------|---------| +| `gridcal-state-estimation-findings.md` | GridCal | Native WLS SE; scaling limits at large networks | +| `matpower-state-estimation-findings.md` | MATPOWER | SE in extras (dormant); requires MATLAB | +| `pandapower-state-estimation-findings.md` | pandapower | Native WLS SE; distribution-focused | +| `powermodels-state-estimation-findings.md` | PowerModels.jl | Ecosystem SE package (PowerModelsStateEstimation.jl) | +| `powersimulations-state-estimation-findings.md` | PowerSimulations.jl | No SE capability | +| `pypsa-state-estimation-findings.md` | PyPSA | No SE capability | + +## Landscape Analysis + +| File | Scope | +|------|-------| +| `state-estimation-investigation.md` | Master synthesis across all tools + recommendations | +| `se-landscape-academic.md` | Academic SE research (PMU, hybrid, ML-based) | +| `se-landscape-python.md` | Python SE tool survey (power-grid-model, ANDES, etc.) | +| `se-landscape-powsybl.md` | PowSyBl assessment (confirms no SE capability) | +| `se-landscape-remaining.md` | Non-Python tools (GridPACK, InterPSS, PSAT, DPsim) | diff --git a/phase2-research/gridcal-state-estimation-findings.md b/phase2-research/gridcal-state-estimation-findings.md new file mode 100644 index 00000000..a20b0365 --- /dev/null +++ b/phase2-research/gridcal-state-estimation-findings.md @@ -0,0 +1,182 @@ +# GridCal State Estimation Investigation + +## Summary + +GridCal (now VeraGrid/VeraGridEngine) implements a WLS-based state estimation framework +with four solver algorithms, support for multiple measurement types, and nascent +observability analysis / pseudo-measurement augmentation. The implementation follows +Monticelli's textbook approach but has significant gaps: bad data detection is coded +but disabled, the decoupled solver has known issues, Issue #419 (observability + +pseudo-measurements) remains open with only a partial PR, and there is no time-series +SE driver. The feature is functional for small textbook-style grids but not +production-grade. + +## Algorithm Details + +The `StateEstimationDriver` class (in `state_stimation_driver.py` -- note the typo in +the filename) delegates to four WLS solver implementations in `state_estimation.py`: + +| Solver | Function | Notes | +|--------|----------|-------| +| Newton-Raphson | `solve_se_nr()` | Solves (H'WH)^-1 H'W(z-h) directly; simplest impl | +| Levenberg-Marquardt | `solve_se_lm()` | Regularized (G'G + mu*I)dx = G'g with adaptive damping | +| Gauss-Newton | `solve_se_gauss_newton()` | Step-size limiting (+-0.3 rad angles, +-0.2 pu voltage) + regularization | +| Decoupled LU | `decoupled_state_estimation()` | P-theta / Q-V decoupling via `splu`; uses relaxed tolerance (100x tol) | + +All solvers minimize `(z - h(x))' W (z - h(x))` where `W = diag(1/sigma^2)`. + +**Measurement types supported:** +- Bus: P injection, Q injection, Vm, Va +- Generator: Pg, Qg +- Branch: Pf, Pt, Qf, Qt (from/to), If, It (current magnitude, handled as squared internally) + +**Jacobian construction** (`Jacobian_SE` function): Uses `dSbus_dV_matpower`, +`dSbr_dV_matpower`, `dIbr_dV_matpower` to build the H matrix. Current measurements +use `d|I|^2 = 2*Re(diag(conj(I))*dI/dx)`. + +**Configuration options** (`StateEstimationOptions`): +- `solver` (default NR), `tol` (1e-8), `max_iter` (100) +- `prefer_correct` vs deletion for bad data +- `c_threshold` (4.0) for bad data confidence +- `fixed_slack` -- must be False for convergence (documented in Issue #443) +- `run_observability_analyis`, `add_pseudo_measurements`, `pseudo_meas_std` +- `run_measurement_profiling` + +**Reference:** Monticelli, "State Estimation in Electric Power Systems." + +## Observability Analysis + +Observability analysis is implemented as a two-phase approach in the driver: + +1. `check_for_observability_and_return_unobservable_buses()` -- identifies unobservable + buses and generates profiling data +2. `add_pseudo_measurements_for_unobservable_buses()` -- synthesizes measurements using + linearized power flow equations when `add_pseudo_measurements=True` + +**Current state:** The test file `test_observability_analysis_and_pseudo_meas.py` confirms +that the system can detect unobservable buses and that enabling pseudo-measurements +allows SE to converge on a 3-bus system. However, the test is incomplete: +- Multiple measurement definitions are commented out +- No validation of solution accuracy (only convergence checked) +- The test asserts unobservable buses remain flagged even after pseudo-measurements are + added, which seems contradictory (possibly intentional for diagnostic tracking) + +**Redundancy profiling:** Mentioned in Issue #419 requirements but not fully implemented. +The issue requests global and local redundancy mapping plus critical measurement +identification. + +## Issue #419 Status + +**Title:** [STATE-ESTIMATION] Observability analysis including redundancy information +for measurements & dealing with pseudo measurements + +- **State:** OPEN (as of 2026-03-27) +- **Created:** 2025-08-25 by AnkurArohi (collaborator) +- **Last comment:** 2025-08-26 + +**What was requested:** +1. Observability analysis with measurement profiling (global/local redundancy, criticality) +2. Pseudo-measurement generation at locations critical for SE convergence using + linearized power flow equations +3. Bad data identification and elimination with re-analysis + +**What was delivered:** +A branch (`419_state_estimation_obsevability_analysis`) with a single commit adding a +3-bus test and pseudo-measurement feature. The comparison page shows only 2 files changed. +The branch has not been merged. + +**Assessment:** The feature request is substantive (Monticelli/Abur textbook-level +capabilities) but the implementation effort appears minimal. The issue has been open for +7 months with no merge activity. + +## Known Limitations + +1. **Bad data detection disabled:** The b-test (Monticelli & Garcia 1983) is fully coded + but commented out in all solvers. Users cannot detect or remove bad measurements + automatically. + +2. **Decoupled solver broken:** SanPen confirmed in Issue #443 that "all solvers pass + [the 3-bus test] except the Decoupled_LU, which might ignore the `fixed_slack` + setting." + +3. **Unit scaling bugs (Issue #353, #443):** SE results were returned in per-unit while + power flow results used MVA, causing 100x discrepancies. Issue #353 (fixed 2025-04-01) + addressed input/output in MVA. Issue #443 (closed 2025-10-08) was a user confusion + but revealed the Sbase handling is fragile -- the results writer is shared with + power flow and "not consistently handled" per collaborator AnkurArohi. + +4. **No time-series SE:** `get_measurements_and_deviations()` ignores the time parameter + (`t=None` always). There is no `StateEstimationTimeSeriesDriver` equivalent. + +5. **Current measurement handling:** Code supports both squared and direct magnitudes but + defaults to squared. A comment notes direct approach is "more stable" but it is not + the default. + +6. **Critical measurements:** Flagged in code with "Do not delete" comment but not + properly handled in the bad data pipeline. + +7. **No PMU support:** Phase angle measurements (Va) are supported in the Jacobian but + there is no dedicated PMU device model or linear SE formulation. + +8. **Filename typo:** `state_stimation_driver.py` (missing 'e') -- minor but indicative + of limited review. + +9. **Observability + pseudo-measurements incomplete:** Issue #419 open, branch not merged, + test is a stub. + +10. **No multi-area SE:** Single-area estimation only; islands processed independently. + +## Recent Development Activity + +| Date | Event | Actor | +|------|-------|-------| +| 2025-03-31 | Issue #353 opened (MVA scaling) | SanPen | +| 2025-04-01 | Issue #353 closed | SanPen | +| 2025-08-25 | Issue #419 opened (observability + pseudo-meas) | AnkurArohi | +| 2025-08-26 | Branch with 3-bus test pushed for #419 | AnkurArohi | +| 2025-10-07 | Issue #443 opened (SE convergence/scaling) | Ferranbd (external) | +| 2025-10-08 | Issue #443 closed (user error, p.u. confusion) | SanPen | + +**Development pattern:** SE work is sporadic. Most activity comes from collaborator +AnkurArohi, with SanPen providing direction. No SE-related commits found in recent +(late 2025 / early 2026) history. The #419 branch has stalled for 7 months. + +**Bus factor concern:** SE development appears to depend on a single collaborator +(AnkurArohi) under the direction of the sole maintainer (SanPen). This compounds the +overall project bus factor of 1. + +## Production Readiness Assessment + +| Criterion | Status | Notes | +|-----------|--------|-------| +| Core WLS solver | Functional | NR, LM, GN work on textbook cases | +| Bad data detection | Not functional | Coded but commented out | +| Observability analysis | Partial | Detects unobservable buses; no redundancy profiling | +| Pseudo-measurements | Partial | Branch exists, not merged, minimal testing | +| Time-series SE | Missing | No driver exists | +| PMU/linear SE | Missing | No dedicated support | +| Decoupled solver | Broken | Fails tests per maintainer | +| Unit handling | Fragile | Fixed but shared code path with PF is a risk | +| Test coverage | Minimal | 3-bus textbook cases only | +| Documentation | Sparse | One-paragraph doc page + Monticelli reference | + +**Overall:** GridCal's SE is a textbook-quality reference implementation suitable for +educational use and small proof-of-concept work. It is not production-ready for real +grid operations. The missing bad data detection alone disqualifies it -- SE without +bad data handling is unusable on real measurement sets. The stalled Issue #419 and +single-developer dependency add further risk. + +## Sources + +- [VeraGrid GitHub Repository](https://github.com/SanPen/VeraGrid) +- [Issue #419: Observability analysis + pseudo measurements](https://github.com/SanPen/VeraGrid/issues/419) (OPEN) +- [Issue #443: State Estimation scaling](https://github.com/SanPen/VeraGrid/issues/443) (CLOSED 2025-10-08) +- [Issue #353: SE powers in MVA](https://github.com/SanPen/VeraGrid/issues/353) (CLOSED 2025-04-01) +- [Issue #139: SE example does not run](https://github.com/SanPen/VeraGrid/issues/139) (CLOSED 2021-11-22) +- [SE source: state_estimation.py](https://github.com/SanPen/VeraGrid/blob/master/src/VeraGridEngine/Simulations/StateEstimation/state_estimation.py) +- [SE driver: state_stimation_driver.py](https://github.com/SanPen/VeraGrid/blob/master/src/VeraGridEngine/Simulations/StateEstimation/state_stimation_driver.py) +- [SE example: state_estimation_run.py](https://github.com/SanPen/VeraGrid/blob/master/examples/state_estimation_run.py) +- [SE test: test_observability_analysis_and_pseudo_meas.py](https://github.com/SanPen/VeraGrid/blob/master/src/tests/StateEstimation/test_observability_analysis_and_pseudo_meas.py) +- [VeraGrid State Estimation Documentation](https://veragrid.readthedocs.io/en/stable/) +- [GridCal SE Module Docs](https://gridcal.readthedocs.io/en/latest/_modules/GridCalEngine/Simulations/StateEstimation/state_estimation.html) +- Monticelli, A. "State Estimation in Electric Power Systems" (referenced in GridCal docs) diff --git a/phase2-research/matpower-state-estimation-findings.md b/phase2-research/matpower-state-estimation-findings.md new file mode 100644 index 00000000..81b66ec2 --- /dev/null +++ b/phase2-research/matpower-state-estimation-findings.md @@ -0,0 +1,234 @@ +# MATPOWER State Estimation Investigation + +## Summary + +MATPOWER ships two separate state estimation (SE) implementations in its `extras/` +directory, both community-contributed and neither integrated with the modern MP-Core +(`mp.extension`) architecture introduced in MATPOWER 8. The newer `se/` package by +Rui Bo provides a classical Weighted Least Squares (WLS) estimator with observability +analysis and IEEE 14-bus test validation. The older `state_estimator/` by James S. +Thorp adds chi-squared bad data detection but is explicitly marked "under +construction." Neither module supports PMU measurements, robust estimation, or +large-scale network deployment. These are research/teaching tools, not +production-grade state estimators. + +## SE Extras Overview + +### `extras/se/` (mx-se, Rui Bo) + +Master repository: [MATPOWER/mx-se](https://github.com/MATPOWER/mx-se), included +in `matpower-extras` as a git subrepo. + +**Core functions:** + +| Function | Purpose | +|---|---| +| `run_se(casename, measure, idx, sigma, type_initialguess, V0)` | Top-level entry point; loads case, builds admittance matrices, calls `doSE` | +| `doSE(baseMVA, bus, gen, branch, Ybus, Yf, Yt, V0, ref, pv, pq, measure, idx, sigma)` | WLS Newton iteration engine (max 100 iterations) | +| `isobservable(H, pv, pq)` | Rank-based observability test on the Jacobian H; diagnoses unobservable variables | +| `checkDataIntegrity(...)` | Validates measurement/index/sigma consistency | +| `getV0(...)` | Generates initial voltage profile (flat start or from power flow) | +| `outputsesoln(...)` | Formats and prints SE solution | + +**Measurement types supported (8 categories):** +PF (branch active from), PT (branch active to), PG (generator active injection), +Va (voltage angle), QF (branch reactive from), QT (branch reactive to), +QG (generator reactive injection), Vm (voltage magnitude). + +**Test cases:** +- `test_se` -- general validation +- `test_se_14bus` -- IEEE 14-bus system with 39 measurements across all categories +- `test_se_14bus_err` -- error handling validation +- `case3bus_P6_6.m` -- minimal 3-bus test case + +**Documentation:** `se_intro.pdf` is included in the repository. No API reference +beyond MATLAB help text in function headers. + +### `extras/state_estimator/` (James S. Thorp) + +Older, simpler implementation bundled directly in matpower-extras. + +**Core functions:** + +| Function | Purpose | +|---|---| +| `runse(casedata, mpopt, fname, solvedcase)` | Runs Newton power flow first, then calls `state_est` | +| `state_est(branch, Ybus, Yf, Yt, Sbus, V0, ref, pv, pq, mpopt)` | WLS Newton estimator with chi-squared bad data detection (threshold 6.25) | + +**Key difference from mx-se:** This module runs a full power flow *first*, then +uses the PF solution to generate synthetic measurements with added noise for +state estimation. It produces comparison plots (PF vs SE) for voltage angles, +magnitudes, and power flows. This is a teaching/demonstration tool. + +## Algorithm Details + +Both implementations use the same fundamental algorithm: + +1. **Weighted Least Squares (WLS)** formulation: + minimize `(z - h(x))^T W (z - h(x))` where `z` = measurements, `h(x)` = + measurement model, `W` = diagonal weight matrix (`1/sigma^2`) + +2. **Newton-Raphson iteration:** construct Jacobian `H = dh/dx`, solve the normal + equation `(H^T W H) dx = H^T W (z - h(x))`, update state vector `x += dx` + +3. **Convergence:** iterate until residual change is below tolerance or max + iterations (100 for mx-se) + +4. **Observability check (mx-se only):** rank test on H matrix before solving; + if `rank(H) < n_states`, reports which variables lack measurement support + +5. **Bad data detection (state_estimator only):** chi-squared test with threshold + 6.25; iteratively removes suspect measurements and re-solves + +**Not supported by either module:** +- PMU / synchrophasor measurements (voltage/current phasors) +- Robust estimation (LAV, M-estimation, GM-estimation) +- Decoupled or fast-decoupled SE formulations +- Topology error detection +- Multi-area or distributed SE +- Real-time streaming measurement interfaces +- Sparse matrix optimizations for large networks + +## Documentation & Examples + +**mx-se (`extras/se/`):** +- `se_intro.pdf` provides a brief mathematical introduction +- MATLAB help text in each `.m` file with function signatures +- Three runnable test scripts serve as usage examples +- Reference docs hosted at matpower.org (e.g., [doSE](https://matpower.org/docs/ref/matpower5.0/extras/se/doSE.html), + [run_se](https://matpower.org/docs/ref/matpower5.0/extras/se/run_se.html), + [isobservable](https://matpower.org/docs/ref/matpower5.0/extras/se/isobservable.html)) +- No tutorial, no user guide beyond the intro PDF +- No README in the mx-se repository + +**state_estimator:** +- No dedicated documentation file +- MATLAB help text only +- `runse` generates comparison plots that serve as visual validation +- Referenced in [MATPOWER Extras appendix](https://matpower.app/manual/matpower/matpowerExtras.html) + +**Documentation quality overall:** Minimal. Adequate for a researcher who already +understands WLS state estimation theory, but insufficient for a newcomer. No +step-by-step tutorials, no API design docs, no discussion of when to use which +module. + +## Maintenance Status + +### mx-se Repository + +- **Created:** 2009 (initial commit from Rui Bo) +- **Total commits:** 19 +- **Last commit:** 2019-06-07 (URL updates) +- **Last substantive code change:** 2018-04-05 (argument order fix for `dSbus_dV`) +- **Stars:** 10, Forks: 1 +- **Open issues:** 3 (all filed 2024, none addressed): + - #1: "Fix problem in state estimation" + - #2: "add active & reactive load as measurements" + - #3: "possible issue with computation of branch flow estimates" + +The mx-se code has been **effectively dormant since 2019**. The three open issues +from 2024 suggest users are encountering bugs that are not being fixed. + +### matpower-extras Repository + +- **Latest release:** 8.1 (July 2025) +- **Total commits:** 255 +- The `se/` subrepo has not been pulled/updated since at least 2019 +- The `state_estimator/` code dates from the MATPOWER 5 era with no recent changes +- CI/CD workflows exist but focus on sdp_pf and other actively maintained extras + +### Conclusion on Maintenance + +Both SE modules are **legacy code in maintenance-only (or abandoned) status**. +They receive compatibility updates when MATPOWER core APIs change (e.g., the +2018 `dSbus_dV` argument reorder) but no feature development, no bug fixes for +reported issues, and no adaptation to MATPOWER 8's architecture. + +## Integration Quality + +### Relationship to MATPOWER Core + +Both SE modules operate as **standalone scripts** that happen to use MATPOWER's +data structures and utility functions: + +- They import case data via `loadcase()` and build admittance matrices via + `makeYbus()` from MATPOWER core +- They use MATPOWER's bus/gen/branch matrix conventions and internal indexing +- They do NOT use `mp.extension`, `mp.task`, or any MP-Core class hierarchy +- They do NOT register as extensions or plug into MATPOWER's solve pipeline +- They cannot be invoked via `run_pf()`, `run_opf()`, or any standard entry point + +### Legacy userfcn vs Modern mp.extension + +The `state_estimator/runse` function uses MATPOWER's `runpf()` internally, +inheriting whatever power flow solver is configured. However, neither module +uses the `add_userfcn` callback API either -- they are truly standalone. + +The modern `mp.extension` API (MATPOWER 8+) provides a clean path to integrate +SE as a first-class task type: +- A hypothetical `mp.task_se` could orchestrate data model -> network model -> + math model for state estimation +- `mp.dm_element` subclasses could represent measurement devices +- `mp.mm_element` subclasses could formulate the WLS objective and constraints +- Extensions could add PMU elements, topology processing, or bad data detection + +None of this integration exists. The SE extras remain at the MATPOWER 5-era +API level, requiring users to manually manage data flow between MATPOWER core +functions and the SE solver. + +### What Would Full Integration Look Like? + +A properly integrated SE module would: +1. Accept measurements as a data model layer alongside bus/gen/branch +2. Support `run_se('case14', measurements, mpopt)` as a top-level function +3. Use `mp.extension` to register SE-specific element types +4. Participate in MATPOWER's options system (`mpoption`) +5. Output results through MATPOWER's standard print/save pipeline +6. Be testable via MATPOWER's `t_run_tests` framework + +Currently, none of these integration points are implemented. + +## Production Readiness Assessment + +| Criterion | mx-se (`extras/se/`) | state_estimator | +|---|---|---| +| Algorithm | WLS Newton-Raphson | WLS Newton-Raphson | +| Observability analysis | Yes (rank-based) | No | +| Bad data detection | No | Yes (chi-squared, basic) | +| PMU support | No | No | +| Scalable to large networks | No (dense matrices) | No (dense matrices) | +| Validated test cases | IEEE 14-bus, 3-bus | IEEE 9-bus (default) | +| Active maintenance | No (dormant since 2019) | No (dormant since ~2013) | +| Open bugs | 3 unaddressed | Unknown | +| Documentation | Minimal (intro PDF + help text) | Minimal (help text only) | +| MP-Core integration | None | None | +| Production deployable | No | No | + +**Bottom line:** These SE extras are suitable for academic demonstrations and +small-network teaching exercises. They implement the textbook WLS algorithm +correctly for simple cases but lack the robustness, scalability, measurement +diversity, and maintenance commitment required for production or even serious +research use. For production-grade state estimation in MATLAB, users would need +to look at commercial tools (e.g., PSS/E, PowerWorld) or build a custom +implementation on top of MATPOWER's network model infrastructure. + +For the evaluation context: the existence of SE extras demonstrates that +MATPOWER's data structures and admittance matrix utilities provide a viable +foundation for building SE tools, but the extras themselves are not evidence +of mature SE capability. They are better characterized as community-contributed +examples that have not kept pace with MATPOWER's architectural evolution. + +## Sources + +- [MATPOWER/mx-se GitHub repository](https://github.com/MATPOWER/mx-se) -- 19 commits, 3 open issues, last commit 2019 +- [MATPOWER/matpower-extras GitHub repository](https://github.com/MATPOWER/matpower-extras) -- contributed/unsupported extras collection +- [MATPOWER Extras documentation](https://matpower.app/manual/matpower/matpowerExtras.html) -- official description of se and state_estimator modules +- [doSE function reference](https://matpower.org/docs/ref/matpower5.0/extras/se/doSE.html) +- [run_se function reference](https://matpower.org/docs/ref/matpower5.0/extras/se/run_se.html) +- [isobservable function reference](https://matpower.org/docs/ref/matpower5.0/extras/se/isobservable.html) +- [state_est function reference](https://matpower.org/docs/ref/matpower5.0/extras/state_estimator/state_est.html) +- [runse function reference](https://matpower.org/docs/ref/matpower5.0/extras/state_estimator/runse.html) +- [test_se_14bus function reference](https://matpower.org/docs/ref/matpower6.0/extras/se/test_se_14bus.html) +- [MATPOWER Extension API How-To](https://matpower.org/documentation/howto/extension.html) -- mp.extension class documentation +- [MATPOWER 8 Legacy Framework documentation](https://matpower.org/documentation/ref-manual/legacy/index.html) +- [MATPOWER 8.0 Release Notes](https://github.com/MATPOWER/matpower/blob/master/docs/relnotes/MATPOWER-Release-Notes-8.0.md) diff --git a/phase2-research/pandapower-state-estimation-findings.md b/phase2-research/pandapower-state-estimation-findings.md new file mode 100644 index 00000000..a68683a9 --- /dev/null +++ b/phase2-research/pandapower-state-estimation-findings.md @@ -0,0 +1,222 @@ +# pandapower State Estimation Investigation + +## Summary + +pandapower provides the most full-featured open-source Python state estimation module +available, with four algorithm families, bad data detection, zero-injection handling, +and a novel AF-WLS estimator for non-observable distribution grids. However, it remains +primarily an academic/research tool: the documentation itself warns that bad data +removal "is not very robust at this time," convergence problems surface on networks +above ~90 buses, there are no known production deployments, and three-phase SE is +unsupported. + +## Algorithms + +| Algorithm | Key | Description | Added | +|-----------|-----|-------------|-------| +| Weighted Least Squares | `wls` | Classical Newton-Gauss WLS. Baseline algorithm. Supports zero-injection constraints and current magnitude measurements. | v1.x | +| Iteratively Reweighted WLS | `irwls` | Robust estimator supporting WLS and SHGM (Schweppe-Huber Generalized M-estimator) weighting. Based on Mili et al. (1996). | v2.0.1 | +| Linear Programming | `lp` | LAV (Least Absolute Value) estimator. Independent of measurement weights, making it more robust to poorly scaled data. | v2.0.1 | +| Scipy Optimization | `opt` | Flexible framework supporting WLS, LAV, QL, and QC estimators via scipy.optimize. Documentation warns it "could collapse in some cases with flat start." | v2.0.1 | +| Allocation Factor WLS | `af-wls` | Designed specifically for non-observable distribution grids with sparse metering. Uses allocation factors to avoid pseudo-measurements. Based on IEEE paper by e2nIEE authors (IEEE TPWRS, 2024). | v3.0.0 | + +The default optimization method was changed from `OptAlgorithm` to `"Newton-CG"` in +v3.1.x. Most algorithms follow Abur & Exposito, *Power System State Estimation: +Theory and Implementation* (CRC Press, 2004). + +## Bad Data Detection + +Two complementary methods, combined in a single wrapper: + +1. **Chi-squared test** (`chi2_analysis()`): Detects *presence* of bad data in the + measurement set. Returns a boolean. Default false alarm probability: 0.05. + +2. **Largest normalized residual test** (`remove_bad_data()`): Identifies and removes + *specific* faulty measurements. Default threshold: `rn_max = 3.0`. + +**Maturity warning**: The documentation explicitly states: +> "The bad data removal is not very robust at this time. Please treat the results +> with caution!" + +Known issues: +- [#1451](https://github.com/e2nIEE/pandapower/issues/1451) (open since 2022-01): + `remove_bad_data()` fails with "linear algebra methods" error on some measurement + sets where `estimate()` alone succeeds. +- The chi-squared test can also fail when the underlying WLS estimation does not + converge, since it depends on a successful estimation run first. + +## Observability Analysis + +pandapower implements a basic observability check: + +- **Minimum measurement rule**: `m_min = 2n - k` (n = bus count, k = slack buses). + This is necessary but not sufficient -- isolated branches/islands can still make + the system unobservable even if the count is met. +- **Practical recommendation**: ~4n measurements for robust performance. +- **No formal topological observability analysis** (island detection, + observable island identification) exists as a standalone function. +- **AF-WLS workaround** (v3.0.0+): For distribution grids that fail the + observability criterion, AF-WLS can produce estimates without pseudo-measurements + by using allocation factors derived from the network topology and available + upstream measurements. +- **Ill-conditioning detection** (v3.0.0+): Matrix conditioning is now computed, + with a warning issued for ill-conditioned Jacobians -- a common symptom of + observability problems. + +## Scalability + +**Tested network sizes from documentation and community reports:** + +| Network | Buses | Result | +|---------|-------|--------| +| case9 | 9 | Converges correctly | +| case30 | 30 | Converges correctly | +| case39 | 39 | Converges correctly | +| case89pegase | 89 | Convergence problems reported (wrong solution point, ~4-5% voltage error, >50% reactive power error) | +| SimBench networks | ~1000+ | Failures reported ([#923](https://github.com/e2nIEE/pandapower/issues/923), [#2364](https://github.com/e2nIEE/pandapower/issues/2364)) | + +**Performance improvements in v3.1.2** (2025-06-16): +- Sparse matrix conversion for internal SE matrices +- Optimized Jacobian creation (skips computations for non-existing measurements) +- Reduced RAM usage +- Optimized merge computations for co-located measurements + +**Known scaling problems:** +- Measurement weight scaling across MW-to-kW ranges causes numerical ill-conditioning + ([openmod forum discussion](https://forum.openmod.org/t/convergence-problems-on-large-net-models-in-pandapower-state-estimation-module/2706)) +- Even with correct power flow results as warm start, the estimator can iterate + away from the correct solution on larger networks +- LAV estimator reported as "extremely slow" ([#1210](https://github.com/e2nIEE/pandapower/issues/1210)) + +No published benchmarks exist for SE execution time vs. network size. The power flow +solver handles 10,000+ bus networks, but SE appears untested beyond a few hundred buses +in practice. + +## Measurement Support + +### Conventional SCADA measurements + +| Type code | Measurement | Elements | Unit | +|-----------|-------------|----------|------| +| `v` | Voltage magnitude | bus | p.u. | +| `p` | Active power injection/flow | bus, line, trafo, trafo3w | MW | +| `q` | Reactive power injection/flow | bus, line, trafo, trafo3w | MVar | +| `i` | Current magnitude | line, trafo, trafo3w | kA | + +### PMU / phasor measurements + +| Type code | Measurement | Elements | +|-----------|-------------|----------| +| `va` | Voltage angle | bus | +| `ia` | Current angle | line, trafo, trafo3w | + +PMU-type measurements (`va`, `ia`) are supported through the same `create_measurement()` +API. There is a dedicated test file `test_pmu.py` in the test suite, though issue +[#2524](https://github.com/e2nIEE/pandapower/issues/2524) noted that `test_pmu_case14` +was incorrectly implemented (using case9 data instead of case14). This was closed but +suggests limited PMU testing rigor. + +### Three-phase SE + +**Not supported.** pandapower has three-phase power flow (`runpp_3ph`) for unbalanced +networks, but the state estimation module operates on the single-phase positive-sequence +equivalent only. There is no documented plan or open issue tracking three-phase SE +development. + +### Real SCADA data handling + +The `create_measurement()` function accepts arbitrary numeric values and standard +deviations, so real SCADA data can be fed in. The CIM/CGMES converter (v3.0.0+) can +extract measurements from CIM data models into `net.measurement`, providing a path +from utility data formats. However, no purpose-built SCADA ingestion pipeline or +real-time measurement interface exists. + +## Known Limitations + +### Open bugs (as of 2026-03-27) + +| Issue | Status | Description | +|-------|--------|-------------| +| [#2700](https://github.com/e2nIEE/pandapower/issues/2700) | Open | `zero_injection="no_inj_bus"` produces IndexError | +| [#1451](https://github.com/e2nIEE/pandapower/issues/1451) | Open (since 2022) | `remove_bad_data()` fails with linear algebra errors | +| [#2918](https://github.com/e2nIEE/pandapower/issues/2918) | Closed | numpy 1/2 compatibility in SE (fixed) | + +### Structural limitations + +1. **No topological observability analysis** -- only a measurement count heuristic +2. **No three-phase state estimation** -- single-phase positive-sequence only +3. **No DC state estimation** -- issue [#95](https://github.com/e2nIEE/pandapower/issues/95) requested this in 2018, never implemented +4. **Zero-injection handling is fragile** -- historically required "fake" zero measurements with high weights ([#243](https://github.com/e2nIEE/pandapower/issues/243)); automatic creation added in v3.0.0/v3.1.2 but still has bugs ([#2700](https://github.com/e2nIEE/pandapower/issues/2700)) +5. **Bus-bus switch handling** -- SE merges switched buses automatically but can lose measurements in the process ([#253](https://github.com/e2nIEE/pandapower/issues/253)) +6. **Disabled branch handling** -- branch mapping changes when branches are disabled, causing SE errors ([#248](https://github.com/e2nIEE/pandapower/issues/248)) +7. **Scaling sensitivity** -- measurement weight imbalances across orders of magnitude cause convergence failures on real-world-sized networks +8. **Bad data detection fragility** -- explicitly documented as "not very robust" + +## Recent Development Activity + +### v3.0.0 (2025-03-06) -- Major SE additions +- AF-WLS for non-observable distribution grids +- Zero-injection measurement creation in WLS +- Ill-conditioning detection and warning +- WLS flat-start divergence fix for highly loaded grids +- Current magnitude measurement handling fix +- Shunt element estimation results +- Power injection results fix +- CIM converter measurement extraction (`load`, `sgen`, `gen`, `shunt`, `ext_grid`, `ward`, `xward`) + +### v3.1.0-v3.1.2 (2025-05-26 to 2025-06-16) -- SE optimization focus +- Sparse matrix conversion for internal SE matrices +- RAM usage optimization +- Calculation speed-up +- Jacobian creation optimization (skip non-existing measurements) +- Debug mode for WLS iterations +- Multiple options for automatic zero-injection measurement creation +- AF-WLS bug fixes +- Automatic test creation bug fixes +- Default optimization method changed to Newton-CG + +### v3.2.0-v3.4.0 (2025-10-08 to 2026-02-09) -- No SE-specific changes +The v3.2.0, v3.3.0, and v3.4.0 releases focused on other areas (plotting, converters, +FACTS, lightsim2grid). One SE-adjacent fix: numpy 1/2 compatibility in SE ([#2918](https://github.com/e2nIEE/pandapower/issues/2918), closed 2026-03-14). + +### Development trajectory +SE received concentrated attention in v3.0.0-v3.1.2 (March-June 2025) but development +appears to have paused since. The AF-WLS paper and implementation represent the most +novel contribution. Core robustness issues (bad data detection fragility, large-network +convergence) remain unaddressed. + +## Production Readiness Assessment + +**Verdict: Research/prototyping tool, not production-ready for utility-grade SE.** + +| Criterion | Assessment | +|-----------|------------| +| Algorithm breadth | Strong -- 5 algorithm families covering classical and robust approaches | +| AF-WLS innovation | Notable -- addresses real distribution grid observability gap | +| Bad data detection | Weak -- docs self-describe as "not very robust" | +| Scalability | Weak -- convergence problems reported above ~90 buses | +| PMU support | Basic -- measurement types exist, testing is thin | +| Three-phase SE | Absent | +| DC SE | Absent | +| Production deployments | None known | +| Real SCADA integration | Possible via CIM converter, but no turnkey pipeline | +| Active maintenance | Moderate -- SE got focused development in mid-2025, now stalled | +| Test coverage | Moderate -- test suite exists but has known gaps (PMU test bug) | + +For Phase 2 Stage 2 evaluation purposes: pandapower SE is the strongest Python-native +SE implementation available in the open-source ecosystem, suitable for research, +education, and small-network prototyping. It would require significant hardening +(scaling fixes, robust bad data detection, observability analysis) before use in +an operational control center or real-time monitoring context. + +## Sources + +- [pandapower SE documentation (v3.3.0/latest)](https://pandapower.readthedocs.io/en/latest/estimation.html) +- [pandapower SE documentation (v3.4.0/stable)](https://pandapower.readthedocs.io/en/stable/estimation.html) +- [GitHub doc/estimation.rst (develop branch)](https://github.com/e2nIEE/pandapower/blob/develop/doc/estimation.rst) +- [GitHub CHANGELOG.rst](https://github.com/e2nIEE/pandapower/blob/develop/CHANGELOG.rst) +- [GitHub releases page](https://github.com/e2nIEE/pandapower/releases) +- [OpenMod forum: Convergence problems on large net models](https://forum.openmod.org/t/convergence-problems-on-large-net-models-in-pandapower-state-estimation-module/2706) +- [AF-WLS IEEE paper (IEEE TPWRS, 2024)](https://ieeexplore.ieee.org/document/10497141/) +- [About pandapower](https://www.pandapower.org/about/) +- GitHub issues: [#2700](https://github.com/e2nIEE/pandapower/issues/2700), [#1451](https://github.com/e2nIEE/pandapower/issues/1451), [#2524](https://github.com/e2nIEE/pandapower/issues/2524), [#1269](https://github.com/e2nIEE/pandapower/issues/1269), [#253](https://github.com/e2nIEE/pandapower/issues/253), [#248](https://github.com/e2nIEE/pandapower/issues/248), [#243](https://github.com/e2nIEE/pandapower/issues/243), [#923](https://github.com/e2nIEE/pandapower/issues/923), [#2364](https://github.com/e2nIEE/pandapower/issues/2364), [#1210](https://github.com/e2nIEE/pandapower/issues/1210), [#95](https://github.com/e2nIEE/pandapower/issues/95), [#2918](https://github.com/e2nIEE/pandapower/issues/2918), [#277](https://github.com/e2nIEE/pandapower/issues/277) diff --git a/phase2-research/powermodels-state-estimation-findings.md b/phase2-research/powermodels-state-estimation-findings.md new file mode 100644 index 00000000..3e8029db --- /dev/null +++ b/phase2-research/powermodels-state-estimation-findings.md @@ -0,0 +1,189 @@ +# PowerModels.jl State Estimation Investigation + +## Summary + +PowerModels.jl core has **no native state estimation** capability. The primary SE package +in its ecosystem is **PowerModelsDistributionStateEstimation.jl (PMDSE)**, a third-party +extension built on PowerModelsDistribution.jl. It targets distribution networks only and +is a research-grade tool from KU Leuven with 5 contributors and a last release in Oct 2023. +A separate, unrelated Julia package -- **JuliaGrid.jl** -- provides transmission-focused +state estimation with WLS, LAV, PMU support, and bad data detection, but it is outside +the PowerModels ecosystem entirely. No `PowerModelsStateEstimation.jl` package exists for +transmission-level SE within the PowerModels family. + +## PowerModelsDistributionStateEstimation.jl + +Repository: + +### Maturity Metrics + +| Metric | Value | +|--------|-------| +| GitHub stars | 40 | +| Forks | 13 | +| Contributors | 5 | +| Open issues | 3 | +| License | BSD-3-Clause | +| Created | 2020-01-29 | +| Last commit | 2025-01-29 | +| Last release | v0.7.0 (2023-10-03) | +| Last push | 2025-02-07 | +| Language | Julia | +| Primary developers | Marta Vanin, Tom Van Acker (KU Leuven / Electa group) | + +The package is academically maintained. The gap between the last tagged release (Oct 2023) +and ongoing commits (early 2025) suggests incremental maintenance without formal releases. +Five contributors and 30 commits in the default API page indicate a small, focused project. + +### Algorithm Support + +**Gaussian criteria:** + +- **WLS** (Weighted Least Squares) -- Euclidean norm (p=2) +- **rWLS** (Relaxed WLS) -- second-order cone constraint formulation +- **WLAV** (Weighted Least Absolute Value) -- absolute value norm (p=1) +- **rWLAV** (Relaxed WLAV) -- exact linear relaxation with inequality constraints + +**Non-Gaussian criteria:** + +- **MLE** (Maximum Likelihood Estimation) -- connects residuals to log-pdf; supports + non-normal distributions + +**Supported measurement distributions:** + +- Normal, Log-Normal, Exponential, Weibull, Gamma, Beta, Extended Beta +- Gaussian Mixture Models (GMM) +- Each measurement can use an individual criterion or a uniform criterion across all + +**Power flow formulations for SE:** + +- Exact: ACP, ACR, IVR (and reduced variants) +- Linear approximation: LinDist3Flow + +### Transmission vs Distribution + +PMDSE is **distribution-only**. It extends PowerModelsDistribution.jl (not PowerModels.jl), +which models unbalanced, multi-phase distribution networks. It cannot be applied to +balanced single-phase transmission models without significant modification. The package +explicitly describes itself as targeting "Power Distribution Network State Estimation." + +There is no corresponding `PowerModelsStateEstimation.jl` for transmission networks in the +PowerModels ecosystem. + +### Bad Data Detection + +Three methods are implemented (added in v0.4.0): + +1. **Chi-squared test** -- detection only (yes/no); compares weighted squared residual sum + against chi-squared threshold; default false-positive probability 0.05 +2. **Largest normalized residuals** -- detection and identification; threshold typically 3.0; + flags specific measurements exceeding threshold +3. **LAV residual analysis** -- inherently robust to bad data; large residuals identify + suspect measurements after estimation + +All methods are **post-estimation** (require a completed SE run first). Chi-squared detects +presence but not location. Effectiveness varies by scenario. + +### Known Limitations + +- **Distribution only** -- no transmission network support +- **Research-focused** -- explicitly states the goal is not fastest algorithms but a + benchmarking framework; "if faster solution times are crucial, a customized algorithm + can be developed afterwards" +- **Small maintainer base** -- 5 contributors, primarily two lead developers +- **Release cadence slowing** -- last tagged release Oct 2023, though commits continue +- **No real-time / dynamic SE** -- static state estimation only +- **No PMU-specific algorithms** -- does not distinguish between SCADA and PMU measurement + models in the way transmission SE tools do + +## Other JuMP/PowerModels SE Packages + +### JuliaGrid.jl + +Repository: + +| Metric | Value | +|--------|-------| +| GitHub stars | 48 | +| Forks | 5 | +| Contributors | 2 | +| License | MIT | +| Last push | 2026-02-06 | +| Created | 2020-04-07 | + +JuliaGrid is an **independent** Julia framework (not part of the PowerModels ecosystem) that +provides comprehensive state estimation for **transmission networks**. It was the subject +of a 2025 academic paper (arXiv:2502.18229). + +**SE capabilities:** + +- Nonlinear SE (polar coordinates, legacy + PMU measurements) +- Linear SE (PMU-only, rectangular coordinates) +- DC state estimation (voltage angles only) +- WLS, LAV, and orthogonal WLS estimators +- Observability analysis +- Bad data detection via normalized residual tests +- Optimal PMU placement (integer LP via JuMP) + +**JuMP integration:** Uses JuMP for OPF and LAV estimator formulations; compatible with +Ipopt and Gurobi solvers. + +**Performance benchmarks (from 2025 paper):** + +- 10,000-bus systems: competitive execution times +- 70,000-bus systems: handles 577,242 measurements; bad data analysis in ~1.2 seconds +- Comparable or superior ACPF times vs MATPOWER + +**Limitations:** Only 2 contributors (essentially single-author). Distribution network +support is limited compared to transmission. + +### PowerModelsStateEstimation.jl + +**Does not exist.** No such package was found on GitHub, JuliaHub, or the Julia General +registry. The PowerModels ecosystem has no transmission-level SE extension. + +### rosetta-opf + +Repository: + +The ROSETTA project (by LANL-ANSI, same group behind PowerModels.jl) benchmarks AC-OPF +implementations across NLP modeling frameworks. It uses PowerModels.jl for data parsing +and PGLib-OPF benchmarks. **It has no state estimation component** -- it is purely an +OPF benchmarking effort. + +## Production Readiness Assessment + +| Criterion | PMDSE | JuliaGrid | +|-----------|-------|-----------| +| Scope | Distribution only | Transmission focused | +| Algorithm breadth | WLS, WLAV, MLE + relaxations | WLS, LAV, orthogonal WLS | +| Bad data detection | Yes (3 methods) | Yes (normalized residuals) | +| PMU support | No dedicated PMU model | Yes (dedicated linear SE) | +| Observability analysis | Not documented | Yes | +| Scale tested | Small distribution networks | Up to 70,000-bus | +| Maintainer base | 5 contributors (2 active) | 2 contributors (1 active) | +| Release maturity | v0.7.0, slowing cadence | v0.5.5, actively maintained | +| JuMP dependency | Yes (via PowerModelsDistribution) | Yes (for OPF and LAV) | +| Production use evidence | None found; research tool | None found; research tool | + +**Neither package is production-ready for utility-scale state estimation.** Both are +academic/research tools. For the PowerModels ecosystem specifically, there is a clear gap: +no transmission-level SE package exists, and PMDSE is limited to distribution networks. +JuliaGrid partially fills the transmission SE gap but is outside the PowerModels family +and has an even smaller contributor base. + +For production SE needs, commercial tools (e.g., PSS/E, PowerWorld, EMS/SCADA vendor +packages) or MATPOWER's SE module remain the practical choices. + +## Sources + +- [PowerModelsDistributionStateEstimation.jl - GitHub](https://github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl) +- [PMDSE SE Criteria Documentation](https://electa-git.github.io/PowerModelsDistributionStateEstimation.jl/stable/se_criteria/) +- [PMDSE Bad Data Documentation](https://electa-git.github.io/PowerModelsDistributionStateEstimation.jl/v0.4/bad_data/) +- [PMDSE Power Flow Formulations](https://electa-git.github.io/PowerModelsDistributionStateEstimation.jl/stable/formulations/) +- [JuliaGrid.jl - GitHub](https://github.com/mcosovic/JuliaGrid.jl) +- [JuliaGrid.jl Documentation](https://mcosovic.github.io/JuliaGrid.jl/stable/) +- [JuliaGrid: An Open-Source Julia-Based Framework for Power System SE (arXiv 2025)](https://arxiv.org/html/2502.18229v1) +- [PowerModels.jl - GitHub](https://github.com/lanl-ansi/PowerModels.jl) +- [rosetta-opf - GitHub](https://github.com/lanl-ansi/rosetta-opf) +- [PowerModelsDistribution.jl - GitHub](https://github.com/lanl-ansi/PowerModelsDistribution.jl) diff --git a/phase2-research/powersimulations-state-estimation-findings.md b/phase2-research/powersimulations-state-estimation-findings.md new file mode 100644 index 00000000..03ef230e --- /dev/null +++ b/phase2-research/powersimulations-state-estimation-findings.md @@ -0,0 +1,79 @@ +# PowerSimulations.jl State Estimation Investigation + +## Summary + +PowerSimulations.jl and the broader NREL Sienna (formerly SIIP) ecosystem have **no state +estimation capabilities**. None of the 57 repositories in the NREL-Sienna GitHub organization +address state estimation. No GitHub issues mentioning state estimation exist across +PowerSimulations.jl, PowerSystems.jl, or PowerSimulationsDynamics.jl. No NREL publications +were found combining Sienna tools with state estimation. + +State estimation in the Julia power systems world is handled by entirely separate ecosystems: +the PowerModels family (LANL/KU Leuven) and JuliaGrid.jl. + +## Native SE Support + +**None.** PowerSimulations.jl is scoped to power system *operations simulation* -- production +cost modeling, unit commitment, economic dispatch, and multi-stage scheduling. Its optimization +formulations are forward-looking decision problems (what should generators do?), not inverse +estimation problems (what is the current system state given measurements?). + +The full Sienna package suite (57 repos) was enumerated via the GitHub API. Relevant packages +and their scope: + +| Package | Scope | SE relevance | +|---------|-------|--------------| +| PowerSimulations.jl | Operations scheduling / PCM | None | +| PowerSimulationsDynamics.jl | Transient/dynamic simulation | None | +| PowerSystems.jl | Data model / system representation | None (data layer only) | +| PowerFlows.jl | Steady-state power flow solvers | None (solves known systems) | +| PowerNetworkMatrices.jl | Network matrix representations | None | +| PowerAnalytics.jl | Post-simulation analytics | None | +| HydroPowerSimulations.jl | Hydro unit modeling | None | +| StorageSystemsSimulations.jl | Storage modeling | None | +| PowerSystemsInvestments.jl | Capacity expansion | None | + +## SIIP Ecosystem SE Packages + +**None exist.** A targeted search of the NREL-Sienna GitHub organization for "state estimation" +returned zero matching repositories. No issues or PRs across the core repos mention state +estimation. + +### Julia SE packages outside Sienna + +Two Julia packages provide state estimation, but neither is part of the Sienna/SIIP ecosystem: + +1. **PowerModelsDistributionStateEstimation.jl** (Electa-Git / KU Leuven) + - Extension of PowerModelsDistribution.jl (LANL PowerModels family) + - Supports WLS, WLAV, and maximum likelihood estimation + - Distribution-network focused + - No dependency on or integration with any NREL-Sienna package + - Repository: https://github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl + +2. **JuliaGrid.jl** (mcosovic) + - Standalone framework for power system state estimation + - Nonlinear SE, PMU-based linear SE, and DC SE models + - WLS with conventional, orthogonal, and Peters-Wilkinson methods + - No connection to Sienna or PowerModels ecosystems + - Repository: https://github.com/mcosovic/JuliaGrid.jl + +## NREL Research Connections + +No NREL publications were found that combine Sienna/SIIP tools with state estimation. The +primary PowerSimulations.jl publication (Lara et al., IEEE Trans. Power Systems, 2024; +arXiv:2404.03074) focuses exclusively on operations simulation and does not mention state +estimation. + +NREL's Sienna development effort is oriented toward planning and operations optimization +(unit commitment, economic dispatch, capacity expansion, reliability assessment via PRAS), +not grid observability or measurement-based estimation. + +## Sources + +- [NREL-Sienna GitHub organization](https://github.com/NREL-Sienna) -- 57 repositories, none SE-related +- [PowerSimulations.jl repository](https://github.com/NREL-Sienna/PowerSimulations.jl) +- [PowerSimulations.jl paper (arXiv)](https://arxiv.org/html/2404.03074v1) +- [PowerModelsDistributionStateEstimation.jl](https://github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl) +- [JuliaGrid.jl (arXiv paper)](https://arxiv.org/html/2502.18229v1) +- [JuliaGrid.jl documentation](https://mcosovic.github.io/JuliaGrid.jl/stable/) +- [NREL-Sienna Julia packages listing](https://juliapackages.com/u/nrel-sienna) diff --git a/phase2-research/pypsa-state-estimation-findings.md b/phase2-research/pypsa-state-estimation-findings.md new file mode 100644 index 00000000..7d7e8a2f --- /dev/null +++ b/phase2-research/pypsa-state-estimation-findings.md @@ -0,0 +1,131 @@ +# PyPSA State Estimation Investigation + +## Summary + +PyPSA has **no native state estimation (SE) capability** and there are **no known community +efforts** (issues, PRs, forks, or plugins) to add one. The PyPSA project explicitly +acknowledges SE as a missing feature relative to pandapower. However, PyPSA's numerical +building blocks (Newton-Raphson solver, sparse Jacobian construction, scipy/numpy linear +algebra) provide a plausible foundation for implementing WLS state estimation as a custom +extension. Two other Python-ecosystem tools -- pandapower and power-grid-model -- already +ship production-grade SE and could serve as references or interop targets. + +## Native SE Support + +**None.** PyPSA's feature set covers: + +- Full nonlinear AC power flow (Newton-Raphson) +- Linearised DC power flow +- Linear, quadratic, and mixed-integer optimal power flow (via linopy) +- Multi-period investment and dispatch optimisation +- Contingency analysis (N-1) + +The original PyPSA paper (Brown et al. 2018) and all subsequent documentation explicitly +position PyPSA as a power flow and optimisation tool. State estimation, short-circuit +analysis, and three-winding transformer modelling are listed as features present in +pandapower but absent from PyPSA. + +## Community Efforts (Issues/PRs/Forks) + +| Source | Search result | +|--------|--------------| +| GitHub Issues (`PyPSA/PyPSA`) | Zero issues matching "state estimation" | +| GitHub PRs (`PyPSA/PyPSA`) | Zero PRs matching "state estimation" | +| GitHub forks | No fork found adding SE functionality | +| PyPSA Google Group | No threads found on SE | +| GitHub Discussions | No discussions found on SE | +| PyPSA-Eur / PyPSA-USA / PyPSA-Earth | Regional model extensions; none add SE | +| GitHub `state-estimation` topic (Python) | Lists pandapower and power-grid-model; no PyPSA-based repo | + +There is no evidence of any community member requesting, proposing, or implementing state +estimation within the PyPSA ecosystem. + +## Academic Integration + +No academic papers were found that combine PyPSA with state estimation. The literature +on Python-based power system SE consistently references: + +- **pandapower** (Thurner et al. 2018) -- WLS SE with chi-squared and normalised residual + bad-data detection +- **power-grid-model** (Alliander) -- iterative linear WLS SE with voltage phase angle + correction across iterations +- Various standalone implementations (e.g., `nbhusal/Power-System-State-Estimation` on + GitHub) + +PyPSA appears exclusively in optimisation and planning literature, not in estimation or +monitoring contexts. + +## Theoretical Feasibility + +WLS state estimation is fundamentally a nonlinear least-squares optimisation problem: + + min_x (z - h(x))^T W (z - h(x)) + +where z is the measurement vector, h(x) maps state to measurements, and W is the +inverse-covariance weight matrix. This is solved iteratively via the Gauss-Newton method, +which requires: + +1. **Admittance matrix (Y-bus)** -- PyPSA builds this already for power flow. +2. **Jacobian of measurement functions** -- PyPSA builds a power-flow Jacobian + (dP/dtheta, dP/dV, dQ/dtheta, dQ/dV) for Newton-Raphson. The SE Jacobian + (dh/dx) is structurally similar but includes rows for voltage magnitude, line + flow, and injection measurements rather than just bus power mismatches. +3. **Sparse linear solve** -- PyPSA uses scipy.sparse with UMFPACK (same solver + as MATPOWER) for its Newton-Raphson iterations. +4. **Measurement model** -- Not present in PyPSA. Would need to be built: measurement + types (P_inj, Q_inj, V_mag, P_flow, Q_flow), noise/weight specification, + placement topology. + +**What could be reused from PyPSA:** +- Network data model (buses, lines, transformers, generators, loads) +- Y-bus construction routines +- Sparse matrix infrastructure +- Newton-Raphson iteration scaffolding (convergence checks, iteration limits) + +**What would need to be built from scratch:** +- Measurement data model (type, location, value, variance) +- SE-specific Jacobian (h(x) and dh/dx for each measurement type) +- WLS normal equations solver (gain matrix G = H^T W H) +- Bad-data detection (chi-squared test, largest normalised residual) +- Observability analysis (rank check of H matrix) +- Optional: PMU measurement handling, topology error detection + +**Estimated effort:** Moderate. A basic WLS SE for bus injection and voltage measurements +could be prototyped in ~500-1000 lines of Python leveraging PyPSA's network model and +scipy.sparse. A production-quality implementation with all measurement types, bad-data +detection, and observability analysis would be significantly more work, and at that point +using pandapower's existing SE module or power-grid-model would be more practical. + +**Linopy/optimisation path:** PyPSA's linopy-based optimiser supports quadratic +objectives (QP), so a WLS SE could theoretically be formulated as a QP. However, SE +requires nonlinear measurement equations h(x), making a pure LP/QP formulation +incomplete -- you would still need iterative linearisation, which is essentially +reimplementing Gauss-Newton. This path offers no real advantage over direct +implementation with scipy. + +## Alternative Approaches for Phase 2 + +Given the absence of native SE in PyPSA, the Phase 2 evaluation has several options: + +1. **Use pandapower for SE tasks** -- pandapower has mature WLS SE with bad-data + detection. Networks can be converted between PyPSA and pandapower formats. +2. **Use power-grid-model** -- Alliander's C++-backed tool with Python bindings; + fast iterative linear WLS SE. Designed for distribution grids but applicable + to transmission. +3. **Build a minimal SE on PyPSA's network model** -- Feasible as a proof of concept + but significant effort for a robust implementation. +4. **Acknowledge as a gap** -- Document that PyPSA does not support SE and evaluate + this dimension using pandapower or PowerModels.jl instead. + +## Sources + +- [PyPSA: Python for Power System Analysis (Brown et al. 2018)](https://openresearchsoftware.metajnl.com/articles/10.5334/jors.188) +- [PyPSA GitHub repository](https://github.com/PyPSA/PyPSA) +- [PyPSA documentation](https://docs.pypsa.org/latest/) +- [PyPSA arXiv preprint](https://arxiv.org/abs/1707.09913) +- [pandapower paper (Thurner et al. 2018)](https://arxiv.org/pdf/1709.06743) +- [pandapower website](https://www.pandapower.org/) +- [power-grid-model documentation -- state estimation](https://power-grid-model.readthedocs.io/en/v1.5.14/user_manual/calculations.html) +- [GitHub state-estimation topic (Python)](https://github.com/topics/state-estimation?l=python) +- [Linopy GitHub repository](https://github.com/PyPSA/linopy) +- [PyPSA optimization with linopy example](https://docs.pypsa.org/v0.27.1/examples/optimization-with-linopy.html) diff --git a/phase2-research/se-landscape-academic.md b/phase2-research/se-landscape-academic.md new file mode 100644 index 00000000..e8b3164a --- /dev/null +++ b/phase2-research/se-landscape-academic.md @@ -0,0 +1,369 @@ +# Academic & Cutting-Edge SE Landscape + +> Research compiled 2026-03-27. Focused on open-source state estimation (SE) tools +> and recent academic work relevant to transmission-level SE for target ISO grid modeling. + +## Summary + +The open-source SE landscape is dominated by a handful of tools, each with different +maturity levels and scope: + +| Tool | Language | SE Scope | PMU Support | Scale Validated | License | +|------|----------|----------|-------------|-----------------|---------| +| **JuliaGrid** | Julia | AC, DC, PMU-only | Yes (native) | 70,000 bus | MIT | +| **pandapower** | Python | AC (WLS, robust) | No native PMU SE | Medium-scale | BSD-3 | +| **GridCal** | Python | AC (WLS) | Limited | Medium-scale | LGPL | +| **ANDES (CURENT LTB)** | Python | Static + dynamic | Via LTB platform | Transmission-scale | GPL-3 | +| **PMDSE.jl** | Julia | Distribution SE | No | Distribution only | BSD-3 | +| **MATPOWER** | MATLAB/Octave | AC (WLS) | No | Medium-scale | BSD-3 | + +**Key finding:** JuliaGrid (published Feb 2025) is currently the most complete open-source +SE framework. It is the only tool that natively supports AC SE, DC SE, and linear +PMU-based SE with observability analysis, bad data detection (chi-squared + largest +normalized residual), and validation at 70,000-bus scale. For a target ISO-scale +implementation, JuliaGrid is the strongest candidate as a computational core. + +pandapower remains the most accessible Python option but lacks PMU-specific SE +formulations, observability analysis, and has reported issues with bad data analysis +at scale (tens of thousands of buses). + +No single open-source tool provides a production-ready, real-time hybrid SCADA+PMU +state estimator out of the box. This remains a gap that would require custom +integration work. +## PMU-Based State Estimation + +### Why PMU SE Matters + +PMU-based (linear) SE exploits the fact that synchrophasor measurements provide +direct voltage/current phasors, making the measurement-to-state relationship +**linear**. This eliminates the iterative Gauss-Newton process required for +traditional SCADA-based (nonlinear) SE, enabling: +- Solve times fast enough for PMU reporting rates (up to 60 Hz) +- No convergence issues (direct linear solve) +- Higher accuracy from synchronized time-stamped measurements + +### Open-Source Tools with PMU SE + +**JuliaGrid** is the standout. It implements: +- **PMU State Estimation** as a dedicated linear model using rectangular coordinates + (real/imaginary parts of bus voltages and branch current phasors) +- WLS estimator for PMU-only SE with robust variants (orthogonal method, + Peters-Wilkinson method) for ill-conditioned measurement sets +- Observability analysis specific to PMU placement +- Bad data detection for PMU measurements + +Reference: Cosovic et al., "JuliaGrid: An Open-Source Julia-Based Framework for +Power System State Estimation," arXiv:2502.18229, Feb 2025. Published in +SoftwareX (2025/2026). + +**OpenPMU** is an open-source PMU hardware+software platform (Python-based phasor +estimator) that produces synchrophasors from sampled values. It is a data +acquisition tool, not an SE solver, but could serve as a PMU data source for +an SE pipeline. + +**ComEd DLSE** (distribution linear SE) demonstrated PMU-rate SE (60 Hz solve rate) +in a real utility deployment, described in a 2024 Springer publication. The +implementation itself is not open-source but validates the feasibility of +PMU-rate linear SE at utility scale. + +### PMU Data Resources + +Texas A&M provides **synthetic PMU data** for their ACTIVSg test cases, and an +open-source library of 1,694 real transmission-level PMU events was published in +IEEE Transactions on Power Systems (2023), providing the largest public dataset +for benchmarking PMU-based SE algorithms. +## Hybrid SE (SCADA + PMU) + +Hybrid SE combines slow SCADA measurements (2-10 second scan rates, unsynchronized) +with fast PMU measurements (30-60 Hz, GPS-synchronized). This is the realistic +operational scenario for any modern grid including target ISO. + +### Approaches in the Literature + +1. **Two-stage sequential**: Run traditional WLS SE on SCADA, then refine with + PMU measurements in a linear post-processing step. Most common in production + EMS implementations (e.g., GE, Siemens, ABB/Hitachi). + +2. **Extended Kalman Filter (EKF) fusion**: Treats the different measurement rates + as a multi-rate estimation problem. A 2024 paper in Energies proposes EKF-based + fusion that handles SCADA/PMU rate mismatch natively. + +3. **Unified WLS with mixed measurements**: Augments the traditional SE + measurement vector with PMU phasor measurements. Requires careful handling of + different coordinate systems (polar for SCADA, rectangular for PMU). + +### Open-Source Status + +**No mature open-source hybrid SE implementation exists.** The closest options: + +- **JuliaGrid** provides separate AC SE (for SCADA-type measurements) and PMU SE + modules. A hybrid pipeline could be built by running both and combining results, + but there is no built-in fusion mechanism. + +- **ORNL** published work on a hybrid SCADA/PMU online state estimator (Parashar + et al., IEEE PES 2013), but no open-source release accompanied it. + +- **SCADA BR** is an open-source web-based SCADA system that has been demonstrated + with PMU integration in a lab setting (IEEE 2014 paper), but it is a monitoring + platform, not an SE solver. + +- **Co-simulation frameworks** (e.g., OpenDSS + OMNET++ at IIT Madras) have + demonstrated hybrid SE in research settings but are not packaged as reusable + SE tools. + +### Implication for target ISO + +Building a hybrid SE for target ISO would likely require: +1. JuliaGrid (or custom Julia/Python code) as the SE computational core +2. Custom data fusion logic for SCADA + PMU measurement streams +3. Integration with a real-time data pipeline (e.g., streaming PMU via C37.118) +## Transmission-Scale SE Tools + +For target ISO grid modeling, the SE tool must handle transmission-scale networks +(thousands of buses, HV/EHV voltage levels). + +### Tools Validated at Transmission Scale + +**JuliaGrid** -- Validated on 10,000, 20,000, and 70,000-bus systems. The Feb 2025 +paper includes benchmarks showing competitive performance with commercial tools. +Julia's JIT compilation and sparse linear algebra (KLU factorization) provide +the performance needed for large-scale SE. + +**ANDES / CURENT LTB** -- Designed for transmission-scale simulation. ANDES +supports power flow via Newton-Raphson and includes a state estimation routine, +though SE is not its primary focus. The LTB platform architecture (ANDES + +DiME messaging + AGVis visualization) is designed for closed-loop real-time +simulation including SE in the loop. + +**pandapower** -- Built on PYPOWER (Python port of MATPOWER). Handles +transmission-scale power flow well but the SE module has reported numerical +issues at scale. The JuliaGrid paper notes pandapower's bad data analysis +"may encounter issues when applied to large-scale power systems." + +**MATPOWER** -- The SE module (`runse`) uses WLS via MATLAB/Octave. Mature and +well-tested but limited to basic WLS without PMU support or robust estimators. +Not designed for real-time operation. + +### Tools NOT Suitable for Transmission SE + +**PowerModelsDistributionStateEstimation.jl (PMDSE)** -- Explicitly scoped to +distribution networks. Built on PowerModelsDistribution.jl which models +unbalanced three-phase systems. Not applicable to balanced transmission SE. + +**GridLAB-D** -- Distribution-focused agent-based simulator (PNNL). No +transmission SE capability. + +**OpenDSS** -- Distribution system simulator (EPRI). No built-in SE; some +community attempts exist but are not maintained. +## Real-Time SE Implementations + +Real-time SE requires solving the estimation problem within the measurement +scan cycle (seconds for SCADA, milliseconds for PMU-rate). + +### Current State + +**No open-source tool provides a production-ready real-time SE pipeline.** All +existing tools are batch/offline solvers that take a measurement snapshot and +return an estimate. The real-time wrapper (data ingestion, time alignment, +triggering, result publication) must be built separately. + +**JuliaGrid** has the best potential for real-time use due to: +- Julia's compiled performance (competitive with C/Fortran for numerical code) +- Automatic detection and reuse of computed data structures between solves +- Linear PMU SE that avoids iterative convergence (deterministic solve time) + +**CURENT LTB** is the closest to a real-time platform. Its architecture includes: +- DiME (Distributed Messaging Environment) for streaming data between components +- ANDES running in a simulation loop with measurement injection +- A state estimator module that receives measurements and returns estimates +- The platform has been demonstrated in hardware-in-the-loop settings + +**ComEd's DLSE** (proprietary) demonstrated 60 Hz SE solve rate using PMU data, +proving real-time PMU-rate SE is feasible. The algorithmic approach (linear SE) +is straightforward to replicate with JuliaGrid's PMU SE module. + +### Architecture Implications + +A real-time SE system for target ISO would need: +1. **Data layer**: C37.118 PDC (phasor data concentrator) for PMU streams, + ICCP/DNP3 for SCADA +2. **Time alignment**: Buffer and align SCADA snapshots with PMU windows +3. **SE solver**: JuliaGrid or equivalent, called per scan cycle +4. **State publication**: Push estimated states to downstream applications + (contingency analysis, SCOPF, visualization) +## ML-Based Approaches + +Machine learning for SE is an active research area with several promising +directions, though all remain at the research stage. + +### Physics-Informed Neural Networks (PINNs) + +PINNs embed power flow equations as loss function constraints, allowing the +network to learn SE while respecting Kirchhoff's laws and Ohm's law. + +- **Physics-informed GNN for SE** (Li et al., Applied Energy, 2024): Combines + graph neural networks with physical power flow constraints. Validated on IEEE + 14, 57, and 118-bus systems. Achieves >20% lower MSE than conventional WLS + in scenarios with high measurement noise or missing data. + +- **PINNs for accelerated SE** (arXiv:2310.03088, 2023): Demonstrates 87x + speedup over conventional Gauss-Newton SE by using a trained neural network + as a warm-start or direct estimator. Most beneficial when the network + topology is stable and measurements arrive at high frequency. + +- **Open source**: github.com/gmisy/Physics-Informed-Neural-Networks-for-Power-Systems + provides a framework for PINN-based power system applications including SE. + +### Graph Neural Networks (GNNs) + +GNNs naturally map to power network topology (buses = nodes, branches = edges). + +- **GNN for SE** (Kundacina et al., arXiv:2201.04056): Uses IGNNITION framework + to train a GNN that estimates complex bus voltages from PMU measurements. + Training data generated synthetically via JuliaGrid's linear WLS solver. + Open source: github.com/ognjenkundacina/graph-neural-network-state-estimation + +- **Topology-aware GNN** (IEEE Trans. Power Systems, 2024): Handles topology + changes and PMU data loss in real-time, combining graph convolution with + multi-head attention layers. + +- **Master's thesis** (Univ. Freiburg, 2024): Comprehensive GNN-based SE with + benchmarks against WLS on IEEE test cases. + +### Practical Assessment + +ML-based SE is **not ready for production** on a target ISO-scale grid: +- Training requires large labeled datasets (synthetic or historical SE solutions) +- Topology changes (switching, outages) invalidate trained models unless the + architecture handles dynamic graphs +- Regulatory/reliability requirements demand explainable, deterministic methods +- Most promising near-term use: warm-starting conventional SE, or providing + fast approximate estimates between SCADA scan cycles + +ML approaches may become relevant as a **complement** to traditional SE +(e.g., filling in pseudo-measurements for unobservable areas, detecting bad +data, or providing fast approximate SE between scan cycles). +## Key Research Groups & Projects + +### CURENT (Center for Ultra-Wide-Area Resilient Electric Energy Transmission) +- **Affiliation**: University of Tennessee Knoxville (UTK), formerly with Cornell +- **Key output**: ANDES, AMS, LTB platform +- **SE relevance**: LTB includes SE in its closed-loop simulation architecture; + ANDES has a state estimation routine for transmission systems +- **GitHub**: github.com/CURENT +- **Note**: GPL-3 license on ANDES may limit commercial use + +### Fraunhofer IEE + University of Kassel +- **Key output**: pandapower (500,000+ downloads as of 2024) +- **SE relevance**: WLS SE with robust estimators (IRWLS, Huber, SHGM, QL, QC), + chi-squared and normalized residual bad data detection +- **Limitation**: No PMU SE, no observability analysis, scale issues +- **GitHub**: github.com/e2nIEE/pandapower + +### TU Berlin (Digital Transformation in Energy Systems, ENSYS) +- **Key output**: PyPSA framework +- **SE relevance**: PyPSA does NOT include state estimation. Focus is on + capacity expansion and optimal power flow for planning studies. +- **Relevance to project**: PyPSA is in our evaluation set but not for SE + +### University of Sarajevo (Mcosovic group) +- **Key output**: JuliaGrid +- **SE relevance**: Most complete open-source SE framework. AC SE, DC SE, PMU SE, + observability analysis, bad data detection, LAV estimator +- **GitHub**: github.com/mcosovic/JuliaGrid.jl + +### KU Leuven / Electa Group +- **Key output**: PowerModelsDistributionStateEstimation.jl +- **SE relevance**: Flexible distribution SE with multiple formulations (WLS, WLAV, + MLE) and power flow approximations. Research-oriented benchmarking tool. +- **GitHub**: github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl + +### LANL (Los Alamos National Laboratory) +- **Key output**: PowerModels.jl, PowerModelsDistribution.jl +- **SE relevance**: Underlying optimization framework used by PMDSE.jl. + PowerModels.jl itself focuses on OPF, not SE. +- **GitHub**: github.com/lanl-ansi + +### NREL (National Renewable Energy Laboratory) +- **Key output**: Sienna ecosystem (PowerSystems.jl, PowerSimulations.jl, + PowerFlows.jl) +- **SE relevance**: No SE module in Sienna. Focus is on production cost modeling + and dynamic simulation. However, PowerSystems.jl provides excellent data + infrastructure for Julia-based power system tools. + +### Texas A&M (Overbye group) +- **Key output**: ACTIVSg synthetic grid test cases (200 to 70,000 bus), + synthetic PMU data +- **SE relevance**: Provides the benchmark networks and PMU datasets needed to + validate SE at scale. The 2,000-bus and 10,000-bus cases with time-series + data are particularly useful for SE testing. +- **Repository**: electricgrids.engr.tamu.edu +## IEEE Benchmarks + +### Standard IEEE Test Cases for SE + +These are the workhorses for SE algorithm validation: + +| Case | Buses | Generators | Lines | Notes | +|------|-------|-----------|-------|-------| +| IEEE 14 | 14 | 5 | 20 | Minimum viable test; too small for scale testing | +| IEEE 30 | 30 | 6 | 41 | Common in textbook examples | +| IEEE 57 | 57 | 7 | 80 | Moderate complexity | +| IEEE 118 | 118 | 19 | 186 | Most common SE benchmark in papers | +| IEEE 300 | 300 | 69 | 411 | Largest classic IEEE case | + +### Texas A&M ACTIVSg Synthetic Grids + +For realistic large-scale SE validation: + +| Case | Buses | Footprint | PMU Data | Time Series | +|------|-------|-----------|----------|-------------| +| ACTIVSg200 | 200 | Illinois | No | No | +| ACTIVSg500 | 500 | South Carolina | No | No | +| ACTIVSg2000 | 2,000 | Texas | Yes | Yes (1 year) | +| ACTIVSg10k | 10,000 | Western US | No | Yes (1 year) | +| ACTIVSg25k | 25,000 | Northeast US | No | No | +| ACTIVSg70k | 70,000 | Eastern US | No | No | + +The ACTIVSg2000 case is particularly relevant for target ISO-scale SE testing because +it includes synthetic PMU data and represents a realistic transmission grid. + +### UW Power Systems Test Case Archive + +The University of Washington maintains the original IEEE test case archive +(labs.ece.uw.edu/pstca/) in MATPOWER-compatible format. These are the canonical +versions used by most open-source tools. + +### MATPOWER Case Files + +MATPOWER ships with 60+ test cases including all IEEE standards plus Polish, +PEGASE (European), and RTE (French) systems up to 13,659 buses. These are +available in the `data/networks/` directory of this evaluation project. +## Sources + +### Papers +- Cosovic et al., "JuliaGrid: An Open-Source Julia-Based Framework for Power System State Estimation," arXiv:2502.18229, Feb 2025 / SoftwareX 2025 +- Thurner et al., "pandapower -- An Open Source Python Tool for Convenient Modeling, Analysis and Optimization of Electric Power Systems," IEEE Trans. Power Systems, 2018 +- Kundacina et al., "State Estimation in Electric Power Systems Leveraging Graph Neural Networks," arXiv:2201.04056, 2022 +- Li et al., "Physics-informed graphical neural network for power system state estimation," Applied Energy, 2024 +- Vanin et al., "PowerModelsDistribution.jl: An Open-Source Framework for Exploring Distribution Power Flow Formulations," EPSR, 2020 +- Birchfield et al., "ACTIVSg synthetic grids" and "Synthetic PMU data," Texas A&M, 2016-2023 +- Open-source PMU data library, IEEE Trans. Power Systems, 2023 + +### Repositories +- JuliaGrid: https://github.com/mcosovic/JuliaGrid.jl +- pandapower: https://github.com/e2nIEE/pandapower +- PMDSE.jl: https://github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl +- CURENT LTB: https://github.com/CURENT +- GNN SE: https://github.com/ognjenkundacina/graph-neural-network-state-estimation +- PINNs for Power Systems: https://github.com/gmisy/Physics-Informed-Neural-Networks-for-Power-Systems +- OpenPMU: https://github.com/OpenPMU/OpenPMUdocs +- Texas A&M test cases: https://electricgrids.engr.tamu.edu/ +- UW PSTCA: https://labs.ece.uw.edu/pstca/ +- best-of-ps (curated list): https://github.com/ps-wiki/best-of-ps + +### Documentation +- JuliaGrid docs: https://mcosovic.github.io/JuliaGrid.jl/stable/ +- pandapower SE docs: https://pandapower.readthedocs.io/en/latest/estimation.html +- Sienna: https://nrel-sienna.github.io/Sienna/ +- CURENT LTB: https://curent.github.io/ diff --git a/phase2-research/se-landscape-powsybl.md b/phase2-research/se-landscape-powsybl.md new file mode 100644 index 00000000..55be57ff --- /dev/null +++ b/phase2-research/se-landscape-powsybl.md @@ -0,0 +1,227 @@ +# PowSyBl State Estimation Investigation + +_Research date: 2026-03-27_ + +## Summary + +PowSyBl (Power System Blocks) is an open-source Java framework for power system modeling +and simulation, initiated by RTE (the French TSO) and contributed to LF Energy in 2019. +It is the most production-deployed open-source grid analysis framework in Europe, now +underpinning cross-border capacity calculation for 30+ TSOs. + +**State estimation verdict: PowSyBl does NOT currently provide a state estimation +implementation.** The grid model (IIDM) includes observability extensions to store SE +results (added 2022), but no SE solver exists in any public PowSyBl repository. SE does +not appear on the 2026 roadmap. The framework's strengths are load flow, security +analysis, sensitivity analysis, and remedial action optimization -- not SE. + +## Ecosystem Overview + +PowSyBl is a modular ecosystem of ~20 repositories under `github.com/powsybl/`. Key +components: + +| Component | Purpose | +|---|---| +| **powsybl-core** | Grid model (IIDM), exchange format importers/exporters (CGMES, MATPOWER, PSS/E, UCTE, IEEE-CDF, PowerFactory), simulation APIs | +| **powsybl-open-loadflow** | AC Newton-Raphson and DC load flow, security analysis, sensitivity analysis (KLU sparse solver) | +| **powsybl-open-rao** | Remedial action optimization engine | +| **powsybl-dynawo** | Dynamic simulation via Dynawo (DynaFlow steady-state, DynaWaltz time-domain) | +| **powsybl-entsoe** | ENTSO-E-specific processes (GLSK, merging, flow-based) | +| **powsybl-metrix** | Multi-variant network simulation | +| **powsybl-optimizer** | Optimal power flow | +| **powsybl-diagram** | Single-line and network-area diagram generation | +| **powsybl-network-store** | Network model persistence (Cassandra-backed) | +| **powsybl-network-viewer** | TypeScript/web visualization components | +| **pypowsybl** | Python bindings via GraalVM native image | +| **powsybl.jl** | Julia bindings (network I/O and element access) | + +## State Estimation Capabilities + +### What exists + +- **Observability extensions in IIDM** (powsybl-core, since September 2022, issue #1787): + The grid model can store SE _results_ via `InjectionObservability` and + `BranchObservability` extensions. These record whether an element is observable, + plus standard deviations and redundancy indicators for P, Q, V measurements per + side. There is also a `ThreeWindingsTransformerPhaseAngleClock` extension and + tap-changer estimability flags. +- **No SE solver**: No `StateEstimation` class, API, or implementation exists in + powsybl-open-loadflow, powsybl-core, or any other public PowSyBl repository. GitHub + code search across the entire `powsybl` org returns zero results for + `StateEstimation` or `state_estimation`. +- **Not on the roadmap**: The 2026 roadmap (quarterly releases 2026.0 through 2026.3) + covers load flow improvements, HVDC/TCSC simulation, operator strategies, and + diagram enhancements. State estimation is not mentioned. The 2027+ "best effort" + bucket also does not include SE. + +### What this means + +PowSyBl was designed for planning and operational security analysis, not for real-time +state estimation. RTE likely uses proprietary or commercial SE tools (e.g., from GE or +Siemens EMS) for their control center, while PowSyBl handles the offline/planning +workloads. The observability extensions exist so that SE results from external tools +can be annotated onto the IIDM grid model. + +## Open Load Flow Details + +powsybl-open-loadflow v2.1.x provides: + +- **AC load flow**: Full Newton-Raphson with KLU sparse solver (native code). Supports + voltage regulation, phase shifters, slack distribution, reactive limits. +- **DC load flow**: Linear DC approximation. +- **Security analysis**: N-1 and N-k contingency analysis. Benchmarked at 5-29 ms per + contingency on RTE 6515-bus network (single core). +- **Sensitivity analysis**: Active/reactive power flow sensitivities to injections, PSTs, + HVDC setpoints. + +### Benchmark numbers (Dell Precision 5680, i7-13700H, single core) + +| Network | AC Load Flow (basic) | AC Load Flow (standard) | +|---|---|---| +| IEEE 14 | 179 us | 188 us | +| IEEE 118 | 1.37 ms | 1.88 ms | +| IEEE 300 | 3.5 ms | 5.9 ms | +| RTE 1888 (French EHV) | 24.7 ms | 30.7 ms | +| RTE 6515 (French EHV+HV) | 118 ms | 191 ms | + +These are competitive with pandapower and faster than MATPOWER for large networks, +though LightSim2Grid (C++ Newton-Raphson for Grid2Op) is reportedly 4-7x faster than +pypowsybl for specific benchmarks. + +## pypowsybl (Python Bindings) + +pypowsybl wraps the Java framework via GraalVM native image compilation. Available +modules (as of the latest docs): + +- `pypowsybl.network` -- grid model creation, import/export (CGMES, MATPOWER, PSS/E, etc.) +- `pypowsybl.loadflow` -- AC and DC load flow +- `pypowsybl.security` -- security analysis +- `pypowsybl.sensitivity` -- sensitivity analysis +- `pypowsybl.rao` -- remedial action optimization +- `pypowsybl.flowdecomposition` -- flow decomposition +- `pypowsybl.dynamic` -- dynamic simulation (via Dynawo) +- `pypowsybl.shortcircuit` -- short circuit analysis +- `pypowsybl.voltage_initializer` -- voltage initialization + +**No `pypowsybl.state_estimation` module exists.** SE is not exposed in the Python API. + +### pypowsybl interoperability + +- Imports: CGMES, MATPOWER (.m), PSS/E (.raw/.rawx), IEEE-CDF, UCTE, PowerFactory +- Exports: CGMES, XIIDM, JIIDM, BIIDM, UCTE +- Includes a pandapower-to-PowSyBl network converter +- ANDES (dynamic simulation) has a `to_pypowsybl()` bridge for diagram generation +- No direct PyPSA converter exists in either direction + +## Scale & Production Use + +### RTE and the French grid + +RTE operates ~100,000 km of transmission lines, ~2,500 substations, 63 kV to 400 kV. +PowSyBl has been validated on: + +- **RTE 1888**: French EHV (Extra-High Voltage) system +- **RTE 6515**: Full French EHV + HV system +- **RTE 7000**: Published on HuggingFace (`rte-france/RTE7000`), ~7,000 buses + representing the complete French transmission network in node-breaker topology + +### European production deployments + +PowSyBl is in production use by: + +- **RTE** (France, initiator) -- grid planning and security analysis +- **Elia** (Belgium) -- grid analysis +- **CORESO** (pan-European RCC) -- operational security via CorNet program +- **TSCNET** (RCC) -- operational coordination +- **Baltic RCC** -- operational grid security studies +- **SeleneCC** -- European Merging Function implementation + +Major milestone (December 2024): CorNet go-live of the **European Merging Function**, +consolidating individual TSO grid models into a unified Common Grid Model for 30+ TSOs. +PowSyBl Open Load Flow, Cost Sharing, and OpenRAO are the core engines. + +### Vendors and contractors + +Artelys (French optimization firm), AIA, Power Info, and CRESYM contribute to the +ecosystem. Artelys specifically improved Open Load Flow robustness under contract to RTE. + +## License & Governance + +- **License**: Mozilla Public License 2.0 (MPL-2.0) -- file-level copyleft, compatible + with proprietary integration +- **Foundation**: LF Energy (Linux Foundation Energy) +- **Lifecycle stage**: Early Adoption (as of May 2023) -- focused on industry adoption, + ready for production consideration. Not yet Graduated. +- **Security**: OpenSSF Best Practices silver badge (2023), security audit completed (2024) +- **Language**: Java (core), Python (pypowsybl via GraalVM), Julia (powsybl.jl) +- **Governance**: Technical Steering Committee (TSC) with representatives from RTE, Elia, + and community contributors + +## PyPSA Integration Path + +There is **no direct PyPSA-PowSyBl converter** in either ecosystem. However, several +indirect paths exist: + +1. **MATPOWER format bridge**: Both PyPSA (via pandapower import) and PowSyBl natively + read MATPOWER `.m` files. This is the simplest interchange format for static network + data. + +2. **CGMES as interchange**: PowSyBl has the most mature open-source CGMES + importer/exporter. PyPSA does not natively support CGMES, but there is community + discussion about adding it (OpenMod forum, Dec 2024). A pypowsybl CGMES import + followed by MATPOWER export could bridge the gap. + +3. **pandapower bridge**: pypowsybl includes a pandapower-to-PowSyBl converter. + pandapower has limited PyPSA interop. This is a lossy two-hop path. + +4. **Custom scripting**: Both pypowsybl and PyPSA expose DataFrames for network elements. + A custom Python script mapping between the two models is feasible but requires manual + effort for each network type. + +For target ISO SE specifically, CGMES is not the natural format (target ISO uses CIM but in a +different profile). The MATPOWER bridge or custom conversion would be more practical. + +## Production Readiness Assessment + +| Criterion | Assessment | +|---|---| +| **State estimation** | Not available. No SE solver, no roadmap item. | +| **Load flow** | Production-grade. Validated on real European grids by multiple TSOs. | +| **Scale** | Proven at 7,000+ bus transmission scale (French grid). | +| **Maturity** | LF Energy Early Adoption. In production at RTE, Elia, CORESO, etc. | +| **Python API** | pypowsybl is functional but SE is not exposed. | +| **target ISO relevance** | Limited. European-centric (CGMES, ENTSO-E processes). No native support for target ISO-specific data formats or market structures. | +| **SE alternative** | Would need to be paired with a separate SE tool (pandapower, JuliaGrid, or custom WLS implementation). | + +### Bottom line for target ISO SE evaluation + +PowSyBl is not a candidate for state estimation. It is an excellent load flow and +security analysis framework with unmatched European TSO adoption, but its scope +explicitly excludes SE. For the target ISO transmission-scale SE use case, the relevant +open-source options remain: + +- **pandapower** -- has WLS SE, but Python-only and scaling concerns +- **JuliaGrid** -- purpose-built SE framework with WLS, LAV, PMU support +- **PowerModelsStateEstimation.jl** -- Julia SE on PowerModels +- **Custom WLS on PowSyBl load flow** -- theoretically possible (use PowSyBl for + Jacobian computation, implement WLS externally) but no existing implementation + +## Sources + +- [PowSyBl Open Load Flow - GitHub](https://github.com/powsybl/powsybl-open-loadflow) +- [PowSyBl Core - GitHub](https://github.com/powsybl/powsybl-core) +- [pypowsybl - GitHub](https://github.com/powsybl/pypowsybl) +- [PowSyBl Benchmark - GitHub](https://github.com/powsybl/powsybl-benchmark) +- [Observability Extensions Issue #1787](https://github.com/powsybl/powsybl-core/issues/1787) +- [PowSyBl Roadmap Wiki](https://github.com/powsybl/.github/wiki/Roadmap) +- [PowSyBl - LF Energy](https://lfenergy.org/projects/powsybl/) +- [LF Energy Project Lifecycle](https://tac.lfenergy.org/process/lifecycle.html) +- [PowSyBl European Grid Sovereignty - LF Energy](https://lfenergy.org/powsybl-a-community-led-open-source-project-for-european-grid-sovereignty/) +- [PowSyBl Case Study - Linux Foundation Europe](https://linuxfoundation.eu/resources/powsybl-open-source-powering-europe) +- [Artelys improves PowSyBl for RTE](https://www.artelys.com/news/grid-power-flow-powsybl-open-source-rte/) +- [LF Energy PowSyBl Release Announcement](https://lfenergy.org/latest-lf-energy-powsybl-release-offers-enhancements-to-load-flow-accuracy-sensitivity-analysis-and-security/) +- [pypowsybl API Reference](https://github.com/powsybl/pypowsybl/blob/main/docs/reference/index.rst) +- [PowSyBl Open Load Flow Docs](https://powsybl.readthedocs.io/projects/powsybl-open-loadflow/en/stable/) +- [pypowsybl Interface in ANDES](https://docs.andes.app/en/latest/examples/pypowsybl.html) +- [PyPSA and CGMES Discussion - OpenMod Forum](https://forum.openmod.org/t/pypsa-and-the-cim-cgmes-does-it-make-sense-to-go-down-this-road/5033) +- [RTE Wikipedia](https://en.wikipedia.org/wiki/R%C3%A9seau_de_Transport_d'%C3%89lectricit%C3%A9) diff --git a/phase2-research/se-landscape-python.md b/phase2-research/se-landscape-python.md new file mode 100644 index 00000000..0ec6ffec --- /dev/null +++ b/phase2-research/se-landscape-python.md @@ -0,0 +1,233 @@ +# Python State Estimation Landscape + +Research date: 2026-03-27 + +## Summary + +PyPSA has no native state estimation (SE). The Python ecosystem offers a small number of +production-grade SE implementations. The two strongest candidates for Phase 2 integration are +**power-grid-model** (LF Energy / Alliander) and **pandapower** (Fraunhofer IEE). ANDES +(CURENT) lists SE as a feature but is primarily a transient dynamics tool. Everything else in +the landscape is either academic one-off code, unmaintained, or distribution-only with +restrictive licensing. + +| Tool | SE Algorithms | Stars | License | Last Release | Maintained | Transmission? | +|------|--------------|-------|---------|-------------|------------|---------------| +| power-grid-model | WLS (Newton-Raphson), iterative linear | 211 | MPL-2.0 | 2026-03-26 | Very active (800+ releases) | No (distribution) | +| pandapower | WLS + chi-squared / normalized residual bad-data | 1,100 | BSD-3 | 2026-03-26 | Active | Yes | +| ANDES | Listed but underdocumented | 346 | GPL-3.0 | 2026-03-12 (v2.0.0) | Active | Yes | +| OpenPy-DSSE | Hybrid WLS (traditional + PMU) | 14 | CC BY-NC-SA 4.0 | 2022-12 | No | No (distribution) | +| PYPOWER | None | ~300 | BSD-3 | 2025-07 | Low activity | Yes (no SE) | +| Roseau Load Flow | Unconfirmed (tagged but not documented) | 63 | Proprietary (free <=10 buses) | 2026-03 | Active | No (distribution) | +| OpenDSS (via opendssdirect.py / py-dss-interface) | None natively | N/A | BSD / EPRI | Active | Active | No (distribution) | + +## Tool Survey + +### power-grid-model (LF Energy / Alliander) + +- **Repository**: https://github.com/PowerGridModel/power-grid-model +- **PyPI**: `pip install power-grid-model` (or via conda) +- **Architecture**: C++ core with Python bindings -- very high performance +- **License**: MPL-2.0 (permissive, compatible with commercial use) +- **Activity**: 10,360+ commits, 800+ releases, v1.13.31 as of 2026-03-26. Backed by + Alliander (Dutch DSO) and hosted under Linux Foundation Energy. + +**State Estimation Capabilities:** +- Two SE calculation methods: `newton_raphson` and `iterative_linear` +- Newton-Raphson SE added in v1.7 (production-ready) +- Supports separate specification of active/reactive power measurement error margins +- Measurement types: voltage magnitude, power injection, power flow, current magnitude +- Full three-phase asymmetric calculation support +- Native parallel computing for batch calculations + +**Limitations for Phase 2:** +- Designed for **distribution** grids, not transmission. The LF Energy project page + explicitly directs transmission users to PowSyBl Open Load Flow instead. +- No direct PyPSA import/export. `power-grid-model-io` supports pandapower and Vision + formats but not PyPSA natively. Integration path: PyPSA -> pandapower -> PGM. +- Focused on steady-state; no dynamic SE (EKF/UKF). + +### ANDES (CURENT, University of Tennessee) + +- **Repository**: https://github.com/CURENT/andes +- **PyPI**: `pip install andes` (v2.0.0, 2026-03-12) +- **License**: GPL-3.0+ (copyleft -- viral license, problematic for proprietary integration) +- **Stars**: 346 +- **Activity**: 4,810 commits, 18 releases. Active development. + +**State Estimation:** +- Listed as one of five analysis routines alongside power flow, time-domain simulation, + eigenvalue analysis, and continuation power flow. +- However, SE documentation is sparse. The main use case and community focus is on + transient dynamics simulation (DAE-based models with 100+ device models). +- Reads PSS/E RAW/DYR, MATPOWER, JSON, Excel formats. +- Returns results as NumPy arrays / Pandas DataFrames. + +**Assessment:** +- ANDES is a serious tool for dynamics but SE appears to be a secondary feature. +- GPL-3.0 license is a significant constraint for any proprietary deployment. +- Could be valuable if the Phase 2 SE needs to be tightly coupled with dynamic simulation. + +### pandapower (Fraunhofer IEE) -- reference only + +Already evaluated separately in this project, but included here for completeness as the +strongest Python SE implementation. + +- **WLS state estimation** with full tutorial and API documentation +- Measurement types: voltage magnitude, active/reactive power (bus, line, transformer), + current magnitude +- Bad-data detection: chi-squared test and normalized residual test +- Direct PyPSA interoperability via `pypsa.Network.import_from_pandapower_net()` (beta) +- BSD-3 license, 1,100+ stars, very active + +### PYPOWER + +- **Repository**: https://github.com/rwl/PYPOWER +- **PyPI**: `pip install PYPOWER` (v5.1.19, 2025-07-10) +- **License**: BSD-3 +- **Features**: DC/AC power flow (Newton-Raphson, Fast Decoupled), DC/AC OPF +- **State Estimation**: **None**. PYPOWER is a Python port of MATPOWER but does not include + MATPOWER's SE module. +- Low development activity. pandapower supersedes it for all practical purposes. + +### OpenDSS Python Interfaces + +Two Python packages provide access to OpenDSS: + +1. **opendssdirect.py** (DSS-Extensions): cross-platform Python bindings to an alternative + OpenDSS engine. BSD license. Active development. +2. **py-dss-interface** (EPRI): Python bindings to official EPRI OpenDSS. Active. + +**State Estimation**: OpenDSS itself has **no built-in SE**. It is a distribution system +simulator focused on time-series power flow, harmonics, and fault analysis. SE would need +to be implemented externally using OpenDSS as the network model backend. + +### OpenPy-DSSE + +- **Repository**: https://github.com/jlara6/OpenPy-DSSE +- **PyPI**: `pip install py-open-dsse` +- **License**: CC BY-NC-SA 4.0 (non-commercial, share-alike -- **not usable commercially**) +- **Stars**: 14, **Last commit**: 2022-12, **Not maintained** + +**Features:** +- Hybrid WLS combining traditional measurements and D-PMU (distribution PMU) data +- Solution methods: nonlinear WLS, linear PMU, nonlinear PMU +- Measurement types: voltage magnitude, branch power flow, current magnitude, smart meter, + zero injection, pseudo-measurements, phasor measurements +- Communicates with OpenDSS for network modeling + +**Assessment**: Academically interesting but non-commercial license and abandoned +development make it unsuitable for production use. The hybrid WLS + PMU approach is +worth studying for algorithm design inspiration. + +### Standalone WLS / EKF Libraries + +No dedicated Python library exists for power system SE using EKF or UKF. The options are: + +- **FilterPy** (`pip install filterpy`): General-purpose Bayesian filtering library with + EKF, UKF, particle filter implementations. Could be used to build a custom dynamic SE + on top of a power system model, but requires writing all the power-system-specific + measurement functions and Jacobians. +- **statsmodels WLS**: General weighted least squares regression, not power-system-aware. +- Academic GitHub repos (IEEE 14-bus WLS implementations): single-commit, no tests, no + maintenance, no documentation. Not suitable for production. + +### GridCal and pandapower + +Both are covered in separate evaluations in this project and excluded from this survey. + +## PyPSA Integration Feasibility + +### Path 1: PyPSA + pandapower SE (Recommended) + +1. Build / maintain the network model in PyPSA (OPF, unit commitment, market clearing) +2. Export to pandapower via `pypsa.Network.export_to_pandapower()` or build a parallel + pandapower net from the same data source +3. Inject SCADA/PMU measurements into the pandapower measurement tables +4. Run `pandapower.estimation.estimate()` for WLS SE with bad-data detection +5. Map estimated voltages/flows back to PyPSA components + +**Pros**: Mature SE, BSD license, documented API, existing PyPSA<->pandapower bridge. +**Cons**: Bridge is beta (missing 3-winding transformers, switches, tap positions). +Maintaining two parallel network representations adds complexity. + +### Path 2: PyPSA + power-grid-model SE + +1. Build network in PyPSA +2. Convert PyPSA -> pandapower -> power-grid-model via `power-grid-model-io` +3. Run PGM SE (Newton-Raphson or iterative linear) +4. Map results back + +**Pros**: C++ performance, LF Energy backing, asymmetric three-phase support. +**Cons**: Two-hop conversion (PyPSA->pp->PGM), distribution-grid focus may miss +transmission-level modeling needs for target ISO, no direct PyPSA converter. + +### Path 3: Custom SE on PyPSA network model + +1. Extract bus admittance matrix (Y-bus) from PyPSA network +2. Implement WLS SE using scipy.sparse + numpy (Jacobian construction, Gauss-Newton + iteration, chi-squared bad-data detection) +3. Optionally use FilterPy for dynamic SE (EKF/UKF) wrapper + +**Pros**: Full control, no license constraints, tailored to target ISO transmission topology. +**Cons**: Significant development effort (2-4 weeks for a basic WLS, longer for dynamic SE), +requires power systems expertise for correct Jacobian derivation and numerical stability. + +### Path 4: PyPSA + ANDES SE + +Feasible in principle (both read MATPOWER cases), but GPL-3.0 license creates viral +licensing concerns, and ANDES SE documentation is insufficient to assess production +readiness. Not recommended without further investigation. + +## Recommended Candidates for Phase 2 + +### Tier 1: pandapower SE via PyPSA bridge + +**Best fit for Phase 2 Stage 2.** Mature WLS implementation with bad-data detection, +permissive license, active maintenance, and an existing (beta) PyPSA conversion path. +The primary risk is the beta status of the PyPSA-pandapower bridge for complex network +topologies (target ISO has 3-winding transformers and phase shifters). + +**Action items:** +- Test the PyPSA->pandapower bridge with the target ISO network topology +- Verify measurement injection workflow in pandapower +- Benchmark SE solve time for target ISO-scale networks (~3,000 buses) + +### Tier 2: power-grid-model (if distribution-level SE is needed) + +Best-in-class performance for distribution grids. If Phase 2 requires distribution-level +SE (e.g., behind-the-meter DER visibility), PGM is the strongest option. Not ideal for +target ISO transmission-level SE due to its distribution focus. + +### Tier 3: Custom WLS SE on PyPSA + +Fall-back if pandapower bridge proves inadequate for the target ISO topology. Building a +lightweight WLS SE directly on PyPSA's Y-bus avoids the conversion overhead but requires +dedicated development time. + +### Not recommended + +- **ANDES**: GPL-3.0 license, underdocumented SE, dynamics-focused +- **OpenPy-DSSE**: Non-commercial license, abandoned +- **PYPOWER**: No SE capability +- **OpenDSS**: No SE capability +- **Roseau Load Flow**: Proprietary license, distribution-only, SE unconfirmed + +## Sources + +- [ANDES GitHub](https://github.com/CURENT/andes) +- [ANDES PyPI](https://pypi.org/project/andes/) +- [power-grid-model GitHub](https://github.com/PowerGridModel/power-grid-model) +- [power-grid-model LF Energy page](https://lfenergy.org/projects/power-grid-model/) +- [power-grid-model v1.7 SE announcement](https://lfenergy.org/power-grid-model-v1-7-now-available-adding-the-newton-raphson-calculation-method-for-enhanced-state-estimation/) +- [power-grid-model-io pandapower converter](https://power-grid-model-io.readthedocs.io/en/stable/converters/pandapower_converter.html) +- [pandapower SE documentation](https://pandapower.readthedocs.io/en/v2.3.1/estimation.html) +- [pandapower SE tutorial notebook](https://github.com/e2nIEE/pandapower/blob/master/tutorials/state_estimation.ipynb) +- [PyPSA import_from_pandapower_net](https://docs.pypsa.org/v0.30.2/api/_source/pypsa.Network.import_from_pandapower_net.html) +- [PYPOWER PyPI](https://pypi.org/project/PYPOWER/) +- [OpenDSSDirect.py GitHub](https://github.com/dss-extensions/OpenDSSDirect.py) +- [py-dss-interface PyPI](https://pypi.org/project/py-dss-interface/) +- [OpenPy-DSSE GitHub](https://github.com/jlara6/OpenPy-DSSE) +- [GitHub state-estimation topic (Python)](https://github.com/topics/state-estimation?l=python) +- [Roseau Load Flow GitHub](https://github.com/RoseauTechnologies/Roseau_Load_Flow) +- [FilterPy EKF docs](https://filterpy.readthedocs.io/en/latest/kalman/ExtendedKalmanFilter.html) diff --git a/phase2-research/se-landscape-remaining.md b/phase2-research/se-landscape-remaining.md new file mode 100644 index 00000000..6edefe2b --- /dev/null +++ b/phase2-research/se-landscape-remaining.md @@ -0,0 +1,321 @@ +# Remaining Non-Python SE Landscape + +## Summary + +This document surveys state estimation (SE) capabilities in non-Python open-source power +system tools that were not covered in the prior Python-focused or academic landscape reports. + +| Tool | Language | Has SE? | License | Maturity | +|------|----------|---------|---------|----------| +| InterPSS | Java | No | Apache 2.0 | Medium -- active but narrow scope | +| PSAT | MATLAB/Octave | No (native) | GPL | High -- widely used in academia | +| HELM-based tools | Various | No dedicated SE | Various | Low -- power flow only | +| GridPACK | C++ (HPC) | Yes | BSD 2-Clause | Medium-High -- PNNL-backed | +| DPsim | C++ | No (native) | MPL 2.0 | Medium -- real-time simulation focus | +| RTDS | Proprietary HW+SW | N/A (commercial) | Commercial | High -- industry standard for HIL | +| JuliaGrid | Julia | Yes -- comprehensive | MIT | Medium -- recent (2025), strong SE | +| PowerModelsDistSE | Julia | Yes -- distribution | BSD | Medium -- research prototype | +| GridAPPS-D SE | C++ | Yes -- distribution | BSD | Medium -- DOE/PNNL-backed | +| PowSyBl | Java | No (not yet) | MPL 2.0 | High -- but SE on roadmap only | +| OpenDSS | Delphi/COM | Partial -- via external | BSD | High -- EPRI-backed, SE via COM | + +**Key findings:** +- The strongest non-Python SE implementations are **JuliaGrid** (transmission-scale WLS/LAV + with PMU support, tested to 70k buses) and **GridPACK** (C++ HPC SE with Kalman filter). +- **GridAPPS-D** provides a C++ WLS distribution SE within the DOE platform. +- Most other tools (InterPSS, PSAT, DPsim, PowSyBl) focus on power flow and dynamics, + with SE either absent or only achievable through external integration. +- RTDS is fully commercial and not applicable to an open-source evaluation. + +--- + +## InterPSS (Java) + +**State estimation: No** + +InterPSS (Internet technology-based Power System Simulator) is a Java-based open-source +simulator developed by an international team (US, Canada, China). It uses an Eclipse-based +plugin architecture. + +**Implemented capabilities:** +- AC and DC load flow +- Short circuit analysis +- Transient stability simulation +- Distribution system analysis +- DC power supply system analysis + +**Planned (not yet implemented):** relay coordination, harmonics, dynamic (small-signal) +stability, reliability. + +State estimation is not mentioned in the documentation, GitHub repositories +(ipss-common, ipss-plugin, ipss-odm, ipss20, ExtendedPiecewiseAlgo), or the project +overview. The plugin architecture could theoretically support SE as an extension, but +no such plugin exists. + +- **GitHub:** https://github.com/InterPSS-Project +- **Website:** https://sites.google.com/a/interpss.org/interpss/Home +- **License:** Apache 2.0 (per GitHub) +- **Last active:** 2026 (repositories show recent commits) + +--- + +## PSAT (MATLAB/Octave) + +**State estimation: No (native)** + +PSAT (Power System Analysis Toolbox) by Federico Milano is one of the most widely used +open-source power system toolboxes in academia. It runs on MATLAB and GNU Octave. + +**Core capabilities:** +- Power flow (Newton-Raphson) +- Continuation power flow (CPF) +- Optimal power flow (OPF) +- Small-signal stability analysis (eigenvalue) +- Time-domain simulation +- N-1 contingency analysis +- PMU placement analysis +- FACTS and wind turbine models +- Simulink-based network editor + +**Regarding SE:** Despite some third-party academic papers using PSAT in conjunction with +state estimation research, PSAT itself does not include a built-in SE module. The +documentation (version 2.1.11) lists power flow, CPF, OPF, small-signal stability, and +time-domain simulation as the supported routines -- SE is not among them. Researchers have +used PSAT's COM/scripting interface to feed network models into external SE algorithms +(typically in MATLAB), but this is user-implemented, not a PSAT feature. + +- **Website:** http://faraday1.ucd.ie/psat.html (cert expired as of 2026-03) +- **GitHub mirror:** https://github.com/cuihantao/PSAT +- **License:** GPL +- **Status:** Mature but largely in maintenance mode; last documented version 2.1.11 + +--- + +## HELM-based SE + +**State estimation: No** + +The Holomorphic Embedding Load Flow Method (HELM) is a mathematically guaranteed +convergent power flow technique (no iterative divergence risk). Open-source +implementations exist: + +- **HELMpy** (Python 3) -- power flow solvers only (HELM + Newton-Raphson). No SE. +- **JosepFanals/HELM** (Python) -- HELM power flow implementation. No SE. +- **GridCal** includes a HELM power flow solver adapted from ASU research, but GridCal's + SE module uses conventional WLS, not HELM-based estimation. + +No open-source project implements state estimation using the holomorphic embedding +approach. HELM remains a power flow technique; its mathematical properties (analytic +continuation, Pade approximants) have not been adapted for SE in any publicly available +code. + +--- + +## GridPACK (C++ HPC) + +**State estimation: Yes** + +GridPACK is a C++ framework from Pacific Northwest National Laboratory (PNNL) for +developing power grid applications on high-performance computing (HPC) platforms. It is +one of the few non-Python tools with a mature, purpose-built SE module. + +**SE capabilities:** +- Weighted Least Squares (WLS) state estimation +- Kalman filter dynamic state estimation (added as a separate application module) +- Designed for distributed/parallel execution on HPC clusters using MPI +- Demonstrated scaling on the IEEE 118-bus system in distributed SE prototype + +**Other applications:** +- AC power flow +- Dynamic simulation (transient stability) +- Contingency analysis +- Real-time path rating + +**Architecture:** GridPACK provides a component-based framework where network topology is +distributed across MPI processes. Custom bus/branch components define the SE measurement +model. Mappers convert the network model into sparse algebraic systems solved via PETSc +or other backends. + +- **GitHub:** https://github.com/GridOPTICS/GridPACK +- **Docs:** https://gridpack.readthedocs.io/en/latest/ +- **License:** BSD 2-Clause +- **Backed by:** US DOE / PNNL +- **Language:** C++ (93.9%), with Python wrappers available + +--- + +## DPsim (C++) + +**State estimation: No (native)** + +DPsim is a real-time capable dynamic power system simulator developed at RWTH Aachen +(Institute for Automation of Complex Power Systems). The simulation core is C++ with +Python bindings. + +**Core capabilities:** +- Electromagnetic transient (EMT) simulation +- Dynamic phasor (DP) simulation +- Steady-state power flow (for initialization) +- Real-time execution (time steps down to 50 microseconds) +- CIM/CGMES model import +- VILLASnode interface for hardware-in-the-loop + +**Regarding SE:** DPsim itself does not implement state estimation. However, it is part +of the SOGNO platform (sogno.energy), which pairs DPsim with **pyVolt** -- a separate +Python package that performs SE using CIM network models. In the SOGNO architecture, +DPsim acts as the real-time simulator providing synthetic measurements, and pyVolt +consumes those measurements for SE. This is an integration pattern, not a native DPsim +feature. + +- **GitHub:** https://github.com/sogno-platform/dpsim +- **License:** MPL 2.0 +- **pyVolt (companion SE):** https://github.com/sogno-platform/pyvolt (Python, separate package) + +--- + +## RTDS + +**State estimation: N/A (commercial product)** + +RTDS (Real-Time Digital Simulator) is a **commercial** hardware+software platform from +RTDS Technologies Inc. (Winnipeg, Canada). It is not open source. + +**What it is:** +- Custom FPGA-based hardware running electromagnetic transient simulations in real time +- Industry standard for hardware-in-the-loop (HIL) testing of protection relays, HVDC + controls, and FACTS devices +- Used by utilities, equipment manufacturers, and research labs worldwide + +**Regarding SE:** RTDS does not perform state estimation itself. It is used as a +real-time simulation environment to *test and validate* external SE algorithms. +Researchers have connected RTDS to MATLAB-based SE via software-in-the-loop (SIL), +feeding simulated RTU/PMU measurements to external estimators. RTDS provides the +"ground truth" simulation, not the estimation. + +- **Website:** https://www.rtds.com/ +- **License:** Commercial (proprietary hardware + software) +- **Open-source components:** None. Some researchers use open-source tools (OpenModelica, + ATP-EMTP) alongside RTDS for model development, but RTDS itself is closed. + +--- + +## Other Tools Found + +### JuliaGrid (Julia) -- Noteworthy + +**State estimation: Yes -- comprehensive** + +JuliaGrid is an open-source Julia package specifically designed for power system state +estimation. Published in a 2025 paper (arXiv:2502.18229), it is the most feature-complete +open-source SE framework outside of Python. + +**SE algorithms:** +- Nonlinear WLS (polar coordinates, Gauss-Newton) +- Robust WLS (orthogonal method, Peters-Wilkinson method) +- Least Absolute Value (LAV) estimator +- Linear SE with PMUs only (rectangular coordinates) +- DC state estimation (voltage angles only) + +**Measurement support:** +- SCADA legacy: bus voltage magnitude, branch current magnitude, active/reactive power + flows and injections +- PMU: voltage and current phasors (polar or rectangular), correlated error handling + +**Additional features:** +- Observability analysis (flow islands, maximal observable islands) +- Observability restoration via pseudo-measurements +- Optimal PMU placement +- Bad data detection via normalized residuals +- Sparse inverse for efficient residual computation + +**Scale tested:** 10,000 / 25,000 / 70,000 bus systems. On a 70,000-bus system, +processed 577,242 measurements with bad data analysis completing in ~1.2 seconds. + +- **GitHub:** https://github.com/mcosovic/JuliaGrid.jl +- **Docs:** https://mcosovic.github.io/JuliaGrid.jl/stable/ +- **License:** MIT +- **Paper:** https://arxiv.org/abs/2502.18229 + +### PowerModelsDistributionStateEstimation.jl (Julia) + +**State estimation: Yes -- distribution systems** + +Extension of PowerModelsDistribution.jl (LANL) for three-phase unbalanced distribution +network SE. Research-oriented flexible framework. + +**Capabilities:** +- Multiple power flow formulations for SE (AC, LinDist, SDP relaxation) +- WLS and other estimation criteria +- Three-phase unbalanced models +- Designed for benchmarking SE formulations, not production speed + +- **GitHub:** https://github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl +- **License:** BSD +- **Affiliation:** KU Leuven / Electa research group + +### GridAPPS-D State Estimator (C++) + +**State estimation: Yes -- distribution systems** + +A C++ WLS state estimator built as a core service in the DOE GridAPPS-D platform for +advanced distribution management systems. + +**Capabilities:** +- Weighted Least Squares estimation +- Processes voltage, current, power, and switch status measurements +- Integrated with CIM-based distribution network models +- Real-time streaming via ActiveMQ message bus + +- **GitHub:** https://github.com/GRIDAPPSD/gridappsd-state-estimator +- **License:** BSD +- **Backed by:** US DOE / PNNL +- **Dependencies:** SuiteSparse, ActiveMQ-CPP + +### PowSyBl (Java) -- No SE Yet + +PowSyBl is a major Java-based open-source framework from RTE (French TSO), hosted under +LF Energy. It supports load flow, security analysis, sensitivity analysis, short-circuit, +and dynamic simulation. State estimation is **not yet implemented** but appears on the +project roadmap (observability extensions issue #1787 in powsybl-core). Given PowSyBl's +industrial backing and the CIM/CGMES data model support, SE could eventually appear. + +- **Website:** https://www.powsybl.org/ +- **License:** MPL 2.0 + +### OpenDSS (Delphi/COM) -- Partial + +OpenDSS (EPRI) supports distribution state estimation conceptually through its COM +interface. The recommended approach is to extract the system Y-matrix and voltage data +via COM, then run SE in an external program (MATLAB, Python). OpenDSS provides the +detailed feeder model and load allocation ("calibration") but not a self-contained SE +solver. Third-party libraries like OpenPy-DSSE (Python) bridge this gap. + +- **Website:** https://www.epri.com/pages/sa/opendss +- **License:** BSD + +--- + +## Sources + +- [InterPSS Overview](https://sites.google.com/a/interpss.org/interpss/Home/overview) +- [InterPSS GitHub](https://github.com/InterPSS-Project) +- [InterPSS arXiv paper](https://arxiv.org/pdf/1711.10875) +- [PSAT GitHub mirror](https://github.com/cuihantao/PSAT) +- [PSAT documentation (Amazon)](https://www.amazon.com/Power-System-Analysis-Toolbox-Documentation/dp/B091F4NGZ8) +- [HELMpy GitHub](https://github.com/HELMpy/HELMpy) +- [GridPACK PNNL](https://www.pnnl.gov/projects/gridpacktm-open-source-framework-developing-high-performance-computing-simulations-power) +- [GridPACK GitHub](https://github.com/GridOPTICS/GridPACK) +- [GridPACK Distributed SE paper](https://www.pnnl.gov/publications/distributing-power-grid-state-estimation-hpc-clusters-system-architecture-prototype) +- [DPsim GitHub](https://github.com/sogno-platform/dpsim) +- [SOGNO State Estimation example](https://sogno.energy/docs/examples/state-estimation/) +- [pyVolt GitHub](https://github.com/sogno-platform/pyvolt) +- [RTDS Technologies](https://www.rtds.com/) +- [JuliaGrid paper (arXiv)](https://arxiv.org/abs/2502.18229) +- [JuliaGrid GitHub](https://github.com/mcosovic/JuliaGrid.jl) +- [JuliaGrid docs](https://mcosovic.github.io/JuliaGrid.jl/stable/) +- [PowerModelsDistributionStateEstimation.jl](https://github.com/Electa-Git/PowerModelsDistributionStateEstimation.jl) +- [GridAPPS-D State Estimator](https://github.com/GRIDAPPSD/gridappsd-state-estimator) +- [GridAPPS-D docs](https://gridappsd.readthedocs.io/en/master/hosted_applications/) +- [PowSyBl](https://www.powsybl.org/) +- [PowSyBl roadmap](https://github.com/powsybl/.github/wiki/Roadmap) +- [OpenDSS and State Estimation](https://opendss.epri.com/OpenDSSandStateEstimation.html) diff --git a/phase2-research/state-estimation-investigation.md b/phase2-research/state-estimation-investigation.md new file mode 100644 index 00000000..9694c418 --- /dev/null +++ b/phase2-research/state-estimation-investigation.md @@ -0,0 +1,162 @@ +# State Estimation Tooling Investigation + +> Cross-cutting investigation for Issue #115 Item 8. +> Research date: 2026-03-28. + +## Summary + +**None of the six evaluated tools provide production-ready state estimation (SE) for transmission-scale grids.** Two tools (pandapower, GridCal) have native SE implementations, but both have critical limitations that prevent production use at target ISO scale. Two others (MATPOWER, PowerModels ecosystem) have SE in extras or ecosystem packages, but these are dormant or distribution-only. PyPSA and PowerSimulations.jl have no SE capability at all. + +Outside the six evaluated tools, **JuliaGrid.jl** (MIT license, University of Sarajevo) is the most complete open-source SE framework — validated at 70,000-bus scale with AC SE, DC SE, PMU-only linear SE, observability analysis, and bad data detection. **power-grid-model** (Alliander/LF Energy, MPL-2.0) provides fast C++-backed SE but targets distribution grids. + +No open-source tool provides hybrid SCADA+PMU state estimation or a production-ready real-time SE pipeline. + +## Per-Tool Findings + +### pandapower — Native SE, Not Production-Ready at Scale + +**Status:** Most feature-rich Python SE implementation available. + +| Aspect | Assessment | +|--------|------------| +| Algorithms | WLS, IRWLS (SHGM robust), LP/LAV, scipy optimization, AF-WLS (novel, for non-observable distribution grids) | +| Bad data detection | Chi-squared + largest normalized residual — docs warn "not very robust at this time"; open bug since 2022 (#1451) | +| Observability analysis | Measurement count heuristic only (2n−k); no topological observability analysis | +| Scalability | Convergence failures reported above ~89 buses (case89pegase); SimBench ~1000-bus networks fail | +| PMU support | `va`/`ia` measurement types accepted; thin testing, known test bug (#2524) | +| Three-phase SE | Not supported | +| Production deployments | None known | +| Recent activity | Concentrated in v3.0.0–v3.1.2 (Mar–Jun 2025); no SE changes since | + +**Verdict:** Suitable for research and small-network prototyping. Would require significant hardening (scaling fixes, robust bad data detection, observability analysis) for target ISO-scale operational use. + +### GridCal — Native SE, Educational Quality + +**Status:** WLS framework exists but has critical gaps. + +| Aspect | Assessment | +|--------|------------| +| Algorithms | 4 WLS solvers: Newton-Raphson, Levenberg-Marquardt, Gauss-Newton, Decoupled LU (broken) | +| Bad data detection | Coded (b-test) but **entirely commented out** in all solvers | +| Observability analysis | Can detect unobservable buses; no redundancy profiling. Issue #419 (open 7 months, stalled) | +| Scalability | Untested beyond textbook cases | +| PMU support | None | +| Filename | `state_stimation_driver.py` (typo — indicative of limited review) | + +**Verdict:** Educational/textbook quality only. Missing bad data detection alone disqualifies it for real grid operations. + +### MATPOWER — Dormant Community Extras + +**Status:** Two SE modules in `extras/`, both academic legacy code. + +| Module | Author | Last Active | Key Feature | +|--------|--------|-------------|-------------| +| `extras/se/` (mx-se) | Rui Bo | 2019 | WLS + observability analysis (`isobservable`) | +| `extras/state_estimator/` | J.S. Thorp | ~2013 | WLS + chi-squared bad data detection | + +Both use dense matrices (no sparse optimization), have no PMU support, no robust estimation, and zero integration with MATPOWER 8's `mp.extension` API. Three open bugs on mx-se (filed 2024) remain unaddressed. + +**Verdict:** Academic demonstrations only. Not maintained. + +### PowerModels.jl — Distribution-Only Ecosystem Package + +**Status:** No native SE in PowerModels core. + +**PowerModelsDistributionStateEstimation.jl (PMDSE):** 40 stars, BSD-3, KU Leuven. Supports WLS, WLAV, MLE with relaxed variants and 3 bad data detection methods. **Distribution networks only** — cannot be applied to transmission SE without major modification. Last release v0.7.0 (Oct 2023). + +No `PowerModelsStateEstimation.jl` exists for transmission networks. + +**Verdict:** Not applicable for target ISO transmission-level SE. + +### PyPSA — No SE + +**Status:** Zero SE capability, zero community efforts (no issues, PRs, forks, or discussions). + +PyPSA's numerical building blocks (Y-bus, Newton-Raphson, scipy sparse) could theoretically support a custom WLS SE implementation (~500–1000 lines for a basic version), but this would be building from scratch. + +**Verdict:** Confirmed gap. SE must come from a companion tool. + +### PowerSimulations.jl — No SE + +**Status:** Zero SE capability across the entire NREL Sienna ecosystem (57 repositories checked). Sienna is scoped to operations simulation (unit commitment, economic dispatch), which is a fundamentally different problem class. + +**Verdict:** No SE, no plans for SE. + +## Open-Source SE Landscape + +### Tier 1: JuliaGrid.jl — Most Complete Open-Source SE + +| Metric | Value | +|--------|-------| +| Repository | github.com/mcosovic/JuliaGrid.jl | +| Stars | 48 | +| License | MIT | +| Language | Julia | +| Last push | 2026-02-06 | +| Contributors | 2 (essentially single-author) | +| Publication | arXiv:2502.18229, Feb 2025 / SoftwareX | + +**SE capabilities:** +- Nonlinear AC SE (polar coordinates, SCADA-type measurements) +- Linear PMU-only SE (rectangular coordinates — deterministic, no convergence issues) +- DC state estimation +- WLS, LAV, orthogonal WLS estimators +- Observability analysis (for both SCADA and PMU configurations) +- Bad data detection (chi-squared + largest normalized residual) +- Optimal PMU placement (integer LP via JuMP) + +**Scale validation:** 10,000, 20,000, and 70,000-bus systems benchmarked. Bad data analysis on 70k buses (577,242 measurements) completes in ~1.2 seconds. + +**Limitations:** Single-author academic project (bus factor = 1). No hybrid SCADA+PMU fusion. Julia language adds a deployment dependency. + +### Tier 2: power-grid-model (Alliander/LF Energy) + +| Metric | Value | +|--------|-------| +| Repository | github.com/PowerGridModel/power-grid-model | +| Stars | 211 | +| License | MPL-2.0 | +| Language | C++ core, Python bindings | +| Activity | 10,360+ commits, 800+ releases | + +**SE capabilities:** Newton-Raphson and iterative linear SE, three-phase asymmetric support, native batch parallelism. + +**Limitation:** Designed for **distribution grids**, not transmission. LF Energy project page explicitly directs transmission users to PowSyBl Open Load Flow. No direct PyPSA converter (requires PyPSA → pandapower → PGM two-hop conversion). + +### Tier 3: ANDES / CURENT LTB + +| Metric | Value | +|--------|-------| +| Repository | github.com/CURENT/andes | +| Stars | 346 | +| License | GPL-3.0+ (copyleft — problematic for proprietary use) | +| Language | Python | + +SE listed as one of five analysis routines, but documentation is sparse and the tool's strength is transient dynamics simulation. LTB platform (ANDES + DiME messaging) is the closest thing to a real-time SE platform in open source, but it's a research platform, not deployable. + +### Notable Gaps in the Landscape + +1. **No hybrid SCADA+PMU SE** exists in open source — all tools provide separate traditional or PMU-based SE +2. **No production-ready real-time SE pipeline** — all tools are batch/offline solvers +3. **ML-based SE** (PINNs, GNNs) shows promise but is not production-ready + +## Recommendation for Phase 2 Stage 2 + +Since PyPSA (the recommended Phase 2 tool) has no SE capability, state estimation will require either a companion tool or custom development. Three viable paths: + +### Path A: pandapower SE via PyPSA bridge (simplest) +- Use PyPSA for OPF/planning, export to pandapower for SE +- Existing (beta) `pypsa.Network.import_from_pandapower_net()` bridge +- **Risk:** pandapower SE convergence fails above ~89 buses; bridge is beta for complex topologies + +### Path B: JuliaGrid.jl as SE core (most capable) +- JuliaGrid provides the most complete SE feature set (PMU support, observability, bad data, 70k-bus scale) +- Would require Julia in the deployment stack and a data bridge to PyPSA +- **Risk:** Single-author project, Julia deployment dependency + +### Path C: Custom WLS SE on PyPSA's network model (most control) +- Build a lightweight WLS SE using PyPSA's Y-bus + scipy sparse +- Estimated effort: 2–4 weeks for basic WLS, longer for bad data detection + observability +- **Risk:** Significant development effort; reinventing existing solutions + +**Bottom line:** No evaluated tool provides production-ready SE for target ISO-scale transmission grids. Phase 2 Stage 2 will require dedicated SE tooling work regardless of which path is chosen. The decision should be deferred to Phase 2 scoping, informed by the data pipeline architecture and measurement availability (SCADA vs PMU). diff --git a/report/docs/index.mdx b/report/docs/index.mdx index c37636c1..714cebe4 100644 --- a/report/docs/index.mdx +++ b/report/docs/index.mdx @@ -37,7 +37,7 @@ Six open-source power system modeling tools (PyPSA, PowerModels.jl, PowerSimulat PowerModels.jl ranks second on the strength of its Strong Extensibility and its JuMP foundation, which provides the most flexible constraint injection API among all evaluated tools. It demonstrated native piecewise-linear cost curves and PTDF computation with sub-1e-11 error at 10,000 buses. PowerModels.jl was not selected for three reasons: -1. **Adequate Expressiveness reflects the cost of JuMP flexibility.** 6 of 11 expressiveness tests required user-assembled code averaging 269 lines per test, compared to PyPSA's native API calls. +1. **Adequate Expressiveness reflects the cost of JuMP flexibility.** Half of the expressiveness tests required user-assembled JuMP code averaging 487 lines per test, compared to PyPSA's native API calls. 2. **Adequate Maturity carries sustainability risk.** 82.4% of commits come from a single contributor (bus factor of 1), raising concerns about long-term maintenance. 3. **Julia adoption barrier.** Julia remains a niche language for many analyst teams, imposing a workforce ramp-up cost that Python-based tools avoid. diff --git a/report/docs/results/expressiveness.mdx b/report/docs/results/expressiveness.mdx index 3aa774da..30a5ac29 100644 --- a/report/docs/results/expressiveness.mdx +++ b/report/docs/results/expressiveness.mdx @@ -30,7 +30,7 @@ reveal API limitations that would surface in production use. | Tool | Tier | Rationale | |------|------|-----------| -| **PyPSA** | **Strong** | 8 of 11 tests pass natively, including SCOPF (A-9), lossy DC OPF (A-10), and multi-period storage (A-12). One blocking gap: distributed slack OPF (A-11). | +| **PyPSA** | **Strong** | 8 of 10 tests pass natively, including SCOPF (A-9), lossy DC OPF (A-10), and multi-period storage (A-12). One blocking gap: distributed slack OPF (A-11). | | **pandapower** | **Weak** | 4 of 10 tests pass; 4 independent blocking failures in SCUC (A-5), lossy OPF (A-10), distributed slack (A-11), and multi-period storage (A-12). Design scope limited to single-period steady-state analysis. | | **GridCal** | **Adequate** | Covers core PF/OPF and SCOPF (A-9) via native LODF formulation. Two blocking formulation bugs: battery energy balance sign error (A-12) and distributed slack hardcoded off (A-11). Soft branch constraints in DC OPF (A-3) reduce formulation fidelity. | | **PowerModels.jl** | **Adequate** | Strong native OPF with LMP decomposition (A-10) and multi-period storage (A-12). No native SCUC (A-5) and no distributed slack (A-11). SCOPF achievable through the JuMP extension API (A-9). | @@ -239,7 +239,7 @@ linear cost constraints. ## Summary PyPSA is the clear leader in expressiveness, earning a Strong tier with 8 of -11 tests passing natively and the broadest coverage of market-operations +10 tests passing natively and the broadest coverage of market-operations formulations (SCOPF, lossy OPF, multi-period storage). Its sole blocking gap, distributed slack OPF (A-11), is shared by every evaluated tool except MATPOWER (where it is achievable via post-processing). diff --git a/report/docs/results/head-to-head.mdx b/report/docs/results/head-to-head.mdx index 42ccc7cd..bc82a741 100644 --- a/report/docs/results/head-to-head.mdx +++ b/report/docs/results/head-to-head.mdx @@ -100,7 +100,7 @@ in translation. Phase 2 requires support for at least PSS/E v31 format. | PowerSimulations | Workaround | PowerSystems.jl PTI parser supports v33/v35 only; v31 fixed-width column support is absent (P2-1, G-FNM-1). | | pandapower | Gap | No PSS/E parser of any kind. A production converter is estimated at 2-4 weeks of development effort (P2-1, G-FNM-1). | | GridCal | Workaround | Parser declares v29-v35 support but is hardcoded to v35 field layout; v31 files fail (P2-1, G-FNM-1). | -| MATPOWER* | **Native** | `psse2mpc()` supports PSS/E Rev 23-33+ natively. The reference DCPF verification confirms correct FNM ingestion at 27,862 buses (P2-1, G-FNM-3). | +| MATPOWER* | **Native** | `psse2mpc()` supports PSS/E Rev 23-33+ natively. The reference DCPF verification confirms correct FNM ingestion at ~28,000 buses (P2-1, G-FNM-3). | diff --git a/report/docs/results/scalability.mdx b/report/docs/results/scalability.mdx index 0cb3f62c..3d8ef82d 100644 --- a/report/docs/results/scalability.mdx +++ b/report/docs/results/scalability.mdx @@ -228,7 +228,7 @@ MATLAB runtime. Grades reflect protocol validation performance only. MATPOWER demonstrates strong scalability for power flow at SMALL scale: DCPF in 0.10s (C-1), ACPF in 0.17s (C-2), DC OPF in 0.51s (C-3). FNM -evidence confirms DCPF scales to the 27,862-bus network in 0.22s, +evidence confirms DCPF scales to the ~28,000-bus network in 0.22s, suggesting MEDIUM PF performance would be competitive. However, the Weak tier reflects a cascading failure: the GLPK exit flag integration bug blocks SCUC result extraction at SMALL (C-4 fail), which triggers the diff --git a/report/docs/tools-evaluated.mdx b/report/docs/tools-evaluated.mdx index a827fcda..2e02e838 100644 --- a/report/docs/tools-evaluated.mdx +++ b/report/docs/tools-evaluated.mdx @@ -44,7 +44,7 @@ Tools are ordered by final rank. MATPOWER is included as a reference benchmark o #### Key Strengths -1. **Broadest native expressiveness.** PyPSA passes 8 of 11 expressiveness tests across all target problem types, including lossy DC OPF with LMP decomposition (A-10) and multi-period storage OPF (A-12). It is the only tool to achieve Strong on the highest-priority criterion. +1. **Broadest native expressiveness.** PyPSA passes 8 of 10 expressiveness tests across all target problem types, including lossy DC OPF with LMP decomposition (A-10) and multi-period storage OPF (A-12). It is the only tool to achieve Strong on the highest-priority criterion. 2. **Zero-friction extensibility.** The Linopy `extra_functionality` callback enables custom constraint injection with full dual extraction in minimal code. All 8 extensibility tests pass (B-1 through B-9), including solver swap, PTDF extraction, and graph-based network analysis. #### Key Weaknesses diff --git a/report/selection-report-v10.md b/report/selection-report-v10.md index dde99d18..c256a900 100644 --- a/report/selection-report-v10.md +++ b/report/selection-report-v10.md @@ -59,7 +59,7 @@ GridCal's Maturity grade of C triggers disqualification (bus factor 1, zero CI t **Scenario 3 — Scalability as top priority** (Scalability → Expressiveness → Extensibility → Accessibility → Maturity): -GridCal's B Scalability leads the field. PyPSA drops to #4 because its C+ Scalability reflects a HiGHS single-threaded MILP timeout on the SCUC SMALL test (C-4 fail). This grade is solver-bound, not an architectural ceiling: PyPSA ACPF scales to 10K buses (C-5 MEDIUM pass) and FNM DCPF runs exact at 27,862 buses in 31.3s (G-FNM-3 pass). PowerModels vs PowerSimulations: tied on Scalability (B-), Expressiveness (B-), and Extensibility (A-); Accessibility breaks the tie — PowerModels B- > PowerSimulations C+ → PowerModels #2, PowerSimulations #3. PyPSA C+ vs pandapower C+: Expressiveness PyPSA B+ > pandapower C+ → PyPSA #4, pandapower #5. +GridCal's B Scalability leads the field. PyPSA drops to #4 because its C+ Scalability reflects a HiGHS single-threaded MILP timeout on the SCUC SMALL test (C-4 fail). This grade is solver-bound, not an architectural ceiling: PyPSA ACPF scales to 10K buses (C-5 MEDIUM pass) and FNM DCPF runs exact at ~28,000 buses in 31.3s (G-FNM-3 pass). PowerModels vs PowerSimulations: tied on Scalability (B-), Expressiveness (B-), and Extensibility (A-); Accessibility breaks the tie — PowerModels B- > PowerSimulations C+ → PowerModels #2, PowerSimulations #3. PyPSA C+ vs pandapower C+: Expressiveness PyPSA B+ > pandapower C+ → PyPSA #4, pandapower #5. | Rank | Tool | |------|------| diff --git a/sweep-data/v10-to-v11/.progress.yaml b/sweep-data/v10-to-v11/.progress.yaml deleted file mode 100644 index 2239e5f0..00000000 --- a/sweep-data/v10-to-v11/.progress.yaml +++ /dev/null @@ -1,8 +0,0 @@ -source_version: v10 -target_version: v11 -tools_available: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] -issues_in_scope: 0 -completed_states: [INIT, SWEEP, PROBE, AGGREGATE, GENERATE, VALIDATE] -current_state: DONE -timestamp: "2026-03-14T00:00:00Z" -notes: "All evaluations on main branch — results accessed via sweep worktree at same path" diff --git a/sweep-data/v10-to-v11/aggregation/comparison-matrices.md b/sweep-data/v10-to-v11/aggregation/comparison-matrices.md deleted file mode 100644 index 06571ea2..00000000 --- a/sweep-data/v10-to-v11/aggregation/comparison-matrices.md +++ /dev/null @@ -1,233 +0,0 @@ -# Cross-Tool Comparison Matrices — v10-to-v11 Sweep - -**Key:** -- `P` = pass -- `QP` = qualified_pass -- `F` = fail -- `I` = informational (not graded) -- `—` = skipped / not applicable / blocked by gate -- `?` = test not run / unknown - -**Dominant factor abbreviations:** capability (C), infrastructure (I), network (N), test_design (TD) - -**Signal levels:** High (H), Medium (M), Low (L) - -**Outcome spread:** Number of distinct graded outcomes across tools (P/QP/F; excludes —, I, ?) - ---- - -## Suite G — Gate Ingestion Tests - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| G-1 TINY | P | P | P | P | P | P | 1 | L | — | -| G-2 SMALL | P | P | P | P | P | P | 1 | L | — | -| G-3 MEDIUM | P | P | P | P | P | P | 1 | L | — | - -**Notes:** Universal pass across all tools. Low signal by design. See T-14 and PC-14. Retain as gates but exclude from pass rate statistics. - ---- - -## Suite A — Problem Expressiveness - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| A-1 DCPF | P | P | P | P | P | P | 1 | L | — | -| A-2 ACPF | P | P | P | P | QP | P | 2 | M | C | -| A-3 DCOPF | P | P | P(soft) | P | P | P | 1* | H | TD | -| A-4 ACPF (loaded) | P | P | P | P | P | P | 1 | L | — | -| A-5 SCUC | QP | F | P | QP | P | QP | 3 | H | N/TD | -| A-6 SCED | QP | F | QP | QP | QP | P | 3 | M | N/TD | -| A-7 | — | — | — | — | — | — | — | — | — | -| A-8 | — | — | — | — | — | — | — | — | — | -| A-9 SCOPF | QP | F | QP | QP | QP | F | 3 | M | N/TD | -| A-10 Lossy DCOPF | P | F | QP | F | F | F | 3 | H | C | -| A-11 Dist. Slack OPF | QP(block) | F | QP | QP | P | QP | 3 | M | C | -| A-12 Multi-Period Storage | P | F | P | P | QP | QP | 3 | H | C | - -**A-3 note:** gridcal DCOPF uses soft branch flow constraints (probe-005 confirmed_issue). The 'P' for gridcal is misleading — branch 2_3_1 reaches 103.5% loading. If hard-constraint enforcement were required, gridcal A-3 = F. See T-10, PC-10. - -**A-5 note:** Spread of 3 is substantive, but cycling evidence is weak across all tools (T-02). SCUC formulation completeness is demonstrated; binding behavioral proof is not (PC-02). - -**A-9 note:** All QP/F results reflect network-level N-1 infeasibility or radial-topology limitations rather than tool capability gaps (T-16, PC-16). probe-009: inconclusive (PowerModels Benders). - ---- - -## Suite B — Extensibility - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| B-1 Custom constraints | QP | QP | QP | P | P | QP | 2 | M | C | -| B-2 Custom cost function | P | P | P | P | P | P | 1 | L | — | -| B-3 Contingency sweep | P | P | P | P | P | P | 1 | L | — | -| B-4 Stochastic scenario | P | P | QP | P | P | P | 2 | L | C | -| B-5 Interoperability | P | P | P | P | P | QP | 2 | L | TD | -| B-6 Architecture audit | P | P | P | P | P | P | 1 | L | — | -| B-7 | — | — | — | — | — | — | — | — | — | -| B-8 Ref. bus config | P | P | P | P | P | P | 1 | L | TD | -| B-9 PTDF extraction | P | P | P | P | P | P | 1 | L | — | - -**B-1 note:** All QP outcomes arise from different friction points (PyPSA: linopy internals, pandapower: PYPOWER monkey-patch, gridcal: API verbosity, matpower: OPF extension API complexity). The spread=2 is meaningful. - -**B-4 note:** gridcal QP reflects TapPhaseControl enum bug (single-version bug, likely fixable). - -**B-5 note:** matpower QP is arguably ambiguous — 3 lines meets the <5 LOC criterion for minimal export; 12 lines for production-quality (matpower-F09). See PC-07 for qualified_pass severity discussion. - -**B-8 note:** All P outcomes are vacuous for DC OPF — LMP is invariant to slack bus choice. See T-11, PC-11. - ---- - -## Suite C — Scalability - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| C-1 DCPF MEDIUM | — | — | P | P | P | — | 1 | L | TD | -| C-2 ACPF MEDIUM | — | — | P | P | P | — | 1 | L | TD | -| C-3 DCOPF MEDIUM | — | — | P | P | QP | — | 2 | M | C | -| C-4 SCUC SMALL | F | F | QP | F | P | F | 3 | H | C/N | -| C-5 ACPF SMALL | P | P | P | P | P | P | 1 | L | — | -| C-5 ACPF MEDIUM | P | — | P | P | P | — | 1 | L | — | -| C-7 Solver swap MEDIUM | — | — | P | P | P | — | 1 | L | I | -| C-8 SCOPF SMALL | — | — | P | P | QP | P | 2 | M | C | -| C-8 SCOPF MEDIUM | — | — | P(vac) | P(nc) | QP(crash) | — | 2 | L | N/TD | -| C-9 PTDF MEDIUM | — | — | P | P | P | — | 1 | L | — | -| C-10 Dist. Slack MEDIUM | — | — | P | P | P | — | 1 | L | — | - -**C-1, C-2, C-3 note:** pypsa, pandapower, matpower have these skipped by C-SMALL gate despite evidence of MEDIUM-scale capability (e.g., pypsa/pandapower pass G-FNM-3 at 27K buses; pandapower solves DCPF at 28K in 0.4s). Gate design conflates MILP and LP/PF scalability (T-01, PC-01). - -**C-3 note:** powersimulations QP because StaticBranchUnbounded removed all branch flow limits — the 'DCOPF' is actually an unconstrained ED (powersimulations-F01). See T-03. - -**C-4 note:** Highest spread (3) and highest signal. Genuine capability differentiation. gridcal QP via snapshot workaround (no inter-temporal coupling). powersimulations P at 404s single-threaded (may be much faster multi-threaded; powersimulations-F16). - -**C-8 MEDIUM note:** 'vac' = vacuous pass (zero redispatch on uncongested network); 'nc' = non-converged (1 Benders iteration); 'crash' = HiGHS OTHER_ERROR. None represent genuine SCOPF capability evidence. See T-03, PC-03. - ---- - -## Suite D — Accessibility - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| D-1 Install timing | I | I | I | I | I | I(est) | — | L | — | -| D-2 Documentation | I | I | I | I | I | I | — | M | C | -| D-3 Examples | I | I | I | I | I | I | — | M | C | -| D-4 Error quality | I | I | I | I | I | I | — | H | C | -| D-5 API ergonomics | I | I | I | I | I | I | — | M | C | - -**D-4 note:** gridcal D-4 'poor' rating is partially confounded by soft-constraint formulation design (gridcal-F07). The test uses zero-rated branches expecting infeasibility detection, but soft constraints absorb the violation. True LP infeasibility (load > capacity) was not tested. - -**D-3 note:** powersimulations 0/10 example pass rate inflated by PowerSystemCaseBuilder dependency gap (powersimulations-F13). Cross-tool comparison on D-3 should account for ecosystem packaging differences between Python and Julia tools. - ---- - -## Suite E — Maturity & Sustainability - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| E-1 Release cadence | I | I | I | I | I | I | — | M | — | -| E-2 CI/CD | I | I | I | I | I | I | — | M | — | -| E-3 Test coverage | I | I | I | I | I | I | — | M | — | -| E-4 Issue response | I | I | I | I | I | I | — | M | — | -| E-5 Deprecation policy | I | I | I | I | I | I | — | L | — | -| E-6 Core maintainers | I | I | I | I | I | I | — | H | — | -| E-7 Operational adoption | I | I | I | I | I | I | — | H | — | - -**E-7 note:** gridcal's claimed adoption (Redeia, Schneider Electric, GE Vernova) originates primarily from project's own documentation (gridcal-F11). Unverified from public sources. - ---- - -## Suite F — Supply Chain - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| F-1 Dependency count | P | P | P | P | P | P | 1 | L | — | -| F-2 Transitive deps | P | P | P | P | P | P | 1 | L | — | -| F-3 License audit | P | P | P | QP(ZIB) | P | P | 2 | H | I | -| F-4 CVE history | P | P | P | P | P | P | 1 | L | — | -| F-5 Build reproducibility | P | P | P | P | P | P | 1 | L | — | -| F-6 Source inspectability | P | P | P | P | P | P | 1 | L | — | -| F-7 Native extensions | P | P | P | P | P | P | 1 | L | — | -| F-8 Solver dependency | P | P | P | P(wrong) | P | P | 1 | H | I | -| F-9 Container isolation | P | P | P | P | P | P | 1 | L | — | - -**F-3 / F-8 note:** PowerModels F-3 correctly classifies SCIP_jll v0.2.1 as ZIB Academic. F-8 incorrectly claims Apache 2.0 — probe-010 confirmed this is wrong (Apache 2.0 switch was at SCIP 8.0.3, not 8.0.0). F-8 P for powermodels should be QP. See T-13, PC-13. - ---- - -## Suite G-FNM — FNM Ingestion Suite - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | Spread | Signal | Dom. Factor | -|------|-------|------------|---------|-------------|-----------------|---------|--------|--------|-------------| -| G-FNM-1 PSS/E ingestion | F | F | F | QP | ? | F | 2 | H | I | -| G-FNM-2 Field coverage | P | ? | QP | P | ? | — | 2 | M | C | -| G-FNM-3 DCPF accuracy | P | F | QP | F | F | P | 3 | H | C/I | -| G-FNM-4 ACPF convergence | ? | F | F | F | ? | I | 2 | H | C/I | -| G-FNM-5 Supplemental data | ? | ? | ? | ? | ? | I | — | M | C | - -**G-FNM-1 note:** 4 of 6 tools have no PSS/E v31 ingestion path (pypsa: no importer, pandapower: no CSV import, gridcal: v35-hardcoded parser, powermodels: v31 header crash). matpower ingested directly (native format). All non-matpower tools used MATPOWER fallback. See T-09, PC-09. - -**G-FNM-3 note:** -- pypsa P: probe-001 confirmed actual deviations are 1.07e-8 deg (floating-point noise); PASS grade correct. -- gridcal QP: 326 branches with deviations up to 562,955% (transformer-adjacent pattern, probe-007 plausible_with_caveats). Material DCPF limitation for real networks. -- powermodels F: DCPPowerModel ignores transformer taps; 2.43% pass rate. DCMPPowerModel not tested (test design constrained to solve_dc_pf; pm-F04). -- powersimulations F: PowerFlows.jl simplified B-matrix ignores tap ratios; 86.8% bus angles outside 1-deg tolerance. -- matpower P: Self-referential (reference generated by MATPOWER itself; matpower-F02). -- pandapower F: 596.6% max branch flow deviation; 101-bus cluster with 14-21 deg angle bias; classified as data_ingestion_error. - ---- - -## P2 Readiness (Informational Only) - -| Test | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | -|------|-------|------------|---------|-------------|-----------------|---------| -| P2-1 PSS/E parsing | I | I | I | I | — | — | -| P2-2 AC OPF feasibility | I | I | I | I | — | — | -| P2-3 Multi-area coordination | I | I | I | I | — | — | - -P2 tests are informational only and not graded. Not included in cross-tool comparison statistics. - ---- - -## Summary Table: Graded Outcomes by Test Suite - -| Suite | pypsa | pandapower | gridcal | powermodels | powersimulations | matpower | -|-------|-------|------------|---------|-------------|-----------------|---------| -| G (gate) | 3P | 3P | 3P | 3P | 3P | 3P | -| A (express.) | 8P 3QP 0F | 4P 0QP 6F | 6P 4QP 0F | 5P 5QP 2F | 7P 4QP 1F | 7P 2QP 1F | -| B (extend.) | 6P 2QP 0F | 7P 1QP 0F | 7P 1QP 0F | 8P 0QP 0F | 8P 0QP 0F | 7P 1QP 0F | -| C (scale) | 2P 0QP 1F (+7—) | 2P 0QP 0F (+8—) | 9P 2QP 0F | 9P 1QP 1F | 8P 1QP 1F (+1—) | 4P 1QP 1F (+8—) | -| F (supply) | 9P 0QP 0F | 9P 0QP 0F | 9P 0QP 0F | 8P 1QP 0F | 9P 0QP 0F | 9P 0QP 0F | -| G-FNM | 1P 0QP 1F (+3?) | 0P 0QP 2F (+3?) | 1P 1QP 1F (+2?) | 1P 1QP 2F (+1?) | 0P 0QP 1F (+4?) | 2P 0QP 1F (+2I) | - -**Footnote:** '—' = skipped by gate; '?' = not run / unknown; 'I' = informational. -D and E suites are all informational across all tools and omitted from this summary. - ---- - -## High-Signal Test Summary - -Tests with High signal and spread >= 2 — the tests most likely to produce useful cross-tool differentiation: - -| Test | Signal | Spread | Key Pattern | -|------|--------|--------|------------| -| C-4 SCUC SMALL | H | 3 | Genuine capability split: tools with/without native SCUC | -| G-FNM-3 DCPF | H | 3 | B-matrix tap-ratio formulation matters at real-grid scale | -| A-10 Lossy DCOPF | H | 3 | Only pypsa passes; lossy formulation rare in this tool set | -| A-12 Multi-Period Storage | H | 3 | Differentiated by multi-period OPF + storage API | -| F-3 License audit | H | 2 | SCIP ZIB Academic for PowerModels (probe-010 confirmed) | -| A-3 DCOPF (hard constraint) | H | — | GridCal soft constraints confirmed (probe-005); requires protocol fix | -| G-FNM-1 PSS/E ingestion | H | 2 | Universal v31 parser gap; infrastructure finding | - ---- - -## Probe Integration Summary - -| Probe | Tool | Test | Classification | Impact on Matrix | -|-------|------|------|---------------|-----------------| -| probe-001 | pypsa | G-FNM-3 | claim_debunked (weak) | P grade confirmed; report precision is display artifact | -| probe-003 | pandapower | A-3 | claim_supported | 46/46 shadow prices are real; sweep concern refuted | -| probe-005 | gridcal | A-3 | confirmed_issue | GridCal A-3 P is misleading; soft constraints confirmed | -| probe-007 | gridcal | G-FNM-3 | classification_plausible_with_caveats | QP defensible but understates severity; no magnitude cap | -| probe-009 | powermodels | A-9 | inconclusive | Benders mechanism real; convergence never demonstrated | -| probe-010 | powermodels | F-3/F-8 | claim_supported (F-3) | SCIP ZIB Academic at pinned version; F-8 P should be QP | -| probe-013 | powersimulations | A-2 | claim_debunked | Iteration count at @info; return type = convergence guarantee; A-2 QP overstates limitation | -| probe-016 | matpower | A-5 | claim_debunked | GLPK failure is genuine (GLP_ETMLIM, status=-1); not an exit-flag mapping bug | diff --git a/sweep-data/v10-to-v11/aggregation/deferred-issues.yaml b/sweep-data/v10-to-v11/aggregation/deferred-issues.yaml deleted file mode 100644 index 41055837..00000000 --- a/sweep-data/v10-to-v11/aggregation/deferred-issues.yaml +++ /dev/null @@ -1,12 +0,0 @@ ---- -# Deferred Issues — v10-to-v11 Sweep -# GitHub issues that were open during the sweep period and required integration. -# Source: sweep-data/v10-to-v11/github-issues.yaml - -deferred_issues: [] - -note: >- - The github-issues.yaml query (2026-03-14) returned zero issues with the - 'protocol' label. No GitHub issue integration was required for this sweep. - All protocol improvement proposals are generated purely from per-tool findings - and probe results. diff --git a/sweep-data/v10-to-v11/aggregation/low-signal-tests.yaml b/sweep-data/v10-to-v11/aggregation/low-signal-tests.yaml deleted file mode 100644 index 99cd7831..00000000 --- a/sweep-data/v10-to-v11/aggregation/low-signal-tests.yaml +++ /dev/null @@ -1,133 +0,0 @@ ---- -# Low-Signal Tests — v10-to-v11 Sweep -# Tests identified as providing insufficient discriminative information -# for cross-tool comparison, despite being technically valid. -# Categorized by signal_level: Low and the pattern observed. - -low_signal_tests: - - - test_id: G-1 - test_name: "Network ingestion — TINY (case39)" - signal_level: low - reason: universal_pass - description: >- - All six evaluated tools pass G-1. The test verifies element counts (39 buses, - 46 branches, 10 generators) after loading case39.m. Any tool with a working MATPOWER - importer passes trivially. For tools where MATPOWER is the native format (MATPOWER - itself), the test is definitionally passed. The only informative outcome would be a - FAIL, which would indicate a broken importer — not a capability distinction. - affected_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - finding_refs: [pypsa-F11, pandapower-F09, gridcal-F12, powersimulations-F15, matpower-F07] - proposed_change: PC-14 - recommendation: >- - Retain as minimum-bar gate. Exclude pass outcomes from cross-tool pass rate - statistics. Add optional data quality sub-check (cost function import completeness). - - - test_id: G-2 - test_name: "Network ingestion — SMALL (ACTIVSg2000)" - signal_level: low - reason: universal_pass - description: >- - All evaluated tools pass G-2. The SMALL 2000-bus network adds scale but the - ingestion test remains a minimum-bar check. The only discriminative finding - would be a meaningful load time difference, but load times are dominated by - JIT compilation (Julia tools) or import overhead rather than parser performance, - making cross-tool timing comparison misleading. - affected_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - finding_refs: [pypsa-F11, pandapower-F09, gridcal-F12, powersimulations-F15, matpower-F07] - proposed_change: PC-14 - recommendation: >- - Same as G-1. Retain as gate. Exclude from pass rate numerator/denominator. - - - test_id: G-3 - test_name: "Network ingestion — MEDIUM (ACTIVSg10k)" - signal_level: low - reason: universal_pass - description: >- - All evaluated tools pass G-3. The 10,000-bus MEDIUM network ingestion is again - a minimum-bar check. Timing information (e.g., pypsa 27s vs pandapower 3s) is - more meaningful at this scale but is not part of the pass criterion. - affected_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - finding_refs: [pypsa-F11, pandapower-F09, gridcal-F12, powersimulations-F15, matpower-F07] - proposed_change: PC-14 - recommendation: >- - Same as G-1/G-2. Consider making load time at MEDIUM a recorded metric (not graded) - for cross-tool informational comparison. - - - test_id: C-5 - test_name: "AC feasibility progressive relaxation" - signal_level: low - reason: no_binding_constraints - description: >- - The C-5 progressive relaxation test (0%, 10%, 20% thermal limit relaxation) produces - identical ACPF solutions at all relaxation levels because the SMALL network - (ACTIVSg2000) and MEDIUM network (ACTIVSg10k) have no binding thermal constraints - at base loading. For matpower: 0 of 3,206 branches are binding at SMALL. For - powersimulations: thermal limit relaxation does not affect NR-based ACPF at all. - The test measures ACPF convergence capability (which passes) but provides no - diagnostic signal about the tool's ability to handle progressively relaxed constraints - because no constraints bind to begin with. - affected_tools: [matpower, powersimulations] - finding_refs: [matpower-F08, powersimulations-F14] - proposed_change: null - recommendation: >- - Run C-5 on a network with pre-confirmed binding constraints, or increase loading - until at least one thermal constraint is active. Alternatively, use voltage - constraints (Q-limits) which are more likely to bind at base loading. - - - test_id: B-6 - test_name: "Code architecture audit" - signal_level: low - reason: no_runtime_execution - description: >- - B-6 is a static documentation and source code review with no runtime execution. - The test produces accurate architectural findings (e.g., PowerModels' four-layer - structure) but these findings are reproducible by any source code inspection. - The pass/fail status does not differentiate tool capability in any runtime-measurable - way. The test contributes to the extensibility grade but the 'pass' outcome is - effectively guaranteed for any tool with documented layered architecture. - affected_tools: [powermodels] - finding_refs: [pm-F11] - proposed_change: null - recommendation: >- - Retain B-6 as a documentation quality assessment. Its findings (layering, - composability, customization surface area) remain useful qualitative signal. - Do not weight B-6 equally with runtime extensibility tests (B-1, B-2, B-3) - in the extensibility grade. - - - test_id: B-8 - test_name: "Reference bus configurability" - signal_level: low - reason: vacuous_pass_condition - description: >- - B-8 requires LMP variation across slack configurations, but in standard DC OPF, - LMPs are invariant to slack bus choice — any correctly implemented DC OPF will - show zero LMP variation (or uniform shift for distributed slack), which trivially - satisfies 'changes consistently.' The test verifies API configurability (can you - set a different reference bus without error?) but the LMP variation component of - the pass condition is vacuous for all correct implementations. Only AC OPF or - lossy DC OPF formulations would show meaningful LMP sensitivity to slack bus. - affected_tools: [pypsa, matpower] - finding_refs: [pypsa-F01, matpower-F06] - proposed_change: PC-11 - recommendation: >- - Redesign pass condition to test API configurability explicitly. Remove LMP - variation requirement for DC OPF. Add AC OPF variation test as optional if - the tool supports AC OPF. - - - test_id: D-1 - test_name: "Install-to-first-solve timing" - signal_level: low - reason: measurement_not_executable - description: >- - D-1 requires measuring wall-clock time from clean environment to first DCPF solve. - For matpower, the result records timing_source: estimated with wall_clock_seconds: - null. An estimated timing is not comparable to measured timings from other tools. - The qualitative friction findings (interactive installer, PATH dependencies) are - useful, but the timing claim itself provides no valid cross-tool comparison data. - affected_tools: [matpower] - finding_refs: [matpower-F10] - proposed_change: PC-15 - recommendation: >- - Require actual stopwatch timing from a clean devcontainer state. Results with - timing_source: estimated should be flagged as incomplete for cross-tool comparison. diff --git a/sweep-data/v10-to-v11/aggregation/proposed-changes.yaml b/sweep-data/v10-to-v11/aggregation/proposed-changes.yaml deleted file mode 100644 index 2ecb8136..00000000 --- a/sweep-data/v10-to-v11/aggregation/proposed-changes.yaml +++ /dev/null @@ -1,399 +0,0 @@ ---- -# Proposed Protocol, Rubric, and Skill Changes — v10-to-v11 Sweep -# Evidence threshold: 3+ tools for protocol/rubric changes; 2+ for skill-only changes. -# All proposals are framed as improvements to the evaluation framework, not judgments about tools. - -proposed_changes: - - - id: PC-01 - type: redesign_test - target: C-SMALL gate (C-4 dependency for C-1, C-2, C-3, C-7, C-9, C-10) - title: "Decouple LP/PF MEDIUM tests from the MILP C-SMALL gate" - rationale: >- - The C-SMALL gate ties all MEDIUM tests to C-4 (SCUC on SMALL), which requires MILP - capability. LP and power-flow MEDIUM tests (C-1 DCPF, C-2 ACPF, C-3 DC OPF, C-9 - PTDF, C-10 distributed slack) are independent of MILP and provide essential - scalability evidence that is currently suppressed for any tool failing C-4. A tool - that lacks SCUC (pandapower, powermodels) or whose MILP solver fails at scale - (pypsa, matpower) cannot have its LP/PF scalability measured, producing structurally - incomplete scalability grades that undersell tools with genuinely strong LP performance. - evidence_tools: [pypsa, pandapower, matpower, powermodels] - evidence_findings: [pypsa-F09, pandapower-F01, matpower-F01, pm-F13] - priority: high - implementation: >- - The C-SMALL gate should gate only tests that depend on MILP: C-4 itself (SCUC SMALL) - and any future MILP MEDIUM tests. LP/PF MEDIUM tests (C-1, C-2, C-3, C-7, C-9, C-10) - should run unconditionally. C-8 SCOPF (which uses LP) should also be gated only by - C-3 (DCOPF MEDIUM) rather than C-4. - - - id: PC-02 - type: modify_test - target: A-5 - title: "Add min up/down binding verification to SCUC pass condition" - rationale: >- - A-5's 2-cycling-generator threshold is easily met on the 10-generator 39-bus network - but does not verify that min up/down time constraints are actually binding. With a - 1.18x capacity-to-load ratio and generous headroom, the optimizer can satisfy the - cycling count by cost-driven shutdowns without the temporal constraints ever being - active. This means a tool could pass A-5 using an LP relaxation that happens to - produce near-binary outputs. Adding a binding check (re-run with min_up=min_down=0 - and compare commitment schedules) would verify behavioral correctness, not just - formulation completeness. - evidence_tools: [pypsa, powermodels, powersimulations, matpower, gridcal] - evidence_findings: [pypsa-F02, pm-F09, powersimulations-F12, matpower-F05, gridcal-F04] - priority: high - implementation: >- - Pass condition addition: "At least one generator's commitment schedule must change when - min_up_time=min_down_time=0 (binding verification). The MIP gap must be extractable - from the solver and reported." The TINY network's capacity margin may also need to - be reduced (e.g., remove 2 of the highest-capacity generators) to force more cycling. - - - id: PC-03 - type: redesign_test - target: C-8 (SCOPF MEDIUM) - title: "Replace ACTIVSg10k with a congested MEDIUM network for SCOPF" - rationale: >- - The ACTIVSg10k MEDIUM network is uncongested (max branch loading ~84.72%), so - N-1 contingencies produce no post-contingency violations requiring redispatch. - Any correctly functioning SCOPF produces results numerically identical to base-case - DCOPF on this network. C-8 MEDIUM therefore provides no signal about SCOPF - expressiveness or the quality of security-constrained redispatch — only about - solve time and solver robustness. Three of three tools that ran C-8 MEDIUM - confirmed this vacuousness independently. - evidence_tools: [gridcal, powermodels, powersimulations] - evidence_findings: [gridcal-F02, pm-F02, powersimulations-F09] - priority: high - implementation: >- - Option A: Load-scale the ACTIVSg10k network to ~120% of its base case until at - least 5% of branches are above 95% loading. Option B: Use ACTIVSg2000 (SMALL) as - the benchmark network for SCOPF (which has confirmed congestion at C-8 SMALL level). - Option C: Use a synthetic stressed version of ACTIVSg10k with reduced thermal - ratings on critical paths. The pass condition must include minimum redispatch - magnitude (e.g., at least 5 MW aggregate dispatch change vs base DCOPF). - - - id: PC-04 - type: modify_test - target: A-2 (ACPF), C-2 (ACPF MEDIUM) - title: "Formalize convergence verification hierarchy for tools without residual API" - rationale: >- - The protocol requires reporting the convergence residual, but at least three tools - have no public API path to extract it. Probe-013 (powersimulations) showed that the - iteration count IS available at @info log level and the return type structurally - guarantees convergence — the evaluation's proxy was weaker than the API actually - provides. The protocol should establish a tiered evidence hierarchy: (1) residual - value preferred, (2) iteration count via any API (including logging), (3) binary - convergence indicator via return type or exception, (4) voltage profile proxy as - last resort. Each tier should carry a corresponding documentation note. - evidence_tools: [pandapower, powermodels, powersimulations] - evidence_findings: [pandapower-F02, pm-F05, powersimulations-F02] - probe_evidence: [probe-013] - priority: medium - implementation: >- - Add a convergence_evidence_quality field to result files with values: - residual_reported | iteration_count_reported | binary_convergence_api | proxy_voltage. - Adjust A-2/C-2 pass conditions to accept any tier while documenting the tier achieved. - Tools with proxy-only convergence evidence should receive a mild notation in the - accessibility dimension (convergence diagnostic quality). - - - id: PC-05 - type: modify_test - target: A-6 (SCED) - title: "Add binding ramp constraint evidence requirement to A-6" - rationale: >- - A-6's pass condition accepts ramp constraint enforcement based on constraint count - (formulation) rather than binding evidence (behavioral). RTS-GMLC technology-median - ramp rates are 100-1000x larger than the dispatch changes driven by the 39-bus load - profile, ensuring no ramp constraints ever bind. This means any tool that can - formulate multi-period OPF without crashing passes A-6, with no discrimination - between tools that correctly enforce tight ramp limits and those that merely include - the constraints. Three or more tools confirmed zero binding ramp constraints. - evidence_tools: [matpower, powermodels, powersimulations] - evidence_findings: [matpower-F04, pm-F13, powersimulations-F08] - priority: medium - implementation: >- - Option A: Scale ramp rates down by a factor of 10x in gen_temporal_params.csv to - force binding. Option B: Add a mandatory re-run with 50% ramp rate tightening and - verify that at least one generator shows a different dispatch pattern. The pass - condition should require: "At least one ramp constraint dual value > 0 (binding), - with the generator index, hour, and dual value reported." - - - id: PC-06 - type: modify_test - target: A-6 (SCED) - title: "Distinguish SCED-with-UC from ED-only when SCUC is absent" - rationale: >- - When a tool cannot do SCUC, A-6 reduces to a pure economic dispatch with ramp - constraints. The rubric criterion for A-6 is Security-Constrained Economic Dispatch - (includes UC stage and security constraints), but the test result is granted the - same qualified_pass label regardless of whether UC was performed. This inflates - the apparent SCED capability for tools that only perform ED. The distinction matters - for the expressiveness grade, which should reflect the full SCED capability. - evidence_tools: [powermodels, pandapower, powersimulations] - evidence_findings: [pm-F13, pandapower-F07, powersimulations-F08] - priority: medium - implementation: >- - Add an explicit sub-category to A-6 results: sced_mode: full_sced | ed_only | - ed_with_security. Score full_sced as pass, ed_only as fail (with context that - the tool does not support the UC stage), ed_with_security as qualified_pass. - This requires the skill to check A-5 outcome before scoring A-6. - - - id: PC-07 - type: scoring_change - target: "qualified_pass status definition (rubric section: grading standards)" - title: "Add severity tiers to qualified_pass: qualified_pass vs partial_pass vs constrained_pass" - rationale: >- - The qualified_pass label is applied to results ranging from 'stable workaround - with minimal friction' to 'blocking architectural impossibility' to 'non-converged - result' to 'solver crash'. The workaround_class field documents the distinction, but - status aggregation collapses it. For cross-tool comparison matrices, qualified_pass - vs qualified_pass appears equivalent even when one is a near-pass and another is a - near-fail. This affects expressiveness and scalability grades. - evidence_tools: [pypsa, powermodels, powersimulations, pandapower, matpower] - evidence_findings: - - pypsa-F12 # A-6 stable vs A-11 blocking both qualified_pass - - pm-F02 # C-8 non-converged awarded pass - - pm-F06 # A-12 three workarounds awarded pass - - powersimulations-F09 # C-8 solver crash awarded qualified_pass - - pandapower-F10 # B-9 vs B-1 inconsistent internal-path classification - priority: high - implementation: >- - Introduce three outcome categories to replace qualified_pass: - qualified_pass: workaround exists, stable, low friction, full capability demonstrated - partial_pass: capability partially demonstrated, non-trivial workaround, some criterion unmet - constrained_pass: capability demonstrated under constraints that affect generalizability - workaround_class: blocking should map to partial_pass or fail, never qualified_pass. - Non-converged SCOPF runs should be constrained_pass. - Solver crash on grade_network should be fail with context (not qualified_pass). - - - id: PC-08 - type: modify_test - target: G-FNM-3 (and all deviation-reporting result files) - title: "Require full float64 precision for deviation reporting in G-FNM-3" - rationale: >- - The 6-decimal-place rounding in result files causes sub-1e-6 deviations to display - as 0.000000. Probe-001 confirmed that PyPSA's reported 0.0 bus angle deviation is - actually 1.07e-8 degrees — non-zero floating-point noise that rounds to zero. - While the PASS grade is unaffected, the display format conflates numerical noise - with true machine-zero agreement, which is misleading for cross-tool comparison. - A tool that uses identical code paths as the reference (true machine zero) would - be indistinguishable from a tool with correct but independent code (float64 noise). - evidence_tools: [pypsa, matpower] - evidence_findings: [pypsa-F06, matpower-F02] - probe_evidence: [probe-001] - priority: low - implementation: >- - Result files should store deviation values in scientific notation (:.6e format) rather - than fixed-point (:.6f). The skill should output max_deviation_deg using Python's - f'{value:.6e}' or Julia's @sprintf("%.6e", value) format. Pass/fail thresholds - remain unchanged; only reporting precision changes. - - - id: PC-09 - type: redesign_test - target: G-FNM-1 (FNM ingestion gate) - title: "Separate PSS/E format support assessment from power-system capability assessment in FNM suite" - rationale: >- - G-FNM-1 failure for PSS/E ingestion is a data-format support gap (interoperability - dimension) but cascades to affect G-FNM-2 (field coverage) and G-FNM-3 (DCPF - accuracy) through the MATPOWER fallback path. This conflates PSS/E format support - (which varies by tool for format-version and parser completeness reasons) with - power-system modeling capability (which should be assessed independently). All - four tools that attempted PSS/E ingestion failed, meaning the FNM suite actually - tests only MATPOWER-preprocessed input across all tools, making cross-tool comparison - conditionally equivalent rather than format-varied. - evidence_tools: [pypsa, pandapower, gridcal, powermodels] - evidence_findings: [pypsa-F05, pandapower-F13, gridcal-F08, pm-F07] - priority: high - implementation: >- - Option A: Add explicit ingestion_path metadata to all G-FNM results (native_csv | - matpower_fallback | native_psse | other) and report it in cross-tool comparisons. - G-FNM-1 FAIL should not cascade to G-FNM-2 and G-FNM-3; field coverage and DCPF - accuracy should always be assessed via the best available path. - Option B: Provide the FNM in both PSS/E and MATPOWER formats as named protocol - inputs, making the MATPOWER fallback a first-class path (not a fallback). - Option C: Separate G-FNM-1 into two sub-tests: G-FNM-1a (PSS/E format support, - informational) and G-FNM-1b (can ingest the FNM via any supported path). - - - id: PC-10 - type: modify_test - target: A-3 (DC OPF) - title: "Add hard constraint enforcement verification to DCOPF pass condition" - rationale: >- - The A-3 pass condition verifies LMP extractability and shadow price availability - but does not verify that branch thermal limits are enforced as hard constraints. - Probe-005 confirmed that GridCal's DCOPF uses soft branch flow constraints - (LP slack variables), allowing branches to exceed their thermal limits in the - optimal solution. The A-3 pass was awarded despite branch loading of 103.5%. - Standard DCOPF definitions require hard thermal limit enforcement; any tool with - soft constraints should be distinguished from one with hard constraints, as this - affects market clearing validity and congestion revenue adequacy. - evidence_tools: [gridcal] - evidence_findings: [gridcal-F06] - probe_evidence: [probe-005] - priority: high - implementation: >- - Add to A-3 pass condition: "No branch may exceed its derated thermal limit in the - optimal solution (max_loading <= 1.0 + epsilon, where epsilon = 1e-4 for numerical - tolerance). If the formulation uses soft constraints, classify as partial_pass and - document the penalty coefficient used." This requires the skill to check - max(res_line.loading_percent) after solve. Soft-constraint DCOPF is not inherently - wrong but must be explicitly labeled. - - - id: PC-11 - type: redesign_test - target: B-8 (reference bus configuration) - title: "Redesign B-8 to test API configurability rather than LMP variation" - rationale: >- - B-8 requires that 'LMP values change consistently across configurations' when - the slack bus is moved. In standard DC OPF, LMPs are invariant to slack bus - choice (the angle reference cancels out of all KCL dual variables). The pass - condition cannot be satisfied in any meaningful way for any correct DC OPF - implementation — either LMPs are invariant (correct, test passes trivially) or - they vary (incorrect behavior, test would fail). The test actually measures - whether the API allows setting a different reference bus without error, which - is a valid API configurability test but should be stated as such. - evidence_tools: [pypsa, matpower] - evidence_findings: [pypsa-F01, matpower-F06] - priority: medium - implementation: >- - Rewrite pass condition: "The tool allows setting an arbitrary reference bus without - error or model reconstruction. Three distinct configurations must complete without - exception. For DC OPF: LMP values are expected to be invariant across configurations - (this is physically correct behavior). For AC OPF: LMPs may vary; document any - variation. The test is primarily an API surface test, not an LMP sensitivity test." - - - id: PC-12 - type: add_test - target: G-FNM-3 (cross-reference validation sub-test) - title: "Add cross-tool reference independence check to G-FNM-3" - rationale: >- - MATPOWER generates the G-FNM-3 reference DCPF solution from its own format - (.m file) using its own solver. This makes MATPOWER's G-FNM-3 result self-referential - (zero deviation is mathematically guaranteed). Any tool with identical B-matrix - construction as MATPOWER would also show machine-noise-level deviation even if - the formulation were incorrect for certain branch types. The reference needs an - independent cross-check to validate that MATPOWER's solution is correct for the - full network including transformer branches. - evidence_tools: [matpower] - evidence_findings: [matpower-F02] - probe_evidence: [probe-001] - priority: low - implementation: >- - Generate a secondary reference using an independently implemented DCPF (e.g., - from PSS/E, if accessible, or from a validated OpenDSS or PSCAD solution for - a subset of buses). Alternatively: cross-validate the MATPOWER reference against - bus injection power balance (sum of all branch flows at each bus = net injection) - as a necessary condition for solution validity. This is tool-independent and - computationally trivial. - - - id: PC-13 - type: scoring_change - target: F-3, F-8 (supply chain license audit) - title: "Fix SCIP license classification and add JLL binary-vs-wrapper license guidance" - rationale: >- - Probe-010 confirmed that F-3 (ZIB Academic for SCIP_jll v0.2.1 = SCIP 8.0.0) is - correct and F-8 (Apache 2.0 claim) is incorrect. The Apache 2.0 switch happened at - SCIP 8.0.3, not 8.0.0. The F-8 supply chain grade upgrade based on the incorrect - Apache 2.0 determination should be reverted. More broadly, JLL packages in the - Julia ecosystem bundle binaries whose license may differ from the wrapper package - license; the evaluation skill should check the binary artifact license, not the - Julia package metadata. - evidence_tools: [powermodels] - evidence_findings: [pm-F08] - probe_evidence: [probe-010] - priority: high - implementation: >- - Skill update: When auditing JLL packages, record both the Julia wrapper package - license (from Pkg metadata) and the bundled binary artifact license (from the - JuliaBinaryWrappers release README and the upstream source tarball). If they differ, - the binary license governs the supply chain classification. For SCIP_jll v0.2.1: - F-3 should remain qualified_pass (commercial excluded); F-8 should be corrected - to qualified_pass. The net F-8 score should reflect ZIB Academic, not Apache 2.0. - - - id: PC-14 - type: scoring_change - target: G-1, G-2, G-3 (gate ingestion tests) - title: "Exclude gate test pass outcomes from cross-tool pass rate comparisons" - rationale: >- - All six evaluated tools pass all three gate tests, making them universally positive - noise in pass rate statistics. Including them in per-tool pass rate numerators and - denominators produces a floor that uniformly inflates apparent capabilities. The - tests serve a valid minimum-bar purpose (a fail would be informative) but pass - outcomes have zero discriminative signal. Multiple per-tool findings confirm this - pattern independently. - evidence_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - evidence_findings: [pypsa-F11, pandapower-F09, gridcal-F12, powersimulations-F15, matpower-F07] - priority: low - implementation: >- - Add a test_category: gate_minimum_bar tag to G-1/G-2/G-3 result schema. Cross-tool - comparison matrices and summary statistics should exclude gate_minimum_bar tests from - the pass rate denominator and numerator. These tests remain in the evaluation as - preconditions only. Optional enhancement: add a data quality sub-check (does cost - data import at all? does gen reactive limit data survive?) that would provide - discriminative signal if any tool fails it. - - - id: PC-15 - type: modify_test - target: C-4 (SCUC scalability), D-1 (installation timing) - title: "Standardize solver thread count reporting in scalability tests" - rationale: >- - powersimulations C-4 records 404s for SCUC with HiGHS at 1 thread on a 32-core - machine. MILP performance scales significantly with thread count for branch-and-bound. - Without thread normalization, cross-tool timing comparisons for MILP tests are - invalid: a tool that runs multi-threaded by default will appear faster than one - constrained to single-thread, even if both are using HiGHS. The D-1 installation - timing issue (matpower records an estimate, not a measurement) is a related - protocol compliance gap. - evidence_tools: [powersimulations, matpower] - evidence_findings: [powersimulations-F16, matpower-F10] - priority: medium - implementation: >- - Add cpu_threads used and cpu_threads available to all C-suite scalability result - frontmatter. For MILP tests (C-4, C-8), report both 1-thread and max-thread wall - clock if the solver supports parallel execution. D-1 should require actual stopwatch - measurement from a clean devcontainer state, not a retrospective estimate. - - - id: PC-16 - type: redesign_test - target: A-9 (SCOPF TINY) - title: "Provide a feasible N-1 SCOPF test scenario for TINY network" - rationale: >- - The IEEE 39-bus network under the Modified Tiny load profile is N-1 infeasible at - nominal ratings, and N-1 infeasible under various derating strategies. This means - no tool can demonstrate multi-iteration Benders convergence on this network — - the algorithm terminates after 1 iteration at an infeasible subproblem. Probe-009 - confirmed the finding is accurately documented but left the Benders convergence - claim inconclusive. The test design gap (no feasible SCOPF scenario) affects all - tools that attempt iterative SCOPF. - evidence_tools: [pypsa, powermodels, powersimulations] - evidence_findings: [pypsa-F04, pm-F03, powersimulations-F04] - probe_evidence: [probe-009] - priority: medium - implementation: >- - Pre-compute and publish a feasible N-1 SCOPF configuration for case39 (e.g., by - reducing the contingency set to the 10 most critical lines, or by using a reduced - load profile at 80% of peak). The protocol should provide this configuration in - eval-config.yaml so all tools use the same feasible subset. The pass condition - should require: "Benders/iterative algorithm converges to a security-constrained - dispatch in >= 2 iterations OR demonstrates the joint N-1 problem is feasible and - solved in 1 iteration." - - - id: PC-17 - type: skill_change - target: A-5 skill (SCUC evaluation) - title: "Require binary commitment variable verification in SCUC skill" - rationale: >- - GridCal's A-5 infers commitment from generator power output (gen_power > 0.1 MW - threshold), not from a binary commitment variable. If the solver is running an LP - relaxation that happens to produce near-binary values, the commitment schedule - appears valid but is not guaranteed to be MILP-generated. The skill should - attempt to extract the binary variable directly from the solver model, and if - unavailable, document it as a formulation transparency limitation. - evidence_tools: [gridcal, powermodels] - evidence_findings: [gridcal-F04, pm-F01] - priority: medium - implementation: >- - Add to the A-5 skill: after solving, attempt to read binary commitment variables - from the solver model. If not directly readable, add MIP gap extraction attempt. - If MIP gap is unavailable, add probe: re-solve with LP relaxation (all binaries - relaxed) and compare commitment schedule — if identical, flag as potential LP - relaxation rather than true MILP solve. diff --git a/sweep-data/v10-to-v11/aggregation/themes.md b/sweep-data/v10-to-v11/aggregation/themes.md deleted file mode 100644 index 342932ee..00000000 --- a/sweep-data/v10-to-v11/aggregation/themes.md +++ /dev/null @@ -1,410 +0,0 @@ -# Cross-Tool Themes — v10-to-v11 Sweep Narrative Analysis - -This document provides full narrative context for each cross-cutting theme identified in the sweep. Themes describe patterns in the evaluation protocol and rubric design — they are framed as opportunities to improve evaluation accuracy and cross-tool comparability, not as criticisms of individual tools. - -**Evidence threshold:** Protocol and rubric changes require 3+ tools. Skill-only changes require 2+ tools. - -**Probe integration:** Eight probes were executed during this sweep. Their classifications (claim_debunked, claim_supported, confirmed_issue, inconclusive) are integrated into the relevant theme narratives below. - ---- - -## T-01: C-SMALL Gate Cascades Block LP/PF MEDIUM Tests When Only MILP Fails - -**Affected tools:** pypsa, pandapower, matpower, powermodels -**Signal level:** High | **Dominant factor:** test_design -**Proposed change:** PC-01 - -### Pattern - -The C-SMALL gate requires C-4 (SCUC on 2000-bus SMALL network) to pass before any MEDIUM-tier scalability tests execute. This design made sense when the gate was introduced: SCUC is a representative stress test that exercises the full optimization stack at scale. If a tool cannot run SCUC at SMALL scale, the implicit assumption was that MEDIUM-scale LP and power flow would also be stressed. - -In practice, C-4 failures in this evaluation arise from three distinct causes, each with different implications for LP/PF scalability: - -1. **SCUC absent from tool scope** (pandapower, powermodels): These tools have no MILP unit commitment formulation. Their C-4 failure is a scope boundary, not a scale failure. pandapower's pandapower-F01 is explicit: "The SMALL gate failure was a cascaded failure from A-5 (SCUC unsupported), not a scale-related failure." The tool demonstrably solves 28,000-bus DCPF in 0.4 seconds in G-FNM-3. - -2. **MILP solver timeout at scale** (pypsa): PyPSA's HiGHS single-threaded cannot solve a 39,168-variable binary problem within 600 seconds. This is a MILP scalability limitation. But PyPSA's LP and power-flow scalability are demonstrably strong: G-FNM-3 passes DCPF at 27,862 buses. The gate prevents formally recording LP/PF MEDIUM capability because the MILP test failed. - -3. **SCUC genuinely infeasible for the solver** (matpower/GLPK): Probe-016 debunked the original matpower finding (matpower-F01 claimed a GLPK exit-flag mapping bug). The actual situation is that GLPK returns GLP_ETMLIM (time limit, errnum=9, status=-1) — no feasible integer solution was found. The SCUC problem is genuinely failing, not producing a solution that cannot be extracted. This means matpower's C-4 fail is a real solver capability gap on this problem instance, but MATPOWER's LP capability (which is excellent) is still obscured by the gate cascade. - -**Consequence:** Seven to eight MEDIUM-tier tests per affected tool receive 'skipped' status rather than outcomes, producing structurally incomplete scalability grades. Cross-tool comparison at MEDIUM scale is only possible for gridcal, powermodels, and powersimulations — the three tools that pass C-4 (or have an alternative path to MEDIUM tests). - -### Probe integration - -Probe-016 (matpower, claim_debunked) is critical here: it established that the GLPK failure in matpower is genuine (GLP_ETMLIM, status=-1, no feasible solution), not a one-line exit-flag-mapping fix as claimed in matpower-F01. This means: - -- The original matpower finding was overclaiming about the nature of the C-4 failure (suggesting a fixable bug was obscuring a solved result) -- The cascade to MEDIUM tests remains appropriate for matpower — GLPK genuinely cannot solve the SCUC -- But the gate design still blocks LP/PF MEDIUM measurement for matpower, where MIPS is demonstrably capable - -### Protocol implication - -The fix (PC-01) is structural: C-SMALL gate should conditionally block MILP MEDIUM tests only. LP/PF MEDIUM tests (C-1, C-2, C-3, C-9, C-10) should run unconditionally. This preserves the gate's minimum-bar function (a tool that fails MILP at small scale should not get credit for MILP at MEDIUM) while allowing LP and power-flow scalability evidence to be collected regardless of MILP outcome. - ---- - -## T-02: SCUC Test Network Insufficient to Verify Min Up/Down Binding - -**Affected tools:** pypsa, powermodels, powersimulations, matpower, gridcal -**Signal level:** High | **Dominant factor:** network -**Proposed change:** PC-02 - -### Pattern - -A-5's pass condition requires at least two generators to cycle (commit/decommit) in the 24-hour TINY horizon. All five tools that attempt SCUC on the 39-bus network meet this threshold. However, no tool's result verifies that the min up/down time constraints are actually binding — that is, that removing them would change the commitment schedule. - -The structural reason is the 10-generator IEEE 39-bus network's characteristics: 7,367 MW total capacity against 6,254 MW peak load (1.18x ratio). With this headroom, the optimizer can satisfy peak demand with the cheapest generators committed and simply shut down expensive units during load valleys. The cycling pattern observed (typically 2-3 gas CC units shutting down during hours 3-8 and restarting at hour 9) is driven by economic dispatch logic, not by min up/down constraints binding. The temporal constraints are present in the MILP formulation but not necessarily active at the optimal solution. - -**Specific evidence:** -- powermodels (pm-F09): "Total shutdowns: 3. Total startups: 0 (generators only decommit toward end of horizon, no recommitment needed within 24h)." This is exactly the pattern expected when min up/down constraints are not binding. -- matpower (matpower-F05): The qualified_pass was validated on ex_case3b (3 buses, 5 generators) rather than case39 due to the solver failure. The 3-bus result had only 1 cycling generator, below the protocol's 2-generator threshold. -- gridcal (gridcal-F04): "MIP gap note: The MIP gap is not directly extractable from the OptimalPowerFlowTimeSeriesResults object." Commitment is inferred from generator power output thresholding, not from binary variables. This creates ambiguity about whether the MILP is genuinely solved or an LP relaxation is producing near-binary values. - -### Protocol implication - -PC-02 proposes a binding verification sub-step: re-run SCUC with min_up_time=min_down_time=0 and compare the commitment schedule. If the schedule changes, at least one constraint was binding. This is feasible computationally (two solves on TINY) and would provide behavioral evidence rather than formulation completeness evidence. The TINY network may also need capacity margin reduction to force genuine cycling under temporal constraints. - ---- - -## T-03: SCOPF MEDIUM Result Is Vacuous — ACTIVSg10k Is Uncongested - -**Affected tools:** gridcal, powermodels, powersimulations -**Signal level:** High | **Dominant factor:** network -**Proposed change:** PC-03 - -### Pattern - -C-8 (SCOPF on MEDIUM) is intended to test whether a tool can solve a large-scale security-constrained OPF with N-1 contingency constraints and produce meaningful redispatch relative to the base-case DCOPF. The ACTIVSg10k network has a maximum branch loading of ~84.72% at base case. With this much headroom, no N-1 contingency creates post-contingency violations — the network is too lightly loaded to stress any security constraint. - -All three tools that executed C-8 MEDIUM confirmed this independently: -- **gridcal (gridcal-F02):** "The maximum dispatch difference is 2.67e-07 MW and the maximum LMP difference is 1.49e-08 $/MWh — effectively zero. No generators show dispatch changes above 1 MW, and no branches become binding." Status: pass (technically correct — the SCOPF ran to completion, produced a valid result, and it happens to equal the base case because there are no security constraints to enforce). -- **powermodels (pm-F02):** SCOPF awarded pass despite Benders decomposition completing only 1 iteration in 595 seconds. The non-convergence is not flagged in the pass condition because the C-8 protocol states it is a 'measurement test.' But a non-converged SCOPF on an uncongested network is doubly uninformative. -- **powersimulations (powersimulations-F09):** HiGHS crashes with OTHER_ERROR after 438 seconds on MEDIUM. The SCOPF crash is on a problem with no meaningful security constraints to enforce. - -**Consequence:** C-8 MEDIUM currently differentiates tools by solver robustness on an uncongested problem rather than by SCOPF expressiveness or security-constrained redispatch quality. A SCOPF on an uncongested network is equivalent to an LP that produces no active constraints — the only discrimination is whether the solver completes without error. - -### Protocol implication - -PC-03 proposes replacing ACTIVSg10k with a congested MEDIUM network for C-8, or stressing the current network until at least 5% of branches exceed 95% loading. The pass condition must include a minimum redispatch magnitude requirement. The SMALL SCOPF (C-8 SMALL on ACTIVSg2000) does produce meaningful redispatch (gridcal: up to 164 MW) and should serve as the model for the MEDIUM redesign. - ---- - -## T-04: Convergence Accepted Without Residual Across ACPF Implementations - -**Affected tools:** pandapower, powermodels, powersimulations -**Signal level:** Medium | **Dominant factor:** capability -**Proposed change:** PC-04 - -### Pattern - -The protocol requires reporting the convergence residual for ACPF solves. At least three tools have no public API path to extract the Newton-Raphson residual value after solve completion. Result files record convergence_residual: null and justify convergence via indirect proxies: - -- **pandapower (pandapower-F02):** Iteration count via private `net._ppc["iterations"]` attribute; residual reported as "below 1e-8 (tolerance_mva setting; exact value not extractable)." -- **powermodels (pm-F05):** `compute_ac_pf` returns only a Bool termination_status. No NR iteration count or residual exposed in the result dictionary. -- **powersimulations (powersimulations-F02):** PowerFlows.jl `solve_powerflow` returns a `Dict{String, DataFrame}` on convergence, `missing` on non-convergence. No residual in the return value. - -**Probe-013 (powersimulations, claim_debunked)** is the most important finding here. The claim that "PowerFlows.jl provides no convergence diagnostics" is falsified: - -1. The iteration count IS emitted at @info log level: `[ Info: The NewtonRaphsonACPowerFlow solver converged after 1 iterations. ]` — the original evaluation set `global_logger(ConsoleLogger(stderr, Logging.Error))`, suppressing @info messages. -2. The return type structurally guarantees convergence: `solve_powerflow` returns `missing` on non-convergence. A returned `Dict` is a binary convergence indicator. -3. The powersimulations A-2 qualified_pass annotation overstates the tool limitation — the API provides stronger convergence guarantees than the evaluation recognized. - -This probe result means the powersimulations A-2 status should arguably be upgraded from qualified_pass to pass, as the convergence verification methodology was unnecessarily weak. - -**The underlying pattern remains real** for pandapower (tolerance_mva bug unfixed, residual inaccessible) and powermodels (Bool-only return, NR internals not exposed). The protocol's residual requirement assumes all tools have this API capability, which is false for the Julia ecosystem tools and pandapower. - -### Protocol implication - -PC-04 proposes a tiered evidence hierarchy: (1) residual value, (2) iteration count via any API including logging, (3) binary convergence indicator via return type or exception, (4) voltage profile proxy as last resort. Evaluators should attempt all tiers and document which tier was achieved. The skill should enable Julia @info logging during ACPF solves and capture the iteration count from the log stream. - ---- - -## T-05: Ramp Constraints Not Binding in A-6/SCED - -**Affected tools:** matpower, powermodels, powersimulations -**Signal level:** Medium | **Dominant factor:** network -**Proposed change:** PC-05 - -### Pattern - -The A-6 SCED pass condition requires that ramp rate constraints are "demonstrably enforced between consecutive dispatch intervals." In all tools that attempt A-6 with ramp constraints, the constraints are formulated correctly but never bind. The root cause is that RTS-GMLC technology-median ramp rates (used to populate gen_temporal_params.csv) are derived from large generating units and are orders of magnitude larger than the dispatch changes required by the 39-bus load profile. - -**Specific evidence:** -- matpower (matpower-F04): Most constrained generator (G1) uses 48.5 MW/hr of a 62,400 MW/hr ramp limit. "No generators hit binding ramp limits." -- powersimulations (powersimulations-F08): "460 ramp constraints added. 1 binding ramp constraint observed." The single binding constraint is not identified; LMP extraction failed. -- powermodels (pm-F13): A-6 reduced to pure ED (UC absent), so ramp constraints bind only if the LP dispatch changes between hours — which requires some other constraint to force a discontinuous dispatch pattern. - -The test demonstrates that multi-period OPF formulation includes ramp terms without numerical error, but any tool that successfully runs multi-period OPF would pass this criterion trivially. - -### Protocol implication - -PC-05 proposes scaling ramp rates down (by 10x or more) in gen_temporal_params.csv to force binding. The pass condition should require at least one ramp constraint dual value greater than zero with identification of the constrained generator, hour, and dual magnitude. - ---- - -## T-06: SCED Degrades to ED-Only When SCUC Is Absent - -**Affected tools:** powermodels, pandapower, powersimulations -**Signal level:** Medium | **Dominant factor:** test_design -**Proposed change:** PC-06 - -### Pattern - -When A-5 SCUC fails or is unsupported, A-6 SCED loses the UC stage: the tool assumes all generators are committed and performs only the multi-period economic dispatch. The rubric criterion for A-6 is Security-Constrained Economic Dispatch (which includes commitment-status inputs from SCUC), but the result receives the same qualified_pass label regardless of whether UC was performed. The 'security-constrained' component (binding N-1 contingency constraints on dispatch) is also absent for most tools attempting A-6. - -This means a tool performing full two-stage SCED (UC → ED with security constraints) and a tool performing pure unconstrained multi-period ED both receive qualified_pass for A-6 — which conflates genuinely different capability levels. - -### Protocol implication - -PC-06 proposes explicit sub-categorization: `sced_mode: full_sced | ed_only | ed_with_security`. Only `full_sced` should map to pass; `ed_only` should map to fail with context. The skill needs to check A-5 outcome before setting A-6 grade. - ---- - -## T-07: qualified_pass Applied at Inconsistent Severity Levels - -**Affected tools:** pypsa, powermodels, powersimulations, pandapower, matpower -**Signal level:** High | **Dominant factor:** test_design -**Proposed change:** PC-07 - -### Pattern - -The qualified_pass status label is the most overloaded outcome in the current protocol. It is applied to: - -- **Stable workarounds with minimal friction:** pypsa A-6 (two-stage dispatch pattern, undocumented but functional), gridcal C-4 (snapshot workaround, explicitly documented) -- **Blocking architectural impossibilities:** pypsa A-11 (distributed slack DC OPF is impossible in optimize() path) — the workaround_class is 'blocking' but the label is the same qualified_pass -- **Non-converged results:** powermodels C-8 (1 Benders iteration, time budget, not converged) — awarded 'pass', not even qualified_pass, but the underlying concern about grade inflation is the same -- **Solver crashes on the grade network:** powersimulations C-8 (HiGHS OTHER_ERROR on MEDIUM) — qualified_pass despite the solver crashing before producing any result on the primary test network -- **Three simultaneous non-trivial workarounds:** powermodels A-12 (SCIP solver switch, cyclic SoC manual injection, two-phase LMP extraction) — awarded 'pass', not qualified_pass, despite three compounding non-obvious workarounds that a new user would not discover - -The workaround_class field (stable / fragile / blocking) documents the severity within qualified_pass, but this information is not surfaced in the aggregated outcome that drives grade calculations. A cross-tool comparison matrix entry of 'QP' is equally opaque for a near-pass and a near-fail. - -### Protocol implication - -PC-07 proposes three severity tiers to replace or supplement qualified_pass: - -- **qualified_pass:** Stable workaround exists, full capability demonstrated, low friction. Current meaning. -- **partial_pass:** Non-trivial workaround, some criterion unmet, moderate friction. Maps to workaround_class: fragile or blocking. Should score lower than qualified_pass in grade calculations. -- **constrained_pass:** Capability demonstrated under constraints that affect generalizability (e.g., uncongested network, single-threaded solver, simplified formulation). Informative but weaker evidence than qualified_pass. - -Additionally: non-converged SCOPF runs should be constrained_pass at best; solver crashes on the grade network should be fail with context; workaround_class: blocking should never map to qualified_pass — it should map to partial_pass or fail. - ---- - -## T-08: G-FNM Result Precision — 6-Decimal Rounding Obscures Floating-Point Accuracy - -**Affected tools:** pypsa, matpower -**Signal level:** Low | **Dominant factor:** test_design -**Proposed change:** PC-08 - -### Pattern - -Result files store deviation values with `round(..., 6)` (6 decimal places). Probe-001 (pypsa G-FNM-3, claim_debunked) confirmed that the reported 0.0 deviation values are display artifacts: the actual maximum bus angle deviation is 1.07e-8 degrees (non-zero floating-point noise from two independently implemented DCPF solvers that round to 0.000000 at 6 decimal places). - -The PASS grade for pypsa is fully warranted — the deviations are at the limit of float64 precision for values of this scale, and all buses pass the 1.0-degree tolerance by a factor of ~100 million. The validation-report.md discrepancy (which listed G-FNM-3 as FAIL) was confirmed to be a stale artifact from before the shared matpower_loader fix was applied. - -The probe also resolved the discrepancy between the result file (PASS) and validation-report.md (FAIL): the validation report reflects an older intermediate state and was not regenerated after the loader fix. The result file is the authoritative source. - -The display precision issue is minor but matters for cross-tool comparison: a tool with true machine-zero agreement (identical code paths as MATPOWER) would be indistinguishable from a tool with correctly equivalent but independently coded formulations (float64 noise at ~1e-8). The former would indicate something potentially concerning (shared code dependency), the latter is expected and correct. - -### Protocol implication - -PC-08 proposes requiring scientific notation (:.6e format) for deviation values in G-FNM results. The change is cosmetic and does not affect grades. - ---- - -## T-09: PSS/E v31 Parser Failures Force MATPOWER Fallback Across the FNM Suite - -**Affected tools:** pypsa, pandapower, gridcal, powermodels -**Signal level:** High | **Dominant factor:** infrastructure -**Proposed change:** PC-09 - -### Pattern - -The FNM data is provided as a PSS/E v31 RAW file with an intermediate CSV export derived from PSS/E v31 record types. Zero of the six evaluated tools successfully ingested the FNM via the PSS/E native path: - -- **pypsa:** No PSS/E importer exists in the codebase (architecture gap) -- **pandapower:** No intermediate CSV import capability; no PSS/E import path -- **gridcal:** PSS/E parser hardcoded to v35 field counts; v31 RAW triggers "1 elements expected, 18 expected" exceptions -- **powermodels:** PSS/E v31 parser crashes on the Case Identification header line - -All non-MATPOWER tools fall back to the fnm_main_island.m MATPOWER file (a pre-cleaned main-island subset with documented record deficits: bus -8.1%, load -42.7% vs the PSS/E manifest). This means all G-FNM test results measure tool performance on a MATPOWER-preprocessed representation of the FNM, not on the FNM directly. - -**Consequences:** - -1. The FNM suite was designed to test format interoperability (G-FNM-1) and power-system modeling capability (G-FNM-2 through G-FNM-5) as distinct dimensions. The universal PSS/E failure collapses these: all tools receive the same pre-processed MATPOWER input, making the "does the tool support the format?" question separately assessable only for MATPOWER. - -2. Cross-tool G-FNM-3 DCPF comparisons are conditioned on the same fallback preprocessor, making them more comparable than if different tools had ingested different format versions — but this comparability comes at the cost of measuring the preprocessor rather than the tools. - -3. G-FNM-4 ACPF non-convergence findings for pandapower and gridcal are attributed partly to "PPC import path loses AC-critical transformer data" — a characteristic of the fallback path, not necessarily of the tool's ACPF solver. - -### Protocol implication - -PC-09 proposes separating PSS/E format support assessment from power-system capability assessment: G-FNM-1 should be a pure format support test (informational) that does not cascade to block G-FNM-2 through G-FNM-5. Field coverage and DCPF accuracy should always be assessed via the best available path, with the path explicitly tagged in results. Providing the FNM in both PSS/E and MATPOWER formats as first-class named inputs (not fallbacks) would normalize this across tools. - ---- - -## T-10: GridCal DCOPF Uses Soft Branch Flow Constraints (Confirmed by Probe) - -**Affected tools:** gridcal -**Signal level:** High | **Dominant factor:** capability -**Proposed change:** PC-10 - -### Pattern - -Probe-005 (gridcal A-3, confirmed_issue) provided definitive evidence that GridCal's `linear_opf` uses soft branch flow constraints: explicit LP slack variables `flow_slacks_pos` and `flow_slacks_neg` per branch. The source code inspection (`VeraGridEngine/Simulations/OPF/opf_driver.py` line 170-171) and runtime verification (branch 2_3_1 reaches 103.5% loading in the optimal solution — 103.5% is only achievable with soft constraints) together establish this as a confirmed formulation characteristic, not an inference. - -The A-3 pass verdict is misleading: the three pass conditions (convergence, LMP extractability, binding branch count) do not verify that thermal limits are enforced as hard constraints. A standard DCOPF enforces hard thermal limits — this is universally required for correct market clearing (hard limits produce LMPs reflecting true congestion; soft limits produce LMPs reflecting penalty costs that may not correspond to marginal congestion relief). - -The D-4 error quality test corroborates this: setting all branch ratings to zero was expected to produce an infeasible LP, but GridCal instead produced a 'feasible' solution where all branches violated their limits via slack absorption. This is internally consistent with the soft-constraint formulation but was incorrectly interpreted as an error reporting quality failure — partially unfair to GridCal's design choice (soft constraints are a legitimate numerical stabilization technique) while also missing the harder question of whether they are appropriate defaults for market applications. - -This is currently a GridCal-specific confirmed issue — no other evaluated tool uses soft branch flow constraints by default in DCOPF. However, the protocol gap (A-3 does not verify hard constraint enforcement) is a cross-tool issue: any tool that used a soft-constraint OPF by default would also pass A-3 as currently written. - -### Protocol implication - -PC-10 proposes adding a hard constraint enforcement check to A-3: `max(loading_percent) <= 1.0 + epsilon`. Soft-constraint DCOPF should be explicitly classified as `partial_pass` with the penalty coefficient documented. This provides discriminative signal that the current pass condition cannot provide. - ---- - -## T-11: B-8 LMP Variation from Slack Reconfiguration Is Mathematically Vacuous for DC OPF - -**Affected tools:** pypsa, matpower -**Signal level:** Medium | **Dominant factor:** test_design -**Proposed change:** PC-11 - -### Pattern - -B-8 requires that "LMP values change consistently across configurations" when the slack bus is moved. In standard DC OPF, the dual variables of KCL constraints (which equal the LMPs) are invariant to the choice of reference bus — the angle reference cancels out. Any correctly implemented DC OPF will show identical LMPs across all slack configurations. - -pypsa A-3 (pypsa-F01): All three slack configurations produce identical objectives (spread 0.0) and identical LMPs (spread 0.0). The evaluator correctly explains this is mathematically expected, but the pass condition language ("LMP values change consistently") is ambiguous — it could mean "change consistently between buses" (which they do, reflecting cost gradients) or "change between configurations" (which they don't, which is correct). - -matpower A-11 distributed slack (matpower-F06): The distributed slack formulation produces only a uniform LMP shift across all buses (std dev $0.00), which the result acknowledges is correct for lossless DC OPF. - -The test successfully measures API configurability (can the tool set a different reference bus without error?), which is a valid dimension. But the LMP variation component of the pass condition is vacuous for all correct DC OPF implementations. - -### Protocol implication - -PC-11 proposes rewriting the pass condition to explicitly test API configurability (three configurations must complete without error) and to remove the LMP variation requirement for DC OPF. For tools supporting AC OPF, an optional AC OPF sub-test where slack choice does affect LMPs could provide genuine discriminative signal. - ---- - -## T-12: G-FNM-3 Reference Is Self-Generated for the Reference Tool - -**Affected tools:** matpower -**Signal level:** Medium | **Dominant factor:** test_design -**Proposed change:** PC-12 - -### Pattern - -The MATPOWER reference solution for G-FNM-3 DCPF is generated by MATPOWER itself from the same .m file it will use as input. This makes MATPOWER's zero-deviation result mathematically guaranteed — a round-trip consistency check, not an accuracy verification. - -Probe-001 (pypsa, claim_debunked) provides indirect validation: PyPSA's deviations from the MATPOWER reference are at float64 noise levels (~1e-8 degrees), confirming the reference is correctly computing a DCPF solution for the network's line-dominated segments. However, the reference cannot independently validate transformer-branch flows, which is where both gridcal (probe-007) and powersimulations show systematic failures. - -### Protocol implication - -PC-12 proposes adding a cross-reference independence check to G-FNM-3: either a bus-injection power balance check (necessary condition for solution validity, tool-independent) or comparison against an independently implemented DCPF for a subset of buses. This would validate that the MATPOWER reference is correct for transformer branches, not just line segments. - ---- - -## T-13: SCIP License Conflict — ZIB Academic at Pinned Version, Not Apache 2.0 - -**Affected tools:** powermodels -**Signal level:** High | **Dominant factor:** infrastructure -**Proposed change:** PC-13 - -### Pattern - -Two PowerModels supply chain tests reach conflicting conclusions about the SCIP license: -- **F-3** classifies SCIP_jll v0.2.1 as ZIB Academic (non-commercial only) — correct per probe-010. -- **F-8** claims SCIP_jll v0.2.1 is Apache 2.0, explicitly overriding F-3's classification — incorrect. - -**Probe-010 (powermodels, claim_supported)** resolved this definitively by checking the JuliaBinaryWrappers release timeline: - -| SCIP_jll version | SCIP version | License | -|-----------------|--------------|---------| -| v0.2.1+0 (pinned) | 8.0.0 | ZIB Academic | -| v800.0.300+0 | 8.0.3 | Apache 2.0 | - -The Apache 2.0 license switch happened at SCIP 8.0.3 (December 2022), not at SCIP 8.0.0. F-8's claim that "SCIP v8.0 switched to Apache 2.0" is factually wrong — the JLL package versioning scheme changed at the same time, and `SCIPversion()` returning "8.0" maps to 8.0.0, not 8.0.3+. - -**Supply chain implication:** The pinned SCIP_jll v0.2.1 = SCIP 8.0.0 is ZIB Academic. Any commercial deployment using this manifest requires either a SCIP commercial license or exclusion of SCIP from the solver stack. F-8's upgrade of the supply chain grade to pass is unwarranted. - -**Broader implication:** In the Julia JLL ecosystem, binary artifact licenses frequently differ from Julia package wrapper licenses. The SCIP_jll wrapper package uses MIT for its Julia code; the bundled SCIP binary uses ZIB Academic for the artifact. The binary license governs actual deployments. The evaluation skill should explicitly check both. - ---- - -## T-14: Gate Tests (G-1/2/3) Are Low-Signal Universal Passes - -**Affected tools:** all six -**Signal level:** Low | **Dominant factor:** test_design -**Proposed change:** PC-14 - -### Pattern - -All six tools pass all three gate tests. The tests verify element counts after loading a standard MATPOWER .m file — a trivially satisfied minimum bar for any tool that has reached the maturity level required to participate in this evaluation. Five of six per-tool finding files explicitly note this. The gate tests serve their intended purpose (a fail would be informative) but their pass outcomes have zero discriminative signal and uniformly inflate apparent pass rates. - -### Protocol implication - -PC-14 proposes tagging gate tests as `test_category: gate_minimum_bar` and excluding their pass outcomes from cross-tool pass rate numerators and denominators. An optional enhancement: add a data quality sub-check for cost function import completeness, which would provide discriminative signal (pypsa-F11 notes that PyPSA's `import_from_pypower_ppc` silently drops gencost data). - ---- - -## T-15: C-4 Timing Comparability Compromised by Solver Thread Count Variation - -**Affected tools:** powersimulations -**Signal level:** Low | **Dominant factor:** infrastructure -**Proposed change:** PC-15 - -### Pattern - -powersimulations C-4 records 404 seconds for 2K-bus SCUC with HiGHS at 1 thread on a machine with 32 available cores. MILP solvers scale significantly with thread count — multi-threaded HiGHS on the same problem would likely solve in 15-30 seconds. The 404s result is reproducible and technically correct, but it is not representative of practical deployment on the same hardware. - -This is a single-tool finding but exposes a protocol gap: without standardized thread count specification (or at minimum, thread count reporting), MILP timing results are not comparable across tools. - -### Protocol implication - -PC-15 proposes requiring cpu_threads used and cpu_threads available in all scalability test frontmatter. For MILP tests (C-4, C-8), both 1-thread and max-thread timings should be reported. - ---- - -## T-16: SCOPF TINY Network Too Radially Connected for Meaningful Contingency Coverage - -**Affected tools:** pypsa, powermodels, powersimulations -**Signal level:** Medium | **Dominant factor:** network -**Proposed change:** PC-16 - -### Pattern - -The IEEE 39-bus TINY network is insufficiently meshed for SCOPF evaluation: -- powersimulations (powersimulations-F04): 27 of 34 line contingencies filtered as near-radial (|LODF| >= 0.95). Only 7 non-trivial contingencies remain. -- pypsa (pypsa-F04): Progressive fallback from 35 lines to 19 lines at <50% utilization due to base-case infeasibility of N-1 sets. -- powermodels: Probe-009 (inconclusive) confirmed the network is N-1 infeasible under Modified Tiny load profile at nominal ratings. The Benders mechanism is demonstrated via API (1 iteration, correct infeasibility detection) but multi-iteration convergence is never observed. - -The consequence is that A-9 SCOPF results across all tools reflect the network's topological limitations (radial structure, tight base-case loading) rather than the tools' SCOPF formulation quality. - -**Probe-009 integration:** The PowerModels A-9 result accurately documents the N-1 infeasibility and 1-iteration behavior. The evaluator's framing ("physical property of the network, not a code limitation") is correct. The qualified_pass for demonstrating the API mechanism is defensible. But the test design gap means no tool demonstrated multi-iteration Benders convergence during this evaluation — not because any tool lacks the capability, but because no feasible SCOPF scenario was provided. - -### Protocol implication - -PC-16 proposes pre-computing and publishing a feasible N-1 SCOPF configuration for case39 in eval-config.yaml. The pass condition should require either multi-iteration convergence OR explicit demonstration that the joint N-1 problem is infeasible (with the infeasibility correctly detected). This would allow SCOPF mechanism verification to be distinguished from SCOPF convergence verification. - ---- - -## Probe Integration Summary - -Eight probes were executed during this sweep. Their outcomes and impact on the themes are: - -| Probe | Classification | Theme Impact | -|-------|---------------|-------------| -| probe-001 (pypsa G-FNM-3) | claim_debunked (weak) | T-08: 0.0 is a display artifact; PASS grade is correct; validation report is stale | -| probe-003 (pandapower A-3) | claim_supported | T-04 context: shadow prices are real (min 8.79 $/MWh); sweep concern about artifacts is refuted | -| probe-005 (gridcal A-3) | confirmed_issue | T-10: soft branch constraints confirmed in source code and runtime; A-3 P is misleading | -| probe-007 (gridcal G-FNM-3) | classification_plausible_with_caveats | T-09: formulation_difference classification is defensible but has two design flaws (no magnitude cap, adjacency proxy) | -| probe-009 (powermodels A-9) | inconclusive | T-16: Benders mechanism real; no feasible SCOPF scenario tested; test design gap confirmed | -| probe-010 (powermodels SCIP) | claim_supported (F-3) | T-13: SCIP 8.0.0 = ZIB Academic; Apache 2.0 switch at 8.0.3; F-8 P should be QP | -| probe-013 (powersimulations A-2) | claim_debunked | T-04: iteration count at @info; return type = convergence guarantee; A-2 QP overstates limitation | -| probe-016 (matpower GLPK) | claim_debunked | T-01: GLPK failure is GLP_ETMLIM with status=-1 (no feasible solution); not an exit-flag mapping bug | - ---- - -## GitHub Issues - -The github-issues.yaml query (2026-03-14) returned zero issues with the 'protocol' label. No issue integration was performed. diff --git a/sweep-data/v10-to-v11/aggregation/themes.yaml b/sweep-data/v10-to-v11/aggregation/themes.yaml deleted file mode 100644 index a54df433..00000000 --- a/sweep-data/v10-to-v11/aggregation/themes.yaml +++ /dev/null @@ -1,427 +0,0 @@ ---- -# Cross-Tool Themes — v10-to-v11 Sweep -# Synthesized from per-tool findings.yaml, probe results, and protocol/rubric review. -# Evidence threshold: 3+ tools for protocol/rubric changes; 2+ for skill-only changes. - -themes: - - - id: T-01 - title: "C-SMALL gate cascades block LP/PF MEDIUM tests when only MILP fails" - description: >- - The C-SMALL gate requires C-4 (SCUC on 2000-bus SMALL) to pass before any - MEDIUM-tier scalability tests execute. When a tool fails C-4 — whether because - it lacks SCUC entirely (pandapower, powermodels), because the MILP solver cannot - solve at that scale (pypsa), or because the SCUC is genuinely infeasible for the - solver (matpower/GLPK) — all MEDIUM tests are skipped. This includes tests that - measure LP and power-flow scalability (C-1 DCPF, C-2 ACPF, C-3 DC OPF, C-9 - PTDF, C-10 distributed slack), which are entirely independent of MILP capability. - The effect is that tools with strong LP/PF performance but weak MILP receive the - same "unknown" MEDIUM-tier verdict as tools with genuine large-scale gaps. In - several cases (pypsa, pandapower, matpower) there is strong FNM-suite or SMALL-tier - evidence of MEDIUM-scale LP/PF capability that the gate prevents from being formally - counted. - affected_tools: - - pypsa # F09: HiGHS single-threaded MILP timeout; 7 MEDIUM tests skipped - - pandapower # F01: SCUC absent (scope); 8 MEDIUM tests skipped - - matpower # F01: GLPK cannot solve SCUC (infeasible); 8 MEDIUM tests skipped - - powermodels # F13: UC absent; C-SMALL gate blocks MEDIUM - probe_evidence: - - probe-016: claim_debunked — matpower GLPK failure is genuine SCUC infeasibility, - not an exit-flag bug; confirms the gate loss is real for matpower - signal_level: High - dominant_factor: test_design - proposed_action: PC-01 - notes: >- - powersimulations also faces MEDIUM-test gaps (C-8 MEDIUM solver crash) but via a - different mechanism (solver error, not gate design). gridcal passes C-4 at SMALL - via workaround so does not trigger the gate. - - - id: T-02 - title: "SCUC test network insufficient to verify min up/down binding" - description: >- - The 10-generator IEEE 39-bus TINY network has a capacity-to-peak ratio of ~1.18x, - which provides enough headroom that the optimizer typically shuts down only the most - expensive generators near load valleys. The minimum 2-cycling-generator threshold - is met in all tools that attempt the test, but no tool's A-5 result verifies that - min up/down constraints are actually binding (i.e., that removing them would change - the commitment schedule). Without binding evidence, A-5 confirms MILP formulation - completeness but not MILP behavioral correctness under tight temporal constraints. - A network with tighter capacity margins and diverse min up/down times is needed. - affected_tools: - - pypsa # F02: cycling observed but binding not verified - - powermodels # F09: 3 shutdowns, 0 startups, all end-of-horizon - - powersimulations # F12: cycling matches load amplitude, not cost structure - - matpower # F05: validated on 3-bus ex_case3b, not case39 - - gridcal # F04: MIP gap not extractable; commitment inferred from power, not binary - signal_level: High - dominant_factor: network - proposed_action: PC-02 - notes: >- - pandapower lacks SCUC so does not encounter this issue. The pattern is consistent - across all five SCUC-capable or SCUC-attempted tools. - - - id: T-03 - title: "SCOPF MEDIUM result is vacuous — ACTIVSg10k is uncongested" - description: >- - The 10,000-bus ACTIVSg10k MEDIUM network has a base-case maximum branch loading - of ~84.72%. No N-1 contingency creates post-contingency violations requiring - redispatch. As a result, any correctly functioning SCOPF produces results - numerically identical to the base-case DCOPF. The C-8 SCOPF MEDIUM test provides - a measurement of solve time and solver robustness but no signal about SCOPF - expressiveness or the quality of security-constrained redispatch. Tools that pass - C-8 MEDIUM did so on a trivially uncongested problem; tools that fail (timeout, - crash) failed on a problem with no physical security constraint. - affected_tools: - - gridcal # F02: C-8 MEDIUM objective delta 1.49e-8 $/MWh — effectively zero - - powermodels # F02: C-8 1 Benders iteration, time budget; awarded 'pass' - - powersimulations # F09: C-8 MEDIUM HiGHS OTHER_ERROR crash - signal_level: High - dominant_factor: network - proposed_action: PC-03 - notes: >- - pypsa, pandapower, and matpower all have MEDIUM tests skipped (T-01 cascade), so - this pattern cannot be observed for them. The vacuousness is confirmed for all three - tools that ran C-8 MEDIUM. - - - id: T-04 - title: "Convergence accepted without residual across ACPF implementations" - description: >- - Multiple tools have no public API path to extract the Newton-Raphson convergence - residual after ACPF. The evaluation protocol requires reporting the convergence - residual, but the result files for at least four tools record convergence_residual: - null and accept convergence via indirect proxy (voltage profile differs from flat - start, boolean termination flag, or boolean return value). This creates a cross-tool - consistency gap: convergence 'quality' is not comparable because different tools - use different indirect proxies. probe-013 (powersimulations) debunked the strongest - form of this claim — PowerFlows.jl does emit the iteration count at @info level and - the return type structurally guarantees convergence — but the residual value remains - inaccessible for all Julia tools, and the pandapower/powermodels residual gaps are - tool-level API limitations. The pattern identifies a protocol insufficiency: the - protocol requires residual reporting but has no specified fallback for tools where - this is architecturally unavailable. - affected_tools: - - pandapower # F02: convergence_residual null; tolerance_mva bug unfixed - - powermodels # F05: compute_ac_pf returns only Bool; no NR count or residual - - powersimulations # F02: residual not in return value; iteration count at @info only - probe_evidence: - - probe-013: claim_debunked — PowerFlows.jl iteration count IS available at @info; - return type (missing vs Dict) is a binary convergence guarantee. - The qualified_pass on A-2 overstates the limitation but the underlying - convergence is genuine. - signal_level: Medium - dominant_factor: capability - proposed_action: PC-04 - notes: >- - pypsa exposes convergence residual via n.convergence (post-solve attribute) so does - not fall into this pattern. gridcal also reports convergence boolean. The issue is - most material for the Julia ecosystem tools and pandapower. - - - id: T-05 - title: "Ramp constraints not binding in A-6/SCED due to over-generous ramp parameters" - description: >- - A-6 SCED requires that ramp rate constraints be demonstrably enforced between - consecutive dispatch intervals. In all tools that attempt A-6, the RTS-GMLC - technology-median ramp rates are orders of magnitude larger than the dispatch - changes required by the 39-bus load profile (which varies by ~33% over 24 hours). - The most constrained generator observed uses less than 1% of its ramp capacity. - The tests confirm that ramp constraints are formulated without error, but cannot - verify they bind correctly when tight. Any tool that implements multi-period OPF - passes this condition trivially, which eliminates its discriminative value. - affected_tools: - - matpower # F04: G1 uses 48.5 MW/hr of 62,400 MW/hr limit; 0 binding - - pypsa # F12 context: A-6 qualified_pass (stable workaround) but no binding reported - - powermodels # F13: ED-only because UC absent; 1 binding constraint of 460 added - - powersimulations # F08: constraint count only; LMP extraction failed - signal_level: Medium - dominant_factor: network - proposed_action: PC-05 - notes: >- - gridcal and pandapower have different A-6 failure modes so are not included in the - binding-evidence count, but the underlying network issue would affect them too. - - - id: T-06 - title: "SCED degrades to ED-only when SCUC is absent" - description: >- - When a tool cannot perform unit commitment (A-5), A-6 SCED loses the - security-constraint component: all generators are assumed committed, reducing - the test to a pure multi-period economic dispatch. This means A-6 results for - tools without SCUC measure a strictly weaker capability than the rubric intends. - The qualified_pass applied to these results implies SCED capability that does - not include the UC stage, and the 'security-constrained' component of SCED - (binding N-1 contingency constraints on dispatch) is also typically absent for - the same tools. - affected_tools: - - powermodels # F13: explicit scope reduction to ED; documented as stable workaround - - pandapower # SCUC absent; A-6 not runnable as full SCED - - powersimulations # A-6: UC bypassed; 1 binding ramp constraint of 460 - signal_level: Medium - dominant_factor: test_design - proposed_action: PC-06 - notes: >- - This is structurally related to T-01 (SCUC absence cascades) but specifically about - the A-6 test design, not the scalability gate. - - - id: T-07 - title: "qualified_pass applied at inconsistent severity levels" - description: >- - The qualified_pass status is used across all tools to label results ranging from - 'stable workaround, low friction' to 'blocking architectural impossibility' to - 'solver crash on the grade network'. The workaround_class field (stable / fragile / - blocking) provides finer-grained information, but the aggregated qualified_pass - status in cross-tool matrices collapses this into one outcome. Specific examples: - PyPSA A-11 (blocking: distributed slack DC OPF architecturally impossible) and - PyPSA A-6 (stable: working undocumented API) both receive qualified_pass. PowerModels - C-8 receives pass despite non-convergence within the time budget. PowerSimulations - C-8 receives qualified_pass despite a HiGHS OTHER_ERROR crash on the grade network. - PowerModels A-12 receives pass despite three simultaneous non-trivial workarounds. - The protocol does not provide a severity tier within qualified_pass. - affected_tools: - - pypsa # F12: A-6 (stable) and A-11 (blocking) both qualified_pass - - powermodels # F02: C-8 pass with non-converged result; F06: A-12 pass with 3 workarounds - - powersimulations # F09: C-8 qualified_pass despite MEDIUM solver crash - - pandapower # F10: B-9 clean pass vs B-1 fragile qualified_pass — same internal path - - matpower # F09: B-5 qualified_pass debatable given <5 LOC criterion met (3 lines) - signal_level: High - dominant_factor: test_design - proposed_action: PC-07 - notes: >- - This is a scoring rubric issue rather than a protocol design issue. The fix is to - add severity tiers to qualified_pass or to add explicit fail categories for - blocking workarounds and solver crashes. - - - id: T-08 - title: "G-FNM result precision: 6-decimal rounding obscures floating-point accuracy" - description: >- - The standard result reporting format stores deviation values with round(..., 6) - (6 decimal places). probe-001 confirmed that PyPSA's G-FNM-3 reports 0.0 degree - max bus angle deviation, but the actual float64 maximum is 1.07e-8 degrees — it - rounds to 0.000000 at 6 decimal places. The same rounding artifact affects branch - flow deviations. While the actual deviations are sub-physical (numerical noise), - the reported 0.0 values are misleading for downstream comparison: they cannot - be distinguished from true machine-zero agreement (which would indicate identical - code paths) from approximated machine-noise agreement (which indicates correctly - equivalent but independently coded formulations). The issue is likely present - wherever deviation magnitudes fall below 5e-7 in the G-FNM suite. - affected_tools: - - pypsa # F06: G-FNM-3 reports 0.0 (actual 1.07e-8); probe-001 confirmed - - matpower # F02: G-FNM-3 reports 0.0 (self-referential test; mathematically exact) - probe_evidence: - - probe-001: claim_debunked (weak) — deviations are 1.07e-8 deg max (numeric noise), - not true zeros. PASS grade is correct; rounding is a display artifact. - signal_level: Low - dominant_factor: test_design - proposed_action: PC-08 - notes: >- - The fix is simple: require full float64 reporting (:.6e scientific notation) for - all deviation values in G-FNM-3 and related tests. The PASS grade for PyPSA is - confirmed correct by probe-001. - - - id: T-09 - title: "PSS/E v31 parser failures force MATPOWER fallback across the FNM suite" - description: >- - The FNM test network is provided as a PSS/E v31 RAW file with an intermediate CSV - export. No tool successfully ingested the FNM via the PSS/E native path: pypsa has - no PSS/E importer, gridcal's parser is hardcoded to v35 field counts, powermodels - crashes on the Case Identification header, and pandapower has no CSV import path. - All FNM tests therefore run on the MATPOWER .m fallback (fnm_main_island.m), which - is a pre-cleaned main-island subset with documented record deficits. The FNM suite - results therefore measure each tool's performance on a MATPOWER-preprocessed - network representation, not on the original FNM. This confounds format support - (supply chain / interoperability criterion) with power-system modeling capability - (expressiveness criterion) and makes cross-tool FNM comparison partially invalid - since all tools see the same preprocessed input. - affected_tools: - - pypsa # F05: no PSS/E import path - - pandapower # F13: no CSV import path; MATPOWER fallback only - - gridcal # F08: PSS/E v31 field count mismatch (v35 hardcoded) - - powermodels # F07: PSS/E v31 parser crashes on Case ID header - signal_level: High - dominant_factor: infrastructure - proposed_action: PC-09 - notes: >- - powersimulations and matpower also use the MATPOWER fallback but do not have explicit - findings logged about it as a cross-tool pattern. matpower's G-FNM-3 is self- - referential (it generated the reference) so the fallback is the native path. The - core issue is that the FNM data provision format (PSS/E v31) is not supported by any - evaluated tool in its native parser. - - - id: T-10 - title: "GridCal DCOPF uses soft branch flow constraints (confirmed by probe)" - description: >- - GridCal's linear_opf formulation uses explicit slack variables (flow_slacks_pos / - flow_slacks_neg) for branch thermal limits. probe-005 confirmed this via source - code inspection and runtime evidence: branch 2_3_1 reaches 103.5% loading in the - optimal solution, which is only possible with soft constraints. The A-3 pass verdict - is therefore misleading — a standard DCOPF enforces hard thermal limits. The D-4 - error quality test further confirmed this behavior: setting all branch ratings to - zero produces a 'feasible' (converged=True) solution that maximally violates all - limits via slack absorption rather than detecting infeasibility. This is a - GridCal-specific issue (no other evaluated tool uses soft constraints by default - in DCOPF), but it has protocol implications: the A-3 pass condition does not - verify that thermal limits are enforced as hard constraints, so the issue was not - caught by the standard evaluation. - affected_tools: - - gridcal # F06: 112% branch loading in DCOPF pass; D-4 confirms soft constraints - probe_evidence: - - probe-005: confirmed_issue — flow_slacks_pos/flow_slacks_neg LP variables in - source code; 103.5% loading confirmed in optimal solution. A-3 pass - is misleading. This is a formulation deficiency, not a workaround. - signal_level: High - dominant_factor: capability - proposed_action: PC-10 - notes: >- - The D-4 error quality test that uses zero-rated branches conflates two distinct - questions: (1) does the tool use hard or soft constraints? and (2) does the tool - detect genuine LP infeasibility? The current D-4 design confounds these and produced - a 'poor' error quality grade that is partly unfair to GridCal's design choice (soft - constraints are a legitimate design option), but the A-3 pass criterion gap (not - verifying hard constraint enforcement) is the primary protocol issue to fix. - - - id: T-11 - title: "B-8 LMP variation from slack reconfiguration is mathematically vacuous for DC OPF" - description: >- - B-8 tests reference bus reconfigurability by requiring that 'LMP values change - consistently across configurations' when the slack bus is moved. In standard DC OPF, - the LMP vector is invariant to slack bus choice (dual variables of KCL constraints - are independent of the reference angle). All tools implementing correct DC OPF - produce zero LMP spread across slack configurations, satisfying the pass condition - trivially (if the condition is interpreted as 'change consistently from one another' - = they are all equal). The test verifies API configurability (the ability to set - a different reference bus) but cannot measure meaningful LMP sensitivity via DC OPF - for any well-functioning tool. LMP sensitivity to slack bus is only observable in - AC OPF or lossy DC OPF formulations. - affected_tools: - - pypsa # F01: zero LMP spread across all 3 slack configs; pass condition is vacuous - - matpower # F06: uniform LMP shift confirmed for distributed slack (same structural issue) - - pandapower # F04: B-8 shows 8.58 $/MWh maximum LMP change — but this is across configs, - # not zero; possibly due to AC configuration or different test setup - signal_level: Medium - dominant_factor: test_design - proposed_action: PC-11 - notes: >- - The pandapower B-8 result shows non-zero LMP change (8.58 $/MWh max), which may - indicate the test uses a different formulation for that tool. Cross-tool consistency - in the B-8 test design needs review. The zero-variation result was reported by pypsa - and is the structural expectation for any correct DC OPF. - - - id: T-12 - title: "G-FNM-3 reference is self-generated for the reference tool" - description: >- - The MATPOWER reference solution for G-FNM-3 (DCPF on FNM) is generated by MATPOWER - itself from the same .m file. This means MATPOWER's G-FNM-3 result (zero deviation) - is mathematically guaranteed regardless of MATPOWER's DCPF accuracy — it is a - round-trip consistency check, not a cross-tool accuracy verification. Other tools are - tested against this reference, but the reference itself has no independent validation. - If the MATPOWER DCPF reference were incorrect for certain branch types (e.g., transformer - tap representation), all tools would be scored against a wrong reference and tools with - the same limitation as MATPOWER could receive passing scores. - affected_tools: - - matpower # F02: self-referential G-FNM-3; documented in result file - probe_evidence: - - probe-001 (pypsa): confirms MATPOWER reference is effectively exact for lines; - PyPSA deviations are float64 noise, confirming the reference - is independently validated to machine precision for that tool. - signal_level: Medium - dominant_factor: test_design - proposed_action: PC-12 - notes: >- - The self-referential issue is documented in the matpower result file. An independent - cross-check (e.g., comparing MATPOWER with PowerWorld or PSS/E reference on a - subset of buses) would validate the reference. For the current evaluation, the - PyPSA probe provides indirect validation that the MATPOWER reference is correct for - line-dominated segments. - - - id: T-13 - title: "SCIP license conflict: ZIB Academic at pinned version, not Apache 2.0" - description: >- - PowerModels evaluation contains a material conflict between F-3 and F-8 regarding - the SCIP license. probe-010 resolved this: SCIP_jll v0.2.1 wraps SCIP 8.0.0, which - is ZIB Academic License. The Apache 2.0 switch happened at SCIP 8.0.3 (December 2022), - not at 8.0.0. F-8's upgrade of the supply chain grade based on an incorrect Apache 2.0 - determination is not warranted. For any commercial deployment using the pinned manifest, - SCIP requires a commercial license or must be excluded from the solver stack. - affected_tools: - - powermodels # F08: F-3 correct (ZIB Academic); F-8 incorrect (claims Apache 2.0) - probe_evidence: - - probe-010: claim_supported (F-3) — SCIP_jll v0.2.1+0 wraps SCIP 8.0.0 (ZIB Academic). - Apache 2.0 switch was at SCIP 8.0.3 (December 2022). F-8 supply chain - upgrade is unwarranted. - signal_level: High - dominant_factor: infrastructure - proposed_action: PC-13 - notes: >- - This is a single-tool finding (powermodels) but the license classification methodology - has broader implications: JLL wrapper package license (MIT) vs bundled binary license - (ZIB Academic) is a common source of confusion in the Julia ecosystem. The protocol - should clarify that binary artifact licenses govern, not wrapper package licenses. - - - id: T-14 - title: "Gate tests (G-1/2/3) are low-signal universal passes" - description: >- - All evaluated tools pass the three gate ingestion tests (G-1 TINY, G-2 SMALL, - G-3 MEDIUM). The tests verify element counts (buses, branches, generators) after - loading a standard MATPOWER .m file. Any tool that has reached maturity sufficient - for evaluation will pass these tests. They serve a valid minimum-bar purpose (a tool - that cannot load a MATPOWER file should not proceed) but contribute no comparative - signal and inflate reported pass rates uniformly across tools. Multiple per-tool - findings explicitly note this. - affected_tools: - - pypsa # F11: gate tests low discriminative value - - pandapower # F09: gate tests are universal passes - - gridcal # F12: gate tests unanimous pass; inflate pass rate - - powermodels # (implicit from universal pass) - - powersimulations # F15: gate tests pass/fail with no tool differentiation - - matpower # F07: gate tests trivially satisfied for native-format tool - signal_level: Low - dominant_factor: test_design - proposed_action: PC-14 - notes: >- - The gate tests should be retained as minimum-bar gates (a fail would be meaningful) - but their pass outcomes should be excluded from cross-tool pass rate comparisons. - An enhancement: add a data quality sub-check (does cost data import correctly?) - that would provide discriminative signal. - - - id: T-15 - title: "C-4 timing comparability compromised by solver thread count variation" - description: >- - powersimulations C-4 records 404s wall-clock for 2K-bus SCUC with CPU threads = 1 - on a 32-core machine. Modern MILP solvers (HiGHS, SCIP, Gurobi) scale significantly - with thread count for branch-and-bound problems. Other tools did not reach C-4 - (gate skip) or use different solvers, so thread-count normalization cannot be applied - retroactively. The 404s figure is reproducible and technically correct, but it is - not representative of practical deployment capability on the same hardware. This - is noted as an extraordinary claim in the powersimulations findings. - affected_tools: - - powersimulations # F16: 404s at 1 thread; 32 cores available - signal_level: Low - dominant_factor: infrastructure - proposed_action: PC-15 - notes: >- - This is a single-tool finding but points to a protocol gap: solver thread count - should be normalized across tools, or results should be reported both at 1-thread - (reproducible) and max-thread (practical) configurations. - - - id: T-16 - title: "SCOPF TINY network is too radially connected for meaningful contingency coverage" - description: >- - The 39-bus TINY network used for A-9 SCOPF has limited meshing: powersimulations - reports 27 of 34 line contingencies filtered as near-radial (|LODF| >= 0.95). - Only 7 contingencies are non-trivial. pypsa reduced to 19 of 35 lines at <50% base - utilization. powermodels found the joint N-1 set infeasible (probe-009 confirmed). - All A-9 results are therefore based on a reduced contingency set that may not - adequately stress SCOPF formulations. The SMALL network would provide more meshing - and more non-trivial contingencies. - affected_tools: - - pypsa # F04: 19-line reduced contingency set due to base-case infeasibility - - powermodels # F03: N-1 infeasible; 1-iteration Benders; probe-009 inconclusive - - powersimulations # F04: 27 of 34 contingencies filtered as near-radial - probe_evidence: - - probe-009: inconclusive — Benders mechanism is verified but convergence on a feasible - SCOPF instance was never demonstrated; test design gap, not tool failure. - signal_level: Medium - dominant_factor: network - proposed_action: PC-16 - notes: >- - gridcal and matpower have different A-9 failure modes (soft constraints, solver - unavailability) so do not add to the network-insufficiency count for SCOPF, but - would benefit from a more meshed test network regardless. diff --git a/sweep-data/v10-to-v11/github-issues.yaml b/sweep-data/v10-to-v11/github-issues.yaml deleted file mode 100644 index e28d30ff..00000000 --- a/sweep-data/v10-to-v11/github-issues.yaml +++ /dev/null @@ -1,3 +0,0 @@ -issues: [] -note: "gh issue list queried 2026-03-14 — no open issues with 'protocol' label found" -issues_in_scope: 0 diff --git a/sweep-data/v10-to-v11/per-tool/gridcal/findings.md b/sweep-data/v10-to-v11/per-tool/gridcal/findings.md deleted file mode 100644 index 71b9713c..00000000 --- a/sweep-data/v10-to-v11/per-tool/gridcal/findings.md +++ /dev/null @@ -1,504 +0,0 @@ -# gridcal — Sweep Findings (v10) - -## Summary - -The gridcal (VeraGridEngine 5.6.28) evaluation is well-documented, code-executed throughout, -and timing measurements are consistently marked as measured rather than estimated. The -primary sweep concerns are: (1) two network insufficiency problems where the test network -is too uncongested or too small to exercise the capability under test (SCOPF on MEDIUM, -SCUC cycling at SMALL); (2) a misleading qualified_pass on lossy DCOPF where the loss -approximation produces results indistinguishable from zero; (3) an unverified -formulation-difference classification for 326 branches with extreme flow deviations in the -FNM DCPF test; and (4) an unverified claim that DCOPF branch limits are enforced as hard -constraints given evidence of soft-constraint behavior. Three extraordinary claims are -flagged for probe: the soft-constraint nature of DCOPF thermal limits, the FNM branch flow -deviation classification, and unverified operational adoption by named utilities. - ---- - -## Finding Details - -### gridcal-F01: SCUC pass on TINY masks complete absence of inter-temporal commitment cycling at SMALL - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-5, C-4 - -A-5 (SCUC on 10-generator IEEE 39-bus) passes with 6 of 10 generators cycling, demonstrating -genuine unit commitment behavior under differentiated costs. At SMALL scale (C-4, 544 -generators, ACTIVSg 2000-bus), no generator cycling is observed across 24 hours under either -solver — all 430 dispatchable generators remain committed for all 24 hours. - -Two compounding factors produce this outcome: the TapPhaseControl enum bug forces a -sequential snapshot workaround that structurally cannot enforce inter-temporal constraints -(min up/down times, ramp rates), and the base-case generator costs in the SMALL network are -insufficiently differentiated to drive decommitment even in snapshot mode. The C-4 result -is scored qualified_pass with "stable" workaround classification, but the workaround -eliminates the distinguishing capability of SCUC (inter-temporal commitment decisions). - -From C-4: *"No generator cycling observed. The SMALL network's base-case generator costs -are not sufficiently differentiated to drive decommitment... the workaround loses -inter-temporal UC coupling (min up/down times, ramp rates are not enforced across hours)."* - -The test demonstrates 24 independent hourly OPF solves at SMALL scale, not scaled SCUC. -This finding is likely cross-tool — other tools' SCUC scale tests may face similar -cost-differentiation gaps in the SMALL network that prevent meaningful UC cycling. - -**Cross-tool relevance:** likely -**Proposed action:** redesign_test — Augment SMALL network with differentiated costs -and evaluate whether true multi-period SCUC (with inter-temporal coupling) is tractable -within a time budget, rather than accepting snapshot workarounds as qualified passes. - ---- - -### gridcal-F02: SCOPF on MEDIUM is vacuous — zero binding constraints, zero redispatch - -**Category:** network_insufficiency | **Severity:** high -**Tests:** C-8 - -C-8 SCOPF on the 10,000-bus ACTIVSg network is scored pass with solve time 29.3s. However, -the SCOPF and base-case DCOPF produce results numerically identical to 1.49e-08 $/MWh: -no generators change dispatch above 1 MW, no branches become binding, and the maximum LMP -difference between DCOPF and SCOPF is essentially zero. - -From C-8: *"This is consistent with the ACTIVSg10k network being uncongested (max loading -84.72%). Since no branches are near their limits in the base case, N-1 contingencies do not -create post-contingency violations that would require redispatch. The SCOPF correctly -determines that the base-case dispatch is already N-1 secure."* - -The test validates that the SCOPF formulation runs to completion on a large network — it -does not measure whether the tool can produce security-driven redispatch at scale. The SMALL -SCOPF (C-8 SMALL) does produce meaningful results (up to 164 MW redispatch, LMP spread -$17–$19/MWh), but is not the scale-stress test. The MEDIUM SCOPF result is effectively a -timing-only measurement for an uncongested network. - -This finding is confirmed cross-tool — the ACTIVSg10k network's lack of congestion at -base loading will produce the same vacuous SCOPF results for any tool that implements -SCOPF correctly. The protocol should specify a minimum congestion criterion (e.g., at -least 5 binding branches in base case, or at least one branch >90% loading) for SCOPF -scale tests. - -**Cross-tool relevance:** confirmed -**Proposed action:** redesign_test — Add minimum congestion criterion for SCOPF scale -test networks. Apply additional derating (e.g., 50% branch capacity) before running -SCOPF at MEDIUM to ensure security constraints are active. - ---- - -### gridcal-F03: Lossy DCOPF qualified_pass obscures a near-nonfunctional loss approximation - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-10 - -A-10 is scored qualified_pass because `add_losses_approximation=True` produces non-zero -losses and a non-zero LMP difference. However, the computed losses (0.055 MW, 0.0009% of -load) are orders of magnitude below the expected 0.5–3% range and far below the ACPF -reference (43.6 MW, 0.7% of load). - -From A-10: *"The loss factor formula uses R * rate / V^2 where rate is the branch thermal -rating rather than the actual power flow. For the case39 network, where branch resistances -are very small (0.0002–0.007 pu) and nominal voltages are in the 100s of kV, this produces -negligible loss factors."* - -The loss factor formula is structurally broken for the test case: using the thermal rating -rather than actual flow means the approximation systematically underestimates losses by a -factor of ~800x (0.0009% vs ~0.7%). LMP decomposition into energy/congestion/loss components -is also entirely absent. The qualified_pass implies a feature that works with caveats; the -actual situation is a formula producing results indistinguishable from lossless DCOPF for -any standard transmission network. A fail with "feature exists but produces negligible -results" would more accurately represent the finding. - -**Cross-tool relevance:** none -**Proposed action:** adjust_scoring — Consider reclassifying A-10 as fail with a note -that the feature exists but the formula is broken for standard transmission parameters. -The 0.5–3% pass criterion should be enforced as a hard threshold. - ---- - -### gridcal-F04: SCUC commitment inferred from dispatch power rather than binary commitment variable - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-5 - -The A-5 SCUC pass is based on a commitment schedule derived by thresholding generator power -output (`gen_power > 0.1` MW) rather than extracting a binary commitment variable from the -MILP solution. The MIP gap is explicitly noted as not directly extractable from the results -object: *"The MIP gap is not directly extractable from the OptimalPowerFlowTimeSeriesResults -object. HiGHS uses a default MIP gap tolerance of 1%."* - -This creates an ambiguity: if GridCal is solving an LP relaxation of the MILP (or if the -MILP binary variables are not properly exposed in the results), the dispatch would appear -binary-like for most generators (due to box constraints and load balance), but with no -guarantee of true integrality. The pass condition requires verifying that the formulation -is a genuine MILP with binary commitment variables — this was not confirmed by extracting -binary solution values. A probe should verify whether `OpfDispatchMode.UnitCommitment` -introduces actual binary variables and that the HiGHS solver is invoked in MILP mode. - -**Cross-tool relevance:** likely — other Python tools may have similar result object -designs that do not expose MIP integrality certificates. -**Proposed action:** add_verification — Protocol should require reporting at least one -of: (a) MIP gap from solver output, (b) binary variable values from solution, (c) solver -log confirmation of MILP solve. - ---- - -### gridcal-F05: FNM DCPF qualified_pass contains 326 branches with deviations up to 562,955% attributed as formulation difference without independent validation - -**Category:** misleading_result | **Severity:** medium -**Tests:** G-FNM-3 - -G-FNM-3 triggers the hard fail condition (max branch deviation > 50%) but is classified -qualified_pass via a "formulation_difference" classification. The classification rests on -the observation that 88.7% of the 326 failing branches are adjacent to transformer buses, -which is attributed to GridCal's simplified B-matrix omitting transformer tap ratio -corrections. - -However, four of the five top-deviating branches are labeled "Line" (not transformer) in -the result table: - -| Branch | Type | GridCal (MW) | Reference (MW) | Dev% | -|--------|------|-------------|----------------|------| -| 1668→88630 | Line | 111,582 | -19.82 | 562,955% | -| 21476→84022 | Line | -68,017 | 12.57 | 541,075% | -| 72100→73053 | Line | -13,365 | 3.23 | 413,787% | -| 1635→92191 | Line | -352,878 | 109.50 | 322,365% | - -Flows on the order of 100,000–350,000 MW on nominal transmission lines are physically -implausible regardless of formulation differences. The "transformer-adjacent" criterion -classifies a branch as affected if either endpoint appears in the transformer terminal bus -set — but this could make many line branches "adjacent" in a highly-meshed network. The -classification was made by inference, not by comparing B-matrix entries for those specific -branches. If these deviations reflect a data ingestion error (e.g., incorrect per-unit -conversion for tap-adjacent bus admittances), the qualified_pass understates a genuine -correctness failure. - -**Cross-tool relevance:** none -**Proposed action:** add_verification — A probe should compare the GridCal B-matrix -entries for the top-5 deviating branches against the MATPOWER reference B-matrix to -confirm whether the deviation is formulation-level (different tap handling) or computation- -level (numerical error or incorrect data ingestion for those branches). - ---- - -### gridcal-F06: DCOPF shows 112% branch overload on a test claiming converged=True — soft constraint not disclosed in pass determination - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** A-3 - -A-3 passes DCOPF with 7 binding branches and meaningful LMP spread ($5–$84/MWh), but branch -2_3_1 shows 112% loading against the derated limit. The result notes: *"This may indicate -the PTDF-based formulation uses soft constraints or penalty functions rather than hard limits -on some branches."* - -The D-4 error quality test confirms that when all branch ratings are set to zero, GridCal -treats them as soft constraints via overload slack variables and reports converged=True. -This establishes that at least some branch thermal limits are enforced as soft constraints. -If the DCOPF formulation systematically uses soft constraints, then all thermal limit -enforcement in A-3, A-9, C-3, C-8 is approximate — the tool may be solving a penalized -dispatch problem rather than a hard-constrained DCOPF. This has material implications for -market applications where hard thermal limits are a legal requirement. - -The pass scoring for A-3 does not address whether the 112% overload is a soft-constraint -design choice (acceptable with disclosure) or a solver infeasibility being suppressed -(unacceptable). The formulation should be audited to determine whether branch limits are -hard inequality constraints or soft penalty terms in GridCal's PTDF-based OPF. - -**Cross-tool relevance:** none -**Proposed action:** add_verification — Probe should inspect the `linear_opf_ts.py` -formulation to determine whether branch flow limits are modeled as hard LP constraints -(`flow <= limit`) or soft penalty terms (`flow <= limit + slack; min slack`). If soft, -the penalty coefficient should be reported and the implications for market dispatch -accuracy should be noted in all DCOPF test results. - ---- - -### gridcal-F07: Error quality test design conflates soft-constraint behavior with infeasibility detection failure - -**Category:** test_design_gap | **Severity:** medium -**Tests:** D-4 - -D-4 test (a) sets all branch ratings to zero and expects infeasibility detection. The test -concludes "Error quality grade: Poor" because the tool reports converged=True. However, -GridCal's DCOPF uses overload slack variables — the zero-rated-branch problem has a feasible -LP solution that includes nonzero slacks. From D-4: *"treats zero-rated branches as soft -constraints via the overload slack variables and produces a feasible dispatch that violates -all branch limits."* - -A tool using soft constraints is behaving correctly (per its own formulation design) when -it finds a feasible solution to a soft-constrained problem. The test conflates two questions: -(1) does the tool enforce branch limits as hard constraints, and (2) does the tool report -solver-level infeasibility when the LP is genuinely infeasible (no feasible dispatch to -meet load balance). A genuine infeasibility test would set load higher than total generation -capacity, not zero branch ratings which a soft-constraint formulation will always handle. -The "poor" error quality rating may be unfair to a deliberately soft-constraint design. - -**Cross-tool relevance:** likely — any tool using soft constraints on branch limits will -fail this test by design. -**Proposed action:** redesign_test — Replace the zero-branch-rating infeasibility test -with a load-exceeds-generation-capacity test that produces genuine LP infeasibility. -Add a separate disclosure test: does the tool document whether branch limits are hard -or soft constraints? - ---- - -### gridcal-F08: FNM ingestion gate failure stems from PSS/e v31 format mismatch — MATPOWER fallback masks the architectural gap - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** G-FNM-1, G-FNM-3, G-FNM-4 - -G-FNM-1 fails because GridCal's PSS/e parser is hardcoded to v35 field counts and v31 -(the FNM format) is absent from the supported versions list. The error is explicit: -*"Exception: PSSe 35 load data came with 1 elements and 18 or 17 were expected."* - -However, P2-1 characterizes this as a "medium effort" fix: *"The parser architecture -already threads a version parameter to device-level parsers. The fix requires auditing -each device parser to accept the correct field counts for v29-v34 formats — mechanical -but tedious."* - -The G-FNM-1 fail is therefore format-version support, not architectural incapability. -The tool supports PSS/e ingestion in principle (v35 works). G-FNM-3 and G-FNM-4 then -run on a MATPOWER fallback that loses area/zone metadata and aggregates multiple loads -per bus (8,624 vs 15,062 loads). The FNM suite results characterize a network loaded via -a degraded representation, yet G-FNM-3/4 results are attributed to the tool's power flow -capability rather than the fallback path's limitations. The protocol should distinguish -between format version gaps (engineering effort) and architectural incapability. - -**Cross-tool relevance:** likely — other tools may also have PSS/e version-specific -parser gaps that cause FNM ingestion to fall back to MATPOWER. -**Proposed action:** add_test — Add a separate test classifying PSS/e version support -range (which versions does the parser handle?) distinct from the CSV-format ingestion -gate. This would allow the evaluation to credit tools for supporting PSS/e v33+ without -penalizing them as heavily for v31 gaps. - ---- - -### gridcal-F09: ACPF false convergence pathway identified in G-FNM-4 but not tested in the D-4 error quality assessment - -**Category:** missing_verification | **Severity:** low -**Tests:** G-FNM-4 - -G-FNM-4 identifies that `retry_with_other_methods=True` causes false convergence: -*"When retry_with_other_methods=True, the solver reports converged=True after 1 iteration -with a residual of 582. This was diagnosed as false convergence — the retry mechanism -falls back to a method that terminates early without achieving actual convergence."* - -The D-4 error quality test does not test this pathway. D-4 uses manually introduced errors -(infeasible OPF, zero costs, no slack bus) but does not test whether the ACPF retry -mechanism correctly reports non-convergence. The false convergence mode (retry reports -success despite residual 582) is a distinct, unscored error quality failure that affects -the accessibility dimension. It should be incorporated into the D-4 error quality rating -or flagged as an untested failure mode. - -**Cross-tool relevance:** likely — retry mechanisms in other tools may have similar -premature termination behaviors. -**Proposed action:** add_test — Add a D-4 subtest for ACPF retry false convergence: -set an impossible initial condition and verify that retry_with_other_methods does not -report success without achieving the convergence tolerance. - ---- - -### gridcal-F10: Single TapPhaseControl bug produces two qualified_pass scores for B-4 and C-4 - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** B-4, C-4 - -B-4 (stochastic scenario DCOPF) and C-4 (SCUC at SMALL) both encounter the same -TapPhaseControl enum bug in VeraGridEngine 5.6.28 and both apply the identical sequential -snapshot workaround. The bug is documented identically: *"ValueError: 0 is not a valid -TapPhaseControl"* on networks with transformers when using time-indexed compilation. - -The two qualified_pass scores arising from a single bug could be collapsed to a single -finding with two affected test IDs. Currently the scoring implies two distinct capability -limitations with stable workarounds, when in fact a single version-specific bug affects -both. If the bug were fixed (as may be the case in v5.6.34, which was not evaluated), -both tests might pass natively. The evaluation does not flag this correlation, which -could influence how the maturity dimension is scored (a single known bug vs. two -independent capability gaps). - -**Cross-tool relevance:** none -**Proposed action:** adjust_scoring — Note in the evaluation that B-4 and C-4 -qualified_pass outcomes share the same root cause. Consider whether fixing the -TapPhaseControl bug would make both tests pass, which would elevate the extensibility -and scalability scores. - ---- - -### gridcal-F11: Operational adoption claims by named utilities are unverified - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** E-7 - -E-7 reports that GridCal claims operational use at Redeia (Spanish TSO), Schneider Electric, -and GE Vernova. These claims appear in the project's own research-context.md documentation. -From E-7: *"direct verification of these claims from public sources is limited — the evidence -comes primarily from the project's own research context documentation and presentations."* - -The only independently verifiable external adoption signal is 52 commits from two Navitasoft -developers. The eRoots website does not list clients publicly. These claims materially affect -the maturity narrative: if verified, they would substantially offset the bus-factor-1 and -zero-CI-testing concerns; if unverifiable, the maturity grade may be too lenient. The -synthesis acknowledges this explicitly (Section 5: "Claims of Redeia, Schneider Electric, -GE Vernova usage are unverified from public sources") but the maturity grade rationale does -not account for the unverified status. - -**Cross-tool relevance:** none -**Proposed action:** add_verification — Probe should search for public references: -case studies, conference presentations, regulatory filings, or press releases from Redeia, -Schneider Electric, or GE Vernova citing GridCal/VeraGrid. The LF Energy landscape -record (mentioned in E-7) may contain verifiable information. - ---- - -### gridcal-F12: Gate ingestion tests are unanimous pass across all tools — low discriminative value - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -The three gate tests (G-1/2/3) verify MATPOWER .m file loading at TINY, SMALL, and MEDIUM -scale. All tools in the evaluation passed these tests. The tests measure a table-stakes -prerequisite (MATPOWER file parsing) that does not differentiate tools. Including them in -the total test count inflates the apparent pass rate for all tools equally without adding -insight. The gate status should be recorded as a binary pass/fail gate criterion rather -than scored tests contributing to capability assessment. - -**Cross-tool relevance:** confirmed — all tools passed. -**Proposed action:** remove_test — Remove G-1/2/3 from the test outcome count. -Retain them as a binary gate check reported separately from capability test results. - ---- - -### gridcal-F13: SCOPF convergence unverified — no optimality gap or residual reported for A-9 or C-8 - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-9, C-8 - -Both A-9 (SCOPF TINY) and C-8 (SCOPF MEDIUM) report pass with no convergence residual or -optimality gap. The YAML frontmatter for both results shows `convergence_residual: null` -and `convergence_iterations: null`. SCOPF is an LP formulation in GridCal — the convergence -check should include LP optimality (dual feasibility, primal feasibility, and duality gap) -rather than just the `converged` flag. - -For C-8 specifically, the near-identical results between DCOPF and SCOPF (differences of -~1e-8) are consistent with a correctly solved LP but also consistent with early termination -on a trivially feasible incumbent. The 29.3s solve time for a 10,000-bus LP with 50 -contingency constraints is plausible but not benchmarked against theoretical expectations. -An optimality gap < 1e-6 should be confirmed via solver output or residual reporting. - -**Cross-tool relevance:** likely — other tools' SCOPF tests may also lack optimality -gap reporting. -**Proposed action:** add_verification — Protocol should require extracting LP dual -feasibility and primal feasibility metrics (or MIP gap for MILP formulations) from -solver output for all OPF-class tests. - ---- - -## Extraordinary Claims - -### A-3: DCOPF passes with a branch at 112% loading — soft constraint status unresolved - -**Concern:** The DCOPF formulation may use soft constraints (overload slack variables) -rather than hard branch thermal limits. If confirmed, all DCOPF/SCOPF/SCUC results in -the evaluation reflect penalized dispatch that allows thermal violations, which has direct -consequences for market dispatch accuracy. The D-4 test confirms soft-constraint behavior -for zero-rated branches. Whether the base DCOPF formulation enforces any hard thermal -limits is unresolved. -**Evidence quality:** moderate — D-4 confirms soft constraints exist; A-3 observes a -violation in practice. The exact formulation structure requires source code inspection. -**Probe type:** formulation_audit — Inspect `linear_opf_ts.py` branch flow constraint -construction to determine whether `flow[i] <= rating[i]` is a hard inequality or -`flow[i] <= rating[i] + slack[i]` with a penalty. - ---- - -### G-FNM-3: 326 branches with deviations up to 562,955% classified as formulation difference without B-matrix inspection - -**Concern:** The top-5 deviating branches include four labeled "Line" with physically -implausible flows (up to 352,878 MW). The "formulation_difference" classification rests -on the transformer-adjacency heuristic, not on direct inspection of the B-matrix entries -for those branches. Flows 3-6 orders of magnitude above reference values on line-type -elements suggest a possible data ingestion error or incorrect per-unit scaling that would -invalidate the qualified_pass. -**Evidence quality:** weak — the result pattern is consistent with the stated hypothesis -but also with alternative explanations involving data ingestion errors. -**Probe type:** formulation_audit — Extract GridCal's B-matrix entries for the top-5 -deviating branches and compare against MATPOWER's `makeBdc()` output for the same branches. - ---- - -### E-7: Operational adoption by Redeia, Schneider Electric, GE Vernova — sourced from project's own documentation - -**Concern:** These are high-credibility claims for a tool with bus factor 1 and no -automated CI testing. If accurate, they substantially mitigate maturity concerns. If -they cannot be independently verified, the maturity grade narrative may require revision. -**Evidence quality:** weak — claims originate from project's own research-context.md; -no public third-party references found. -**Probe type:** claim_verification — Search for public references: conference presentations, -case studies, press releases, or regulatory documents citing GridCal/VeraGrid by Redeia, -Schneider Electric, or GE Vernova. Check the LF Energy landscape record for corroboration. - ---- - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | — | Low signal: all tools pass | -| G-2 | pass | — | Low signal: all tools pass | -| G-3 | pass | — | Low signal: all tools pass | -| A-1 | pass | — | — | -| A-2 | pass | — | — | -| A-3 | pass | — | 112% branch overload; soft constraint status unverified | -| A-4 | pass | — | — | -| A-5 | pass | — | MIP gap not extractable; binary commitment not verified | -| A-6 | pass | stable | Profile manipulation workaround, documented | -| A-9 | pass | — | No optimality gap reported | -| A-10 | qualified_pass | — | Loss approximation produces 0.0009% of load; near-zero signal | -| A-11 | fail | blocking | distributed_slack hardcoded False in OPF formulation | -| A-12 | fail | blocking | Battery energy balance sign error in linear_opf_ts.py | -| B-1 | qualified_pass | fragile | Monkey-patch on internal constraint naming; depends on 3 internal naming conventions | -| B-2 | pass | — | — | -| B-3 | pass | — | — | -| B-4 | qualified_pass | stable | TapPhaseControl enum bug; same root cause as C-4 | -| B-5 | pass | — | — | -| B-6 | informational | — | Monolithic OPF (3146 LOC); no hook points | -| B-8 | pass | — | — | -| B-9 | pass | — | — | -| C-1 | pass | — | — | -| C-2 | pass | — | — | -| C-3 | pass | stable | GLPK absent; SCIP substituted | -| C-4 | qualified_pass | stable | TapPhaseControl bug + monolithic MILP intractable; zero UC cycling | -| C-5 (SMALL) | pass | — | — | -| C-5 (MEDIUM) | pass | — | — | -| C-7 | pass | stable | GLPK absent; SCIP substituted | -| C-8 (SMALL) | pass | — | — | -| C-8 (MEDIUM) | pass | — | Uncongested network; zero SCOPF redispatch; timing measurement only | -| C-9 | pass | — | — | -| C-10 | fail | blocking | Cascaded from A-11 | -| D-1 | informational | — | Fast install; clean first-solve pattern | -| D-2 | informational | — | ReadTheDocs 6 versions behind; 8/10 tests require source reading | -| D-3 | informational | — | All tutorials use deprecated GridCal.Engine imports | -| D-4 | informational | — | OPF reports converged=True on infeasible; test design gap for zero-rate test | -| D-5 | informational | — | Median 183 NBNC LOC | -| E-1 | informational | — | 209 releases/24mo; 28 of 209 with GitHub tags | -| E-2 | informational | — | ~2,357 commits; 8 named contributors in last 12mo | -| E-3 | informational | — | Bus factor 1; 0 of 30 PRs had reviewers | -| E-4 | informational | — | eRoots commercial backing | -| E-5 | informational | — | Median 115 days to close issues; batch-closing pattern | -| E-6 | informational | — | 125 test files; zero run in CI; pylint targets Python 3.8-3.10 | -| E-7 | informational | — | Adoption claims unverified from public sources | -| F-1 | pass | — | MPL-2.0 core | -| F-2 | pass | — | 62 deps; heavy but within gate threshold | -| F-3 | pass | — | 2 LGPL transitive deps (chardet, moocore) | -| F-4 | pass | — | Pure Python core | -| F-5 | pass | — | DCPF path fully traceable | -| F-6 | pass | — | No signing; 28/66 PyPI releases have GitHub tags | -| F-7 | pass | — | ~500-700 MB air-gap bundle | -| F-8 | pass | — | HiGHS bundled; no commercial solver required | -| F-9 | pass | — | Unpinned deps in eval project; no lock file | -| G-FNM-1 | fail | — | PSS/e v31 unsupported; no CSV network import | -| G-FNM-2 | informational | — | Skipped (blocked by G-FNM-1) | -| G-FNM-3 | qualified_pass | — | 326 branches at up to 562,955% deviation; formulation_difference classification unverified | -| G-FNM-4 | informational | — | ACPF infeasible on 27,862-bus FNM; false convergence identified in retry mode | -| G-FNM-5 | informational | — | 83% native contingency; 0% native interface/flowgate model | -| P2-1 | informational | — | PSS/e v31 gap: medium effort to fix | -| P2-2 | informational | — | Piecewise linear costs: not assessed in this sweep | -| P2-3 | informational | — | Commitment injection: not assessed in this sweep | diff --git a/sweep-data/v10-to-v11/per-tool/gridcal/findings.yaml b/sweep-data/v10-to-v11/per-tool/gridcal/findings.yaml deleted file mode 100644 index b4052943..00000000 --- a/sweep-data/v10-to-v11/per-tool/gridcal/findings.yaml +++ /dev/null @@ -1,384 +0,0 @@ -tool: gridcal -source_version: v10 -timestamp: "2026-03-14T00:00:00Z" -evaluation_summary: - total_tests: 37 - # Gate (G-1/2/3): 3 pass - # Expressiveness (A-1..A-12, minus A-7/A-8 which are not in eval-config): 10 tests - # Extensibility (B-1..B-9, minus B-7): 8 tests - # Scalability (C-1..C-10): 11 tests (including 2x C-5, 2x C-8) - # Accessibility (D-1..D-5): 5 informational - # Maturity (E-1..E-7): 7 informational - # Supply chain (F-1..F-9): 9 tests - # FNM (G-FNM-1..G-FNM-5): 5 tests - # P2 readiness (P2-1..P2-3): 3 informational - pass: 20 - fail: 3 - qualified_pass: 5 - informational: 19 - -findings: - - id: gridcal-F01 - category: network_insufficiency - severity: medium - test_ids: [A-5, C-4] - title: SCUC pass on TINY masks complete absence of inter-temporal commitment cycling at SMALL - description: > - A-5 passes SCUC on the 10-generator IEEE 39-bus network, where differentiated costs - drive 6 of 10 generators to cycle. At SMALL scale (C-4, 544 generators), no cycling - is observed across 24 hours under both solvers — all 430 dispatchable generators - remain committed. The SMALL result is scored qualified_pass via a sequential snapshot - workaround that structurally cannot enforce min-up/down or ramp constraints. The - transition from A-5 (native UC, cycling demonstrated) to C-4 (snapshot workaround, - zero cycling) represents a capability cliff that is not reflected in the grading. The - qualified_pass at C-4 implies scaled SCUC is feasible, but inter-temporal coupling - is entirely absent. - evidence: - - file: evaluations/gridcal/results/scalability/C-4_scuc_scale_SMALL.md - excerpt: "Committed gens: 430 (constant). Cycling generators: 0. No generator cycling observed. The SMALL network's base-case generator costs are not sufficiently differentiated to drive decommitment." - - file: evaluations/gridcal/results/expressiveness/A-5_scuc.md - excerpt: "Cycling generators: 6 (of 10). UC formulation correctly decommits expensive generators during low-load hours." - - file: evaluations/gridcal/results/scalability/C-4_scuc_scale_SMALL.md - excerpt: "Workaround loses inter-temporal UC coupling (min up/down times, ramp rates are not enforced across hours). This means the test demonstrates that 24 independent hourly OPF solves complete successfully on the SMALL network, but not that true multi-period SCUC works at scale." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: gridcal-F02 - category: network_insufficiency - severity: high - test_ids: [C-8] - title: SCOPF on MEDIUM is vacuous — zero binding constraints, zero redispatch - description: > - C-8 (SCOPF on 10,000-bus ACTIVSg network) is scored pass, but the SCOPF and base-case - DCOPF produce results that are numerically identical to 1.49e-08 $/MWh precision. The - MEDIUM network is uncongested (max branch loading 84.72%), so no N-1 contingency creates - post-contingency violations requiring redispatch. The test verifies that the SCOPF - formulation runs to completion on a large network — not that it produces security-driven - redispatch. A congested test network is needed to measure SCOPF expressiveness under - stress. The SMALL SCOPF (C-8 SMALL) does produce meaningful redispatch (up to 164 MW), - but the MEDIUM result is a near-zero-signal measurement of solve time only. - evidence: - - file: evaluations/gridcal/results/scalability/C-8_scopf_scale_MEDIUM.md - excerpt: "LMP and dispatch comparison: The SCOPF produces results nearly identical to the base-case DCOPF. The maximum dispatch difference is 2.67e-07 MW and the maximum LMP difference is 1.49e-08 $/MWh — effectively zero. No generators show dispatch changes above 1 MW, and no branches become binding." - - file: evaluations/gridcal/results/scalability/C-8_scopf_scale_MEDIUM.md - excerpt: "This is consistent with the ACTIVSg10k network being uncongested (max loading 84.72%). Since no branches are near their limits in the base case, N-1 contingencies do not create post-contingency violations that would require redispatch." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: gridcal-F03 - category: misleading_result - severity: medium - test_ids: [A-10] - title: Lossy DCOPF qualified_pass obscures a near-nonfunctional loss approximation - description: > - A-10 is scored qualified_pass because the loss approximation feature exists and - produces non-zero losses. However, the computed losses (0.055 MW, 0.0009% of load) - are orders of magnitude below the expected 0.5–3% range and far below the ACPF - reference of 43.6 MW (0.7% of load). The root cause — the loss formula uses branch - thermal ratings rather than actual power flows — means the approximation is - structurally broken for networks where branch resistances are small relative to ratings. - The qualified_pass framing implies a feature that works with caveats; the actual - situation is a feature that produces results indistinguishable from zero losses in - standard IEEE test cases. LMP decomposition is also entirely absent. - evidence: - - file: evaluations/gridcal/results/expressiveness/A-10_lossy_dcopf_lmp.md - excerpt: "Sum branch losses (MW): 0.055. Losses as % of load: 0.0009%. Loss magnitude: far below expected range." - - file: evaluations/gridcal/results/expressiveness/A-10_lossy_dcopf_lmp.md - excerpt: "The loss factor formula uses R * rate / V^2 where rate is the branch thermal rating rather than the actual power flow. This produces negligible loss factors." - - file: evaluations/gridcal/results/expressiveness/A-10_lossy_dcopf_lmp.md - excerpt: "(b) Losses 0.5--3% of load: FAIL — 0.0009% -- far below expected range." - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: gridcal-F04 - category: missing_verification - severity: medium - test_ids: [A-5] - title: SCUC commitment schedule inferred from dispatch power rather than binary commitment variable - description: > - The A-5 SCUC pass is based on a commitment schedule derived by thresholding generator - power output (gen_power > 0.1 MW) rather than reading a binary commitment variable - from the solver. The protocol requires verifying that the MILP formulation produces - binary on/off decisions. The MIP gap is explicitly noted as "not directly extractable" - from the results object. If the solver is running as an LP relaxation rather than MILP, - the commitment indicators would still appear binary-like for most generators due to - box constraints, but with no guarantee of true integrality. The test cannot distinguish - between a genuine MILP solution and an LP relaxation that happens to produce near-binary - dispatch values. - evidence: - - file: evaluations/gridcal/results/expressiveness/A-5_scuc.md - excerpt: "MIP gap note: The MIP gap is not directly extractable from the OptimalPowerFlowTimeSeriesResults object. HiGHS uses a default MIP gap tolerance of 1%, which satisfies the pass condition." - - file: evaluations/gridcal/results/expressiveness/A-5_scuc.md - excerpt: "commitment = (gen_power > 0.1).astype(int) # Commitment derived from generator_power (shape: 24 x 10)" - cross_tool_relevance: likely - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: gridcal-F05 - category: misleading_result - severity: medium - test_ids: [G-FNM-3] - title: FNM DCPF qualified_pass contains 326 branches with deviations up to 562,955% attributed as formulation difference without independent validation - description: > - G-FNM-3 triggers the hard fail condition (max branch deviation > 50%) but is classified - qualified_pass via a "formulation_difference" classification that attributes the 326 - failing branches (up to 562,955% deviation) to GridCal's simplified B-matrix construction - for transformer tap ratios. While the transformer-adjacency concentration (88.7%) is - consistent with this hypothesis, the classification rests on an inference from result - patterns rather than direct verification of the B-matrix formulation against a known - reference. The deviations include branches labeled "Line" (not "Xfmr") in the top-5 - list (4 of 5 are lines), which undermines the pure transformer-tap explanation. The - qualified_pass may understate a genuine data ingestion or formulation error. - evidence: - - file: evaluations/gridcal/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md - excerpt: "Top 5 deviating branches: 1668→88630 (Line, 562,955%), 21476→84022 (Line, 541,075%), 72100→73053 (Line, 413,787%), 180421→36990 (Xfmr, 325,193%), 1635→92191 (Line, 322,365%)." - - file: evaluations/gridcal/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md - excerpt: "Transformer-adjacent: 289 (88.7%). Classification: formulation_difference. The 326 failing branches exhibit extreme flow deviations... consistent with GridCal using a simplified B-matrix construction." - cross_tool_relevance: none - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: gridcal-F06 - category: extraordinary_claim - severity: high - test_ids: [A-3] - title: DCOPF shows 112% branch overload on a test claiming converged=True — soft constraint not disclosed in pass determination - description: > - A-3 is scored pass despite branch 2_3_1 showing 112% loading — a physical violation of - the derated thermal limit. The result notes this "may indicate the PTDF-based formulation - uses soft constraints or penalty functions rather than hard limits on some branches." If - the formulation uses soft constraints, then the DCOPF is not enforcing thermal limits as - hard constraints, which is a fundamental formulation characteristic relevant to market - applications. The pass scoring accepts an overloaded result without confirming whether - this is a soft-constraint formulation (acceptable approximation) or a solver infeasibility - being masked (comparable to the D-4 finding where infeasible OPF reports converged=True). - evidence: - - file: evaluations/gridcal/results/expressiveness/A-3_dcopf.md - excerpt: "Branch 2_3_1: 112% (overloaded). This may indicate the PTDF-based formulation uses soft constraints or penalty functions rather than hard limits on some branches. The overloads result attribute captures this with a value of -42.85 MW." - - file: evaluations/gridcal/results/accessibility/D-4_error_quality.md - excerpt: "The solver does not detect or report the infeasibility. Instead it treats zero-rated branches as soft constraints via the overload slack variables and produces a feasible dispatch that violates all branch limits." - cross_tool_relevance: none - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: gridcal-F07 - category: test_design_gap - severity: medium - test_ids: [D-4] - title: Error quality test omits OPF infeasibility with structural infeasibility (zero-rate branches), conflating soft-constraint design with error reporting - description: > - D-4 test (a) sets all branch ratings to zero and expects infeasibility detection. - However, GridCal's DCOPF uses soft constraints (overload slack variables) by design, - so the formulation is not structurally infeasible — it has a feasible solution that - violates branch limits with nonzero slacks. The test mixes two distinct questions: - (1) does the formulation use hard or soft branch constraints, and (2) does the tool - report infeasibility when the LP is genuinely infeasible. A true infeasibility test - would require a case where no dispatch can satisfy power balance (e.g., load > total - generation capacity), not one where soft-constraint violations are the expected behavior. - The "poor" error quality rating from this test may be partially unfair to GridCal's - design choice. - evidence: - - file: evaluations/gridcal/results/accessibility/D-4_error_quality.md - excerpt: "treats zero-rated branches as soft constraints via the overload slack variables and produces a feasible dispatch that violates all branch limits. The converged=True status is misleading." - - file: evaluations/gridcal/results/accessibility/D-4_error_quality.md - excerpt: "Error quality grade: Poor. No exception, no warning, no infeasibility status." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: gridcal-F08 - category: infrastructure_friction - severity: medium - test_ids: [G-FNM-1, G-FNM-3, G-FNM-4] - title: FNM ingestion gate failure stems from PSS/e v31 format mismatch, not tool architecture — MATPOWER fallback masks the gap - description: > - G-FNM-1 fails because GridCal's PSS/e parser is hardcoded to v35 field counts and - cannot parse v31 RAW files. The intermediate CSV format (17 tables) is also - unsupported. Both gaps are format-specific limitations, not architectural barriers — - the parser already threads a version parameter to device parsers and supports related - versions. The evaluation then proceeds via a MATPOWER fallback that loses area/zone - metadata and loads aggregates at per-bus level (8,624 loads vs 15,062 in RAW). - G-FNM-3 and G-FNM-4 results are therefore evaluated on an impoverished network - representation. The FNM test suite conflates format-version support (a fixable - engineering gap) with fundamental data model capability. - evidence: - - file: evaluations/gridcal/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md - excerpt: "Exception: PSSe 35 load data came with 1 elements and 18 or 17 were expected. Version 31 is notably absent from the supported versions list." - - file: evaluations/gridcal/results/p2_readiness/P2-1_psse_raw_parsing.md - excerpt: "Estimated effort to fix: Medium. The parser architecture already threads a version parameter to device-level parsers. The fix requires auditing each device parser to accept the correct field counts for v29-v34 formats." - - file: evaluations/gridcal/results/fnm_ingestion/G-FNM-1_fnm_ingestion_gate.md - excerpt: "MATPOWER format lost area data: 0 of 49 areas retained. Loads: 8,624 ingested vs 15,062 in RAW." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_test - - - id: gridcal-F09 - category: missing_verification - severity: low - test_ids: [G-FNM-4] - title: ACPF false convergence pathway identified but not quantified across solver configurations - description: > - G-FNM-4 identifies that setting retry_with_other_methods=True causes false convergence - reporting (converged=True with residual 582). This false convergence mode was excluded - from results. The finding implies a second D-4-class error quality problem: the tool's - built-in retry mechanism can report success when the solver has not converged. This - pattern was not tested in D-4 (which used only single-solver, no-retry configurations) - and represents an untested failure mode in the accessibility dimension. - evidence: - - file: evaluations/gridcal/results/fnm_ingestion/G-FNM-4_fnm_acpf_convergence.md - excerpt: "False convergence with retry: When retry_with_other_methods=True, the solver reports converged=True after 1 iteration with a residual of 582. This was diagnosed as false convergence." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_test - - - id: gridcal-F10 - category: scoring_inconsistency - severity: low - test_ids: [B-4, C-4] - title: TapPhaseControl enum bug triggers qualified_pass in both B-4 and C-4 with identical snapshot workaround — counted as separate test signal - description: > - B-4 (stochastic scenario DCOPF) and C-4 (SCUC at SMALL) both trigger the same - TapPhaseControl enum bug and both apply the same sequential snapshot workaround. - The bug is counted as a distinct limitation in both tests, but it is a single - underlying defect producing two qualified_pass scores. If the bug were fixed, both - tests might pass or fail independently. The current scoring treats a single bug's - impact as two separate capability demonstrations, which inflates the qualified_pass - count and potentially the overall capability impression. - evidence: - - file: evaluations/gridcal/results/extensibility/B-4_stochastic_scenario.md - excerpt: "VeraGridEngine 5.6.28 bug: TapPhaseControl enum sparse profile default value (0) is not a valid enum member. The compile_numerical_circuit_at function fails when compiling at a non-None time index for networks with transformers." - - file: evaluations/gridcal/results/scalability/C-4_scuc_scale_SMALL.md - excerpt: "TapPhaseControl enum bug (v5.6.28): The time-series OPF driver crashes with ValueError: 0 is not a valid TapPhaseControl on networks with transformers. This is the same bug documented in B-4." - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: gridcal-F11 - category: extraordinary_claim - severity: medium - test_ids: [E-7] - title: Operational adoption claims (Redeia, Schneider Electric, GE Vernova) are unverified from public sources - description: > - E-7 reports that GridCal claims adoption by Redeia (Spanish TSO), Schneider Electric, - and GE Vernova. These are significant credibility claims for a tool with bus factor 1 - and no CI test execution. The only independently verifiable external adoption signal is - 52 commits from two Navitasoft developers. The eRoots website does not list clients - publicly, and claims appear to originate from the project's own research-context.md - documentation. If these claims are accurate, they would substantially offset the - maturity concerns; if unverifiable, the maturity grade narrative may be too lenient. - evidence: - - file: evaluations/gridcal/results/maturity/E-7_operational_adoption.md - excerpt: "Claimed operational users: Redeia, Schneider Electric, GE Vernova. Verifiable adoption signals: Navitasoft integration (52 commits). eRoots website does not list clients publicly." - - file: evaluations/gridcal/results/maturity/E-7_operational_adoption.md - excerpt: "direct verification of these claims from public sources is limited — the evidence comes primarily from the project's own research context documentation and presentations." - cross_tool_relevance: none - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: gridcal-F12 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: Gate ingestion tests (TINY/SMALL/MEDIUM) are unanimous pass across all tools evaluated — minimal discriminative value - description: > - The three gate tests (G-1, G-2, G-3) verify that MATPOWER .m files can be loaded at - increasing scale. All tools evaluated passed these tests. The tests measure a basic - capability (MATPOWER file parsing) that is a prerequisite for all subsequent tests - and does not differentiate tools. Including them in the total test count and pass rate - inflates the apparent pass rate for all tools equally. - evidence: - - file: evaluations/gridcal/results/gate/G-1_ingest_tiny.md - excerpt: "Status: pass" - - file: evaluations/gridcal/results/gate/G-2_ingest_small.md - excerpt: "Status: pass" - - file: evaluations/gridcal/results/gate/G-3_ingest_medium.md - excerpt: "Status: pass" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: remove_test - - - id: gridcal-F13 - category: missing_verification - severity: medium - test_ids: [A-9, C-8] - title: SCOPF convergence not verified by residual — only converged flag and binding branch count reported - description: > - A-9 (SCOPF on TINY) and C-8 (SCOPF on MEDIUM) both report pass based on the tool's - convergence flag and qualitative dispatch comparison. No residual or optimality gap - is reported for either test. The protocol specifies convergence verification for OPF - results, but no MIP gap or LP duality gap is extracted. For SCOPF on MEDIUM specifically, - the near-zero differences from base-case DCOPF (to 1e-8 precision) are consistent with - a correctly solved LP — but also consistent with the solver terminating early on a - numerically ill-conditioned problem with a trivial incumbent. The lack of a reported - optimality gap prevents distinguishing these cases. - evidence: - - file: evaluations/gridcal/results/scalability/C-8_scopf_scale_MEDIUM.md - excerpt: "convergence_residual: null. convergence_iterations: null. The SCOPF produces results nearly identical to the base-case DCOPF. The maximum dispatch difference is 2.67e-07 MW." - - file: evaluations/gridcal/results/expressiveness/A-9_scopf.md - excerpt: "convergence_residual: null. convergence_iterations: null. SCOPF solve time: 0.067 s." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - -extraordinary_claims: - - test_id: A-3 - claim: > - DCOPF passes with 7 binding branches and valid LMPs despite one branch showing 112% - loading — attributed to soft-constraint formulation without explicit verification. - concern: > - If GridCal's DCOPF uses soft constraints (overload slacks) as a design choice, all - branch thermal limits in the evaluation are unenforced hard constraints. This would - mean every DCOPF/SCOPF/SCUC pass result may reflect soft-constrained optimization - rather than physically feasible dispatch. The D-4 error quality test confirms that - zero-rated branches are handled via soft constraints. The extent to which the base - DCOPF formulation enforces hard limits vs soft limits is not documented and was not - independently verified. - evidence_quality: moderate - probe_recommended: true - probe_type: formulation_audit - - - test_id: G-FNM-3 - claim: > - 326 branches with deviations up to 562,955% classified as "formulation_difference" - (B-matrix simplification for transformer taps) rather than data ingestion error, - earning a qualified_pass. - concern: > - The four top-5 deviating branches labeled "Line" (not transformer) cast doubt on - the purely transformer-tap explanation. Extreme flow values (e.g., 111,582 MW vs - reference -19.82 MW, a factor of ~5,630x) on line-type elements suggest possible - data ingestion corruption or incorrect unit conversion rather than a formulation - sophistication difference. The classification was made by inference from result - patterns, not by comparing B-matrix entries or inspecting the numerical circuit - compilation for those specific branches. - evidence_quality: weak - probe_recommended: true - probe_type: formulation_audit - - - test_id: E-7 - claim: > - GridCal is claimed to be in operational use at Redeia (Spanish TSO), Schneider - Electric, and GE Vernova. - concern: > - These claims originate from the project's own documentation (research-context.md) - and are not independently verifiable from public sources. If accurate, they - substantially offset the bus-factor-1 and no-CI-testing maturity concerns. If - inaccurate, the maturity grade narrative may be too lenient. - evidence_quality: weak - probe_recommended: true - probe_type: claim_verification diff --git a/sweep-data/v10-to-v11/per-tool/matpower/findings.md b/sweep-data/v10-to-v11/per-tool/matpower/findings.md deleted file mode 100644 index e59771ea..00000000 --- a/sweep-data/v10-to-v11/per-tool/matpower/findings.md +++ /dev/null @@ -1,467 +0,0 @@ -# matpower — Sweep Findings (v10) - -## Summary - -MATPOWER's evaluation is largely well-documented with measured timings and executed code -across all non-skipped tests. The most significant structural problem is a cascading gate -failure (matpower-F01): a GLPK exit-flag mapping bug causes MOST to reject a valid integer -solution, which fails C-4, which triggers the Suite C SMALL gate, which blanks all 8 -MEDIUM-tier scalability tests. This single solver integration defect—fixable with a -one-line patch—accounts for the bulk of the evaluation's missing evidence. A second -cross-cutting concern is the self-referential G-FNM-3 verification (matpower-F02), which -reports perfect accuracy because MATPOWER's own output was used as the reference. Two probes -are recommended: (1) patching the GLPK exit-flag mapping and re-running A-5/C-4 on case39, -and (2) auditing whether MOST's contingency table approach (Approach 3 in A-9) would -produce a valid SCOPF before finalizing the fail classification. - ---- - -## Finding Details - -### matpower-F01: GLPK exit flag mapping bug cascades into an 8-test scalability blackout - -**Category:** infrastructure_friction | **Severity:** high -**Tests:** A-5, C-4, and 8 MEDIUM-tier skips (C-1 through C-10 MEDIUM) - -MATPOWER's `miqps_glpk.m` wrapper maps GLPK's MIP-gap-termination code (GLP_EMIPGAP, -`errnum=9`) to a failure exit flag (`-9`). MOST's `most.m` line 2111 checks -`if mdo.QP.exitflag > 0` and skips all post-processing when the flag is negative. This -means GLPK can find and report a feasible integer solution, but MOST cannot extract it. - -In A-5, GLPK solved the 39-bus 24-hour SCUC in 0.68s with MIP gap ~0.8%, but returned -`exitflag=-9` (GLP_EMIPGAP). The solution vector exists in `mdo.QP.x` (3,576 variables) -but cannot be accessed via `most_summary()`. This forces the A-5 validation onto the -bundled 3-bus `ex_case3b` test case (where GLPK returns `exitflag=1` for exact optimality), -producing a qualified_pass on a network far smaller than the protocol target. - -In C-4, GLPK solved a 162,048-variable SCUC on ACTIVSg2000 in 1.112s with the same exit -flag behavior. The Suite C SMALL gate then blocks all 8 MEDIUM-tier tests (C-1, C-2, C-3, -C-5, C-7, C-8, C-9, C-10 on the MEDIUM network). These 8 skips are the largest evidence gap -in MATPOWER's evaluation. - -From `A-5_scuc.md`: *"GLPK finds an integer feasible solution (MIP gap ~0.8%) but returns -errnum=9 (GLP_EMIPGAP), which MATPOWER's miqps_glpk.m maps to exitflag=-9."* - -From `C-4_scuc_scale_SMALL.md`: *"MOST successfully assembled the SCUC problem and GLPK -solved it in 1.1 seconds, but returned exitflag=-10... The solution vector exists (162,048 -variables) but cannot be accessed via most_summary()."* - -The fix is a one-line change in `miqps_glpk.m`: accept `errnum=9` (GLP_EMIPGAP) and -`errnum=10` (GLP_ETMLIM) as feasible outcomes rather than failures. - -**Cross-tool relevance:** likely — other tools using GLPK for MILP in Octave may encounter -the same integration issue. - -**Proposed action:** Patch `miqps_glpk.m` to treat GLP_EMIPGAP as a feasible (non-optimal) -termination, then re-run A-5 on case39 and re-run C-4. If both succeed, reclassify A-5 as -pass/qualified_pass on case39 and reopen C-4 and the MEDIUM scalability suite. The gate -logic should also be reviewed to avoid coupling MEDIUM access to a MILP-specific test. - ---- - -### matpower-F02: G-FNM-3 DCPF pass is self-referential (reference generated by MATPOWER itself) - -**Category:** misleading_result | **Severity:** medium -**Tests:** G-FNM-3 - -G-FNM-3 reports 100% pass with zero deviation at every metric: max VA error = 0.0000 deg, -max branch deviation = 0.00%, all 27,862 buses passing, all 32,532 in-service branches -passing. The result file correctly discloses the reason: *"The reference DCPF solution was -generated by MATPOWER itself from the same .mat file, so this test verifies round-trip -consistency rather than cross-tool accuracy. All zero deviations are expected..."* - -The 0.168s wall-clock timing is legitimately informative as a large-scale performance data -point. However, the pass grade and 100% accuracy metrics, presented in isolation, create -the impression of independently validated correctness at LARGE scale. In a cross-tool -comparison where other tools may be compared against a MATPOWER-generated reference, -this creates a structural advantage for MATPOWER. - -**Cross-tool relevance:** confirmed — the shared FNM reference solution (`buses_dcpf.csv`, -`branches_dcpf.csv`) was generated by MATPOWER, so any tool whose DCPF formulation matches -MATPOWER's conventions will score well, while formulation differences will appear as -deviations regardless of physical correctness. - -**Proposed action:** Generate an independent cross-tool reference using a solver with a -different formulation (e.g., PowerModels.jl ACPowerFlow or pandapower) and reconcile -differences. The timing result should be retained and highlighted as the primary -informational value of G-FNM-3. - ---- - -### matpower-F03: A-9 SCOPF fail may be a solver-environment artifact rather than a tool capability gap - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-9 - -A-9 fails because MIPS (the only QP-capable solver available in the Octave devcontainer) -produces numerical singularity when user constraints are added to DC OPF. The LODF-based -constraint formulation via `mpc.A/l/u` is the correct API mechanism and is fully documented. -The problem is the solver, not the formulation. - -From `A-9_scopf.md`: *"MIPS solver (the only available solver in the devcontainer for QP -problems) cannot handle the N-1 constraint matrix with quadratic costs. HiGHS was -unavailable. A commercial solver (Gurobi, CPLEX) or HiGHS might succeed."* - -Additionally, MOST's contingency framework (Approach 3 in the result file) was not pursued -after Approaches 1 and 2 failed: *"MOST supports contingency states with probability -weighting. However... was not pursued after Approaches 1 and 2 failed, as the test protocol -prioritizes the standard OPF extension mechanism."* If MOST's contingency table produces a -valid SCOPF, the fail classification is incorrect. - -The B-1 custom constraint test (which also uses `mpc.A/l/u`) is classified as -qualified_pass by switching to GLPK with linear costs. The same approach might be viable -for A-9. - -**Cross-tool relevance:** likely — other tools' SCOPF results may similarly reflect solver -availability in the evaluation environment rather than formulation capability. - -**Proposed action:** Before finalizing A-9 as fail, explore (a) whether GLPK with linear -costs can handle a small subset of N-1 constraints (same workaround as B-1), and (b) whether -MOST's contingency table produces a valid SCOPF dispatch. If either produces a result, -reclassify A-9 as qualified_pass with documented workaround. - ---- - -### matpower-F04: A-6 ramp rate constraints are never binding, providing no signal on ramp enforcement quality - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-6 - -The A-6 pass condition requires that "ramp rate constraints are demonstrably enforced between -consecutive dispatch intervals in the ED stage." The result shows no generators hit binding -ramp limits: the most-constrained generator (G1) uses 48.5 MW/hr against a 62,400 MW/hr -limit. The ratio is approximately 1:1285. The ramp rates from RTS-GMLC technology medians -are designed for much larger systems and are orders-of-magnitude larger than the per-hour -dispatch changes on the 10-generator 39-bus case. - -From `A-6_sced.md`: *"No ramp violations detected across all 24 periods... No generators hit -binding ramp limits because the ramp rates from RTS-GMLC technology medians are generous -relative to the dispatch changes driven by the load profile."* - -The result demonstrates that ramp constraints are formulated and do not error out, and that -the formulation respects the GEN_STATUS = 0 constraint for decommitted generators. However, -it provides no evidence about whether binding ramp constraints would be correctly handled. -This test design gap applies across all tools evaluated with the same parametrization. - -**Cross-tool relevance:** confirmed — the same RTS-GMLC ramp rates and the same 39-bus -load profile would produce non-binding ramps for any tool. - -**Proposed action:** Set ramp limits to 50-100% of the maximum per-period dispatch change -for the marginal generator, ensuring at least one generator faces a binding ramp constraint -in each of several consecutive periods. This would make the "demonstrably enforced" language -in the pass condition meaningful. - ---- - -### matpower-F05: A-5 SCUC qualified_pass validated on 3-bus bundled test case, not the specified TINY network - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-5 - -The protocol specifies A-5 on the TINY network (IEEE 39-bus) with the pass condition -requiring "at least 2 generators must cycle (commit/decommit)." The result's qualified_pass -is based on validation against MOST's bundled `ex_case3b` (3 buses, 5 generators, 12 -periods), which shows only 1 generator cycling. The case39 attempt failed due to the GLPK -exit flag bug. - -From `A-5_scuc.md`: *"Validation on standard MOST test case (ex_case3b)... Cycling -generators: 1 (G2)."* The cycling count requirement (>=2) was not met even on the -fallback network. - -The qualified_pass documents MOST's SCUC API correctly and demonstrates the constraint -formulation, but does not demonstrate the required cycling behavior on the required network. -The combination of wrong network + wrong cycling count makes this a weak evidential basis -for the qualified_pass. - -**Cross-tool relevance:** none — this is specific to the GLPK/Octave environment limitation. - -**Proposed action:** Fix the GLPK exit flag mapping and re-run on case39. If that produces -a valid result with >=2 cycling generators, reclassify as pass. If not, the fallback evidence -should be clearly flagged as insufficient for the pass condition. - ---- - -### matpower-F06: A-11 pass condition language does not specify expected behavior for lossless DC OPF - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-11 - -The A-11 pass condition requires that "LMPs differ from single-slack results in a physically -consistent manner (SMEC reflects the distributed reference)." For lossless DC OPF, the -theoretically correct behavior is that distributed slack produces only a uniform shift in all -LMPs (identical to subtracting the weighted-average LMP), with no change in the dispatch or -LMP spreads. The result confirms this: uniform $-180.30/MWh shift, std dev = $0.00. - -The congestion LMP components do change meaningfully ($58.82/MWh max abs change) because the -PTDF matrix changes with the slack reference, which is physically correct behavior. - -The issue is that a tool could produce a non-uniform (incorrect) LMP shift and the current -pass condition does not provide enough specificity to distinguish correct from incorrect -behavior. The result is correct, but the verification criteria should be made explicit. - -From `A-11_distributed_slack_opf.md`: *"In DC OPF, the distributed slack formulation only -changes the LMP reference point, not the dispatch. All LMPs shift by a uniform constant -equal to the negative weighted average of single-slack LMPs. The dispatch (generator Pg -values) is identical regardless of slack formulation."* - -**Cross-tool relevance:** confirmed — the same ambiguity in the pass condition affects all -tools evaluated on A-11. - -**Proposed action:** Add explicit verification criteria: (a) |uniform_shift - weighted_avg_lmp| -< tolerance, (b) dispatch unchanged (Pg vector identical to single-slack), (c) congestion -components differ by the PTDF-reference change. These three criteria would distinguish -correct from incorrect behavior and would be testable across tools. - ---- - -### matpower-F07: Gate tests are expected to pass for all mature tools (low discriminative value) - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -The gate tests verify bus/branch/generator counts after loading native-format case files. -For MATPOWER, which ships the IEEE 39-bus and ACTIVSg cases in its own native distribution, -these tests are trivially satisfied. The gate purpose is correct (establishing a precondition -for further testing), but the tests provide no signal that differentiates tool capability. - -From `G-1_ingest_tiny.md`: *"MATPOWER loads its native .m case format without any conversion -or adaptation. The IEEE 39-bus (New England) test case is a standard MATPOWER distribution -case."* - -**Cross-tool relevance:** confirmed — all tools that have stable parsing of their primary -native format will pass G-1 through G-3. The tests are more informative for tools that -require format conversion. - -**Proposed action:** No change to gate tests (their purpose is precondition, not -differentiation). Cross-tool aggregation should not weight gate test results in capability -comparisons. - ---- - -### matpower-F08: C-5 progressive relaxation test is uninformative when no constraints bind - -**Category:** test_design_gap | **Severity:** medium -**Tests:** C-5 - -C-5 is designed as a diagnostic for networks near their feasibility boundary. Progressive -relaxation (0%, 10%, 20%) should reveal how a tool handles infeasibility. However, the -ACTIVSg2000 SMALL network has zero binding voltage or thermal constraints at base loading, -so all three relaxation levels produce identical solutions (5 NR iterations, identical -voltage profiles, 0 binding branches). - -From `C-5_ac_feasibility_relaxation_SMALL.md`: *"The solution is identical across all three -levels because the base case has no binding constraints — the relaxation has no effect on -the solution. Binding branches: 0 / 3206."* - -The result is correct and the convergence behavior is useful evidence, but the progressive -relaxation mechanism is never exercised. The test passes identically for any tool whose ACPF -solver converges on ACTIVSg2000 from flat start. - -**Cross-tool relevance:** confirmed — the same ACTIVSg2000 network is used across all tools. - -**Proposed action:** Before C-5, increase load by 130-150% until thermal violations appear, -then apply progressive relaxation starting from that infeasible base case. This would -provide meaningful signal on each tool's ability to handle and diagnose constraint violations. - ---- - -### matpower-F09: B-5 qualified_pass scoring depends on unresolved LOC criterion interpretation - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** B-5 - -B-5 protocol states "Trivial — fewer than 5 lines of code beyond the solve." MATPOWER's -minimal export (data only, no column headers) is 3 lines — passing the criterion. The -production-quality export with column headers is 12 lines. The qualified_pass was assigned -based on the production-quality version. The synthesis flags this: *"Verify whether the -'<5 lines' criterion applies to the minimal or production-quality version."* - -If the criterion applies to minimal functionality, B-5 should be a pass (3 lines). The -qualified_pass is conservative but potentially inconsistent with how the same criterion -is interpreted for Python tools that have `DataFrame.to_csv()` (typically 1-2 lines -including header). - -**Cross-tool relevance:** likely — the same ambiguity affects all tools, with Python-based -tools likely scoring better on the production-quality interpretation due to DataFrame -abstractions. - -**Proposed action:** Standardize the criterion as "production-quality export with column -headers in fewer than 5 lines" or "any working CSV export in fewer than 5 lines" uniformly -across all tools. The current ambiguity creates cross-tool comparison noise. - ---- - -### matpower-F10: D-1 install time is a retrospective estimate, not a measured wall-clock time - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** D-1 - -D-1 records `timing_source: estimated` and `wall_clock_seconds: null`. The result states -"approximately 2-3 minutes of human effort" but acknowledges this is not a measured time. -The protocol specifies "Measure wall-clock time from clean environment to successful DCPF -solve on TINY." No measurement was made. - -From `D-1_install_to_first_solve.md` YAML header: *"timing_source: estimated ... wall_clock_seconds: null"* - -The qualitative findings (interactive installer blocks on `input()`, hidden path dependencies -on `mips/lib` and `mp-opt-model/lib`) are well-evidenced and provide useful friction data. -Only the time claim lacks measured support. - -**Cross-tool relevance:** likely — D-1 timing methodology may be similarly estimated across -other tool evaluations. - -**Proposed action:** Re-run D-1 with stopwatch timing from a clean devcontainer (clear all -MATPOWER paths from Octave path, restart, measure time to first `runpf()` success). The -qualitative findings do not require re-run. - ---- - -### matpower-F11: C-3 qualified_pass reflects solver ecosystem gap rather than scalability quality - -**Category:** misleading_result | **Severity:** low -**Tests:** C-3 - -C-3 protocol specifies running DC OPF on MEDIUM with HiGHS and GLPK. MIPS was used instead -(because HiGHS has no Octave binding and GLPK fails with a singular basis matrix on the -2000-bus network). MIPS succeeded in 0.507s. The qualified_pass framing suggests a -scalability concern, but the tool solved the problem correctly and quickly; the limitation -is solver availability, not scalability. - -From `C-3_dcopf_scale_SMALL.md`: *"The test specified HiGHS and GLPK per the config, but -HiGHS is unavailable in Octave. MIPS was used as the primary solver."* - -**Cross-tool relevance:** likely — HiGHS unavailability in Octave affects all Octave-based -tests that specify HiGHS as primary. - -**Proposed action:** Separate solver-availability notes from pass/fail grade. When a tool -solves correctly using its built-in solver and only the specified external solver is -unavailable, the result should be recorded as pass with a solver-environment note rather -than qualified_pass. - ---- - -### matpower-F12: B-9 PTDF extraction is largely redundant with PTDF computation in A-9 and B-3 - -**Category:** redundant_test | **Severity:** low -**Tests:** B-9, A-9, B-3 - -B-9 demonstrates `makePTDF()` accuracy (machine-precision, <1e-12 MW error). A-9 also uses -`makePTDF()` and `makeLODF()` as the primary computational mechanism for SCOPF constraint -formulation. B-3 uses `makeLODF()` (derived from PTDF) for contingency screening. The -additional signal from B-9 is the explicit dimension verification (46x39) and numerical -accuracy quantification, which is not captured in A-9 or B-3. - -For MATPOWER, all three tests produce consistent outcomes (B-9 pass, A-9 uses PTDF -correctly before failing on the solver, B-3 pass). The redundancy is not harmful but reduces -the information density of the extensibility suite. - -**Cross-tool relevance:** likely — any tool with native PTDF support will pass B-9 and use -PTDF in A-9/B-3, making B-9 the least differentiating test in Suite B for capable tools. - -**Proposed action:** Consider consolidating B-9 into A-9 (verify PTDF dimensions and -accuracy before injecting contingency constraints) to reduce test count without losing -signal. - ---- - -## Extraordinary Claims - -### G-FNM-3: 0.168s DCPF solve on 27,862-bus FNM with 100% accuracy - -**Concern:** The 100% accuracy (zero deviation) is guaranteed by self-reference — MATPOWER's -own output was used as the reference. The timing claim (0.168s) is separately measured and -credible, but should not be conflated with validated accuracy. Cross-tool timing comparisons -using this result are valid; cross-tool accuracy comparisons at LARGE scale are not valid -until an independent reference is established. - -**Evidence quality:** moderate (timing is strong; accuracy is vacuous by construction) - -A probe to generate an independent DCPF reference from a different solver would either -confirm MATPOWER's solution or reveal formulation differences (phase-shift transformer -handling, reference bus convention, admittance matrix construction). - ---- - -### A-5: SCUC formulation completeness claimed on a 3-bus test case - -**Concern:** The qualified_pass asserts that MOST's SCUC formulation includes all required -constraint types (min up/down, startup costs, ramp rates, reserves) and produces a valid -commitment schedule. This is demonstrated on MOST's bundled `ex_case3b` (3 buses, -5 generators, 12 periods), which produced only 1 cycling generator (protocol requires >=2). -The case39 attempt failed due to a solver integration bug, not a formulation error. - -**Evidence quality:** weak - -The claim that MOST's SCUC formulation is complete is credible based on API inspection and -the 3-bus validation, but the cycling count requirement and the 39-bus network behavior are -unverified. A probe (fixing the GLPK exit flag and re-running on case39) would resolve this. - ---- - -## Test Outcome Matrix - -| Test ID | Network | Status | Workaround | Key Issue | -|---------|---------|--------|------------|-----------| -| G-1 | TINY | pass | — | — | -| G-2 | SMALL | pass | — | — | -| G-3 | MEDIUM | pass | — | — | -| A-1 | TINY | pass | — | — | -| A-2 | TINY | pass | — | — | -| A-3 | TINY | pass | — | HiGHS unavailable; MIPS used | -| A-4 | TINY | pass | — | — | -| A-5 | TINY | qualified_pass | stable | GLPK exit flag bug; validated on 3-bus fallback | -| A-6 | TINY | pass | stable | Per-period rundcopf instead of MOST CommitKey | -| A-9 | TINY | fail | blocking | MIPS singularity with user constraints; HiGHS unavailable | -| A-10 | TINY | fail | blocking | No internal loss model in DC OPF | -| A-11 | TINY | qualified_pass | stable | Post-processing via makePTDF (not native OPF) | -| A-12 | TINY | qualified_pass | stable | Linear costs used; MIPS singularity on QP | -| B-1 | TINY | qualified_pass | stable | MIPS fails with user constraints; GLPK LP workaround | -| B-2 | TINY | pass | — | — | -| B-3 | TINY | pass | — | — | -| B-4 | TINY | pass | stable | Per-period loop instead of MOST multi-period | -| B-5 | TINY | qualified_pass | stable | No DataFrame; manual fopen/fprintf for headers | -| B-6 | N/A | pass | — | — | -| B-8 | TINY | pass | — | — | -| B-9 | TINY | pass | — | — | -| C-1 | SMALL | pass | — | — | -| C-2 | SMALL | pass | — | — | -| C-3 | SMALL | qualified_pass | — | HiGHS unavailable; GLPK singular basis | -| C-4 | SMALL | fail | — | GLPK exit flag mapping bug (cascaded from A-5) | -| C-5 | SMALL | pass | — | — | -| C-1 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-2 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-3 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-5 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-7 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-8 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-9 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| C-10 | MEDIUM | skip | — | Blocked by C-SMALL-gate (C-4 fail) | -| D-1 | N/A | informational | — | Timing estimated, not measured | -| D-2 | N/A | informational | — | 5/10 Suite A from docs alone | -| D-3 | N/A | informational | — | 6/7 examples work | -| D-4 | N/A | informational | — | Mixed: excellent data validation, poor solver diagnostics | -| D-5 | N/A | informational | — | Median 265 LOC; high vs Python tools | -| E-1 | N/A | informational | — | 2 releases in 24 months | -| E-2 | N/A | informational | — | 121 commits/12mo; 87.6% single contributor | -| E-3 | N/A | informational | — | Bus factor 1; 97.6% lifetime from one maintainer | -| E-4 | N/A | informational | — | Grant funding ended; exploratory commercial | -| E-5 | N/A | informational | — | Median 37d to close; batched triage | -| E-6 | N/A | informational | — | CI on 4 platforms; no coverage metrics | -| E-7 | N/A | informational | — | 800K downloads; no operational deployments | -| F-1 | N/A | pass | — | BSD 3-Clause | -| F-2 | N/A | pass | — | 4 bundled deps, zero external | -| F-3 | N/A | pass | — | All BSD 3-Clause | -| F-4 | N/A | pass | — | Zero compiled extensions | -| F-5 | N/A | pass | — | Zero opaque steps | -| F-6 | N/A | pass | — | Versioned releases with SHA-256 | -| F-7 | N/A | pass | — | Fully air-gap installable | -| F-8 | N/A | pass | — | Built-in MIPS; no commercial solver required | -| F-9 | N/A | pass | — | Version-pinned, immutable artifacts | -| G-FNM-1 | LARGE | fail | — | No CSV import; limited to .m/.mat/PSS/E RAW | -| G-FNM-2 | LARGE | skip | — | Blocked by G-FNM-1 fail | -| G-FNM-3 | LARGE | pass | — | Self-referential reference (zero deviation by construction) | -| G-FNM-4 | LARGE | informational | — | NR fails all 3 relaxation levels; singular Jacobian | -| G-FNM-5 | LARGE | informational | — | 45% native field coverage (20/44 fields) | -| P2-1 | N/A | informational | — | PSS/E RAW v30/v33 supported via psse2mpc() | -| P2-2 | N/A | informational | — | PWL costs supported via poly2pwl() | -| P2-3 | N/A | informational | — | Commitment injection workflow viable | diff --git a/sweep-data/v10-to-v11/per-tool/matpower/findings.yaml b/sweep-data/v10-to-v11/per-tool/matpower/findings.yaml deleted file mode 100644 index 893e7b48..00000000 --- a/sweep-data/v10-to-v11/per-tool/matpower/findings.yaml +++ /dev/null @@ -1,374 +0,0 @@ -tool: matpower -source_version: v10 -timestamp: 2026-03-14T00:00:00Z -evaluation_summary: - total_tests: 49 - # Gate: G-1, G-2, G-3 (3 pass) - # Expressiveness: A-1 pass, A-2 pass, A-3 pass, A-4 pass, A-5 qpass, A-6 pass, A-9 fail, A-10 fail, A-11 qpass, A-12 qpass (10 tests) - # Extensibility: B-1 qpass, B-2 pass, B-3 pass, B-4 pass, B-5 qpass, B-6 pass, B-8 pass, B-9 pass (8 tests) - # Scalability: C-1 SMALL pass, C-2 SMALL pass, C-3 SMALL qpass, C-4 SMALL fail; C-1/2/3/5/7/8/9/10 MEDIUM skip (12 tests incl skips) - # Accessibility: D-1..D-5 informational (5 tests) - # Maturity: E-1..E-7 informational (7 tests) - # Supply Chain: F-1..F-9 pass (9 tests) - # FNM: G-FNM-1 fail, G-FNM-2 skip, G-FNM-3 pass, G-FNM-4 informational, G-FNM-5 informational (5 tests) - # P2 readiness: P2-1, P2-2, P2-3 informational (3 tests, not graded) - pass: 22 - fail: 3 - qualified_pass: 6 - informational: 18 - # skip not a grade outcome; 9 tests skipped (8 MEDIUM + G-FNM-2) - -findings: - - id: matpower-F01 - category: infrastructure_friction - severity: high - test_ids: [A-5, C-4] - title: GLPK exit flag mapping bug cascades into an 8-test scalability blackout - description: >- - MATPOWER's miqps_glpk.m wrapper maps GLPK's MIP-gap-termination code (GLP_EMIPGAP, - errnum=9) to a failure exit flag (-9), causing MOST's post-processing to skip solution - extraction even when GLPK finds a feasible integer solution. This single solver - integration bug cascades: A-5 becomes a qualified_pass (validated on a bundled 3-bus - test case, not case39), C-4 fails (blocking the Suite C SMALL gate), and all 8 MEDIUM - scalability tests are skipped. The 162K-variable SCUC problem was actually solved by - GLPK in 1.1 seconds; only the exit-flag mapping prevents result extraction. - evidence: - - file: evaluations/matpower/results/expressiveness/A-5_scuc.md - excerpt: "GLPK finds an integer feasible solution (MIP gap ~0.8%) but returns errnum=9 (GLP_EMIPGAP), which MATPOWER's miqps_glpk.m maps to exitflag=-9. MOST's most.m line 2111 checks 'if mdo.QP.exitflag > 0' and skips all post-processing when the flag is negative." - - file: evaluations/matpower/results/scalability/C-4_scuc_scale_SMALL.md - excerpt: "GLPK finds a feasible integer solution ... but returned exitflag=-10 (mapped from GLPK's internal termination code)... The solution vector exists (162,048 variables) but cannot be accessed via most_summary()." - - file: evaluations/matpower/results/scalability/C-1_dcpf_scale_MEDIUM.md - excerpt: "C-4 (SCUC on SMALL) failed, triggering the Suite C SMALL gate. All MEDIUM-tier scalability tests are skipped per protocol." - cross_tool_relevance: likely - probe_recommended: true - probe_type: claim_verification - proposed_action: >- - The gate logic tying MEDIUM scalability access to SCUC passing (C-4) conflates a solver - integration bug with a scalability limitation. Protocol should either (a) gate MEDIUM - access on a non-MILP test (e.g., C-1 DCPF), or (b) add an exception for cascaded - infrastructure failures where the underlying solve succeeded. Immediate probe: patch - miqps_glpk.m to accept exitflag=-9/-10 as feasible and re-run A-5 and C-4 on case39 - to verify the SCUC solution is valid. - - - id: matpower-F02 - category: misleading_result - severity: medium - test_ids: [G-FNM-3] - title: G-FNM-3 DCPF pass is self-referential (reference generated by MATPOWER itself) - description: >- - G-FNM-3 reports 100% pass with zero deviation (max VA error = 0.0000 deg, max branch - deviation = 0.00%) because the reference DCPF solution was generated by MATPOWER from - the same .mat file. This is a round-trip consistency check, not a cross-tool accuracy - check. The result confirms determinism and self-consistency but provides no evidence - of correctness relative to an independent reference. The synthesis and result file both - acknowledge this; however, the pass grade and cited 0.168s timing may create a false - impression of independently validated accuracy. - evidence: - - file: evaluations/matpower/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md - excerpt: "The reference DCPF solution was generated by MATPOWER itself from the same .mat file, so this test verifies round-trip consistency rather than cross-tool accuracy. All zero deviations are expected because..." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: >- - The protocol should require a cross-tool reference for G-FNM-3 verification, or - explicitly label the self-referential case as "consistency check" rather than - "verification." As-is, each tool that uses its own native format will trivially - produce perfect scores. The timing result (0.168s) is legitimately meaningful - and should be retained as a scalability data point, but accuracy claims need - the cross-tool context. - - - id: matpower-F03 - category: misleading_result - severity: medium - test_ids: [A-9] - title: A-9 SCOPF fail may be a solver-environment artifact rather than a tool capability gap - description: >- - A-9 fails because MIPS (the only available QP solver in the Octave devcontainer) - produces numerical singularity when user constraints are added to DC OPF. The result - file explicitly notes that HiGHS or commercial solvers would likely succeed, and that - MOST's contingency framework (Approach 3) was not pursued. The protocol specification - calls for HiGHS as the solver for A-9, but HiGHS has no Octave binding in the - devcontainer. A fail grade on A-9 may overstate MATPOWER's actual SCOPF limitations: - the constraint formulation API exists (mpc.A/l/u + LODF), only the solver coupling - is broken in this environment. - evidence: - - file: evaluations/matpower/results/expressiveness/A-9_scopf.md - excerpt: "Durability: blocking -- MIPS solver (the only available solver in the devcontainer for QP problems) cannot handle the N-1 constraint matrix with quadratic costs. HiGHS was unavailable. A commercial solver (Gurobi, CPLEX) or HiGHS might succeed." - - file: evaluations/matpower/results/expressiveness/A-9_scopf.md - excerpt: "Approach 3: MOST contingency framework -- MOST supports contingency states with probability weighting. However... was not pursued after Approaches 1 and 2 failed, as the test protocol prioritizes the standard OPF extension mechanism." - cross_tool_relevance: likely - probe_recommended: true - probe_type: formulation_audit - proposed_action: >- - Probe Approach 3 (MOST contingency table) before finalizing A-9 as a fail. If MOST's - contingency framework produces a valid SCOPF solution, the result should be reclassified - as qualified_pass. Additionally, note in the cross-tool report that several A-9 "fails" - across tools may reflect solver-environment constraints rather than formulation gaps. - - - id: matpower-F04 - category: test_design_gap - severity: medium - test_ids: [A-6] - title: A-6 ramp rate constraints are never binding, providing no signal on ramp enforcement quality - description: >- - The A-6 pass condition requires that "ramp rate constraints are demonstrably enforced - between consecutive dispatch intervals." The result shows no generators hit binding - ramp limits across all 24 periods, with the most-constrained generator (G1) using only - 48.5 MW/hr of a 62,400 MW/hr ramp limit. The ramp rates from RTS-GMLC technology - medians are orders-of-magnitude larger than the dispatch changes driven by the 39-bus - load profile. The test demonstrates that ramp constraints are formulated and do not - error, but provides no signal about whether they would bind correctly if the network - were designed to force them to bind. This is a network insufficiency for the specific - sub-criterion of ramp enforcement quality. - evidence: - - file: evaluations/matpower/results/expressiveness/A-6_sced.md - excerpt: "No ramp violations detected across all 24 periods... No generators hit binding ramp limits because the ramp rates from RTS-GMLC technology medians are generous relative to the dispatch changes driven by the load profile." - - file: evaluations/matpower/results/expressiveness/A-6_sced.md - excerpt: "G1: Max Ramp 48.5 MW/hr, Ramp Limit 62,400 MW/hr, Binding? No" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: >- - The protocol should specify minimum ramp constraint tightness to ensure binding. - For a 39-bus 10-generator network with a 0.7x-1.3x load swing over 24 hours, ramp - limits should be set at no more than 2x the maximum expected per-period dispatch change - for the marginal unit. As designed, this test passes for any tool that can formulate - a multi-period OPF without crashing. - - - id: matpower-F05 - category: network_insufficiency - severity: medium - test_ids: [A-5] - title: A-5 SCUC qualified_pass validated on 3-bus bundled test case, not the specified TINY network - description: >- - A-5's qualified_pass is based on validation against MOST's bundled ex_case3b (3 buses, - 5 generators, 12 periods), not the case39 39-bus network specified by the protocol. - The 3-bus test case has only 1 generator cycling, while the protocol requires at least - 2. The case39 attempt failed due to the GLPK exit flag bug. The qualified_pass therefore - conflates two separate issues: (a) MOST's SCUC formulation is correct (demonstrated on - the 3-bus case), and (b) the case39 result is unavailable. A result on a 3-bus network - is not equivalent evidence for a 39-bus network with differentiated costs and cycling - behavior. - evidence: - - file: evaluations/matpower/results/expressiveness/A-5_scuc.md - excerpt: "Validation on standard MOST test case (ex_case3b): The MOST SCUC formulation was first verified on the bundled ex_case3b test case (3-bus, 5 generators including wind, 12 periods)." - - file: evaluations/matpower/results/expressiveness/A-5_scuc.md - excerpt: "ex_case3b: Cycling generators: 1 (G2). The protocol requires at least 2 generators cycling." - cross_tool_relevance: none - probe_recommended: true - probe_type: claim_verification - proposed_action: >- - Fix the GLPK exit flag mapping (see F01) and re-run A-5 on case39 to obtain the - intended result. Until then, the qualified_pass should note that the cycling count - requirement (>=2 generators) was not verified on the target network. - - - id: matpower-F06 - category: missing_verification - severity: medium - test_ids: [A-11] - title: A-11 distributed slack qualified_pass provides no evidence that the dispatch changed - description: >- - A-11's pass condition requires that "LMPs differ from single-slack results in a - physically consistent manner." The result shows LMPs shift by a uniform constant - (-$180.30/MWh for load-proportional slack) with std dev = $0.00. The result file - acknowledges this is mathematically expected for lossless DC OPF. However, the - protocol pass condition implies the distributed slack should affect something beyond - a rigid shift -- the "SMEC reflects the distributed reference" language. The test - passes because the shift is uniform (which is correct), but the congestion LMP - components do change ($58.82/MWh max abs change), suggesting the test does - distinguish behavior but the explanation could be clearer. The dispatch is identical - to A-3, which is correctly noted but may not be visible to report readers. - evidence: - - file: evaluations/matpower/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "Key insight: In DC OPF, the distributed slack formulation only changes the LMP reference point, not the dispatch. All LMPs shift by a uniform constant equal to the negative weighted average of single-slack LMPs." - - file: evaluations/matpower/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "LMP shift std dev: $0.00/MWh (perfectly uniform)" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: >- - The protocol should clarify that for lossless DC OPF, distributed slack produces - only a uniform LMP shift, and that this is the physically correct behavior. The - test should explicitly verify that (a) the uniform shift magnitude equals the - weighted average of single-slack LMPs, and (b) congestion components change - consistent with the new PTDF reference. Both are met in this result but neither - is explicitly required by the current pass condition language. - - - id: matpower-F07 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: Gate tests are expected to pass for all mature tools and provide no differentiation - description: >- - G-1, G-2, and G-3 test network ingestion with bus/branch/generator count verification. - For a tool like MATPOWER that ships the IEEE 39-bus and ACTIVSg cases as native format - distributions, these tests are trivially satisfied. The tests correctly gate the - evaluation but provide no signal that distinguishes tool capability. All tools that - have reached maturity-level stability are expected to pass these tests. - evidence: - - file: evaluations/matpower/results/gate/G-1_ingest_tiny.md - excerpt: "MATPOWER loads its native .m case format without any conversion or adaptation. The IEEE 39-bus (New England) test case is a standard MATPOWER distribution case." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: >- - Gate tests serve their intended purpose as preconditions. Their low signal is by - design. No change recommended, but cross-tool aggregation should not weight gate - test results in capability comparisons. - - - id: matpower-F08 - category: test_design_gap - severity: medium - test_ids: [C-5] - title: C-5 AC feasibility relaxation test is uninformative when no constraints bind at any relaxation level - description: >- - C-5 progressive relaxation produces identical ACPF solutions at 0%, 10%, and 20% - relaxation because the ACTIVSg2000 base case has no binding voltage or thermal - constraints (all 3,206 branches have 0 binding constraints at base load). The test - was designed as a diagnostic tool for networks near their feasibility boundary, but - the SMALL network is well within limits. The result records "relaxation_level_achieved: 0% - (no relaxation needed)" which is informative but means the test provides no signal - about the tool's ability to handle progressively relaxed constraints. - evidence: - - file: evaluations/matpower/results/scalability/C-5_ac_feasibility_relaxation_SMALL.md - excerpt: "The solution is identical across all three levels because the base case has no binding constraints -- the relaxation has no effect on the solution. Binding branches: 0 / 3206" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: >- - C-5 should be run on a network with pre-confirmed constraint violations at base - loading, or the loading should be increased until at least one thermal or voltage - constraint binds before applying progressive relaxation. The current design produces - uniform pass outcomes for any tool whose ACPF solver converges on the SMALL network. - - - id: matpower-F09 - category: scoring_inconsistency - severity: low - test_ids: [B-5] - title: B-5 qualified_pass classification is ambiguous given the protocol's <5 LOC criterion - description: >- - The protocol pass condition for B-5 states "Trivial -- fewer than 5 lines of code - beyond the solve." The result file documents that a minimal export (no headers) is - 3 lines -- meeting the criterion. A production-quality export with headers is 12 lines. - The qualified_pass is assessed based on the production-quality version. The synthesis - flags this as an open question: "Verify whether the '<5 lines' criterion applies to the - minimal or production-quality version." If the criterion is interpreted as minimal - functionality, B-5 should be a pass. - evidence: - - file: evaluations/matpower/results/extensibility/B-5_interoperability.md - excerpt: "Minimal export (no column headers): 3 lines of code. With column headers (production-quality): requires fopen/fprintf/fclose/dlmwrite pattern -- 4 lines per table (12 lines total)." - - file: evaluations/matpower/results/synthesis.md - excerpt: "Verify whether the '<5 lines' criterion applies to the minimal or production-quality version." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: >- - The protocol should specify whether the LOC criterion applies to minimal functionality - (data written, no headers) or production-quality output (headers included). If the - intent is to measure friction for analyst use, production-quality is the right target. - This ambiguity likely affects multiple tools. - - - id: matpower-F10 - category: extraordinary_claim - severity: medium - test_ids: [D-1] - title: D-1 install time is an estimate, not a measured wall-clock time - description: >- - D-1 reports timing_source: estimated and wall_clock_seconds: null. The result - claims "approximately 2-3 minutes of human effort" from clean environment to first - solve, but this is a retrospective estimate. The protocol specifies "Measure - wall-clock time from clean environment to successful DCPF solve on TINY." No - stopwatch measurement was made. The result is useful qualitative evidence but - does not satisfy the protocol's measurement requirement. - evidence: - - file: evaluations/matpower/results/accessibility/D-1_install_to_first_solve.md - excerpt: "timing_source: estimated ... wall_clock_seconds: null ... Time from clean environment to first successful DCPF solve is approximately 2-3 minutes of human effort" - cross_tool_relevance: likely - probe_recommended: false - probe_type: timing_verification - proposed_action: >- - D-1 should be re-run with actual stopwatch timing starting from a clean devcontainer - state. The qualitative friction findings (interactive installer, hidden path deps) are - well-documented; only the timing claim needs verification. This finding likely applies - across all tools evaluated. - - - id: matpower-F11 - category: misleading_result - severity: low - test_ids: [C-3] - title: C-3 qualified_pass reflects solver availability gap, not tool scalability quality - description: >- - C-3 is scored as qualified_pass because HiGHS (the protocol-specified solver) is - unavailable in Octave, and GLPK failed with a singular basis matrix on the 2000-bus - network. MIPS succeeded in 0.507s. The result is correct but the qualified_pass - framing may suggest a scalability concern when the actual issue is solver ecosystem - limitation. The synthesis correctly identifies this as a "solver ecosystem" cross-cutting - issue rather than a scalability deficiency. - evidence: - - file: evaluations/matpower/results/scalability/C-3_dcopf_scale_SMALL.md - excerpt: "The test specified HiGHS and GLPK per the config, but HiGHS is unavailable in Octave. MIPS was used as the primary solver... MIPS: Wall clock 0.507s, Converged." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: >- - The scoring rubric should distinguish between (a) tool unable to solve the problem - and (b) specified solver unavailable in the evaluation environment. When the tool - solves correctly using its built-in solver (MIPS), a pass with environment note is - more informative than a qualified_pass that suggests a tool deficiency. - - - id: matpower-F12 - category: redundant_test - severity: low - test_ids: [B-9, A-9] - title: B-9 PTDF extraction is substantially redundant with the PTDF computation already done in A-9 - description: >- - B-9 demonstrates makePTDF() with TINY network, achieving machine-precision accuracy - (error < 1e-12 MW). However, A-9's SCOPF attempt already uses makePTDF() and - makeLODF() as the primary computation mechanism, documented with essentially the same - API calls. The additional signal from B-9 is the explicit dimension and accuracy - verification, but the core capability (native PTDF computation) is already well - established from A-9. For MATPOWER specifically, this is not an issue since both - tests pass/fail consistently; for cross-tool comparison, B-9 provides incremental value - in the accuracy quantification. - evidence: - - file: evaluations/matpower/results/extensibility/B-9_ptdf_extraction.md - excerpt: "Used MATPOWER's native makePTDF(baseMVA, bus, branch) function. Max absolute error: 2.27e-12 MW." - - file: evaluations/matpower/results/expressiveness/A-9_scopf.md - excerpt: "Built LODF matrix via makeLODF(branch, PTDF) (documented API). For each N-1 contingency (41 non-radial branches), computed post-contingency flow coefficients..." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: >- - B-9 provides additive value through explicit accuracy quantification. No removal - recommended, but the cross-tool report should note that tools with native PTDF APIs - may appear to have higher differentiation than is warranted between B-9 and A-9/B-3. - -extraordinary_claims: - - test_id: G-FNM-3 - claim: >- - MATPOWER solves DCPF on a 27,862-bus FNM network in 0.168 seconds with 100% bus - angle accuracy (zero deviation) and 100% branch flow accuracy (zero deviation). - concern: >- - The zero-deviation accuracy is mathematically guaranteed (self-referential test) and - not independently verified. The 0.168s timing is measured and credible given sparse - linear algebra on a DCPF, but the implied accuracy claim (cross-tool comparison - potential) is undermined by the self-referential reference. The timing alone is the - extraordinary and valuable claim. - evidence_quality: moderate - probe_recommended: false - probe_type: null - - - test_id: A-5 - claim: >- - MATPOWER's MOST SCUC formulation supports all required constraint types (min up/down, - startup costs, ramp rates, reserves) and produces a valid commitment schedule on a - 24-hour horizon. - concern: >- - The primary validation was on a 3-bus, 12-period bundled test case (ex_case3b), not - on the 39-bus, 24-period case specified by the protocol. The case39 attempt failed - due to a solver integration bug. The 3-bus validation demonstrates API correctness - but is insufficient evidence for the SCUC formulation under realistic cost - differentiation and network topology. - evidence_quality: weak - probe_recommended: true - probe_type: claim_verification diff --git a/sweep-data/v10-to-v11/per-tool/pandapower/findings.md b/sweep-data/v10-to-v11/per-tool/pandapower/findings.md deleted file mode 100644 index 3629d213..00000000 --- a/sweep-data/v10-to-v11/per-tool/pandapower/findings.md +++ /dev/null @@ -1,530 +0,0 @@ -# pandapower — Sweep Findings (v10) - -## Summary - -The pandapower evaluation is thorough, well-evidenced, and self-consistent. All 60 result -files are present, timing measurements are marked as measured rather than estimated -throughout, and fail/pass determinations are accurate. The main quality issues fall into -two categories: protocol-design weaknesses that affect pandapower disproportionately due -to its narrow scope, and a small number of unverified claims in passing tests. The most -significant protocol problem is the C-SMALL gate design, which cascades a feature-absence -failure (SCUC) into 8 MEDIUM-tier scalability skips — for a tool that demonstrably -handles DCPF at 28K buses in 0.4 s. Two probe recommendations are made: one on A-3 branch -shadow prices (all 46 branches claimed binding, inconsistent with loading data) and one on -A-3 DC OPF optimality verification (convergence accepted from boolean flag only, with no -iteration count or residual reported by the PYPOWER solver). - ---- - -## Finding Details - -### pandapower-F01: C-SMALL gate design conflates feature absence with scale failure, muting 8 MEDIUM tests - -**Category:** test_design_gap | **Severity:** high -**Tests:** C-4, C-1, C-2, C-3, C-5 (MEDIUM), C-7, C-8, C-9, C-10 - -The protocol's C-SMALL gate requires C-4 (SCUC on SMALL) to pass before unlocking all -MEDIUM-tier scalability tests. For pandapower, C-4 fails because SCUC is architecturally -absent — not because the tool struggles with scale. The result: - -> "Skipped because Suite C SMALL gate failed (C-4 failed with blocked_by: A-5). The SMALL gate failure was a cascaded failure from A-5 (SCUC unsupported), not a scale-related failure." -> — C-1, C-2, C-3, C-5 MEDIUM (all) - -This skips 8 MEDIUM tests covering DCPF, ACPF, DC OPF, solver swap, SCOPF, PTDF, and -distributed-slack OPF — the entire second layer of scalability evidence. Meanwhile, the -tool has demonstrable scale evidence from FNM Suite G: - -> "FNM DCPF solved 28K buses in 0.4s" -> — synthesis.md - -The gate design was presumably intended to catch scale failures early, not to penalize -tools that lack a specific optimization formulation. The practical effect is that any tool -without SCUC support has its scalability dimension assessed on at most 2 tests (C-4 fail -+ C-5 SMALL pass), regardless of how well it performs at scale for its supported feature -set. - -**Cross-tool relevance:** confirmed — the gate design is in the shared protocol and will -affect any tool without SCUC (e.g., matpower, which also lacks native SCUC). -**Proposed action:** redesign_test — MEDIUM tests that do not depend on SCUC (C-1 DCPF, -C-2 ACPF, C-3 DC OPF, C-9 PTDF) should be independently runnable when C-4 fails due to -feature absence rather than scale failure. The gate should only cascade to SCUC-dependent -scalability tests. - ---- - -### pandapower-F02: ACPF convergence residual not verifiable — protocol pass accepted without residual scalar - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-2, C-5 - -The A-2 pass condition requires the convergence residual to be "reported and below the -tool's stated tolerance." pandapower does not expose the final NR residual: - -> "Convergence residual: below 1e-8 (tolerance_mva setting; exact value not extractable)" -> — A-2 result - -The iteration count (4) is accessible only via `net._ppc["iterations"]` — a private -underscore-prefixed attribute. The convergence_residual frontmatter field is null in both -A-2 and C-5. The pass is accepted based on 100% of buses differing from the 1.0 pu flat -start, which demonstrates convergence indirectly but does not satisfy the protocol's -explicit residual requirement. - -An additional complication: the tolerance_mva parameter has an unfixed unit bug (issue -\#2750) — documented but not treated as a qualification to the pass: - -> "pandapower documents this parameter as MVA but internally compares against per-unit mismatches (known bug \#2750, unfixed in v3.4.0)." -> — A-2 result - -**Cross-tool relevance:** likely — other tools with embedded solvers (e.g., matpower with -PYPOWER heritage) may also not expose residual scalars publicly. -**Proposed action:** add_verification — the protocol should explicitly specify that when a -tool does not expose a residual scalar, indirect evidence (voltage profile divergence + NR -iteration count from any accessible interface) is sufficient, with a required documentation -note. The tolerance_mva unit ambiguity should be flagged as a qualification. - ---- - -### pandapower-F03: DC OPF branch shadow prices extracted from internal _ppc — scored as no-workaround - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-3 - -A-3 is a clean pass (workaround_class: null). The LMP extraction via `res_bus.lam_p` is -correctly no-workaround — it is a public, documented API. However, the branch shadow -prices required to demonstrate "at least 2 branches with non-zero shadow prices" are -extracted from: - -> "Branch shadow prices: Extracted from net.\_ppc\['branch'\]\[:, 13:15\] (MU\_SF, MU\_ST columns)." -> — A-3 result - -The result notes this requires "accessing the internal net._ppc structure" but still -scores the test as no-workaround on the grounds that the pass condition "only requires -LMP/shadow price extractability, which is satisfied by the public API." This interpretation -is technically defensible (LMPs are publicly accessible), but the branch shadow prices — -which are also part of the pass condition — require internal API access. The synthesis -separately documents this as an arch-quality concern (OPF duals discarded during result -extraction), but that finding doesn't feed back into the A-3 score. - -**Cross-tool relevance:** none — specific to the PYPOWER-backed dual extraction pattern. -**Proposed action:** adjust_scoring — the A-3 pass condition references both LMPs and -branch shadow prices. If branch shadow prices require internal API access, the test should -be scored as qualified_pass/stable rather than clean pass, or the pass condition should be -amended to specify which dual values must be accessible via public API. - ---- - -### pandapower-F04: Distributed slack tests on 39-bus TINY — A-11 pass condition untestable when feature is absent - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-11, B-8 - -A-11 correctly fails because distributed slack OPF is not supported. However, the test's -pass condition requires LMPs to "differ from single-slack results in a physically consistent -manner" — which is impossible to verify when the feature is absent. The test therefore -measures only API transparency (does the tool surface an error or silently absorb the -parameter?), not the LMP behavior the rubric intends. - -pandapower's case is particularly instructive because rundcopp() silently absorbs -`distributed_slack=True` with zero effect: - -> "LMPs are identical across all 39 buses. The distributed_slack=True kwarg is silently accepted by rundcopp() via **kwargs but has zero effect." -> — A-11 result - -For tools that do support distributed slack, the TINY network with a single ext_grid -element is a minimal test case. B-8 (reference bus configuration) tests three slack -configurations on the same network, finding a maximum LMP shift of 8.58 $/MWh — a -meaningful difference, but with only 10 generators and a radial topology, it is unclear -whether the LMP shifts are "physically consistent" in the sense the rubric intends or -simply reflect the arithmetic of rerouting reference. - -**Cross-tool relevance:** likely — any tool without distributed slack OPF will have the -same untestable pass condition. -**Proposed action:** add_test — A-11 should have a split pass condition: (a) for tools with -distributed slack, the existing LMP comparison test; (b) for tools without it, a scored -assessment of API transparency (does the tool explicitly indicate the feature is absent vs. -silently fail?). - ---- - -### pandapower-F05: A-3 binding branch count anomalous — all 46 branches reported as shadow-price-nonzero - -**Category:** missing_verification | **Severity:** high -**Tests:** A-3, A-9 - -A-3 reports every branch has a nonzero shadow price under 70% derating: - -> "Binding branches: 46 of 46 | Lines > 95% loading: 7" -> — A-3 result - -This is internally inconsistent. Only 7 branches are above 95% loading, but 46 of 46 -are reported as having nonzero shadow prices. In a correctly solved DC OPF, a shadow price -is nonzero only when a constraint is binding (flow = limit). Having all 46 binding while -only 7 are near their limits suggests either (a) the threshold for "nonzero" was not -applied (interior-point solvers produce numerically small but nonzero duals even for -inactive constraints), or (b) the result is computed from column indices that include both -MU_SF and MU_ST and any nonzero in either direction counts. - -No threshold was documented: - -> "All 46 branches have non-zero shadow prices, far exceeding the 2-branch minimum threshold." -> — A-3 result - -The test passes (minimum threshold is 2), so this discrepancy was not investigated further. -This is a moderate concern for cross-tool comparability: if other tools apply a threshold -(e.g., |dual| > 1e-4) while pandapower counts all numerically nonzero duals, the binding -branch counts will not be comparable. - -**Cross-tool relevance:** likely — interior-point solver dual extraction without -magnitude thresholds is a common issue. -**Proposed action:** add_verification - -**Probe recommended:** Yes -**Probe type:** convergence_check -**Probe description:** Re-run A-3, extract branch shadow prices, apply an explicit -threshold (|dual| > 1e-4 $/MWh), and compare count against branches with loading_percent -> 99%. Expected: ~7-15 branches binding, consistent with the 7 branches >95% loading. -If the threshold-filtered count drops below 2, the test result should be revised to -qualified_pass. - ---- - -### pandapower-F06: G-FNM-3 failure classification as data_ingestion_error — single ingestion path tested - -**Category:** misleading_result | **Severity:** medium -**Tests:** G-FNM-3 - -G-FNM-3 fails due to a localized cluster of ~101 buses with 14-21 degree angle deviations -and a maximum branch flow deviation of 596.6%. The result classifies this as -`data_ingestion_error` because the failing buses are not adjacent to non-unity-tap -transformers (0% transformer adjacency), ruling out a formulation difference: - -> "Classification: data_ingestion_error (not formulation_difference)" -> — G-FNM-3 result - -This classification is plausible but not verified. Only one ingestion path was tested -(MATPOWER PPC fallback). The classification confidence is limited — if the localized -anomaly persists across any alternative ingestion of the same network data, it would be -a solver characteristic rather than a path artifact. For pandapower, there is no -alternative path available (no native CSV import), so the single-path limitation is -inherent to the tool. - -The practical consequence is that the hard-fail penalizes pandapower for an ingestion-path -limitation rather than a solver deficiency. The aggregate metrics are strong (99.64% buses -pass, 99.67% branches pass), and the failure is concentrated in a physically unusual -sub-region (zero-load zero-gen radial cluster with extreme angle swings). - -**Cross-tool relevance:** none — specific to pandapower's MATPOWER-only ingestion path. -**Proposed action:** add_verification — tag FNM test results with path_type -(native_csv | matpower_fallback) so that data-ingestion-path artifacts can be distinguished -from tool capabilities in cross-tool synthesis. - ---- - -### pandapower-F07: Six expressiveness failures are scope-boundary audits, not capability probes - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-5, A-6, A-9, A-10, A-11, A-12 - -Five of the six expressiveness failures follow the same pattern: the test confirms by -API inspection that a feature (SCUC, SCOPF, lossy OPF, distributed slack OPF, multi-period -OPF) is architecturally absent from pandapower's design scope, documents the absence, and -records a blocking fail. The timing entries confirm no solver work was done: - -> "Wall-clock: 0.79 s (import and capability check only — no solve attempted)" — A-5 result -> -> "Wall-clock: 0.78 s (includes lossless baseline OPF solve)" — A-10 result - -These tests are accurate and correctly scored — pandapower genuinely cannot express these -formulations. The issue is at the protocol level: 60% of expressiveness tests for pandapower -measure what the tool is not, while 40% measure what it does. For a focused tool like -pandapower, this makes the expressiveness dimension score primarily a scope-boundary -assessment rather than a quality assessment of the expressiveness within scope. - -The 4 tests pandapower passes (A-1, A-2, A-3, A-4) all pass cleanly with strong results. -The protocol does not distinguish between "tool is in scope and excellent" and "tool is -out of scope for 60% of tests." A broader tool that implements SCUC poorly would score -higher on expressiveness than pandapower, even if pandapower's AC/DC power flow is -technically superior. - -**Cross-tool relevance:** confirmed — any tool with a specialized scope (e.g., matpower) -faces the same protocol imbalance. -**Proposed action:** add_test — add a secondary scoring axis for "expressiveness within -declared scope" to complement the absolute pass/fail count. - ---- - -### pandapower-F08: A-3 DC OPF convergence accepted from boolean flag — no iteration count or residual - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** A-3 - -A-3 reports DC OPF convergence in 0.10 s with objective $156,929, accepted via: - -> "Solver iterations: not reported by PYPOWER DC OPF solver" -> — A-3 result - -The PYPOWER interior-point solver does not expose iteration counts for the DC OPF -formulation. Convergence is verified only from the boolean `net.converged = True`. The -known case9 silent failure (D-3: "case9 DC OPF fails silently") demonstrates that this -boolean is not always reliable. However, the A-3 result includes the full dispatch table -and LMP values, which provide meaningful indirect verification: the LMP spread of -$76.05/MWh across a 39-bus network with differentiated costs and 70% branch derating is -physically plausible. - -The dispatch table shows ext_grid (slack) dispatching at 1,342 MW — above its nominal -Pmax of 1,040 MW. This is noted in the table without comment. The ext_grid (external grid) -in pandapower is unconstrained by default and acts as a slack bus, so dispatch above Pmax -is the expected behavior when Pmax is a "nominal" rather than a hard limit — but this -interpretation is not verified in the result. - -**Cross-tool relevance:** likely — PYPOWER-heritage solvers across multiple tools may -share this convergence-verification limitation. -**Proposed action:** add_verification - -**Probe recommended:** Yes -**Probe type:** convergence_check -**Probe description:** For the A-3 result, verify dual feasibility: at each generator bus, -check that LMP approximately equals the generator's marginal cost when unconstrained, and -equals marginal cost ± branch shadow price when at a flow limit. This is a necessary -condition for a true DC OPF optimum and can be checked with 5-10 lines of arithmetic on -the existing result data. - ---- - -### pandapower-F09: Gate tests are near-universal passes — minimal discriminative value - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -All three gate tests pass for pandapower with the standard MATPOWER .m file loader. The -tests verify bus/branch/gen counts match expected values and that no NaN or infinity values -are present. These are format-compatibility checks, not capability assessments: - -> "Result: PASS — Actual counts: 39 buses / 46 branches / 10 generators. Load time: 0.085s." -> — G-1 result - -For any tool that supports MATPOWER .m file input (all six evaluated tools do), these tests -are expected to pass. A failure would indicate a fundamental format-parsing bug, not a -meaningful capability gap. The load time (0.085 s for TINY) is measured but not used as a -comparative metric. - -**Cross-tool relevance:** confirmed — the gate tests provide essentially no discriminative -information across all evaluated tools. -**Proposed action:** redesign_test — gate tests should verify minimum functional capability -(e.g., DCPF convergence + structured result output) rather than element count matching. - ---- - -### pandapower-F10: B-9 uses internal \_ppc (clean pass) while B-1 uses \_ppc interception (fragile) — consistency gap - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** B-1, B-9 - -B-9 accesses `net._ppc` to get PYPOWER bus/branch arrays for PTDF computation and is -scored as a clean pass with no workaround: - -> "Access to these arrays requires the internal net._ppc attribute (set after solving DCPF), but this is the standard pandapower pattern for accessing PYPOWER-level data." -> — B-9 result - -B-1 accesses the same `_ppc` structure (plus the PYPOWER result dict) but requires -intercepting the solve pipeline and is scored as qualified_pass/fragile: - -> "Durability: fragile — The workaround depends on the internal structure of _optimal_powerflow (undocumented private function), the PYPOWER opf_model.add_constraints interface, and the internal result dict structure." -> — B-1 result - -The distinction is real (read-only post-solve access vs. mid-solve pipeline intercept), but -it is not made explicit in either result file. The scoring rationale should document where -the line falls between "standard pandapower internal access pattern" and "fragile -workaround." Without this documentation, the same `_ppc` access in a future test could be -scored inconsistently. - -**Cross-tool relevance:** likely — any tool with a layered internal/external API distinction -will face this classification question. -**Proposed action:** adjust_scoring — explicitly define in the protocol or per-tool -evaluation notes what constitutes acceptable internal API access (post-solve read-only) -vs. fragile workaround (pipeline intercept or pre-result capture). - ---- - -### pandapower-F11: B-3 100% DCPF convergence rate conflates solver success with topological completeness - -**Category:** missing_verification | **Severity:** low -**Tests:** B-3 - -B-3 reports 3,276 contingency cases with 100% convergence. For DCPF (a direct linear -solve), "convergence" means the linear system was solved without a disconnection error. -With check_connectivity=True, disconnected cases raise an exception rather than producing -a solution. Of the 3,276 cases, 924 (28.2%) had load loss — meaning the network -disconnected in those cases, and the "solution" is for the remaining connected components -only. - -> "Converged cases: 3,276 (100%) | Cases with load loss: 924 (28.2%)" -> — B-3 result - -The 100% convergence figure is not incorrect, but it encompasses cases where the network -was fully or partially islanded. For ACPF-style solvers where convergence means iterative -NR convergence on the full connected network, the comparison is not direct. - -**Cross-tool relevance:** confirmed — DCPF contingency sweep convergence rates are -structurally different from ACPF convergence rates and should not be compared across -tools without this qualification. -**Proposed action:** add_verification — protocol should distinguish DCPF "solve completed" -from ACPF "iterative convergence" in contingency sweep convergence reporting. - ---- - -### pandapower-F12: A-11 FAIL category does not distinguish feature-absent from silent-mismatch - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-11 - -A-11 fails because `rundcopp()` silently absorbs `distributed_slack=True` and produces -identical LMPs to the single-slack baseline. This is categorically different from a tool -that raises `NotImplementedError` (unambiguous absence) or prints a warning: - -> "rundcopp() accepts arbitrary **kwargs and silently ignores distributed_slack=True without raising an error or warning. This could mislead users into thinking the feature is active." -> — A-11 result - -The current protocol records a FAIL in both cases. However, the silent-mismatch failure -mode has accessibility implications beyond expressiveness: a user who passes -`distributed_slack=True` will believe the feature is active and may publish incorrect -results. This is arguably a higher-severity failure than "not implemented." The A-11 result -documents this accurately but the scoring does not reflect the severity difference. - -**Cross-tool relevance:** likely — **kwargs-style parameter absorption without validation -is a common Python API design pattern that could affect other evaluated tools. -**Proposed action:** adjust_scoring — add a secondary observation category -"silent-mismatch" or flag in the FAIL record when a tool accepts a parameter with zero -effect and no warning. This should feed into the accessibility/error-quality dimension -scoring. - ---- - -### pandapower-F13: FNM tests measure MATPOWER fallback path quality, not pandapower ingestion capability - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** G-FNM-1, G-FNM-3, G-FNM-4 - -All FNM Suite G tests for pandapower use the MATPOWER PPC fallback path because -pandapower has no native intermediate CSV import capability. The FNM results therefore -reflect a two-stage process: (1) the MATPOWER fallback converts intermediate format to -PPC, and (2) pandapower ingests the PPC. Issues in either stage are attributed to -pandapower, even when they may be in the conversion step. - -G-FNM-3's localized failure (101 buses with 14-21 degree deviations) is attributed to -"the MATPOWER PPC import path's handling of specific impedance details." G-FNM-4's ACPF -infeasibility is partly attributed to "PPC import path loses AC-critical transformer data." -Both attributions may be correct, but they cannot be confirmed without an alternative -ingestion path. - -> "pandapower has no native CSV import capability. The MATPOWER PPC format is the standard programmatic entry point." -> — G-FNM-3 result - -For cross-tool comparison, this means pandapower's FNM results are not directly comparable -to tools that use native CSV import — they are measuring different ingestion stacks. - -**Cross-tool relevance:** none — specific to pandapower's lack of native FNM CSV import. -**Proposed action:** add_test — tag FNM results with `input_path` (native_csv | -matpower_fallback) and adjust cross-tool FNM comparisons to account for path differences. - ---- - -## Extraordinary Claims - -### A-3: All 46 branches have nonzero shadow prices with only 7 above 95% loading - -**Concern:** Interior-point solvers produce numerically small but nonzero duals on -inactive constraints. The result reports 46/46 binding without specifying a magnitude -threshold, while the loading data shows only 7 branches above 95% utilization. These -are inconsistent: if shadow prices reflect binding constraints, at most ~7-15 branches -should have meaningful (threshold-filtered) duals. - -**Evidence quality:** moderate — the loading_percent vs shadow-price count discrepancy -is documented in the result but not investigated. - -This finding should be verified with a threshold-filtered dual extraction. If the count -drops below 2 with a reasonable threshold (|dual| > 1e-4 $/MWh), the A-3 pass condition -may not be satisfied at the level the rubric intends. - ---- - -### A-3: DC OPF convergence verified by boolean flag only — no iteration count or residual - -**Concern:** The PYPOWER DC OPF solver does not report iteration count or residual. -Convergence is accepted from `net.converged = True`. The known case9 silent failure (D-3) -demonstrates this flag is not always reliable. The dispatch table provides meaningful -indirect verification (plausible LMPs and objective), but dual-feasibility is not checked. - -**Evidence quality:** moderate — the convergence evidence is indirect (dispatch table -plausibility) rather than direct (residual or optimality conditions). - -The probe should check dual feasibility: at unconstrained generator buses, LMP should -equal the generator's marginal cost. This is checkable from the existing A-3 results -without rerunning anything. - ---- - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | — | Low signal — format compatibility check only | -| G-2 | pass | — | Low signal — format compatibility check only | -| G-3 | pass | — | Low signal — format compatibility check only | -| A-1 | pass | — | Clean; timing measured | -| A-2 | pass | — | Convergence residual not extractable (null in frontmatter) | -| A-3 | pass | — | Branch shadow prices via internal _ppc; 46/46 binding anomaly | -| A-4 | pass | — | Clean | -| A-5 | fail | blocking | Correct scope-boundary assessment; no solve attempted | -| A-6 | fail | blocking | Cascaded from A-5; also independently infeasible | -| A-9 | fail | blocking | Correct scope-boundary assessment | -| A-10 | fail | blocking | Correct scope-boundary assessment | -| A-11 | fail | blocking | Silent **kwargs absorption — not just feature absent | -| A-12 | fail | blocking | PandaModels.jl bridge noted but requires Julia runtime | -| B-1 | qualified_pass | fragile | Monkey-patch of _optimal_powerflow; fragile classification well-justified | -| B-2 | pass | — | Clean; exemplary NetworkX bridge | -| B-3 | pass | — | 100% "convergence" conflates DCPF solve success with ACPF-style convergence | -| B-4 | pass | — | 240 solves; scenario variation spread is modest (3.6%) | -| B-5 | pass | — | Clean | -| B-6 | pass | — | Clean | -| B-8 | qualified_pass | stable | Verbose but public-API-only approach; classification appropriate | -| B-9 | pass | — | Uses internal _ppc but scored no-workaround; scoring inconsistency vs B-1 | -| C-1 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-2 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-3 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-4 | fail | blocking | Cascaded from A-5; correct | -| C-5 (SMALL) | pass | — | Clean 2K-bus ACPF; good scale evidence within scope | -| C-5 (MEDIUM) | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-7 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-8 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-9 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| C-10 | skip | C-SMALL-gate | Feature-absence cascade, not scale failure | -| D-1 | informational | — | Correct use of informational status | -| D-2 | informational | — | 3/10 tests completable from docs; 7 require source inspection | -| D-3 | informational | — | 15/16 examples pass; case9 DC OPF silent failure noted | -| D-4 | informational | — | OPF diagnostics poor; silent **kwargs documented here and A-11 | -| D-5 | informational | — | LOC compact for supported features | -| E-1 | informational | — | 19 releases / 24 months; strong | -| E-2 | informational | — | 838 commits / 30 committers; activity pattern consistent with major release cycle | -| E-3 | informational | — | Bus factor 3; reviewer pool concentration noted | -| E-4 | informational | — | Dual Fraunhofer/Uni Kassel anchor; strong | -| E-5 | informational | — | 100% acknowledgment; 29d median; batch-triage pattern | -| E-6 | informational | — | 72% coverage; 28 CI jobs | -| E-7 | informational | — | Hessen DSO study is genuine production use; Grid2Op caveat appropriate | -| F-1 | informational | — | BSD license; clean | -| F-2 | informational | — | 37 packages; depth 3 | -| F-3 | informational | — | All permissive; 1 optional MPL-2.0 | -| F-4 | informational | — | All source-available | -| F-5 | informational | — | Full Python trace to spsolve | -| F-6 | informational | — | Sigstore provenance; strong | -| F-7 | informational | — | Fully air-gap installable | -| F-8 | informational | — | Self-contained PYPOWER solver | -| F-9 | informational | — | Unversioned install in docs; tutorials not in package | -| G-FNM-1 | pass | stable | MATPOWER fallback + zero RATE_A fix; infrastructure_friction applies | -| G-FNM-2 | pass | — | 100% DCPF-critical fields; 55.8% ACPF-critical | -| G-FNM-3 | fail | stable | 99.6% aggregate pass but 596.6% max deviation triggers hard-fail | -| G-FNM-4 | informational | stable | Infeasible at all relaxation levels; DCPF warm-start angles 536.9 deg max | -| G-FNM-5 | informational | — | 34% native; 43% external; lowest native coverage among evaluated tools | -| P2-1 | informational | — | PSS/E RAW not supported | -| P2-2 | informational | — | Piecewise-linear costs natively supported | -| P2-3 | informational | — | Commitment injection via in_service; low friction | diff --git a/sweep-data/v10-to-v11/per-tool/pandapower/findings.yaml b/sweep-data/v10-to-v11/per-tool/pandapower/findings.yaml deleted file mode 100644 index 57888f11..00000000 --- a/sweep-data/v10-to-v11/per-tool/pandapower/findings.yaml +++ /dev/null @@ -1,455 +0,0 @@ -tool: pandapower -source_version: "v10" -timestamp: "2026-03-14T00:00:00Z" - -evaluation_summary: - total_tests: 60 - pass: 24 - fail: 12 - qualified_pass: 2 - informational: 14 - skip: 8 - # Note: skip count (8) reflects cascaded C-SMALL-gate blockage from A-5, not - # independent scale failures. C-5 has separate SMALL (pass) and MEDIUM (skip) files. - -findings: - - id: pandapower-F01 - category: test_design_gap - severity: high - test_ids: [C-4, C-1, C-2, C-3, C-5, C-7, C-8, C-9, C-10] - title: C-SMALL gate design conflates feature absence with scale failure, muting 8 MEDIUM tests - description: > - The protocol requires C-4 (SCUC on SMALL) to pass before unlocking MEDIUM-tier - scalability tests. For pandapower, C-4 fails because SCUC is absent from the - tool's scope, not because of any scale limitation. This design choice causes 8 - MEDIUM tests — covering DCPF, ACPF, DC OPF, solver swap, SCOPF, PTDF, and - distributed-slack OPF at scale — to be reported as skipped rather than executed. - The tool demonstrably handles DCPF at 28K buses in 0.4 s (G-FNM-3) and ACPF at - 2K buses in 1.3 s (C-5 SMALL), yet these observations cannot be compared across - tools at the MEDIUM tier because the gate design blocks execution. The net effect - is that the scalability grade for a tool with a narrow feature scope is always - under-evidenced, regardless of its actual scale performance within its scope. - evidence: - - file: evaluations/pandapower/results/scalability/C-4_scuc_small.md - excerpt: "Skipped because Suite C SMALL gate failed (C-4 failed with blocked_by: A-5). The SMALL gate failure was a cascaded failure from A-5 (SCUC unsupported), not a scale-related failure." - - file: evaluations/pandapower/results/scalability/C-1_dcpf_medium.md - excerpt: "Result: SKIP — Skipped because Suite C SMALL gate failed (C-4 failed with blocked_by: A-5)." - - file: evaluations/pandapower/results/synthesis.md - excerpt: "8 MEDIUM-tier tests skipped due to C-SMALL-gate; no actual scale failures observed; FNM DCPF solved 28K buses in 0.4s" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - # Redesign: C-SMALL gate should only block MEDIUM tests that *depend* on the - # failed feature (e.g., C-4 fail blocks C-SCUC-MEDIUM). Feature-agnostic MEDIUM - # tests (C-1 DCPF, C-2 ACPF, C-3 DC OPF) should be independently runnable. - - - id: pandapower-F02 - category: missing_verification - severity: medium - test_ids: [A-2, C-5] - title: ACPF convergence residual not verifiable — protocol accepts private-attribute iteration count as verification - description: > - The A-2 pass condition requires the convergence residual to be "reported and - below the tool's stated tolerance." pandapower does not expose the final NR - residual as a scalar from the public API; the exact mismatch value is not stored - in any result attribute. The test records "convergence_residual: null" in the - frontmatter and notes "below 1e-8 (tolerance_mva setting; exact value not - extractable)." Iteration count (4) is accessible only via the private - net._ppc["iterations"] attribute. The pass is accepted on the basis of 100% - of buses diverging from the flat-start 1.0 pu, which demonstrates convergence - indirectly but does not satisfy the letter of the pass condition. The tolerance_mva - parameter itself has an unfixed unit-interpretation bug (issue #2750) that is noted - but not flagged as a qualification. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-2_acpf.md - excerpt: "Convergence residual: below 1e-8 (tolerance_mva setting; exact value not extractable)" - - file: evaluations/pandapower/results/expressiveness/A-2_acpf.md - excerpt: "Note on tolerance_mva: pandapower documents this parameter as MVA but internally compares against per-unit mismatches (known bug #2750, unfixed in v3.4.0)." - - file: evaluations/pandapower/results/observations/convergence-quality-expressiveness-A-2_acpf.md - excerpt: "pandapower's Newton-Raphson solver reports iteration count via net._ppc['iterations'] (private attribute) but does not expose the final convergence residual as a scalar." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - # Proposed: Protocol should accept indirect convergence evidence (voltage profile - # divergence from flat start + iteration count) as sufficient when the tool does - # not expose a residual scalar, documenting this as a diagnostic quality limitation - # rather than treating residual-absence as a pass-condition gap. - - - id: pandapower-F03 - category: misleading_result - severity: medium - test_ids: [A-3] - title: DC OPF branch shadow prices extracted from internal _ppc structure, but result counted as "no workaround" - description: > - A-3 is scored as a clean pass with workaround_class: null. The pass condition - requires shadow prices (dual values on flow limits) to be "extractable." The - result shows LMPs are available via the public res_bus.lam_p column, which is - correctly labeled as no-workaround. However, branch shadow prices (MU_SF/MU_ST) - are extracted from net._ppc["branch"][:, 13:15] — a private internal PYPOWER - array requiring knowledge of PYPOWER column indices. The result text notes "Branch - shadow prices require accessing the internal net._ppc structure" but still counts - the test as no-workaround. The synthesis reiterates "OPF duals discarded during - result extraction" as an arch-quality finding. This is accurately documented but - the scoring does not reflect the friction involved in extracting branch duals. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-3_dcopf.md - excerpt: "Branch shadow prices: Extracted from net._ppc['branch'][:, 13:15] (MU_SF, MU_ST columns). With 70% derating, all 46 branches have non-zero shadow prices." - - file: evaluations/pandapower/results/expressiveness/A-3_dcopf.md - excerpt: "Branch shadow prices require accessing the internal net._ppc structure, but the test's pass condition only requires LMP/shadow price extractability, which is satisfied by the public API." - - file: evaluations/pandapower/results/synthesis.md - excerpt: "OPF duals discarded during result extraction: The 6-layer architecture is clean, but constraint multipliers and shadow prices from the PYPOWER result dict are dropped during DataFrame conversion (B-6, severity: medium)" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - # Proposed: Pass condition for A-3 should distinguish between LMP extraction - # (public API) and branch dual extraction (internal API), scoring the branch - # dual extraction as a mild workaround rather than no-workaround. - - - id: pandapower-F04 - category: network_insufficiency - severity: medium - test_ids: [A-11, B-8] - title: Distributed slack tests on 39-bus TINY network — single slack bus and no physical evidence of LMP sensitivity - description: > - A-11 (distributed slack OPF) and B-8 (reference bus configuration) both run on - the 39-bus TINY network. A-11 fails because pandapower does not support distributed - slack OPF — a correct and well-supported finding. However, B-8's comparative LMP - analysis would be poorly designed even for a capable tool: with only 1 ext_grid - (slack) in the base model, the "three slack configurations" test manually - reconstructs the network with different element types. For distributed slack - specifically, the TINY network has no quantitative evidence that LMPs differ in - a "physically consistent manner" because the feature is absent; the protocol's - pass condition for A-11 requires LMP comparison against single-slack results — - impossible to verify when the feature is missing. The network size itself is not - the core issue, but a network with only one existing slack element limits the - ability to stress-test slack configuration behavior for tools that do support it. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "LMPs are identical across all 39 buses. The distributed_slack=True kwarg is silently accepted by rundcopp() via **kwargs but has zero effect on the PYPOWER optimization." - - file: evaluations/pandapower/results/extensibility/B-8_reference_bus_config.md - excerpt: "Maximum LMP change: 8.58 $/MWh (between configs 1 and 2/3)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_test - # Proposed: A-11 pass condition should clarify that for tools not supporting - # distributed slack OPF, the test measures API transparency (does the tool - # surface a clear error or silently absorb the parameter?) rather than LMP - # comparison. A "silent kwarg absorption" outcome should have an explicit scoring - # category distinct from "feature not implemented." - - - id: pandapower-F05 - category: missing_verification - severity: high - test_ids: [A-3, A-9] - title: A-3 binding branch count inflated — all 46 branches reported as shadow-price-nonzero under 70% derating - description: > - A-3's pass condition requires "at least 2 branches with non-zero shadow prices - (binding flow constraints)" with 70% derating. The pandapower result reports - "Binding branches: 46 of 46" — every branch has a nonzero shadow price. This - is anomalous: in a well-posed DC OPF on a 39-bus system with differentiated - costs, far fewer than all 46 branches would typically be at their flow limit - simultaneously. A possible explanation is that the PYPOWER interior-point solver - returns small nonzero duals on inactive constraints (numerical noise) rather - than true structural binding. The result does not distinguish between genuinely - binding constraints and near-zero duals, and there is no reported threshold for - what qualifies as a "non-zero shadow price." The test passes because the minimum - threshold (2 binding) is easily met, but the 46-of-46 result is suspicious and - was not investigated further. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-3_dcopf.md - excerpt: "Binding branches: 46 of 46 | Lines > 95% loading: 7" - - file: evaluations/pandapower/results/expressiveness/A-3_dcopf.md - excerpt: "Branch shadow prices: Extracted from net._ppc['branch'][:, 13:15] (MU_SF, MU_ST columns). With 70% derating, all 46 branches have non-zero shadow prices, far exceeding the 2-branch minimum threshold." - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - # Probe: Re-run A-3 DC OPF, extract branch shadow prices with an explicit - # threshold (e.g., |dual| > 1e-4 $/MWh), compare against the loading_percent - # column for the same branches, and verify that shadow-price-nonzero branches - # are actually at or near (>99%) their thermal limit. - - - id: pandapower-F06 - category: misleading_result - severity: medium - test_ids: [G-FNM-3] - title: G-FNM-3 classified as data_ingestion_error but no alternative ingestion path was tested - description: > - G-FNM-3 fails due to localized angle deviations of 14-21 degrees in ~101 buses, - triggering the hard-fail (596.6% max branch flow deviation). The result classifies - this as "data_ingestion_error" because 0% of the failing buses are adjacent to - transformers with non-unity tap ratios, ruling out formulation difference. However, - the classification is limited: only one ingestion path was tested (MATPOWER PPC - fallback via matpowercaseframes + from_ppc). No attempt was made to ingest the - FNM via intermediate CSV or another route that might avoid the localized anomaly. - The claim that this is an ingestion-path artifact rather than a pandapower solver - characteristic is plausible but not verified by testing an alternative path. The - synthesis acknowledges this as a workaround path but does not flag the - classification confidence limitation. - evidence: - - file: evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md - excerpt: "Deviation pattern: systematic bias (~14-21 degrees) in a connected cluster, not scattered across the network. Classification: data_ingestion_error (not formulation_difference)" - - file: evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md - excerpt: "input_path: matpower — Used pre-cleaned fnm_main_island.m via matpowercaseframes.CaseFrames + from_ppc (same ingestion path as G-FNM-1, using MATPOWER fallback since pandapower has no native CSV import)." - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: add_verification - # Proposed: If intermediate CSV import were attempted (even with manual mapping), - # it would allow distinguishing whether the localized cluster deviation is - # path-dependent. This is a protocol limitation: the FNM fallback path is the - # only viable option for pandapower, so the single-path limitation is inherent. - - - id: pandapower-F07 - category: test_design_gap - severity: medium - test_ids: [A-5, A-6, A-9, A-10, A-11, A-12] - title: Six expressiveness tests are structural scope assessments, not capability probes — all confirm tool-design-boundary in same way - description: > - Five of the six expressiveness failures (A-5 SCUC, A-9 SCOPF, A-10 Lossy OPF, - A-11 Distributed Slack OPF, A-12 Multi-Period Storage OPF) follow the same - pattern: the test checks for a feature that is architecturally outside pandapower's - scope as a steady-state network analysis tool, confirms its absence by API - inspection, and correctly records a blocking fail. While each failure is accurate - and correctly scored, the tests function as scope boundary audits rather than - capability probes. The timing entries for these tests (0.75–2.07 s) are - "import and capability check only" — not solves — and the LOC (73–338) reflects - the length of the audit code rather than any implemented capability. The cumulative - effect is that 60% of expressiveness tests are measuring what pandapower is not, - rather than how well it does what it claims to do. The remaining 4 tests (A-1, - A-2, A-3, A-4) that pandapower passes are all within its declared scope and - pass cleanly. This is a protocol-design issue, not an evaluator error. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-5_scuc.md - excerpt: "Wall-clock: 0.79 s (import and capability check only — no solve attempted)" - - file: evaluations/pandapower/results/expressiveness/A-9_scopf.md - excerpt: "Wall-clock: 0.83 s (includes network loading, base-case DC OPF solve, and contingency analysis attempt)" - - file: evaluations/pandapower/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "Result: FAIL — rundcopp() accepts arbitrary **kwargs and silently ignores distributed_slack=True without raising an error or warning." - - file: evaluations/pandapower/results/synthesis.md - excerpt: "5 independent blocking failures (SCUC, SCOPF, lossy OPF, distributed slack OPF, multi-period OPF)" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_test - # Proposed: Add a sub-dimension or secondary scoring axis for "expressiveness within - # declared scope" to distinguish scope-boundary tools (pandapower, matpower) from - # scope-broader tools (pypsa, gridcal). The current protocol scores all failures - # equally regardless of whether the tool claims the capability. - - - id: pandapower-F08 - category: extraordinary_claim - severity: medium - test_ids: [A-3] - title: DC OPF objective ($156,929) appears plausible but OPF solver iterations not reported — convergence verification relies only on boolean flag - description: > - A-3 reports DC OPF convergence in 0.10 s solve time with objective $156,929. - The PYPOWER interior-point solver does not report iteration count for the DC OPF - case (convergence_iterations: null in frontmatter). The pass is accepted on the - basis of "OPF converged = True." The known case9 DC OPF silent failure mentioned - in D-3 demonstrates that the PYPOWER solver can return converged=True for - degenerate cases. The objective value and LMP spread ($76.05/MWh) are physically - plausible for the 39-bus case with 70% branch derating, and the full dispatch - table is reported, providing indirect verification. However, no residual or - iteration count confirms true optimality. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-3_dcopf.md - excerpt: "Solver iterations: not reported by PYPOWER DC OPF solver | OPF converged: True" - - file: evaluations/pandapower/results/accessibility/D-3_example_verification.md - excerpt: "15/16 examples pass; case9 DC OPF fails silently" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - # Probe: Verify A-3 result by cross-checking the dual feasibility: for a binding - # DC OPF, the LMPs should satisfy the stationarity condition (LMP at generator bus - # equals its marginal cost when at its limit, or the shadow-price-adjusted marginal - # cost when not at its limit). Check 2-3 generator buses to confirm LMP == marginal - # cost relationship rather than just accepting the converged boolean. - - - id: pandapower-F09 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: Gate tests are effectively universal passes across all tools — low discriminative value - description: > - All three gate tests (G-1 TINY, G-2 SMALL, G-3 MEDIUM) pass for pandapower with - standard MATPOWER .m file ingestion. The gate tests only check bus/branch/gen - counts and absence of NaN values. They do not test any formulation capability - or data fidelity. For all tools evaluated, loading a standard MATPOWER case is - expected to succeed; a gate failure would represent a fundamental tool bug or - wrong file format rather than a meaningful discriminator. The load times (0.085 s - for TINY) are not even compared across tools as a performance metric. - evidence: - - file: evaluations/pandapower/results/gate/G-1_ingest_tiny.md - excerpt: "Result: PASS — Actual counts: 39 buses / 46 branches / 10 generators. Load time: 0.085s. Errors/warnings: None." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - # Proposed: Gate tests should verify a minimum functional capability (e.g., DCPF - # convergence + result structure) rather than just element count matching, to provide - # some discriminative signal across tools. - - - id: pandapower-F10 - category: scoring_inconsistency - severity: medium - test_ids: [B-1, B-9] - title: B-9 PTDF extraction via internal net._ppc scores as clean pass while B-1 accessing _ppc for duals scores as fragile qualified_pass - description: > - B-9 (PTDF extraction) is scored as a clean pass (workaround_class: null) even - though it requires accessing net._ppc["bus"] and net._ppc["branch"] to get the - PYPOWER bus/branch arrays. The result notes "Access to these arrays requires the - internal net._ppc attribute (set after solving DCPF), but this is the standard - pandapower pattern for accessing PYPOWER-level data." In contrast, B-1 (custom - constraints) is scored as qualified_pass/fragile partly because it requires - capturing data from the PYPOWER result dict before pandapower discards it. - Both tests access the same _ppc internal structure, but B-9 is treated as - no-workaround while B-1 is fragile. The distinction is that B-9 only reads from - _ppc (post-solve, stable) while B-1 intercepts _ppc during solve (requiring - monkey-patching), but this distinction is not made explicit in the scoring rationale. - evidence: - - file: evaluations/pandapower/results/extensibility/B-9_ptdf_extraction.md - excerpt: "Workarounds: None required. Access to these arrays requires the internal net._ppc attribute (set after solving DCPF), but this is the standard pandapower pattern for accessing PYPOWER-level data." - - file: evaluations/pandapower/results/extensibility/B-1_custom_constraints.md - excerpt: "Workaround: Replicated pandapower's internal _optimal_powerflow function to inject a PYPOWER userfcn callback and capture the PYPOWER result dict. Durability: fragile." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - # Proposed: Explicitly define the line between "standard pandapower pattern for - # internal access" (acceptable, no-workaround) and "fragile internal intercept" - # (workaround-fragile). Reading post-solve _ppc arrays should be documented as - # a stable pattern; intercepting the solve pipeline should remain fragile. - - - id: pandapower-F11 - category: missing_verification - severity: low - test_ids: [B-3] - title: B-3 contingency sweep counts every DCPF call as converged but connectivity may mask topology errors - description: > - B-3 reports "3,276 contingency cases, 100% converged." However, pandapower's - DCPF is a direct linear solve that always "converges" (the system Ax=b either - has a solution or the network is disconnected). The convergence rate is checked - via check_connectivity=True in the rundcpp call, which raises an error for - disconnected networks. The 100% figure therefore reflects "no disconnection - errors" rather than "physically meaningful convergence." For cases with load - loss (924 of 3,276), the DCPF may produce solution values for the connected - components while ignoring the disconnected load — which is correct behavior but - means the result includes some trivial cases. This is an accurate characterization - of DCPF behavior but is not explicitly noted in the result. - evidence: - - file: evaluations/pandapower/results/extensibility/B-3_contingency_sweep.md - excerpt: "Converged cases: 3,276 (100%) | Cases with load loss: 924 (28.2%)" - - file: evaluations/pandapower/results/extensibility/B-3_contingency_sweep.md - excerpt: "Ran pp.rundcpp(net, check_connectivity=True), detected unsupplied buses via top.unsupplied_buses(net)" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_verification - # Proposed: Protocol should clarify that for DCPF contingency sweeps, "converged" - # means "direct solve completed on connected components" and should require - # distinguishing this from ACPF-style iterative convergence. - - - id: pandapower-F12 - category: test_design_gap - severity: medium - test_ids: [A-11] - title: Silent **kwargs absorption is a test-design boundary case — FAIL and feature-absent-with-misleading-API are distinct failure modes - description: > - A-11 fails because pandapower does not support distributed slack OPF. However, - the failure mode is more nuanced than simple feature absence: rundcopp() silently - accepts distributed_slack=True via **kwargs, produces identical results to - single-slack, and raises no warning. This is qualitatively different from a tool - that raises NotImplementedError (unambiguous absence) and different from a tool - that actually implements the feature. The protocol currently has no scoring - category for "feature absent with misleading API behavior" — it records the - same FAIL as a tool that raises an error. The misleading API could cause real - users to believe the feature is active, which is a safety concern for production - use. The A-11 result correctly documents this, but the FAIL status does not - capture the severity distinction. - evidence: - - file: evaluations/pandapower/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "rundcopp() accepts arbitrary **kwargs and silently ignores distributed_slack=True without raising an error or warning. This could mislead users into thinking the feature is active." - - file: evaluations/pandapower/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "OPF LMP comparison: LMPs are identical across all 39 buses. Difference: 0.0 at every bus." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - # Proposed: Add a sub-category for fails: "absent" (tool raises error or lacks - # function) vs "silent-mismatch" (tool accepts parameter, runs, but produces - # incorrect results). Silent-mismatch should carry a penalty in accessibility - # scoring beyond the expressiveness FAIL. - - - id: pandapower-F13 - category: infrastructure_friction - severity: medium - test_ids: [G-FNM-1, G-FNM-3, G-FNM-4] - title: FNM MATPOWER fallback path required for all pandapower FNM tests — tests measure format-compatibility, not pandapower capability - description: > - pandapower has no native intermediate CSV ingestion capability. All FNM Suite G - tests use the MATPOWER PPC fallback path (matpowercaseframes.CaseFrames + from_ppc). - This means the FNM tests for pandapower are measuring the quality of the - MATPOWER fallback preprocessing rather than pandapower's own data model. The - G-FNM-3 failure (localized 596.6% flow deviation) is attributed to "the MATPOWER - PPC import path's handling of specific impedance details" — which may not be a - pandapower problem at all. The G-FNM-4 ACPF infeasibility is partly attributed - to "PPC import path loses AC-critical transformer data." A tool with native CSV - import might handle these cases differently. The FNM tests therefore measure - pandapower's performance on a preprocessed, format-converted version of the FNM - rather than on the FNM directly. - evidence: - - file: evaluations/pandapower/results/fnm_ingestion/G-FNM-1_intermediate_ingestion.md - excerpt: "input_path: matpower" - - file: evaluations/pandapower/results/fnm_ingestion/G-FNM-3_dcpf_verification.md - excerpt: "pandapower has no native CSV import capability. The MATPOWER PPC format is the standard programmatic entry point." - - file: evaluations/pandapower/results/synthesis.md - excerpt: "The ACPF non-convergence is attributable to a combination of factors: 1. Data ingestion path limitations. pandapower ingests the FNM via the MATPOWER PPC path, which flattens transformer-specific data." - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: add_test - # Proposed: G-FNM test results for tools using the MATPOWER fallback path should - # be tagged with a path_type field (native_csv | matpower_fallback) so cross-tool - # comparison of FNM results appropriately accounts for the preprocessing step. - -extraordinary_claims: - - test_id: A-3 - claim: "All 46 branches have non-zero shadow prices with 70% branch derating on the 39-bus TINY network" - concern: > - In a well-posed DC OPF on a 39-bus network with 46 branches and 10 generators, - having every single branch simultaneously at its flow limit (all 46 binding) is - physically unexpected. Interior-point solvers can produce small non-zero duals - on inactive constraints due to complementary slackness relaxation. Only 7 branches - are reported above 95% loading, which is inconsistent with all 46 having binding - shadow prices. The threshold for "non-zero" shadow price is not reported. - evidence_quality: moderate - probe_recommended: true - probe_type: convergence_check - # Probe: Extract branch shadow prices with an explicit magnitude threshold - # (|dual| > 1e-4 or > 0.1 $/MWh), compare against branch loading_percent, - # and verify that claimed-binding branches are actually at or near flow limit. - # Expected: 7-15 branches binding (matching the 7 branches >95% loading), - # not all 46. - - - test_id: B-4 - claim: "240 DC OPF solves at 22.3 ms/solve with 100% convergence rate across all 20 scenarios x 12 hours" - concern: > - 22.3 ms per DC OPF solve on a 39-bus network is physically reasonable. The - concern is the 100% convergence rate across scenarios with differentiated costs - and renewable scenario multipliers. With aggressive scenario multipliers, some - scenarios may produce infeasible dispatch (renewables exceed load or generators - hit limits). The result notes "no infeasibility checks across scenarios were - performed." If all 240 scenarios trivially converge, the test may not be - exercising meaningful stochastic variation. The scenario_multipliers_50x24.csv - details are not shown, so it is unclear how extreme the scenarios are. - evidence_quality: weak - probe_recommended: false - probe_type: null - # Note: The 3.6% spread across hour-1 scenarios suggests scenarios are - # not extreme, making the 100% convergence rate plausible. Low concern. diff --git a/sweep-data/v10-to-v11/per-tool/powermodels/findings.md b/sweep-data/v10-to-v11/per-tool/powermodels/findings.md deleted file mode 100644 index 33539e94..00000000 --- a/sweep-data/v10-to-v11/per-tool/powermodels/findings.md +++ /dev/null @@ -1,491 +0,0 @@ -# powermodels — Sweep Findings (v10) - -## Summary - -The PowerModels.jl evaluation is well-evidenced and thoroughly executed, with all 37 -tests producing result files and measured timings. However, several scoring decisions -warrant review. The most significant issue is that SCUC receives qualified_pass status -at both TINY and SMALL scale despite PowerModels providing zero UC capability: the -entire MILP was assembled by the evaluator from raw JuMP. Two additional misleading -scores affect the scalability grade: C-8 (SCOPF MEDIUM) awards pass to a run that did -not converge within the time budget, and A-12 (multi-period storage) awards outright -pass despite three simultaneous non-obvious workarounds. The FNM DCPF hard-fail may -reflect test design (forcing DCPPowerModel when DCMPPowerModel exists and handles tap -ratios) rather than tool limitation. Two extraordinary claims warrant probe: -SCOPF mechanism is "demonstrated" on an N-1 infeasible network (A-9), and the GLPK -convergence at MEDIUM scale reverses a v9 timeout failure without a confirmed explanation. - -## Finding Details - -### pm-F01: SCUC qualified_pass awarded when PowerModels provides zero UC capability - -**Category:** misleading_result | **Severity:** high -**Tests:** A-5, C-4 - -A-5 (TINY) and C-4 (SMALL) both receive qualified_pass status. The result files are -unambiguous: "PowerModels used ONLY for data parsing ... Everything else is raw JuMP" -(A-5) and "the entire UC formulation (~300 LOC) was user-assembled" (C-4). The workaround -class is recorded as blocking in both cases. The 250/300 lines of user-assembled JuMP -construct binary commitment variables, min up/down time rolling-sum constraints, startup/ -shutdown logic, ramp-rate coupling, nodal power balance with B-theta DCPF, and branch -thermal limits. PowerModels' only contribution is parse_file. - -A qualified_pass with a blocking workaround accurately represents "the JuMP foundation -makes this implementable" but conveys a false impression that PowerModels participates in -the UC formulation. The synthesis itself identifies A-5's status as a human spot-check -item: "Verify whether 'qualified_pass with blocking workaround' is the correct -classification vs. 'fail' for the expressiveness sub-question." The result should either -be reclassified as fail (with a note that JuMP enables manual implementation) or the -scoring rubric for "blocking workaround + qualified_pass" needs explicit criteria -distinguishing "tool provides partial capability" from "tool provides a data format only." - -**Cross-tool relevance:** likely — this scoring pattern may apply to any tool evaluated -with a blocking workaround on a primary capability. -**Proposed action:** adjust_scoring - ---- - -### pm-F02: SCOPF MEDIUM awarded pass despite not converging within the time budget - -**Category:** misleading_result | **Severity:** high -**Tests:** C-8 - -C-8 records status: pass with convergence_iterations: 1. The result file explicitly -states "Benders converged: No (1 iteration, time budget)." The timing breakdown shows -that 50 post-contingency DCPF checks consumed 416s of the 600s budget at ~8.3s/case, -leaving insufficient time for the second iteration. With 17 binding contingencies -identified and 8 blocks added (out of 17 needed), the resulting "optimal" objective -($2,162,360/h) reflects only partial security constraint coverage. - -The protocol's "measurement test" designation is cited to justify pass status regardless -of convergence. However, "measurement test" should mean metrics are recorded even if -performance is poor -- not that incomplete convergence equals a pass. The same algorithm -converged in approximately 110s at SMALL scale (A-9), providing a meaningful -scalability comparison point that the pass designation obscures. A qualified_pass -noting "mechanism demonstrated, not converged within budget" would more accurately -characterize the result. - -**Cross-tool relevance:** likely — the "measurement test" designation for SCOPF may -produce similar misleading pass scores for other tools. -**Proposed action:** adjust_scoring - ---- - -### pm-F03: SCOPF TINY qualified_pass on an N-1 infeasible network - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** A-9 - -A-9 demonstrates the SCOPF two-level API mechanism on the IEEE 39-bus Modified Tiny -network. The test discovers the network is not fully N-1 secure at original ratings -under the Modified Tiny load profile: the full N-1 SCOPF LP (all 46 contingencies -simultaneously) is infeasible, and the iterative Benders algorithm runs exactly 1 -iteration before hitting infeasibility at iteration 2. - -The mechanism verification claim -- that "the PowerModels.jl SCOPF mechanism works -correctly" -- rests on: (a) 46/46 individual single-contingency SCOPFs being OPTIMAL, -and (b) the Benders algorithm correctly adding constraints and producing a higher-cost -dispatch in iteration 1 ($144,663/h vs. $98,091/h base). This is reasonable evidence -for API correctness but does not demonstrate that the algorithm converges to a -security-constrained solution. The rubric criterion for SCOPF presumably requires -demonstrating that the tool produces a feasible security-constrained dispatch; this -test only shows the tool correctly identifies that no such dispatch exists for this -network configuration. A network where N-1 feasibility is achievable (e.g., with -looser thermal limits or a modified load profile) would provide stronger evidence. - -Probe recommended: run the same SCOPF algorithm on the 39-bus network with relaxed -thermal limits (e.g., restoring original 100% ratings with a lighter load profile -that allows N-1 feasibility) to verify Benders convergence. - -**Cross-tool relevance:** likely — the choice of an N-1 infeasible test network -could affect other tools' SCOPF results similarly. -**Proposed action:** add_verification - ---- - -### pm-F04: DCPF verification test mandates solve_dc_pf (DCPPowerModel) when DCMPPowerModel exists and would likely pass - -**Category:** test_design_gap | **Severity:** medium -**Tests:** G-FNM-3 - -G-FNM-3 hard-fails with a 2.43% bus angle pass rate (threshold 95%) and a mean -deviation of 5.1 degrees. The root cause is explicitly identified: DCPPowerModel uses -a simplified B-matrix (`b = -1/x`) that ignores transformer tap ratios, while the -MATPOWER reference uses the full B-matrix via `makeBdc()` which incorporates taps. -The FNM network has 12,501 transformer-connected buses (45% of all buses). - -The evaluator identifies that DCMPPowerModel (which uses the full B-matrix) "would -incorporate taps into the B-matrix but is not available through solve_dc_pf" and -notes "Using solve_pf(data, DCMPPowerModel, optimizer) could potentially produce -results closer to the MATPOWER reference, but this was not tested because the task -specification explicitly requires solve_dc_pf." - -The synthesis flags this as a human spot-check item: "Verify whether using -solve_pf(data, DCMPPowerModel, optimizer) would pass the DCPF verification." The test -design is measuring which default function the tool exposes for DCPF rather than the -tool's maximum DCPF fidelity. A production user working with a transformer-heavy network -would select DCMPPowerModel. The failure is a test design decision, not necessarily a -tool capability boundary. - -Probe recommended: run solve_pf(data, DCMPPowerModel, optimizer) on the FNM network -and compare bus angles against the reference to determine whether the fail is -attributable to DCPPowerModel's simplified B-matrix choice or to deeper tool limitations. - -**Cross-tool relevance:** none (specific to PowerModels' formulation split) -**Proposed action:** redesign_test - ---- - -### pm-F05: ACPF convergence accepted without residual or iteration count due to API gap - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-2, A-4, C-5 - -compute_ac_pf returns only a Bool termination_status with no NR iteration count and no -convergence residual in the result dict. Both convergence_residual and -convergence_iterations are null in the A-2 and C-5 (SMALL) result headers. - -The evaluator validates convergence indirectly: termination_status == true, and 100% of -PQ buses have Vm != 1.0 pu (differ from flat start). This indirect check is reasonable -but cannot distinguish genuine convergence from early termination within tolerance. The -protocol requires reporting both iteration count and residual; neither is structurally -available for the compute_ac_pf path due to NLsolve's callback architecture. - -This is a documented tool limitation (not an evaluator error) but it means the pass and -qualified_pass results for A-2, A-4, and C-5 SMALL are accepted on weaker evidence -than the protocol intends. The Ipopt path (solve_ac_pf) would expose convergence -diagnostics but was used only for the failing MEDIUM tests. - -**Cross-tool relevance:** none -**Proposed action:** add_verification - ---- - -### pm-F06: A-12 awarded pass (not qualified_pass) despite three simultaneous stable workarounds - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** A-12, A-5 - -A-12 records status: pass with three simultaneous stable workarounds: -1. Solver switch: must use SCIP (not HiGHS or Ipopt) for MIQP due to ZeroOne constraints -2. Cyclic SoC: manual JuMP constraint injection required (not native to solve_mn_opf_strg) -3. Two-phase LMP extraction: SCIP cannot return LP duals; requires fix-and-price via HiGHS - -All three are non-obvious. None are documented in the PowerModels quickguide or -solve_mn_opf_strg documentation. The synthesis calls this out as a human spot-check item. - -By contrast, A-5 receives qualified_pass with a single blocking workaround. The -evaluation's workaround taxonomy defines blocking vs. stable but does not specify how -multiple simultaneous stable workarounds should affect status. Three concurrent -non-obvious workarounds on a MIQP storage formulation represent substantially more -friction than a single stable workaround, and the cumulative discovery cost for a new -user would likely exceed the discovery cost for some blocking workarounds on simpler -features. - -**Cross-tool relevance:** likely — the treatment of cumulative stable workarounds -vs. single blocking workarounds may be inconsistent across tools. -**Proposed action:** adjust_scoring - ---- - -### pm-F07: FNM ingestion suite tests MATPOWER fallback capability, not PowerModels PSS/E parsing - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** G-FNM-1, G-FNM-2, G-FNM-3, G-FNM-4 - -The PSS/E v31 RAW parser crashes on the Case Identification header, so the entire FNM -ingestion suite was executed against the MATPOWER fallback file (fnm_main_island.m). -The fallback is a pre-cleaned main-island subset with significant record deficits vs. -the manifest: bus -8.1%, load -42.7%, gen -0.5%, branch -3.6%. - -The consequences: -- G-FNM-2's 100% DCPF-critical field coverage measures MATPOWER PPC format coverage - (5 record types: bus, load, gen, branch, shunt), not PowerModels' data model breadth -- G-FNM-1's qualified_pass measures MATPOWER .m loading, not PSS/E parsing -- The 8% ACPF-critical coverage finding reflects MATPOWER format limitations, not - PowerModels' architecture -- The -42.7% load count deficit means the FNM power balance test is on a different - (lighter) network than the reference - -The PSS/E failure is estimated at 1-2 days to fix, but the FNM suite conclusions as -written characterize MATPOWER ingestion as the primary PowerModels data path for the -FNM, which may not reflect production use. - -**Cross-tool relevance:** none (PowerModels-specific PSS/E parser failure) -**Proposed action:** add_test - ---- - -### pm-F08: SCIP license classification conflict between F-3 (ZIB Academic) and F-8 (Apache 2.0) - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** F-3, F-8 - -F-3 classifies SCIP_jll v0.2.1 / SCIP 8.0.0 as ZIB Academic License and recommends -excluding SCIP from commercial deployments. F-8 classifies the same artifact as -Apache 2.0 and upgrades its assessment from qualified_pass to pass, explicitly stating -"the prior assessment incorrectly identified SCIP as ZIB Academic." - -Both tests reference the same Julia environment. The conflict is material because SCIP -is required for A-12 multi-period storage OPF (the only test requiring MIQP with -ZeroOne constraints). If SCIP carries the ZIB Academic restriction, the A-12 multi-period -storage result is not reproducible in commercial deployment and the supply chain -assessment changes. The synthesis calls this out for human resolution. - -The F-8 evidence (SCIPversion() returning 8.0, GitHub confirming Apache-2.0 for SCIP -9.x, note that Apache-2.0 applies since v8.0) is more specific, but F-3's JLL artifact -LICENSE file reading is direct evidence that should take precedence if the artifact -itself carries the ZIB Academic text. - -**Cross-tool relevance:** likely — other Julia-based tools may use SCIP via the same -SCIP_jll artifact and face the same ambiguity. -**Proposed action:** add_verification - ---- - -### pm-F09: 39-bus network cycling guardrail is weak: only end-of-horizon shutdowns, no restarts - -**Category:** network_insufficiency | **Severity:** low -**Tests:** A-5 - -The A-5 commitment schedule shows 3 generators decommitting in hours H22-H24 with -0 startup events: "Total shutdowns: 3. Total startups: 0 (generators only decommit -toward end of horizon, no recommitment needed within 24h)." The SCUC protocol requires ->= 2 generators cycling; 3 shutdowns meets the threshold but the cycling is economically -trivial -- generators shed at the end of a 24-hour window when there is no future -commitment obligation. - -The 39-bus network's capacity/load ratio (7,367 MW / 6,254 MW = 1.18x) provides limited -economic incentive for mid-cycle unit commitment decisions. A more discriminating test -would show mid-cycle startups (generators coming online for a peak then returning to -offline status), which would exercise min_up and min_down constraints jointly in a -single run. The current result verifies that binary constraint logic is coded correctly -but does not demonstrate that it produces economically significant commitment decisions. - -**Cross-tool relevance:** likely — other tools tested on the same 39-bus network face -the same network insufficiency for SCUC demonstration. -**Proposed action:** redesign_test - ---- - -### pm-F10: GLPK convergence on MEDIUM DCOPF contradicts v9 timeout failure; no confirmed explanation - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** C-3, C-7 - -C-3 reports GLPK solving the 10k-bus MEDIUM DC OPF in 61.86s. The result file notes -this contradicts the v9 evaluation where GLPK timed out at 300s: "the difference may -be due to JIT warm-up improving the model construction or solver configuration differences." -No definitive explanation is offered. - -The most plausible explanation is that v10 linearizes 45.5% of quadratic generator costs -(1,130 of 2,485 generators), reducing the problem from a QP to an LP that GLPK can -handle. GLPK does not support QP (confirmed in F-8); if v9 attempted to solve quadratic -costs with GLPK, GLPK would have rejected the formulation entirely rather than timing -out. The discrepancy may be an artifact of different preprocessing between protocol -versions rather than GLPK performance improvement. - -This matters for cross-tool comparisons: if GLPK performance is reported as 61.86s -based on a linearized LP while other tools solve the original QP, the solver comparison -is on different problem formulations. - -Probe recommended: verify whether v9 applied the same quadratic cost linearization -before attempting GLPK, or whether v9 submitted a QP that GLPK rejected silently (which -would appear as a timeout rather than an explicit error). - -**Cross-tool relevance:** none (PowerModels-specific v9 vs v10 protocol difference) -**Proposed action:** add_verification - ---- - -### pm-F11: Architecture audit (B-6) is documentation-based with no runtime execution - -**Category:** low_signal | **Severity:** low -**Tests:** B-6 - -B-6 records status: pass with wall_clock_seconds: null and loc: null. The test is a -static audit of the source tree structure, identifying the four-layer dispatch -architecture (public API / model lifecycle / formulation build / solver interface). -No code is executed and no runtime behavior is observed. - -The finding -- that PowerModels has a clean four-layer architecture with Julia multiple -dispatch as the extension mechanism -- is accurate and useful context. However, the -result is not differentiated from what any evaluator could derive from reading the -source code, and it carries the same evidentiary weight as a documentation reading. -This pattern is likely consistent across all tools evaluated (B-6 is presumably an -identical audit for each tool) and produces confirmed low signal. - -**Cross-tool relevance:** confirmed — B-6 is identical methodology for all tools. -**Proposed action:** redesign_test - ---- - -### pm-F12: PTDF timing discrepancy: B-9 MEDIUM (106s) vs C-9 MEDIUM (7.55s) for identical API call - -**Category:** extraordinary_claim | **Severity:** low -**Tests:** C-9 - -B-9 ran at MEDIUM scale (10k-bus, identical API call) in 106.44s total. C-9 ran the -same call in 7.55s total, attributed to JIT warm-up. The reported improvement is 14x -for make_basic_network (15.5s → 1.16s) and 15x for calc_basic_ptdf_matrix (35.5s → -2.37s). These are consistent with Julia's JIT behavior for first-invocation vs. -warm-REPL calls, but the magnitudes are at the upper end of typical JIT overhead. - -The result also notes that the synthesis reports the warm PTDF time (2.37s) as the -MEDIUM PTDF benchmark, while the cold-start time (35.5s) would represent the user -experience in a fresh process. For scalability grading purposes, the warm-REPL time -is appropriate for repeated evaluation workflows, but single-invocation workflows -(typical in scripted market operations) would see the 35.5s figure. - -Probe recommended: verify that the C-9 warm-up sequence (case39 solve before MEDIUM -PTDF) was applied consistently, and confirm whether the synthesis-reported PTDF time -(2.37s) or the cold-start time (35.5s) is the appropriate benchmark for the -scalability grade. - -**Cross-tool relevance:** none (Julia JIT specific) -**Proposed action:** add_verification - ---- - -### pm-F13: SCED test bypasses UC stage by design, making it an independent ED test - -**Category:** test_design_gap | **Severity:** low -**Tests:** A-6 - -A-6 skips the SCUC stage and treats all generators as committed, reducing the SCED -test to a multi-period ED LP with ramp constraints. The result correctly documents this -as a "stable scope reduction" and awards qualified_pass. The workaround is clean and -the ED implementation via replicate + build_mn_opf + manual ramp constraint injection -is genuine PowerModels capability. - -However, the rubric criterion for SCED includes commitment decisions as part of the -assessed capability. By bypassing the UC stage, the test grades the ED capability only, -which PowerModels handles well. Additionally, the "Security-Constrained" prefix in SCED -implies contingency constraints on dispatch; A-6 includes ramp constraints but not -N-1 contingency constraints on the dispatch schedule. The security component is absent. - -This is likely a consistent pattern across tools with UC limitations (the test always -degrades to ED-only when UC is unavailable), reducing A-6's discriminative power to -measuring multi-period LP capability rather than SCED capability specifically. - -**Cross-tool relevance:** likely — tools without native UC will all reduce A-6 to ED-only. -**Proposed action:** redesign_test - ---- - -## Extraordinary Claims - -### C-8: SCOPF MEDIUM awarded pass on non-converged Benders run - -**Concern:** The protocol labels C-8 a "measurement test" to justify pass status -regardless of convergence. The run completed 1 of an unknown number of required -iterations; the contingency screening loop alone consumed 70% of the time budget at -8.3s/case for 50 contingencies. The resulting objective ($2,162,360/h) has only 8 of -17 binding contingency blocks enforced. A pass score implies the tool can do MEDIUM -SCOPF; a qualified_pass with "not converged within budget" better reflects the result. - -**Evidence quality:** strong - -A convergence check probe would run the same algorithm with a lighter contingency set -(top 10 branches instead of 50) to verify whether the algorithm converges within the -600s budget, and report the convergence iteration count and final security constraint -violation magnitude. - ---- - -### A-9: SCOPF mechanism "demonstrated" on N-1 infeasible network - -**Concern:** The qualified_pass rests on showing that the tool correctly reports -infeasibility for a network where no N-1 secure dispatch exists. The key question -- -does Benders converge to a security-constrained optimum when one exists? -- is never -answered because the test network selection precludes a positive answer. - -**Evidence quality:** moderate - -A formulation audit probe would test the same SCOPF algorithm on the 39-bus network -with original (100% rated) thermal limits and a modified load profile that allows N-1 -feasibility, documenting the number of Benders iterations to convergence and the -final security-constrained objective. - ---- - -### G-FNM-3: DCPF hard-fail may be a test design constraint, not a tool limitation - -**Concern:** The test requires solve_dc_pf (DCPPowerModel, simplified B-matrix) even -though DCMPPowerModel (full B-matrix, tap-aware) exists in PowerModels and the evaluator -specifically identifies it as a candidate fix. The 97.6% bus failure rate measures -the gap between DCPPowerModel's simplified formulation and MATPOWER's full B-matrix, -not PowerModels' maximum DCPF fidelity on transformer-heavy networks. - -**Evidence quality:** strong - -A formulation audit probe would run solve_pf(data, DCMPPowerModel, highs_opt) on the -FNM network and compare bus angle deviations against the same reference CSV, documenting -whether DCMPPowerModel achieves the 95% bus angle pass threshold that DCPPowerModel fails. - ---- - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | — | — | -| G-2 | pass | — | — | -| G-3 | pass | — | — | -| A-1 | qualified_pass | stable | Branch flows require manual post-processing | -| A-2 | qualified_pass | stable | No NR residual/iterations exposed | -| A-3 | pass | — | — | -| A-4 | pass | — | Branch flows require stable workaround | -| A-5 | qualified_pass (blocking) | blocking | PowerModels not involved in UC formulation (pm-F01) | -| A-6 | qualified_pass | stable | UC stage bypassed; SCED reduces to ED-only (pm-F13) | -| A-9 | qualified_pass | stable | N-1 infeasible network; algorithm completes 1 iteration (pm-F03) | -| A-10 | qualified_pass | stable | Requires Ipopt, not HiGHS | -| A-11 | fail | blocking | No distributed slack formulation exists | -| A-12 | pass | stable (x3) | Three simultaneous non-obvious workarounds (pm-F06) | -| B-1 | pass | — | — | -| B-2 | qualified_pass | stable | No Graphs.jl integration | -| B-3 | pass | — | — | -| B-4 | pass | — | — | -| B-5 | pass | — | — | -| B-6 | pass | — | Documentation-based audit, no runtime (pm-F11) | -| B-8 | qualified_pass | stable | Distributed slack requires ~150 LOC custom OPF | -| B-9 | pass | — | — | -| C-1 | qualified_pass | stable | Branch flows require manual post-processing | -| C-2 | fail | blocking | ACPF diverges at MEDIUM; both NLsolve and Ipopt | -| C-3 | qualified_pass | stable | Cost linearization required; GLPK v9/v10 discrepancy (pm-F10) | -| C-4 | qualified_pass (blocking) | blocking | 300 LOC user-assembled MILP; PowerModels parses only (pm-F01) | -| C-5 (SMALL) | pass | — | — | -| C-5 (MEDIUM) | fail | blocking | Cascaded from C-2 | -| C-7 | qualified_pass | stable | SCIP dual extraction crash; GLPK timing discrepancy | -| C-8 | pass | stable | Non-converged Benders (1 iteration); should be qualified_pass (pm-F02) | -| C-9 | pass | — | Cold vs. warm JIT timing discrepancy (pm-F12) | -| C-10 | fail | blocking | Cascaded from A-11 | -| D-1 | qualified_pass | — | JIT overhead; API signature discovery issue | -| D-2 | qualified_pass | — | Core OPF documented; advanced types absent | -| D-3 | qualified_pass | — | 7/10 examples pass; 2 PSS/E header failures | -| D-4 | qualified_pass | — | INFEASIBLE OK; missing cost key cryptic | -| D-5 | informational | — | Mean 379 NBNCL | -| E-1 | informational | — | 5 releases / 24 months | -| E-2 | informational | — | 24 commits / 12 months | -| E-3 | fail | — | 82.9% single contributor; 100% reviewer concentration | -| E-4 | informational | — | LANL/DOE institutional backing | -| E-5 | informational | — | Median close time ~71.5 days | -| E-6 | informational | — | 93.93% coverage, CI on 3 OS | -| E-7 | informational | — | National-lab/academic adoption; no commercial ISO evidence | -| F-1 | pass | — | BSD 3-Clause (LANL) | -| F-2 | pass | — | 114 packages pinned | -| F-3 | qualified_pass | — | GLPK GPL v3; SCIP ZIB Academic (disputed, pm-F08) | -| F-4 | pass | — | All 35 JLL components have public source | -| F-5 | pass | — | Pure Julia to ccall boundary | -| F-6 | pass | — | Registry SHA verification | -| F-7 | pass | — | Julia depot + offline mode | -| F-8 | pass | — | SCIP license claim Apache 2.0 (contradicts F-3, pm-F08) | -| F-9 | qualified_pass | — | /stable/ links OK; no version pin guidance | -| G-FNM-1 | qualified_pass | — | MATPOWER fallback only; PSS/E parser fails (pm-F07) | -| G-FNM-2 | pass | — | 100% DCPF-critical coverage (MATPOWER format only) | -| G-FNM-3 | fail | — | DCPPowerModel simplified B-matrix; DCMPPowerModel not tested (pm-F04) | -| G-FNM-4 | informational | — | ACPF diverges on 27k-bus FNM | -| G-FNM-5 | informational | — | 39% native supplemental CSV | -| P2-1 | informational | — | PSS/E v33 only; v31 header crash | -| P2-2 | informational | — | PWL convex-only native; SOS2 absent | -| P2-3 | informational | — | Commitment injection ~40 LOC; ramp/reserve need JuMP extension | diff --git a/sweep-data/v10-to-v11/per-tool/powermodels/findings.yaml b/sweep-data/v10-to-v11/per-tool/powermodels/findings.yaml deleted file mode 100644 index 18f48613..00000000 --- a/sweep-data/v10-to-v11/per-tool/powermodels/findings.yaml +++ /dev/null @@ -1,384 +0,0 @@ -tool: powermodels -source_version: "v10" -timestamp: "2026-03-14T00:00:00Z" - -evaluation_summary: - total_tests: 37 - pass: 12 - fail: 5 - qualified_pass: 16 - informational: 4 - -findings: - - - id: pm-F01 - category: misleading_result - severity: high - test_ids: [A-5, C-4] - title: "SCUC qualified_pass awarded when PowerModels provides zero UC capability" - description: > - Both A-5 (TINY) and C-4 (SMALL) receive qualified_pass status despite PowerModels - contributing nothing to the unit commitment formulation beyond file parsing. The - entire MILP -- binary commitment variables, min up/down time constraints, startup - cost terms, ramp coupling -- was assembled by the evaluator from scratch using raw - JuMP. A qualified_pass with a "blocking" workaround class on the tool's most - operationally important expressiveness feature conveys that PowerModels can do SCUC - with effort; the accurate characterization is that PowerModels cannot do SCUC and - the result measures the evaluator's ability to write a JuMP MILP. - evidence: - - file: "evaluations/powermodels/results/expressiveness/A-5_scuc_TINY.md" - excerpt: "PowerModels used ONLY for data parsing ... Everything else is raw JuMP" - - file: "evaluations/powermodels/results/scalability/C-4_scuc_scale_SMALL.md" - excerpt: "PowerModels v0.21.5 does NOT natively support SCUC; the entire UC formulation (~300 LOC) was user-assembled" - - file: "evaluations/powermodels/results/expressiveness/A-5_scuc_TINY.md" - excerpt: "workaround_class: blocking" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F02 - category: misleading_result - severity: high - test_ids: [C-8] - title: "SCOPF MEDIUM awarded pass despite not converging within the time budget" - description: > - C-8 records status: pass even though the Benders decomposition completed only 1 of - an unknown required number of iterations before the 600s time budget was exhausted. - The result file itself states "Benders converged: No (1 iteration, time budget)." - The protocol states C-8 is a "measurement test" where metrics are recorded regardless - of convergence, but awarding pass to a non-converged SCOPF run overstates capability. - A qualified_pass noting incomplete convergence would more accurately represent the - result. The same SCOPF algorithm converged in ~110s at SMALL scale (A-9), so the - MEDIUM result meaningfully limits practical usability. - evidence: - - file: "evaluations/powermodels/results/scalability/C-8_scopf_scale_MEDIUM.md" - excerpt: "Benders converged: No (1 iteration, time budget)" - - file: "evaluations/powermodels/results/scalability/C-8_scopf_scale_MEDIUM.md" - excerpt: "status: pass ... wall_clock_seconds: 595.18" - - file: "evaluations/powermodels/results/synthesis.md" - excerpt: "C-8 SCOPF MEDIUM (pass, 1 iteration, not converged) -- Only 1 Benders iteration completed within 595s budget. Verify whether a non-converged result qualifies as pass vs. qualified_pass" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F03 - category: extraordinary_claim - severity: medium - test_ids: [A-9] - title: "SCOPF TINY qualified_pass on an N-1 infeasible network" - description: > - A-9 demonstrates the two-level API SCOPF mechanism on a network that is provably - N-1 infeasible under the Modified Tiny load profile. The iterative algorithm runs - exactly 1 iteration, then hits infeasibility because the joint N-1 constraint set - is overconstrained. The mechanism verification claim -- that SCOPF "works correctly" - -- rests entirely on confirming that 46/46 individual single-contingency SCOPFs are - feasible and that joint N-1 infeasibility is mathematically correct for this network. - This is plausible but the test does not verify that the Benders algorithm converges - to a secure solution on any network; the only convergence evidence is for the trivial - base-case OPF. The qualified_pass may overstate what was demonstrated. - evidence: - - file: "evaluations/powermodels/results/expressiveness/A-9_scopf_TINY.md" - excerpt: "Full SCOPF LP (all 46 N-1 simultaneously): INFEASIBLE ... Iterative iterations: 1 (hits infeasibility at iteration 2)" - - file: "evaluations/powermodels/results/expressiveness/A-9_scopf_TINY.md" - excerpt: "convergence_iterations: 1" - cross_tool_relevance: likely - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: pm-F04 - category: test_design_gap - severity: medium - test_ids: [G-FNM-3] - title: "DCPF verification test mandates solve_dc_pf (DCPPowerModel) when DCMPPowerModel exists and would likely pass" - description: > - G-FNM-3 hard-fails (2.43% bus angle pass rate) because the test specification - requires solve_dc_pf, which internally uses DCPPowerModel's simplified B-matrix - that ignores transformer tap ratios. The evaluator identifies that DCMPPowerModel - (full B-matrix including taps) exists in PowerModels and "could potentially produce - results closer to the MATPOWER reference" but was not tested because the spec - requires a specific function. The failure reflects a test design choice (constraining - the API to use) rather than a fundamental tool limitation. A production user would - select the appropriate formulation for their network characteristics. - evidence: - - file: "evaluations/powermodels/results/fnm_ingestion/G-FNM-3_dcpf_verification.md" - excerpt: "PowerModels' DCMPPowerModel formulation would incorporate taps into the B-matrix but is not available through solve_dc_pf ... this was not tested because the task specification explicitly requires solve_dc_pf" - - file: "evaluations/powermodels/results/fnm_ingestion/G-FNM-3_dcpf_verification.md" - excerpt: "Bus angle pass rate: 678 (2.43%) ... Root cause: DCPPowerModel uses a simplified B-matrix" - cross_tool_relevance: none - probe_recommended: true - probe_type: formulation_audit - proposed_action: redesign_test - - - id: pm-F05 - category: missing_verification - severity: medium - test_ids: [A-2, A-4, C-5] - title: "ACPF convergence accepted without residual or iteration count due to API gap" - description: > - compute_ac_pf returns only a Bool termination_status with no NR iteration count and - no convergence residual exposed in the result dict. At TINY and SMALL scale the - result file validates convergence indirectly (100% of PQ buses have Vm != 1.0 pu, - Vm range within physical bounds). This indirect check does not verify convergence - residual below the standard tolerance and cannot detect partial convergence where - the solver stopped early due to iteration limits. The protocol requires reporting - both iteration count and residual; both are structurally unavailable for the - compute_ac_pf path. - evidence: - - file: "evaluations/powermodels/results/expressiveness/A-2_acpf_TINY.md" - excerpt: "convergence_residual: null ... convergence_iterations: null ... NR iteration count and convergence residual not exposed" - - file: "evaluations/powermodels/results/expressiveness/A-2_acpf_TINY.md" - excerpt: "Convergence verified indirectly from: termination_status == true ... 100% of PQ buses (29/29) have Vm ≠ 1.0 pu" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: pm-F06 - category: scoring_inconsistency - severity: medium - test_ids: [A-12, A-5] - title: "A-12 awarded pass (not qualified_pass) despite three simultaneous stable workarounds" - description: > - A-12 records status: pass with three simultaneous workarounds -- solver switch to - SCIP for MIQP, manual cyclic SoC constraint injection, and two-phase LMP extraction - via fix-and-price. The evaluation synthesis flags this as a human spot-check item: - "verify whether three simultaneous workarounds should remain pass or be reclassified - as qualified_pass." By contrast, A-5 receives qualified_pass with a blocking - workaround. The distinction between pass-with-three-stable-workarounds and - qualified_pass is not consistently applied. All three workarounds in A-12 are - non-obvious and domain-specific; a new user would not discover any of them from - PowerModels documentation. - evidence: - - file: "evaluations/powermodels/results/expressiveness/A-12_multiperiod_dcopf_storage_TINY.md" - excerpt: "status: pass ... Three workarounds were required: 1. Solver switch (SCIP) ... 2. Cyclic SoC via manual JuMP constraint injection ... 3. Two-phase LMP extraction" - - file: "evaluations/powermodels/results/synthesis.md" - excerpt: "A-12 (pass with 3 stable workarounds) -- SCIP solver switch, cyclic SoC manual constraint, two-phase LMP extraction. Verify whether three simultaneous workarounds should remain pass or be reclassified as qualified_pass" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F07 - category: infrastructure_friction - severity: medium - test_ids: [G-FNM-1, G-FNM-2, G-FNM-3, G-FNM-4] - title: "FNM ingestion suite tests MATPOWER fallback capability, not PowerModels PSS/E parsing" - description: > - The entire FNM ingestion suite was executed against the MATPOWER fallback file - (fnm_main_island.m) because PowerModels' PSS/E v31 parser crashes on the Case - Identification header. The fallback is a pre-cleaned main-island subset with - record deficits vs. the manifest (bus -8.1%, load -42.7%). G-FNM-2's 100% - DCPF-critical field coverage and G-FNM-1's qualified_pass both measure MATPOWER - PPC format ingestion, not PowerModels' claimed PSS/E capability. The PSS/E failure - is characterized as a 1-2 day fix, but its effect on the FNM test suite results - is structural: all FNM test conclusions apply to the MATPOWER fallback path only. - evidence: - - file: "evaluations/powermodels/results/fnm_ingestion/G-FNM-1_intermediate_ingestion.md" - excerpt: "input_path: matpower" - - file: "evaluations/powermodels/results/synthesis.md" - excerpt: "FNM ingestion used the MATPOWER .m fallback because PowerModels' PSS/E v31 RAW parser crashes on the Case Identification header" - - file: "evaluations/powermodels/results/synthesis.md" - excerpt: "Record counts show deficits vs. manifest (bus -8.1%, load -42.7%, gen -0.5%, branch -3.6%) because the fallback is a pre-cleaned main-island subset" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: add_test - - - id: pm-F08 - category: scoring_inconsistency - severity: low - test_ids: [F-3, F-8] - title: "SCIP license classification conflict between F-3 (ZIB Academic) and F-8 (Apache 2.0)" - description: > - F-3 classifies SCIP_jll v0.2.1 (SCIP 8.0.0) as ZIB Academic License and flags it - as non-commercial. F-8 classifies the same artifact as Apache 2.0 and concludes it - is permissive for commercial use, explicitly noting "prior assessment incorrectly - identified SCIP as ZIB Academic." Both tests reference the same Julia environment - and the same binary. The conflict is unresolved in the evaluation and is called out - as a human spot-check item in the synthesis. The A- supply chain grade depends on - whether SCIP (which is required for A-12 multi-period storage OPF) is commercially - usable. - evidence: - - file: "evaluations/powermodels/results/supply_chain/F-3_dependency_license_audit.md" - excerpt: "SCIP_jll v0.2.1 (SCIP 8.0.0): ZIB Academic -- Non-commercial only" - - file: "evaluations/powermodels/results/supply_chain/F-8_solver_dependency_assessment.md" - excerpt: "SCIP v8.0 (November 2021) switched from the ZIB Academic License to Apache 2.0 ... The SCIP_jll v0.2.1 in this manifest wraps SCIP 8.0, confirmed by SCIP.SCIPversion() returning 8.0" - - file: "evaluations/powermodels/results/synthesis.md" - excerpt: "F-3 vs. F-8 SCIP license discrepancy -- F-3 classifies SCIP_jll as ZIB Academic; F-8 reports SCIP 8.0.0 as Apache 2.0. Both reference the same JLL version. Needs definitive resolution" - cross_tool_relevance: likely - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: pm-F09 - category: network_insufficiency - severity: low - test_ids: [A-5] - title: "39-bus network cycling guardrail is weak: only 3 of 10 generators cycle, all at end-of-horizon" - description: > - A-5 demonstrates UC cycling by showing 3 of 10 generators decommitting in hours - H22-H24 of a 24-hour horizon with no recommitment. All transitions are shutdowns - (3 shutdown events, 0 startups). The 39-bus network's high capacity-to-load ratio - (7,367 MW / 6,254 MW peak = 1.18x) means commitment decisions are economically - marginal: the optimizer sheds expensive gas CC units during the last few hours - only. A network with tighter capacity margins or more diverse cost structure would - force more meaningful cycling (mid-cycle startups, alternating unit commitment). - The result verifies the MILP constraint logic but does not demonstrate robust - UC cycling behavior. - evidence: - - file: "evaluations/powermodels/results/expressiveness/A-5_scuc_TINY.md" - excerpt: "Total shutdowns: 3. Total startups: 0 (generators only decommit toward end of horizon, no recommitment needed within 24h)" - - file: "evaluations/powermodels/results/expressiveness/A-5_scuc_TINY.md" - excerpt: "case39 has a high capacity-to-load ratio (7,367 MW capacity vs 6,254 MW peak load)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pm-F10 - category: extraordinary_claim - severity: medium - test_ids: [C-3, C-7] - title: "GLPK convergence on MEDIUM DCOPF contradicts v9 timeout failure; no explanation for improvement" - description: > - C-3 reports GLPK converging on the MEDIUM 10k-bus DC OPF in 61.86s wall-clock - time. The result file notes: "In the v9 evaluation, GLPK hit the 300s time limit - without finding an optimal solution. In this v10 run, GLPK converged in 61.86s. - The difference may be due to JIT warm-up improving the model construction or solver - configuration differences." No definitive explanation is provided. The 5x speedup - (300s+ to 61.86s) without a protocol change is extraordinary and could reflect a - solver configuration difference, a cost linearization change (v10 linearizes 45.5% - of quadratic costs which v9 may not have), or a test artifact. This matters because - the GLPK comparison is part of C-7 solver swap testing and affects the A- supply - chain assessment (GLPK is GPL v3). - evidence: - - file: "evaluations/powermodels/results/scalability/C-3_dcopf_scale_MEDIUM.md" - excerpt: "Improvement from v9: In the v9 evaluation, GLPK hit the 300s time limit without finding an optimal solution. In this v10 run, GLPK converged in 61.86s. The difference may be due to JIT warm-up improving the model construction or solver configuration differences." - - file: "evaluations/powermodels/results/scalability/C-3_dcopf_scale_MEDIUM.md" - excerpt: "Generators cost-linearized: 1,130 (45.5%) ... GLPK wall-clock: 63.20s" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - - - id: pm-F11 - category: low_signal - severity: low - test_ids: [B-6] - title: "Architecture audit (B-6) is documentation-based with no runtime execution" - description: > - B-6 (code architecture) is classified as a pass based on a static audit of the - PowerModels source tree structure and documentation review. No code is executed. - The test contributes to the A- extensibility grade but measures the evaluator's - reading of source code rather than any runtime capability. The four-layer - architecture finding -- while accurate -- would be reproducible through any source - code inspection and does not differentiate PowerModels from other tools in a - runtime-measurable way. This is likely the pattern for all tools in the evaluation. - evidence: - - file: "evaluations/powermodels/results/extensibility/B-6_code_architecture.md" - excerpt: "Test: N/A ... Time: -- ... LOC: --" - - file: "evaluations/powermodels/results/synthesis.md" - excerpt: "B-6 Architecture audit | N/A | pass | -- | -- | -- | -- |" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pm-F12 - category: extraordinary_claim - severity: low - test_ids: [C-9] - title: "PTDF timing discrepancy: B-9 MEDIUM (106s) vs C-9 MEDIUM (7.55s) on identical API call" - description: > - B-9 ran at MEDIUM scale (same 10k-bus network, same calc_basic_ptdf_matrix call) - in 106.44s total. C-9 ran the same call in 7.55s total. The result file attributes - the 14x speedup to JIT warm-up. This implies the JIT compilation overhead for - PTDF computation alone is ~100s, which is extraordinary for a function that - performs dense linear algebra on a known-size matrix. The explanation is plausible - (Julia's JIT for specialized BLAS/sparse operations can be significant) but the - magnitude suggests either the test conditions differ in ways not captured, or the - "warm JIT" claim for C-9 should be verified against the actual test sequence. - evidence: - - file: "evaluations/powermodels/results/scalability/C-9_ptdf_scale_MEDIUM.md" - excerpt: "B-9: make_basic_network 15.5s, calc_basic_ptdf_matrix 35.5s, total 106.44s | C-9: make_basic_network 1.16s, calc_basic_ptdf_matrix 2.37s, total 7.55s ... The large performance improvement is due to JIT warm-up." - - file: "evaluations/powermodels/results/scalability/C-9_ptdf_scale_MEDIUM.md" - excerpt: "timing_source: measured" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - - - id: pm-F13 - category: test_design_gap - severity: low - test_ids: [A-6] - title: "SCED test bypasses UC stage by design, making it an independent ED test" - description: > - A-6 explicitly bypasses the SCUC stage because PowerModels cannot do UC, treating - all generators as committed. This reduces SCED to a pure multi-period ED LP. The - test correctly documents this as a "stable scope reduction" workaround. However, - the rubric criterion for A-6 is SCED (Security-Constrained Economic Dispatch with - commitment), not standalone ED. The resulting qualified_pass grades a capability - PowerModels does have (multi-period LP with ramp constraints) rather than the - capability being assessed. The test result is accurate but the scoring may overstate - SCED capability since the security-constraint component (binding N-1 contingency - constraints on dispatch) is also absent. - evidence: - - file: "evaluations/powermodels/results/expressiveness/A-6_sced_TINY.md" - excerpt: "Since A-5 (SCUC) is a blocking fail, the UC stage is bypassed: all generators are assumed committed across all periods." - - file: "evaluations/powermodels/results/expressiveness/A-6_sced_TINY.md" - excerpt: "workaround_class: stable ... Grade impact: B-level. The two-stage architecture is cleanly separable. The ED stage is fully implemented via documented API. The UC gap is a hard capability boundary" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: redesign_test - -extraordinary_claims: - - - test_id: C-8 - claim: > - SCOPF MEDIUM awarded status: pass after completing only 1 Benders iteration within - the 600s budget, with 17 identified binding contingencies and 8 blocks added, but - without converging to a security-constrained optimal solution. - concern: > - The protocol states C-8 is a "measurement test," but pass status is awarded for a - non-converged run. The remaining 9 binding contingency blocks were not added within - budget, meaning the "optimal" objective ($2,162,360/h) reflects only partial - security constraint coverage. A tool that runs faster on the contingency screening - loop could produce a converged result within the same budget. - evidence_quality: strong - probe_recommended: true - probe_type: convergence_check - - - test_id: A-9 - claim: > - SCOPF TINY mechanism is "fully demonstrated" via a qualified_pass on a network - where the full N-1 SCOPF is provably infeasible, and the iterative algorithm - completed only 1 iteration before terminating. - concern: > - The claim that the mechanism is verified rests on 46 individual single-contingency - SCOPFs being feasible, but the key question -- does the Benders cutting-plane - algorithm converge to a security-constrained dispatch when one exists? -- is - never answered because the network chosen is N-1 infeasible under the evaluation - load profile. A different network configuration or load profile would be needed to - demonstrate convergence. - evidence_quality: moderate - probe_recommended: true - probe_type: formulation_audit - - - test_id: G-FNM-3 - claim: > - DCPF hard-fails on the FNM production network (2.43% bus angle pass rate) due to - DCPPowerModel ignoring transformer tap ratios, but DCMPPowerModel (full B-matrix) - was not tested despite being available in PowerModels. - concern: > - The test result presents a tool failure that may actually be a test design failure: - constraining the test to solve_dc_pf (which hardcodes DCPPowerModel) when - DCMPPowerModel is available and would produce a physically correct result for a - transformer-heavy network. The synthesis calls this out as a human spot-check item. - evidence_quality: strong - probe_recommended: true - probe_type: formulation_audit diff --git a/sweep-data/v10-to-v11/per-tool/powersimulations/findings.md b/sweep-data/v10-to-v11/per-tool/powersimulations/findings.md deleted file mode 100644 index bcf646ae..00000000 --- a/sweep-data/v10-to-v11/per-tool/powersimulations/findings.md +++ /dev/null @@ -1,537 +0,0 @@ -# powersimulations — Sweep Findings (v10) - -## Summary - -The PowerSimulations.jl evaluation is evidence-rich and methodologically sound. All -timing figures are flagged as measured; code outputs are captured with concrete numerical -results. The primary sweep concerns are: (1) two qualified_pass results that mask -fundamentally relaxed or partially failed problems (C-3 loses branch flow limits, A-12 -fails its BESS arbitrage behavioral condition); (2) ACPF convergence accepted without -residuals at any scale, including a 10K-bus case where the solver logged a convergence -warning but still returned results; (3) a single-threaded SCUC timing measurement on a -32-core machine that is likely 20-30x slower than production-representative performance. -Three probes are recommended: convergence verification for A-2/C-2, LMP unit-conversion -verification for A-3, and multi-threaded timing for C-4. - ---- - -## Finding Details - -### powersimulations-F01: DCOPF qualified_pass on MEDIUM masks a fundamentally relaxed problem - -**Category:** misleading_result | **Severity:** high -**Tests:** C-3 - -The C-3 MEDIUM DCOPF qualified_pass is supported by real solver runs with consistent -objectives across HiGHS ($3,659,662.46) and GLPK ($3,659,662.46). However, workaround 4 -of the five stacked workarounds removes all branch flow limit constraints: -`StaticBranchUnbounded replaces StaticBranch` because "branch flow limit constraints -cause numerical infeasibility at 10K scale (basis matrix condition number > 10^15)." -Without flow limits, the problem reduces to unconstrained economic dispatch with DC -network balance constraints — the network is present but congestion is absent. This -is a different problem than a network-constrained DCOPF, which is what C-3 is designed -to assess. - -The result file notes that LMP extraction failed with StaticBranchUnbounded, which is -consistent with an unconstrained problem: "LMPs should be uniform on ACTIVSg10k (no -binding branch constraints at ~84% max loading)." This is correct reasoning, but it also -means the tool cannot produce LMPs at MEDIUM scale — a significant capability gap that -the qualified_pass status does not surface. - -The grading standard should distinguish between "qualified_pass with minor caveats" and -"qualified_pass where a fundamental constraint class is absent from the problem." The -latter is closer to a fail on the congestion-enforcement criterion. - -**Cross-tool relevance:** likely — other tools may face similar numerical conditioning -issues with large LP constraint matrices at 10K bus scale. -**Proposed action:** adjust_scoring - ---- - -### powersimulations-F02: ACPF convergence accepted without residual or iteration count - -**Category:** missing_verification | **Severity:** high -**Tests:** A-2, C-2 - -Neither A-2 (TINY ACPF) nor C-2 (MEDIUM ACPF) can report Newton-Raphson iteration count -or convergence residual — PowerFlows.jl v0.9.0 does not expose these in its public API. -Both results infer convergence from non-flat voltage profiles (100% of buses with non-unity -Vm in A-2; 84% in C-2). - -The protocol requires: "Convergence residual must be reported and below the tool's stated -tolerance. Number of NR iterations must be reported." A-2 explicitly qualifies the pass on -these grounds. C-2 does not mention the convergence residual gap in its status. - -More seriously, the C-2 MEDIUM result documents: "The warm-start NR solver on the first -call logged a convergence warning (NewtonRaphsonACPowerFlow solver failed to converge) but -still returned results." This is a case where the solver itself flagged non-convergence but -the results were accepted and the second call treated as validated. The possibility that -the 10K-bus ACPF result is a partially-converged solution is not ruled out by voltage -profile inspection alone. - -A probe could verify ACPF quality by checking active power balance residual (sum of -mismatches across all buses) and comparing against a reference ACPF solution on case39 -from a tool that exposes residuals. - -**Cross-tool relevance:** none (PowerFlows.jl-specific API limitation) -**Proposed action:** add_verification - ---- - -### powersimulations-F03: SCOPF 17.7% cost increase claim based on only 7 of 34 contingencies - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** A-9 - -The A-9 SCOPF result asserts a 17.7% cost premium for N-1 security on the 39-bus network. -The measurement is real — the solve completes optimally, and the dispatch shift from nuclear -to gas CC is economically coherent. However, the contingency set is severely pruned: 27 of -34 line contingencies are eliminated because they produce near-radial redistribution -(|LODF| >= 0.95). The 7 applied contingencies add 288 constraints to the base DCOPF. - -The 17.7% cost premium reflects security margins against this specific 7-contingency subset. -If the pass/fail filter were relaxed (e.g., |LODF| < 0.99 instead of 0.95), more -contingencies would bind and the cost premium would likely change. The cost-increase -direction is confirmed but the magnitude is contingency-set dependent, which is not -documented in the result. - -This finding has cross-tool relevance because all tools will face the same near-radial -topology issue on case39, producing comparably small effective N-1 sets. - -**Cross-tool relevance:** confirmed -**Proposed action:** add_verification - ---- - -### powersimulations-F04: 39-bus network is too radially connected for meaningful SCOPF evaluation - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-9 - -The protocol specifies "all 46 branches as contingency set" for SCOPF on TINY. In practice, -79% of branch contingencies (27/34 lines) are immediately infeasible on the 39-bus topology -due to near-radial redistribution. The effective test is N-7 SCOPF, not N-46. A tool that -implements SCOPF through brute-force enumeration (like the manual LODF approach used here) -and a tool with built-in intelligent contingency screening would both appear equivalent on -this network — both would filter to approximately 7 effective contingencies. - -A more meshed reference network at the SMALL tier (2K-bus, 3206 branches) would produce -meaningful N-50 or N-100 SCOPF tests where contingency screening methodology matters. -The SCOPF behavioral signal on case39 cannot distinguish between "excellent SCOPF with -screening" and "basic LODF enumeration with the same filter." - -**Cross-tool relevance:** confirmed — this affects all tools evaluated on A-9. -**Proposed action:** redesign_test - ---- - -### powersimulations-F05: A-12 qualified_pass masks explicit BESS arbitrage condition failure - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-12 - -The A-12 result explicitly reports: "Condition 2: BESS Arbitrage (FAIL). Discharge LMP is -NOT greater than charge LMP." The protocol pass condition requires: "Mean LMP at the BESS -bus during discharge hours must exceed mean LMP during charge hours." This condition failed. - -The qualified_pass is justified in the result on the grounds that multi-period DCOPF itself -works (Conditions 1 and 3 pass), and the BESS arbitrage failure stems from using linear -costs (required because HiGHS fails on multi-period QP). This is a reasonable explanation -of causality but does not change the fact that the behavioral condition failed. - -The root cause chain is: HiGHS MIQP limitation → forced linear costs → LP degeneracy → -non-unique LMPs → BESS does not arbitrage. This represents a tool capability limitation -(multi-period QP with HiGHS) that produces a behavioral test failure. The current status -of qualified_pass may over-represent capability. - -**Cross-tool relevance:** likely — other tools using HiGHS for multi-period LP face the -same LP degeneracy with linear costs for storage arbitrage tests. -**Proposed action:** adjust_scoring - ---- - -### powersimulations-F06: A-10 fail is solver-ecosystem mismatch, not an absent formulation - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** A-10 - -A-10 fails because DCPLLPowerModel (the lossy DC OPF formulation in PowerModels.jl, -accessible through PSI) uses `ScalarQuadraticFunction-in-GreaterThan` constraints for its -linearized Ohm's law. HiGHS supports QP objectives but not quadratic constraints. The -protocol specifies HiGHS as the primary solver. - -This is categorically different from A-11 (distributed slack), where no formulation exists -anywhere in the ecosystem. For A-10, the formulation exists; the failure is that the -evaluation's designated solver cannot execute it. Ipopt (an NLP solver also available in -the evaluation environment) would handle DCPLLPowerModel's constraint type. The result -notes this: "Alternatively, Ipopt could be used but it is an NLP solver, not the specified -solver for this test." - -The fail correctly captures the HiGHS-constrained evaluation outcome, but cross-tool -comparison should recognize this as a protocol-solver-selection effect, not an absolute -capability gap. A protocol note distinguishing "no formulation" from "formulation present -but solver-incompatible" would improve cross-tool interpretability. - -**Cross-tool relevance:** confirmed — PowerModels.jl-based tools (PowerModels.jl itself) -face the same HiGHS incompatibility with DCPLLPowerModel. -**Proposed action:** add_test - ---- - -### powersimulations-F07: LMP values produced with undocumented unit conversion not independently verified - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-3, B-1 - -Both A-3 (DCOPF LMPs) and B-1 (custom constraint dual) require dividing raw dual values -by `base_power` (100 MVA) and negating. The D-2 documentation audit confirms this -conversion is not documented: "Dual unit conversion (divide by base_power, negate): No — -Discovered empirically." - -The LMP range reported in A-3 ($7.76-$290.11/MWh) is economically plausible for the cost -structure (hydro $5/MWh to gas CC $40/MWh), which provides informal confirmation. But -the result does not show raw dual values alongside converted LMPs, so an independent -reviewer cannot verify the conversion was applied correctly. A 100x error (missing the -base_power division) would produce $0.078-$2.90/MWh (too low) or $776-$29,011/MWh -(too high) — both easily recognizable as wrong, but only if the reviewer knows the expected -range. Without raw dual capture, the verification chain is incomplete. - -**Cross-tool relevance:** none — PSI-specific dual unit convention. -**Proposed action:** add_verification - ---- - -### powersimulations-F08: A-6 SCED ramp enforcement verified by constraint count, not binding evidence - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-6 - -The A-6 result states: "1 binding ramp constraint observed, confirming ramp limits are -enforced in ED independently of UC." This claim is made in passing without identifying -which generator's ramp is binding, what the before/after dispatch values are, or what -the constraint dual value is. The protocol requires that ramp constraints be "demonstrably -enforced between consecutive dispatch intervals in the ED stage." - -One binding constraint out of 460 added (0.2%) is a weak signal. The ramp parameters -are set from gen_temporal_params.csv and may be loose relative to the load profile -changes — most generators may simply not need to ramp near their limits. Demonstrating -enforcement with a single unnamed binding constraint is insufficient to confirm the -protocol's intent. - -The LMP extraction failure for the ED stage also reduces confidence: "LMPs from ED stage -returned null values. The initialize_model=false + JuMP.optimize!() bypass prevents PSI's -dual tracking from populating." A test that cannot extract dual values for verification -has limited credibility as an economic dispatch validation. - -**Cross-tool relevance:** likely — the ramp enforcement verification gap may affect other -tools' SCED results similarly. -**Proposed action:** add_verification - ---- - -### powersimulations-F09: C-8 SCOPF qualified_pass despite solver crash on grade_network - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** C-8, A-9 - -C-8 receives qualified_pass despite HiGHS encountering OTHER_ERROR on the MEDIUM -(10K-bus) network after 438 seconds. The grade_network per eval-config.yaml for C-8 is -MEDIUM. A solver crash that produces no result is a more severe outcome than a timeout -with an incumbent solution (which is what SMALL produced). Neither outcome demonstrates -that SCOPF at scale is achievable. - -By contrast, A-9 (SCOPF on TINY with manual LODF assembly, which is clearly a workaround -for absent built-in SCOPF) also receives qualified_pass. The scoring system does not -distinguish between: (a) a workaround that produces a correct result, (b) a timeout with -an incumbent, and (c) a solver crash with no result. All receive qualified_pass. - -The OTHER_ERROR crash on MEDIUM should arguably be treated as a fail on the grade_network, -with the SMALL partial result noted as additional context. Using the same qualified_pass -for both a 7-contingency manual LODF solution and a solver crash is a scoring pattern -inconsistency. - -**Cross-tool relevance:** likely — other tools' SCOPF scale results may show similar -outcome diversity under the same qualified_pass label. -**Proposed action:** adjust_scoring - ---- - -### powersimulations-F10: Machine-precision PTDF claim may not generalize to networks with tap transformers - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** B-9 - -B-9 reports: "Max error: 1.15e-14 p.u. All 46 branches match to machine precision." This -is verified against DCPF results on the case39 network, which has no phase-shifting -transformers (all SHIFT=0). The B-9 result explicitly notes this precondition. - -C-9 (PTDF at MEDIUM scale) documents PTDF values outside [-1,1] for the 10K-bus network -due to 776 non-unity tap transformers, but does not perform a flow accuracy check because -the DCPF on MEDIUM also has the simplified B-matrix issue (see G-FNM-3). The machine- -precision claim on case39 cannot be assumed to generalize to networks with off-nominal -taps — which is precisely the real-world scenario where PTDF accuracy matters for -contingency screening. - -This finding also connects to G-FNM-3: if PowerFlows.jl's DCPF uses a simplified B-matrix -that ignores tap ratios, then PTDF-DCPF consistency may hold (both use the same simplified -model) while both diverge from physical reality on real networks. - -**Cross-tool relevance:** confirmed — the protocol's B-9 PTDF verification requirement -applies equally to all tools, and the phase-shifter correction gap affects any tool using -a simplified admittance matrix. -**Proposed action:** add_test - ---- - -### powersimulations-F11: G-FNM-3 formulation-difference classification understates practical impact - -**Category:** misleading_result | **Severity:** low -**Tests:** G-FNM-3 - -G-FNM-3 classifies the 86.8% bus angle failure as a formulation-difference. This is -technically accurate — PowerFlows.jl uses `b = -1/x` (simplified B-matrix) while MATPOWER -uses the full B-matrix with tap ratio correction. The classification is appropriate for -academic comparison, but it may understate the practical consequence for operational -workflows. - -With 2,340 of 2,358 TapTransformers having off-nominal taps (99.2%), the simplified -B-matrix is not a minor edge case on the FNM network — it affects nearly every transformer. -The mean angle deviation (2.66 degrees) and the absence of any API-level fix -("no configuration option in PowerFlows.jl to switch to a full B-matrix") mean this -is not a correctable limitation but a fundamental architectural choice with real impact -on DCPF accuracy for the target use case. - -**Cross-tool relevance:** none — PowerFlows.jl-specific architecture. -**Proposed action:** null - ---- - -### powersimulations-F12: SCUC cycling claim not cross-verified against economic necessity - -**Category:** missing_verification | **Severity:** low -**Tests:** A-5 - -A-5 confirms 3 generators cycle (gen-5 Coal, gen-7 Gas CC, gen-10 Gas CC) over 24 hours, -which satisfies the "at least 2 generators must cycle" protocol requirement. The cycling -pattern is economically coherent (gas CC cycles for peak hours), and MIP gap 0.57% is -within the 1% tolerance. - -However, the load amplitude (4,237 MW valley to 6,254 MW peak, a 48% swing) is unusually -large. With such a strong demand signal, cycling is driven as much by capacity necessity -as by cost optimization. The test does not verify whether cycling is driven by cost -differentiation specifically. On a flatter load profile, the same 10 generators with -the same cost structure might not force meaningful cycling. The protocol's intent is to -verify that the tool correctly uses cost signals to drive commitment decisions, not just -that it can handle a large demand swing. - -**Cross-tool relevance:** confirmed — the same load profile and network are used for -all tools' A-5 tests. -**Proposed action:** null - ---- - -### powersimulations-F13: 0/10 example pass rate inflated by external dependency design choice - -**Category:** infrastructure_friction | **Severity:** low -**Tests:** D-3 - -D-3 reports 0 of 10 official examples run unmodified. This accurately reflects the -out-of-box experience, but all failures at the PSI tutorial level trace to a single root -cause: tutorials use `PowerSystemCaseBuilder.jl`, a separate package not bundled with PSI. - -Python ecosystem tools (PyPSA, pandapower) include built-in test networks that require -no external downloads. The Julia ecosystem chose to separate test data into a dedicated -package for architectural cleanliness. This is a legitimate trade-off, not poor tutorial -quality. The PowerFlows.jl tutorial works correctly with a one-line data source -substitution. - -Cross-tool comparison using D-3 "examples run unmodified" will systematically disadvantage -Julia-ecosystem tools relative to Python tools with bundled test networks. The protocol -should either standardize the data source across all tools or note that the 0/10 figure -reflects a packaging convention rather than tutorial quality. - -**Cross-tool relevance:** likely — other Julia ecosystem tools (PowerModels.jl, Sienna -ecosystem) will show the same pattern. -**Proposed action:** redesign_test - ---- - -### powersimulations-F14: C-5 progressive relaxation is structurally inapplicable to NR power flow - -**Category:** test_design_gap | **Severity:** low -**Tests:** C-5 - -C-5 applies progressive thermal limit relaxation (0%, 10%, 20%) to diagnose ACPF -convergence difficulty. For PowerSimulations.jl using PowerFlows.jl's built-in NR solver, -all three relaxation levels produce identical voltage profiles with identical solve times -(0.34s after JIT). The result correctly explains: "thermal limit relaxation only affects -OPF branch constraints, not the NR power flow equations." - -This is a structural test design issue: progressive relaxation is a meaningful diagnostic -for tools that use OPF-based AC feasibility checks (e.g., Ipopt solving the AC OPF with -relaxed thermal constraints). For NR power flow solvers, thermal limits are not part of -the solve — the NR converges or it does not, independent of thermal ratings. The test -as designed cannot distinguish "tool converges easily at nominal limits" from "tool ignores -thermal limits in NR" — both look identical in the output. - -**Cross-tool relevance:** confirmed — this affects all tools that implement ACPF via -direct NR rather than through an OPF-based feasibility formulation. -**Proposed action:** redesign_test - ---- - -### powersimulations-F15: Gate ingest tests produce no cross-tool signal - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -G-1/G-2/G-3 test MATPOWER ingestion at three network tiers. All tools that reach this -evaluation stage pass these tests — they gate out tools that cannot load MATPOWER files -at all, but do not differentiate among surviving tools. The timing reported (6.44s for -case39) is dominated by Julia JIT compilation, not parser performance, making it -incomparable to Python tool loading times. - -These tests are necessary as gates but should not carry comparative weight in scoring. - -**Cross-tool relevance:** confirmed — universal across all evaluated tools. -**Proposed action:** null - ---- - -### powersimulations-F16: C-4 SCUC timing measured single-threaded on 32-core machine - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** C-4 - -C-4 measures HiGHS solving a 22,608-binary MILP in 404 seconds, single-threaded, on a -machine with 32 available cores. The result explicitly notes "CPU cores used: 1 (32 -available)" without explaining why multi-threading was not used. Modern MILP solvers -including HiGHS achieve near-linear speedup on branch-and-bound with more threads. -A 32-thread run would plausibly reduce solve time to 15-30 seconds, moving C-4 from -"barely within an acceptable time window" to "efficient." - -SCIP hit TIME_LIMIT (600s) single-threaded; multi-threaded SCIP might solve within -the limit. The timing measurement as recorded is reproducible but unrepresentative of -production deployment on the evaluation hardware. Cross-tool comparison of C-4 times -would be misleading if other tools' SCUC tests used multi-threading. - -The protocol does not specify single-threaded measurement for scalability tests. Adding -a multi-threaded measurement as a secondary metric would significantly improve the -interpretability of C-4 results across tools. - -**Cross-tool relevance:** confirmed — SCUC timing comparisons across all tools are -affected if threading discipline is inconsistent. -**Proposed action:** add_verification - ---- - -## Extraordinary Claims - -### A-9: SCOPF 17.7% cost increase from N-1 contingencies - -**Concern:** Only 7 of 34 line contingencies applied after near-radial topology filter. -The cost increase is real but quantified against a minimal effective contingency set on -a topology where 79% of contingencies are infeasible. -**Evidence quality:** moderate - -The 17.7% cost increase direction is confirmed by the economic logic (nuclear→gas CC shift) -and by the mathematical correctness of LODF-based constraint injection. However, the -magnitude is highly sensitive to which contingencies survive the |LODF| < 0.95 filter. -A probe is not recommended because the result is correctly bounded as a TINY-network -finding; the SCOPF capability itself is more meaningfully assessed at scale (C-8). - ---- - -### B-9: Machine-precision PTDF (max error 1.15e-14 p.u.) - -**Concern:** Verified on case39 which has no phase-shifting transformers. C-9 shows -PTDF values outside [-1,1] on the 10K-bus network due to tap transformers, with no -flow accuracy check. The machine-precision claim cannot be assumed to hold at scale -with off-nominal taps. -**Evidence quality:** moderate - -The B-9 measurement itself is credible — the case39 verification is thorough and the -machine-precision result is consistent with known properties of PTDF computation on -lossless networks. The claim should be interpreted as network-specific rather than -as a general characterization of PowerNetworkMatrices.jl's PTDF accuracy. - ---- - -### C-4: 404-second SCUC (single-threaded, 32 cores available) - -**Concern:** Single-threaded HiGHS on a 32-core machine. Multi-threaded performance -would likely be 15-30 seconds. The timing as measured is reproducible but does not -represent production capability. -**Evidence quality:** strong - -This is a straightforward timing understatement. The fix is to re-run with -`set_optimizer_attribute(optimizer, "threads", 32)` and record both timings. This probe -is recommended because C-4 is the primary SCUC scalability data point and the single- -threaded constraint is not explained in the result file. - ---- - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | — | Low signal (gate only) | -| G-2 | pass | — | Low signal (gate only) | -| G-3 | pass | — | Low signal (gate only) | -| A-1 | pass | — | DC/AC result type inconsistency noted | -| A-2 | qualified_pass | stable | No NR residual or iteration count exposed | -| A-3 | pass | stable | LMP unit conversion undocumented; not independently verified | -| A-4 | pass | — | Clean DCOPF→ACPF workflow | -| A-5 | qualified_pass | fragile | PSI initialization bypass; internal API for result extraction | -| A-6 | qualified_pass | fragile | ED LMP extraction fails; 1 binding ramp constraint asserted but not shown | -| A-9 | qualified_pass | stable | Only 7/34 contingencies applied; network too radial for SCOPF signal | -| A-10 | fail | blocking | Formulation exists (DCPLLPowerModel) but HiGHS cannot solve its constraint type | -| A-11 | fail | blocking | No formulation exists at any level in PSI or PowerModels | -| A-12 | qualified_pass | fragile | BESS arbitrage condition explicitly fails; masked by qualified_pass | -| B-1 | pass | — | JuMP model access works cleanly | -| B-2 | qualified_pass | stable | Manual BFS over adjacency matrix | -| B-3 | qualified_pass | stable | LODF superposition approximate for M>1 | -| B-4 | pass | stable | System reconstruction per scenario; linear costs produce degenerate LMPs | -| B-5 | pass | — | 2-LOC export | -| B-6 | pass | — | 5-layer architecture documented | -| B-8 | pass | — | Reference bus config works; LMP invariance confirmed | -| B-9 | pass | — | Machine precision on case39 (no phase shifters); claim limited to simple networks | -| C-1 | pass | — | 0.275s; no workarounds | -| C-2 | pass | — | Convergence warning on first call; residual not verified | -| C-3 | qualified_pass | fragile | Branch flow limits removed; effectively unconstrained economic dispatch | -| C-4 | qualified_pass | fragile | 404s single-threaded; 32 cores available; SCIP times out | -| C-5 SMALL | pass | — | Progressive relaxation inapplicable to NR; all levels identical | -| C-5 MEDIUM | pass | — | Same structural issue as SMALL | -| C-7 | pass | fragile | Inherits all 5 C-3 workarounds; solver swap is 1-LOC | -| C-8 | qualified_pass | fragile | SMALL timeout with incumbent; MEDIUM OTHER_ERROR crash | -| C-9 | pass | — | 1.6s; 68.6% dense matrix; orientation is buses×branches (transposed vs convention) | -| C-10 | fail | blocking | Cascaded from A-11 | -| D-1 | informational | — | 19s first-solve; JIT dominant; 92 warning lines on load | -| D-2 | informational | — | 3/10 tests doable from docs; dual unit conversion undocumented | -| D-3 | informational | — | 0/10 unmodified; root cause is PowerSystemCaseBuilder dependency | -| D-4 | informational | — | Infeasibility: best-in-class; zero-cost silent success: concerning | -| D-5 | informational | — | Median 332 LOC; ~6x Python tools | -| E-1 | informational | — | 21 releases/24mo; strong cadence | -| E-2 | informational | — | 1019 commits/12mo; 12 contributors | -| E-3 | informational | — | Bus factor = 1; jd-lara 70.5% commits, 78% reviews | -| E-4 | informational | — | NREL/DOE backing; strongest funding model | -| E-5 | informational | — | Median 19.8 day TTC | -| E-6 | informational | — | 78% test coverage; 8 CI workflows | -| E-7 | informational | — | No utility/ISO production deployment confirmed | -| F-1 | pass | — | BSD-3-Clause core | -| F-2 | pass | — | 184 packages; large but standard for Julia | -| F-3 | pass | — | GLPK GPL-3.0 removable flag | -| F-4 | pass | — | Solver binaries source-available; Yggdrasil SHA-256 | -| F-5 | pass | — | Full execution path inspectable | -| F-6 | pass | — | No GPG signatures; git-tree-sha1 | -| F-7 | pass | — | Air-gap feasible via depot copy | -| F-8 | pass | — | All use cases on open-source solvers | -| F-9 | pass | — | Tutorials use mutable URLs and external downloads | -| G-FNM-1 | fail | blocking | PSS/E v31 parser fails at line 1; MATPOWER fallback used | -| G-FNM-2 | blocked | — | Blocked by G-FNM-1 PSS/E failure | -| G-FNM-3 | fail | — | Simplified B-matrix; 86.8% bus angles outside tolerance | -| G-FNM-4 | informational | — | ACPF non-convergent at all relaxation levels on 28K-bus FNM | -| G-FNM-5 | informational | — | 50% N / 30% E / 20% X across 44 fields | -| P2-1 | informational | — | v33/v35 only; v31 needs 3-6 weeks effort | -| P2-2 | informational | — | Piecewise linear: SOS2 fully supported | -| P2-3 | informational | — | Commitment injection: 5/9 ops need internal APIs | diff --git a/sweep-data/v10-to-v11/per-tool/powersimulations/findings.yaml b/sweep-data/v10-to-v11/per-tool/powersimulations/findings.yaml deleted file mode 100644 index bfebc2c3..00000000 --- a/sweep-data/v10-to-v11/per-tool/powersimulations/findings.yaml +++ /dev/null @@ -1,423 +0,0 @@ -tool: powersimulations -source_version: v10 -timestamp: "2026-03-14T00:00:00Z" -evaluation_summary: - total_tests: 46 - pass: 17 - fail: 5 - qualified_pass: 13 - informational: 11 - -findings: - - id: powersimulations-F01 - category: misleading_result - severity: high - test_ids: [C-3] - title: "DCOPF qualified_pass on MEDIUM masks a fundamentally relaxed problem" - description: >- - C-3 receives a qualified_pass for DCOPF on the 10K-bus MEDIUM network, but one of the - five stacked workarounds removes all branch flow limit constraints (StaticBranchUnbounded - replaces StaticBranch). The result is an uncongested economic dispatch, not a network- - constrained DCOPF. The qualification notation does not convey that the flow limit - removal invalidates the primary measure the test is meant to assess — whether the tool - can enforce thermal ratings at scale. The objective ($3.66M) and solver agreement are - real, but they reflect an unconstrained problem. - evidence: - - file: "evaluations/powersimulations/results/scalability/C-3_dcopf_scale.md" - excerpt: "StaticBranchUnbounded: Replaced StaticBranch with StaticBranchUnbounded for all branch types. Branch flow limit constraints cause numerical infeasibility at 10K scale (basis matrix condition number > 10^15). This removes branch flow limits but preserves DC network topology." - - file: "evaluations/powersimulations/results/scalability/C-3_dcopf_scale.md" - excerpt: "LMP extraction from duals was not successful with StaticBranchUnbounded — the dual structure differs from StaticBranch. Per the cross-tool watchpoint, LMPs should be uniform on ACTIVSg10k (no binding branch constraints at ~84% max loading), so this is not a material gap." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: powersimulations-F02 - category: missing_verification - severity: high - test_ids: [A-2, C-2] - title: "ACPF convergence accepted without residual or iteration count" - description: >- - PowerFlows.jl does not expose Newton-Raphson iteration count or convergence residual - in its public API. Both A-2 (TINY) and C-2 (MEDIUM) infer convergence solely from - non-flat voltage profiles, which is a necessary but not sufficient condition for true - NR convergence. At 10K scale (C-2), the first ACPF call logged a convergence warning - ("NewtonRaphsonACPowerFlow solver failed to converge") but still returned results — - this silent partial-convergence behavior is particularly concerning and is not - flagged in the result status. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-2_acpf.md" - excerpt: "PowerFlows.jl v0.9.0 does not expose Newton-Raphson iteration count or convergence residual in its public return value. Convergence quality is verified indirectly through non-trivial voltage profiles." - - file: "evaluations/powersimulations/results/scalability/C-2_acpf_scale.md" - excerpt: "The warm-start NR solver on the first call logged a convergence warning (NewtonRaphsonACPowerFlow solver failed to converge) but still returned results. On the timed second call, convergence was clean." - cross_tool_relevance: none - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: powersimulations-F03 - category: extraordinary_claim - severity: medium - test_ids: [A-9] - title: "SCOPF 17.7% cost increase claim based on only 7 of 34 contingencies" - description: >- - The A-9 SCOPF result reports a 17.7% cost increase from N-1 security constraints, - but only 7 of 34 line contingencies were applied. Twenty-seven contingencies were - skipped because they produced |LODF| >= 0.95 on at least one monitored line - (near-radial topology). On the 39-bus case39 network, which has limited meshing, - a 7-contingency SCOPF is not a rigorous test of the full N-1 security margin. - The cost increase is real and directionally correct, but the magnitude depends - heavily on which 7 contingencies the feasibility filter admits. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-9_scopf.md" - excerpt: "Contingencies with flow variables: 34 (lines only). Contingencies skipped (radial/near-radial): 27. Contingencies applied: 7. Constraints added: 288." - - file: "evaluations/powersimulations/results/expressiveness/A-9_scopf.md" - excerpt: "Cost comparison: DCOPF (unconstrained) $155,569.55 vs SCOPF (N-1) $183,119.36, Cost increase +17.7%" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: powersimulations-F04 - category: network_insufficiency - severity: medium - test_ids: [A-9] - title: "39-bus network is too radially connected for meaningful SCOPF evaluation" - description: >- - The IEEE 39-bus network has sufficient meshing for basic power flow tests but is - poorly suited for SCOPF evaluation. With 27 of 34 line contingencies immediately - filtered as near-radial (|LODF| >= 0.95), the SCOPF test effectively exercises - only 7 contingency-monitor pairs. A more meshed network (such as the 2K-bus SMALL - tier) would provide more contingencies where the security constraint meaningfully - binds, producing a more credible cost-increase signal. The 39-bus result cannot - distinguish between "good SCOPF capability" and "the network is too small to create - binding security constraints." - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-9_scopf.md" - excerpt: "Contingencies skipped (radial/near-radial): 27. 27 of 34 contingencies produce |LODF| >= 0.95 on at least one monitored line, indicating near-radial topology." - - file: "evaluations/powersimulations/results/expressiveness/A-9_scopf.md" - excerpt: "convergence-quality: 27 of 34 line contingencies produce near-radial redistribution (|LODF| >= 0.95). On the case39 network, only 7 contingencies are non-trivial. This is expected for a small radial-ish network and does not indicate a tool limitation." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: powersimulations-F05 - category: misleading_result - severity: medium - test_ids: [A-12] - title: "A-12 qualified_pass masks BESS arbitrage condition failure with test design gap" - description: >- - A-12 is scored as qualified_pass despite the BESS arbitrage behavioral pass condition - explicitly failing (discharge LMP $35.68 < charge LMP $36.61). The failure is attributed - to linear costs producing degenerate LP LMPs (non-unique at the margin), which is itself - caused by a separate workaround: HiGHS cannot solve multi-period QP (a tool limitation). - The qualified_pass status obscures a fundamental behavioral failure that is tightly - linked to a compounded solver-formulation mismatch. A reader viewing only the status - sees "pass" where the core economic behavior being tested did not demonstrate correctly. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-12_multiperiod_dcopf_storage.md" - excerpt: "Condition 2: BESS Arbitrage (FAIL). Average charge LMP: $36.61/MWh. Average discharge LMP: $35.68/MWh. Discharge LMP is NOT greater than charge LMP." - - file: "evaluations/powersimulations/results/expressiveness/A-12_multiperiod_dcopf_storage.md" - excerpt: "The LP with linear costs produces non-unique LMPs at the margin, which causes the BESS to behave as a net load rather than an arbitrageur. Quadratic costs would resolve this but require a different solver." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: powersimulations-F06 - category: infrastructure_friction - severity: medium - test_ids: [A-10] - title: "A-10 fail is solver-ecosystem mismatch, not a tool formulation gap" - description: >- - The A-10 fail for lossy DCOPF is correctly recorded as a blocking failure, but the - root cause is a mismatch between the protocol's solver constraint (HiGHS) and the - DCPLLPowerModel formulation's requirement for quadratic constraint support (SOCP/QCP). - DCPLLPowerModel exists in PowerModels.jl and is accessible through PSI. The failure - would not occur if the protocol permitted Ipopt (an NLP solver that handles these - constraints) or a commercial solver. This distinguishes PSI from tools that have no - lossy DC formulation whatsoever — but the current fail status does not capture this - nuance. The finding is cross-tool relevant because other JuMP-based tools face the - same solver-formulation compatibility constraint. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-10_lossy_dcopf_lmp.md" - excerpt: "DCPLLPowerModel discovered: PowerModels.jl does provide a DCPLLPowerModel formulation that includes linearized losses. However, this formulation uses quadratic constraints (ScalarQuadraticFunction-in-GreaterThan) in the Ohm's law constraint." - - file: "evaluations/powersimulations/results/expressiveness/A-10_lossy_dcopf_lmp.md" - excerpt: "This requires a solver supporting SOCP or general quadratic constraints (e.g., Gurobi, CPLEX, or potentially SCIP). The evaluation protocol specifies HiGHS as the solver." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_test - - - id: powersimulations-F07 - category: missing_verification - severity: medium - test_ids: [A-3, B-1] - title: "LMP values produced with undocumented unit conversion not independently verified" - description: >- - Both A-3 (DCOPF LMPs) and B-1 (custom constraint dual) require an undocumented - unit conversion: dividing raw dual values by base_power (100 MVA) and negating. The - D-2 documentation audit confirms this conversion is not documented anywhere in the - official PSI documentation. The LMP values reported ($7.76-$290.11/MWh) are not - independently verified against a reference solution or an alternative extraction - path. A 100x error in either direction (missing base_power division) would produce - plausible-looking but wrong LMPs, and the result files do not show the raw dual - values alongside the converted values to allow independent checking. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-3_dcopf.md" - excerpt: "The raw dual values are in internal units (per-unit basis); conversion to $/MWh requires dividing by base_power (100 MVA) and negating per shadow price sign convention. unit-mismatch: read_variable returns MW but read_dual returns per-unit-based values requiring manual division by base_power and negation — undocumented." - - file: "evaluations/powersimulations/results/accessibility/D-2_documentation_audit.md" - excerpt: "Dual unit conversion (divide by base_power, negate): No — Discovered empirically. A user following only official docs would not produce correct LMP values." - cross_tool_relevance: none - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: powersimulations-F08 - category: test_design_gap - severity: medium - test_ids: [A-6] - title: "A-6 SCED ramp enforcement verified by constraint count, not by binding evidence" - description: >- - The A-6 SCED test claims ramp constraints are "demonstrably enforced" and reports - "1 binding ramp constraint observed." However, verification consists only of counting - the added constraints (460 added) and asserting 1 is binding, without showing the - actual ramp-limited generator, the before/after dispatch values, and the constraint - dual value. The protocol requires ramp constraints to be "demonstrably enforced between - consecutive dispatch intervals" — a single binding constraint (out of 460 added) with - no contextual output is weak evidence. Additionally, the LMP extraction for the ED stage - failed ("returned null values"), further limiting the quality of evidence. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-6_sced.md" - excerpt: "Ramp constraints: 460 added (2 per generator per inter-hour transition). 1 binding ramp constraint observed, confirming ramp limits are enforced in ED independently of UC." - - file: "evaluations/powersimulations/results/expressiveness/A-6_sced.md" - excerpt: "LMP extraction: LMPs from ED stage returned null values. The initialize_model=false + JuMP.optimize!() bypass prevents PSI's dual tracking from populating." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: powersimulations-F09 - category: scoring_inconsistency - severity: medium - test_ids: [C-8, A-9] - title: "C-8 SCOPF qualified_pass despite solver crash on MEDIUM" - description: >- - C-8 receives a qualified_pass despite HiGHS crashing with OTHER_ERROR on the MEDIUM - (10K-bus) network — meaning the test literally did not complete on the grade_network. - The SMALL result timed out (TIME_LIMIT, with incumbent but no optimality proof). By - comparison, A-9 (SCOPF on TINY) receives a qualified_pass for a tool that has no - built-in SCOPF and produces results only via manual LODF constraint injection. The - scoring pattern suggests qualified_pass is applied too broadly — a solver crash on - the grade_network (MEDIUM, as specified in eval-config.yaml) should arguably be - classified as a fail or at least clearly distinguished from a timeout with incumbent. - evidence: - - file: "evaluations/powersimulations/results/scalability/C-8_scopf_scale.md" - excerpt: "MEDIUM finding: HiGHS encountered an internal error (OTHER_ERROR) after 438 seconds on the MEDIUM problem. The LP has 24,113 variables and 539,661 constraints." - - file: "evaluations/powersimulations/results/scalability/C-8_scopf_scale.md" - excerpt: "Solver outcome: SMALL: TIME_LIMIT (incumbent), MEDIUM: OTHER_ERROR" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: powersimulations-F10 - category: extraordinary_claim - severity: medium - test_ids: [B-9] - title: "Machine-precision PTDF claim (1.15e-14 error) may not generalize to networks with phase shifters" - description: >- - B-9 reports machine-precision PTDF verification with max error 1.15e-14 p.u. on the - 39-bus network. The result notes "IEEE 39-bus has no phase-shifting transformers - (all SHIFT=0)" and explicitly states no phase-shifter correction is needed. However, - C-9 (PTDF at MEDIUM scale) finds PTDF values outside [-1,1] on the 10K-bus network - (due to 776 non-unity tap transformers) and documents that the protocol requires - Pbusinj/Pfinj corrections for networks with phase shifters. The B-9 result provides - strong validation on a simple network but cannot establish that the claimed precision - holds on real-world networks with off-nominal taps — which is where PTDF accuracy - matters most for SCOPF applications. - evidence: - - file: "evaluations/powersimulations/results/extensibility/B-9_ptdf_extraction.md" - excerpt: "Max error: 1.15e-14 p.u. (1.15e-12 MW). All 46 branches match to machine precision. IEEE 39-bus has no phase-shifting transformers (all SHIFT=0 in MATPOWER branch data)." - - file: "evaluations/powersimulations/results/scalability/C-9_ptdf_scale.md" - excerpt: "Values outside [-1, 1] indicate the presence of phase-shifting or tap-changing transformers (PTDF entries can exceed unity when phase shifters are present). The ACTIVSg10k network contains 970 TapTransformers, of which 776 have non-unity tap ratios." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_test - - - id: powersimulations-F11 - category: misleading_result - severity: low - test_ids: [G-FNM-3] - title: "G-FNM-3 formulation-difference classification understates the practical impact" - description: >- - The G-FNM-3 DCPF failure (86.8% of bus angles outside 1 degree tolerance) is - classified as a formulation-difference, which is technically accurate: PowerFlows.jl - uses a simplified B-matrix that ignores transformer tap ratios. However, calling this - a formulation difference implies it is a methodological choice rather than a limitation. - In practice, the 2,340 off-nominal tap transformers in the FNM are essential to realistic - power flow computation — and there is no configuration option in PowerFlows.jl to use - the full B-matrix. The classification correctly avoids calling it a bug, but framing - it as a "formulation difference" in the synthesis may understate the practical consequence - for any workflow that needs DCPF accuracy on real-world transmission networks. - evidence: - - file: "evaluations/powersimulations/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md" - excerpt: "PowerFlows.jl uses a simplified B-matrix that ignores transformer tap ratios, while the MATPOWER reference uses the full B-matrix. With 2,340 off-nominal tap transformers in the network, this produces systematic angle deviations (mean 2.66 deg)." - - file: "evaluations/powersimulations/results/fnm_ingestion/G-FNM-3_fnm_dcpf_verification.md" - excerpt: "There is no configuration option in PowerFlows.jl to switch to a full B-matrix for DCPF." - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: null - - - id: powersimulations-F12 - category: missing_verification - severity: low - test_ids: [A-5] - title: "SCUC cycling claim verified by schedule inspection, not economic logic" - description: >- - A-5 reports 3 cycling generators (gen-5, gen-7, gen-10) and MIP gap 0.57%. The cycling - is verified by inspecting the commitment schedule array. However, the protocol also - specifies that the network "must have enough generators with different cost curves to - force unit commitment cycling." The cycling here is primarily driven by the load valley - (4,237 MW) vs peak (6,254 MW) amplitude. It is not verified whether cycling would - still occur if the load profile had less amplitude variation, or whether the 10 generators - and their cost structure genuinely force the commitment cycling through cost optimization - rather than capacity constraints alone. - evidence: - - file: "evaluations/powersimulations/results/expressiveness/A-5_scuc.md" - excerpt: "Cycling generators (3): gen-5 (Coal), gen-7 (Gas CC), gen-10 (Gas CC). The two gas CC units (most expensive) cycle on for peak hours and off during low-load hours, consistent with economic dispatch logic." - - file: "evaluations/powersimulations/results/expressiveness/A-5_scuc.md" - excerpt: "System load profile: 4,237 MW (valley, HR 4) to 6,254 MW (peak, HR 18)." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: powersimulations-F13 - category: infrastructure_friction - severity: low - test_ids: [D-3] - title: "0/10 example pass rate inflated by dependency on external PowerSystemCaseBuilder package" - description: >- - D-3 reports that 0 of 10 official examples run unmodified. However, all PSI tutorials - fail at the data-loading step because they require PowerSystemCaseBuilder.jl — a - separate package not included in a standard PSI installation. This is an ecosystem - design choice (separating test data from the core package) rather than a tutorial - quality failure. One tutorial (PowerFlows.jl DCPF) runs correctly with a one-line - data source substitution. The 0/10 figure accurately reflects the out-of-box - experience but conflates an infrastructure packaging decision with tutorial quality. - Cross-tool comparison using this metric will disadvantage Julia-ecosystem tools - systematically. - evidence: - - file: "evaluations/powersimulations/results/accessibility/D-3_example_verification.md" - excerpt: "Both tutorials begin with: using PowerSystemCaseBuilder; sys = build_system(PSITestSystems, ...). PowerSystemCaseBuilder is a separate package that downloads pre-built test systems from NREL's data repository. It is not a dependency of PowerSimulations.jl." - - file: "evaluations/powersimulations/results/accessibility/D-3_example_verification.md" - excerpt: "Summary: PSI tutorials: 0/2 run unmodified. PowerFlows tutorial: 0/1 unmodified but 1/1 with fixes." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: powersimulations-F14 - category: test_design_gap - severity: low - test_ids: [C-5] - title: "C-5 progressive relaxation test has no discriminative value for ACPF convergence" - description: >- - The C-5 protocol applies progressive thermal limit relaxation (0%, 10%, 20%) as a - diagnostic for ACPF convergence. For PowerSimulations.jl (using PowerFlows.jl NR), - all three relaxation levels converge to identical voltage profiles because thermal - limit relaxation only affects OPF branch constraints, not the NR power flow equations. - The result correctly documents this: "thermal limit relaxation only affects OPF branch - constraints, not the NR power flow equations." The protocol design assumes that thermal - limit relaxation helps convergence (appropriate for OPF-based ACPF using Ipopt), but - is meaningless for direct NR solvers that do not enforce thermal limits during the - power flow solve. The test measures convergence capability but cannot be used to - compare across tools that use fundamentally different ACPF approaches. - evidence: - - file: "evaluations/powersimulations/results/scalability/C-5_ac_feasibility_progressive_MEDIUM.md" - excerpt: "Key finding: ACPF converges at 0% relaxation (nominal thermal limits) on MEDIUM. No progressive relaxation was needed. All three relaxation levels converge to the same voltage profile, consistent with SMALL behavior — thermal limit relaxation only affects OPF branch constraints, not the NR power flow equations." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: powersimulations-F15 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate ingest tests are pass/fail with no differentiation between tools" - description: >- - G-1, G-2, and G-3 test MATPOWER file ingestion at TINY, SMALL, and MEDIUM scale. - All tools that survived this phase of evaluation pass these tests — they establish - a floor (can the tool ingest a MATPOWER file?) rather than differentiating tool - capability. The tests are necessary as gates but contribute no comparative signal. - The PowerSimulations.jl result correctly reports the 6.44s load time for case39, - but this is dominated by Julia JIT compilation rather than parser performance, which - is not tool-comparable without JIT normalization. - evidence: - - file: "evaluations/powersimulations/results/gate/G-1_ingest_tiny.md" - excerpt: "Result: PASS. Actual counts: 39/46/10. Load time: 6.44s." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: powersimulations-F16 - category: extraordinary_claim - severity: medium - test_ids: [C-4] - title: "SCUC 404s wall-clock measured at single-threaded; actual production capability is potentially 20-30x better" - description: >- - C-4 reports HiGHS solving the 2K-bus 24h SCUC (22,608 binary variables) in 404 seconds. - The result notes CPU threads = 1 explicitly: "CPU cores used: 1 (32 available)." Modern - MILP solvers including HiGHS scale near-linearly with thread count on branch-and-bound - problems. A 32-thread run would likely reduce solve time to 15-30 seconds, potentially - changing the grade interpretation for SCUC scalability. The single-threaded measurement - is reproducible but unrepresentative of practical deployment. The result file does not - discuss whether multi-threaded HiGHS was attempted or why it was restricted to 1 thread. - evidence: - - file: "evaluations/powersimulations/results/scalability/C-4_scuc_scale.md" - excerpt: "Termination status: OPTIMAL. Wall-clock: 404.2 s. MIP gap: 0.93%. Threads: 1. CPU cores used: 1 (32 available)." - - file: "evaluations/powersimulations/results/scalability/C-4_scuc_scale.md" - excerpt: "HiGHS solved to 0.93% optimality in 404 seconds... the 22,608 binary variables represent a substantial MILP problem." - cross_tool_relevance: confirmed - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - -extraordinary_claims: - - test_id: A-9 - claim: "SCOPF produces 17.7% cost increase from N-1 contingency constraints on 39-bus network" - concern: >- - Only 7 of 34 contingencies are applied after filtering. The 17.7% cost increase - is real but measured against a minimal contingency set on a small, radially-connected - network. The claim implies robust N-1 security enforcement but the test exercises - only 20% of available contingencies. - evidence_quality: moderate - probe_recommended: false - probe_type: formulation_audit - - - test_id: B-9 - claim: "PTDF matrix achieves machine precision (max error 1.15e-14 p.u.) verified against DCPF" - concern: >- - Machine-precision verification is on the 39-bus case with no phase-shifting - transformers. The C-9 PTDF at 10K scale shows PTDF values outside [-1, 1] due - to tap transformers, with no flow accuracy check performed. The precision claim - may not hold at practical scale. - evidence_quality: moderate - probe_recommended: false - probe_type: claim_verification - - - test_id: C-4 - claim: "SCUC on 2K-bus network solves in 404 seconds at 0.93% MIP gap" - concern: >- - Single-threaded measurement on a 32-core machine. Multi-threaded HiGHS would - likely achieve this in 15-30 seconds. The 404s figure could mislead cross-tool - comparisons if other tools use multi-threading by default. - evidence_quality: strong - probe_recommended: true - probe_type: timing_verification diff --git a/sweep-data/v10-to-v11/per-tool/pypsa/findings.md b/sweep-data/v10-to-v11/per-tool/pypsa/findings.md deleted file mode 100644 index e1a9df00..00000000 --- a/sweep-data/v10-to-v11/per-tool/pypsa/findings.md +++ /dev/null @@ -1,290 +0,0 @@ -# pypsa — Sweep Findings (v10) - -## Summary - -The PyPSA v10 evaluation is well-executed: all 59 test IDs have result files, timings are measured (not estimated), convergence residuals are reported for all Newton-Raphson solves, and workaround classifications are consistently applied. The primary structural issues are a protocol design flaw (the C-SMALL gate conflates MILP and LP/PF scalability, causing 7 MEDIUM tests to be skipped despite strong LP/PF evidence), a test design gap in B-8 (slack reconfiguration cannot produce LMP variation in any standard DC OPF), and a scoring ambiguity in A-11 (qualified_pass with blocking workaround is functionally a fail for the stated test condition). One extraordinary claim — 0.0 deviation on a 27,862-bus DCPF — warrants a probe both for numerical verification and to resolve a discrepancy between the result file and the validation report. Two probes are recommended: one for the G-FNM-3 deviation claim and one for the A-10 LMP decomposition completeness. - -## Finding Details - -### pypsa-F01: B-8 slack reconfiguration produces zero LMP variation — pass condition is vacuous for DC OPF - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** B-8 - -B-8 requires solving DC OPF with three different slack bus configurations and checking that "LMP values change consistently across configurations." The result reports zero LMP variation across all three configurations (objective spread = 0.0, LMP shift mean = 0.0, std = 0.0). The evaluator correctly explains this is mathematically expected: in a standard LP-based DC OPF, the dual variables (LMPs) are invariant to the angle reference bus choice because the formulation uses nodal power balance constraints, not angle-based constraints. The angle reference only affects the primal solution (bus angles), not the dual (LMPs). - -This means the B-8 pass condition as written — "LMP values change consistently" — is unsatisfiable for any tool implementing standard DC OPF. Every tool will pass B-8 on API configurability (the call succeeds, no model reconstruction needed) and fail on LMP variation (because there is none). The test currently measures only whether the API call succeeds, not whether the slack configuration has any computational consequence. - -The result file says: "Objective spread: 0.0000 (identical across all three configs) / LMP spread variation: 0.0000 (identical across all three configs)" and explains the mathematical reason, which is correct. - -**Cross-tool relevance:** confirmed — this is a protocol design issue that affects all tools implementing standard LP-based DC OPF. - -**Proposed action:** redesign_test — either use DCPF (not DC OPF) where the slack bus absorbs power imbalance and produces angle differences, or rewrite the pass condition to check only API configurability (two DataFrame assignments, no model reconstruction, no error raised) without requiring LMP variation. - ---- - -### pypsa-F02: A-5 SCUC on 10-generator TINY network cannot verify that min up/down constraints are binding - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-5 - -A-5 passes the numeric threshold (3 generators cycle, ≥2 required) on the 10-generator IEEE 39-bus with a capacity-to-peak ratio of 1.18. However, the result cannot confirm that min_up_time or min_down_time constraints are actually binding the commitment schedule. With only 10 generators across 4 cost tiers and 18% capacity headroom, the optimizer has substantial freedom to schedule generators without the min time constraints ever being the binding constraint. The cycling pattern shown (G9 shuts down hours 3-8, restarts hour 9, shuts down again hour 22; G6 shuts down hour 20; G3 shuts down hour 23) appears driven by economic merit, not minimum time constraints — the decommits happen over single-digit hour periods that are within the min up/down windows. - -The protocol's intent is to verify that the SCUC formulation is correctly implemented. A claim that min up/down time constraints are active requires either (a) a network with tighter capacity margins forcing the optimizer to keep generators online longer than economics alone would dictate, or (b) an explicit check that removing the min time constraints changes the schedule. - -The result states: "Capacity-to-peak-load ratio: 1.18. Load range: 4,237--6,254 MW." With 10 generators totaling 7,367 MW capacity against a 6,254 MW peak, economic dispatch can always find a valid solution without straining minimum time constraints. - -**Cross-tool relevance:** confirmed — a 10-generator, 39-bus network is too small to stress min up/down time constraints for any tool. This affects all tools' A-5 results. - -**Proposed action:** add_verification — add a supplementary check confirming at least one min up/down time constraint is binding in the A-5 solution, implemented by re-running with min_up_time=min_down_time=1 and comparing against min_up_time=min_down_time=0 to show a schedule difference. - ---- - -### pypsa-F03: A-10 check (d) verifies LMP change, not the three-component additive decomposition required by protocol - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-10 - -The A-10 pass condition requires: "(d) loss component LMPs sum with energy and congestion components to total LMP within 1% tolerance." The result checks four conditions but implements check (d) as "Bus LMPs change between lossy and lossless: PASS (38/39 buses)" — a two-scenario comparison, not a three-component additive decomposition. - -The protocol requires per-bus verification that energy_component + congestion_component + loss_component = total_LMP within 1%. The result provides a two-component decomposition (energy = slack bus LMP, loss = bus LMP - energy), but never isolates the congestion component separately. The congestion rent table shows "Lines with non-zero congestion rent: 35 of 35" but does not verify that these congestion rents, when decomposed to the bus level, add to a congestion LMP component that participates in the three-way additivity check. - -The result file says: "| (d) Bus LMPs change between lossy and lossless | PASS (38/39 buses) |" — this is a different and weaker condition than what the protocol specifies. - -**Cross-tool relevance:** likely — the three-component LMP decomposition check is the same protocol requirement for all tools solving A-10. If this verification was not done for PyPSA, it may not have been done for other tools either. - -**Probe recommended:** claim_verification — compute per-bus loss component as (total LMP - energy component - congestion component) using the PTDF matrix and shadow prices, and verify the three sum to within 1% of total LMP for each bus. If PyPSA's `transmission_losses` formulation does not expose the congestion and loss components separately, that finding should be recorded. - ---- - -### pypsa-F04: A-9 SCOPF protocol specifies all 46 branches but test runs with 19 — reduced contingency set may be insufficient - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-9 - -The A-9 protocol specifies "DC OPF with N-1 contingency flow constraints on TINY (all 46 branches)." The result documents a progressive fallback: (1) all 35 lines → infeasible at any derating; (2) lines <70% utilization → infeasible; (3) lines <50% utilization → feasible with 19 lines. The test passes on this 19-line subset. - -Two distinct issues compound here. First, PyPSA's `optimize_security_constrained()` API does not accept Transformer names, so 11 of the 46 branches (the transformers) are excluded regardless of feasibility. This is a genuine tool limitation. Second, the feasibility reduction from 35 to 19 lines is a consequence of the IEEE 39-bus network topology: 2 lines are already at 100% utilization in the base DC OPF, and any contingency set including them creates unavoidable infeasibility. No tool could pass A-9 as specified (all 46 branches) on this network, because the network itself does not have a feasible N-1 security-constrained dispatch when all branches are contingency candidates. - -The test as designed cannot distinguish between "tool handles SCOPF well" and "network happens to have feasible SCOPF with the contingency subset tried." The result says: "The test used a progressive fallback strategy: all 35 lines -> lines <70% utilization -> lines <50% utilization. The SCOPF became feasible with 19 lines at <50% base-case utilization." - -**Cross-tool relevance:** confirmed — this network topology characteristic affects all tools evaluating A-9. - -**Proposed action:** redesign_test — pre-identify a contingency subset that is guaranteed to produce a feasible SCOPF on IEEE 39-bus (e.g., branches not in the active binding set at base-case OPF) and specify that subset as the protocol-required contingency set for A-9. - ---- - -### pypsa-F05: G-FNM-1 failure blocks G-FNM-2, preventing field coverage assessment despite a complete MATPOWER fallback path - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** G-FNM-1, G-FNM-2 - -G-FNM-1 fails because PyPSA lacks a PSS/E ingestion path. This correctly triggers G-FNM-2 to be skipped. However, G-FNM-5 (supplemental CSV representability) proceeds via the MATPOWER fallback and produces a complete field coverage analysis (73 fields, 20.5% N / 61.6% E / 17.8% X). The protocol gates G-FNM-2 on G-FNM-1, meaning field coverage can only be assessed for tools that support PSS/E ingestion. - -The PSS/E ingestion capability is a format interoperability feature. The field coverage measured in G-FNM-2 is a data model fidelity feature. These are independent: a tool can have a complete data model for all DCPF-critical fields without a PSS/E parser (which PyPSA demonstrates via G-FNM-5). The current gate design systematically disadvantages tools with complete data models but no PSS/E parser by withholding the G-FNM-2 measurement. - -G-FNM-5 proceeds to report "61.6% extension-representable" fields — the same domain knowledge that G-FNM-2 would assess at the DCPF-critical vs. ACPF-critical level. The distinction is that G-FNM-5 uses the supplemental CSVs, while G-FNM-2 uses the core network tables. The result is an information gap that is attributable to protocol design, not tool capability. - -**Cross-tool relevance:** likely — any tool without PSS/E ingestion (PowerModels.jl, pandapower without pypsa-converter, etc.) would face the same gate-induced gap in G-FNM-2 coverage. - -**Proposed action:** add_test — allow G-FNM-2 to run via the MATPOWER fallback path for tools that fail G-FNM-1, with a note that PSS/E-specific field mappings cannot be assessed. The DCPF-critical fields (19 fields) in the FNM are derivable from the MATPOWER .m file regardless of PSS/E ingestion support. - ---- - -### pypsa-F06: G-FNM-3 reports 0.0 deviation on 27,862-bus DCPF and has discrepancy with validation report - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** G-FNM-3 - -G-FNM-3 reports perfect 0.0 agreement (mean and max deviation for both bus angles and branch flows) between PyPSA and the MATPOWER reference on a 27,862-bus network. The reported values are: mean deviation = 0.0 deg, max deviation = 0.0 deg (buses); mean deviation = 0.0%, max deviation = 0.0% (branches). - -Two concerns arise. First, the numerical claim: exact 0.0 agreement across 27,862 buses and 32,532 branches in a floating-point computation requires that both tools use identical B-matrix construction and identical sparse linear algebra. The evaluator's explanation — that both PyPSA and MATPOWER trace to the same formulation (b = 1/(x*tap)) and the shared matpower_loader applies the same branch status patch — is plausible. The 0.0 result would be expected if both tools solve the exact same linear system with the same coefficient matrix. However, the result tables report to one decimal place, so "0.0" could mean any deviation < 0.05 deg or < 0.05%. The raw numerical maximum is not reported. - -Second, the metadata discrepancy: `validation-report.md` lists G-FNM-3 in the Failures table: "G-FNM-3 | DCPF verification failed — systematic impedance conversion differences via MATPOWER fallback." The G-FNM-3 result file has `status: pass` and the synthesis section on G-FNM-3 also records pass. The synthesis methodology notes include: "G-FNM-3 DCPF deviation resolved — Root cause identified: import_from_pypower_ppc ignores MATPOWER BR_STATUS column, including 74 inactive branches in the DCPF." This indicates the validation report was written before the branch status bug was identified and fixed, and was not regenerated after the result was updated. The current state of the result file appears to reflect the corrected evaluation, but the validation report creates ambiguity. - -**Cross-tool relevance:** none — this is specific to PyPSA's implementation. - -**Probe recommended:** convergence_check — re-run the G-FNM-3 DCPF and extract the raw (unrounded) maximum absolute bus angle deviation and maximum branch flow percent deviation. Confirm whether "0.0" is machine-zero or display-rounded. Also verify validation-report.md reflects the current result state. - ---- - -### pypsa-F07: A-11 qualified_pass with blocking workaround is functionally a fail for the stated test condition - -**Category:** misleading_result | **Severity:** low -**Tests:** A-11 - -A-11's pass condition requires: "Solve DC OPF on TINY with distributed slack (load-proportional)." The result confirms: "No Bus-v_ang variable. Distributed slack OPF is NOT achievable." The workaround_class is "blocking" — explicitly noted as architecturally impossible. - -The result is classified qualified_pass because distributed slack works in the AC power flow context (n.pf(distribute_slack=True)). The result correctly demonstrates this capability. However, the test's pass condition is specifically DC OPF with distributed slack. Demonstrating a different capability (AC PF distributed slack) under different solver conditions does not satisfy the stated pass condition. - -The qualified_pass label implies the tool partially meets the requirement, when in fact the requirement is unmet and a related but distinct capability is offered as context. This creates a misleading impression in cross-tool comparison: A-11 counts as neither a clean fail nor a clean pass, potentially inflating PyPSA's expressiveness score relative to a tool that honestly fails the test without offering a partial substitute. - -The synthesis agrees this is a gap: "The meaningful gap is A-11 (distributed slack OPF) which is a blocking architectural limitation — no workaround exists in the OPF context." But the status is still qualified_pass, not fail. - -**Cross-tool relevance:** none — this is specific to PyPSA's scoring. - -**Proposed action:** adjust_scoring — reclassify A-11 as fail (the protocol's pass condition is not met). Document the AC PF distributed slack capability as an observation in the result file, which is appropriate context but not a pass signal. - ---- - -### pypsa-F08: A-12 branch shadow price extraction depends on internal linopy naming — bug-dependent pass - -**Category:** missing_verification | **Severity:** low -**Tests:** A-12 - -A-12 passes by extracting branch shadow prices via `n.model.constraints['Line-fix-s-upper'].dual` rather than the documented `n.lines_t.mu_upper` attribute. The evaluator classifies this as fragile and documents the underlying bug: "The solver log confirms shadow prices were computed ('shadow-prices of the constraints ... were not assigned to the network')." - -Pass condition 1 (congestion reporting: ≥2 branches with non-zero shadow prices in ≥2 of 24 hours) depends entirely on these extracted duals. The result shows "Hours with >=2 binding branches: 24/24" — strong performance. But this result rests on working around a documented bug using an undocumented internal naming convention. If the constraint naming changes in PyPSA 1.2.x, A-12 would silently fail on shadow price extraction while appearing to pass on BESS arbitrage (condition 2) and SoC feasibility (condition 3). - -The same bug is documented in A-3 (shadow price extraction), A-6 (SCED), and the cross-cutting observations section. This is a systematic tool defect that affects multiple tests, not a single-test workaround choice. - -**Cross-tool relevance:** none — this is specific to PyPSA's shadow price assignment bug. - -**Proposed action:** add_verification — document explicitly that A-12's pass on condition 1 depends on a bug workaround. The bug (n.lines_t.mu_upper empty after optimize()) is a tool defect that should be tracked as a separate observation, distinct from the workaround's success. - ---- - -### pypsa-F09: C-SMALL gate triggered by MILP failure blocks 7 LP/PF scalability tests - -**Category:** test_design_gap | **Severity:** medium -**Tests:** C-4, C-1, C-2, C-3, C-7, C-8, C-9, C-10 - -The C-SMALL gate design requires C-4 (SCUC/MILP on 2,000-bus SMALL) to pass before running MEDIUM-tier tests. C-4 fails for PyPSA: HiGHS cannot solve the root LP relaxation of 39,168-variable MILP within 600 seconds. This cascades to skip 7 MEDIUM tests: C-1 (DCPF), C-2 (ACPF), C-3 (DC OPF), C-7 (solver swap), C-8 (SCOPF), C-9 (PTDF), C-10 (distributed slack). - -The 7 skipped tests cover LP and linear algebra problems — not MILP. PyPSA's LP/PF scalability is demonstrably strong from other evidence: C-5 passes ACPF on 10,000-bus MEDIUM in 19s, and G-FNM-3 passes DCPF on 27,862-bus LARGE in 31s. The gate design prevents collecting this evidence in the scalability dimension. - -The MILP scalability ceiling is a real limitation. But conflating MILP scalability (C-4) with LP/PF scalability (C-1, C-2, C-3, C-7, C-9, C-10) in a single gate means that a tool's overall scalability assessment is determined by its weakest problem type, even when the other problem types scale well. The synthesis note says: "C+ is assigned rather than B- because the C-SMALL-gate prevents demonstrating MEDIUM-tier OPF capability." This is a protocol-driven grade depression that is not attributable to any tool failing a test it could have passed. - -The result says: "C-4 SCUC SMALL fails: HiGHS cannot solve root LP relaxation of 544-generator 24hr SCUC within 600s on single thread (39,168 binary variables)." - -**Cross-tool relevance:** confirmed — this gate design issue would affect any tool that fails MILP at SMALL scale but succeeds at LP/PF at MEDIUM or LARGE scale. - -**Proposed action:** redesign_test — separate the MILP gate (C-4) from the LP/PF gate. LP/PF MEDIUM tests (C-1, C-2, C-3, C-7, C-9, C-10) should not be conditional on MILP SMALL success. Only a MILP MEDIUM test (if one exists) should be gated on C-4. - ---- - -### pypsa-F10: G-FNM-1 determination based on API inspection without code execution - -**Category:** missing_verification | **Severity:** low -**Tests:** G-FNM-1 - -The G-FNM-1 result states: "No test script was written. The verification was an API surface inspection confirming the absence of any PSS/E ingestion path in PyPSA's public interface." The conclusion is correct — PyPSA has no PSS/E parser. However, the protocol requires recording a failure_reason (psse_parse_error), which implies an actual parse attempt was made and produced an error. - -An API inspection confirms the absence of the capability but does not produce an execution trace or a concrete error message that would confirm the exact failure mode. For comparison, G-FNM-3 and G-FNM-4 both produce actual execution records with timing and error messages. G-FNM-1 is recorded as a conclusion from inspection, not as an executed result. - -This is a minor deviation from protocol standards. The conclusion is not in question. - -**Cross-tool relevance:** none. - -**Proposed action:** add_verification — a minimal one-line script attempting import_from_csv_folder on the intermediate CSV directory would produce an actual error message and constitute a proper execution record, consistent with the protocol's failure documentation standard. - ---- - -### pypsa-F11: Gate ingestion tests G-1/G-2/G-3 are low-signal — pass/fail outcomes are predictable - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -The gate tests verify only that a tool loads MATPOWER .m files and returns correct counts. These tests would be passed by any tool with a MATPOWER importer. The only signal is in data quality notes — specifically, G-1 notes that PyPSA's import_from_pypower_ppc silently drops generator cost data. This finding is meaningful but is not part of the gate pass/fail outcome. - -G-1 says: "Generator cost data NOT imported — import_from_pypower_ppc does not support gencost." This affects every downstream OPF test (the shared loader's gencost patch corrects this), but the gate test passes regardless. - -**Cross-tool relevance:** confirmed — this is a protocol design issue. The gate tests serve their minimum-bar purpose but provide no discriminative information beyond pass/fail. - -**Proposed action:** add_verification — gate tests should include a data quality checklist (cost data imported, branch flow limits populated, slack bus identified, NaN check) as part of the pass condition. This would make G-1's gencost finding a gated finding rather than an incidental note. - ---- - -### pypsa-F12: qualified_pass status applied to both stable-workaround (A-6) and blocking-limitation (A-11) cases - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** A-6, A-11 - -Both A-6 and A-11 receive qualified_pass status. The workaround_class values differ: "stable" for A-6 (two-stage UC/ED via documented API, ~15 LOC) and "blocking" for A-11 (distributed slack DC OPF architecturally impossible). These represent substantially different degrees of capability gap, but the same outcome label. - -A-6 is a genuine partial pass: the required two-stage UC/ED workflow is achievable using documented public API, just without a convenience method. A-11 is a fail in the OPF context: the required capability (distributed slack DC OPF) cannot be achieved by any workaround because the model lacks bus angle variables. The synthesis correctly describes this distinction but both map to qualified_pass in the status field. - -This inconsistency propagates to cross-tool aggregation where qualified_pass outcomes may be treated uniformly. - -**Cross-tool relevance:** confirmed — the same status ambiguity would affect other tools receiving qualified_pass with blocking workarounds. - -**Proposed action:** adjust_scoring — either introduce distinct status values (qualified_pass vs. capability_gap) or require workaround_class=blocking to map to fail rather than qualified_pass in the grading logic. - ---- - -## Extraordinary Claims - -### G-FNM-3: 0.0 mean and max deviation on 27,862-bus DCPF - -**Concern:** Perfect 0.0 numerical agreement in a floating-point computation at this scale is unusual. Display precision (one decimal place) may round values <0.05 to zero. Additionally, the validation report lists G-FNM-3 as a failure with "systematic impedance conversion differences via MATPOWER fallback," conflicting with the result file's pass status. - -**Evidence quality:** moderate — the explanation (identical B-matrix formulation) is plausible and would predict 0.0 agreement, but the raw unrounded deviations are not reported. - -The probe should: (1) extract raw maximum absolute angle deviation and maximum branch flow percent deviation without display rounding, (2) confirm that the validation report's failure record reflects a superseded intermediate result rather than the current evaluation state. If the deviation is truly machine-zero, this is a strong positive finding (both tools trace to the same B-matrix formulation). If it is display-rounded from a non-zero value, the threshold checks (95% of buses within 1.0 deg, 90% of branches within 10%) may still pass but the "0.0" characterization is misleading. - ---- - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | — | gencost silently dropped (note, not gate failure) | -| G-2 | pass | — | — | -| G-3 | pass | — | — | -| A-1 | pass | — | — | -| A-2 | pass | — | — | -| A-3 | pass | stable | shadow price extraction via linopy internals (bug workaround) | -| A-4 | pass | — | requires two loader paths (DC vs AC); not a workaround | -| A-5 | pass | — | min up/down binding not verified (pypsa-F02) | -| A-6 | qualified_pass | stable | no fix_commitment() API; manual bound manipulation | -| A-9 | pass | — | only 19/46 branches used; transformer contingencies excluded (pypsa-F04) | -| A-10 | pass | — | three-component LMP decomposition check not fully verified (pypsa-F03) | -| A-11 | qualified_pass | blocking | distributed slack DC OPF architecturally impossible; AC PF credited instead (pypsa-F07) | -| A-12 | pass | fragile | branch shadow prices via internal linopy naming (bug workaround) (pypsa-F08) | -| B-1 | pass | — | — | -| B-2 | pass | — | — | -| B-3 | pass | — | load loss = 0 (expected for unconstrained DCPF, documented) | -| B-4 | pass | — | — | -| B-5 | pass | — | — | -| B-6 | pass | — | — | -| B-8 | pass | — | LMPs identical across all slack configs (mathematically expected; pass condition vacuous) (pypsa-F01) | -| B-9 | pass | — | — | -| C-4 | fail | — | HiGHS MILP timeout (39,168 binary vars); SCIP not installed | -| C-5 (SMALL) | pass | — | converges 0% relaxation; 4 NR iterations | -| C-5 (MEDIUM) | pass | — | converges 0% relaxation; 5 NR iterations; 2099 MB peak memory | -| C-1 | skip | C-SMALL-gate | LP/PF unnecessarily blocked by MILP gate (pypsa-F09) | -| C-2 | skip | C-SMALL-gate | LP/PF unnecessarily blocked by MILP gate (pypsa-F09) | -| C-3 | skip | C-SMALL-gate | LP/PF unnecessarily blocked by MILP gate (pypsa-F09) | -| C-7 | skip | C-SMALL-gate | LP/PF unnecessarily blocked by MILP gate (pypsa-F09) | -| C-8 | skip | C-SMALL-gate | blocked by MILP gate (pypsa-F09) | -| C-9 | skip | C-SMALL-gate | LP/linear algebra unnecessarily blocked by MILP gate (pypsa-F09) | -| C-10 | skip | C-SMALL-gate | blocked by MILP gate (pypsa-F09) | -| D-1 | pass | — | — | -| D-2 | informational | — | 5/10 from docs, 3 need source, 2 need trial-and-error | -| D-3 | pass | — | 11/11 examples pass unmodified | -| D-4 | pass | — | — | -| D-5 | informational | — | median 259 LOC; range 111-415 | -| E-1 | pass | — | — | -| E-2 | pass | — | — | -| E-3 | pass | — | — | -| E-4 | pass | — | — | -| E-5 | pass | — | — | -| E-6 | pass | — | — | -| E-7 | pass | — | — | -| F-1 | pass | — | — | -| F-2 | informational | — | ~70 transitive deps, max depth 4; GCS cloud deps unnecessary | -| F-3 | qualified_pass | — | 1 GPL dep (Levenshtein); replaceable with MIT rapidfuzz | -| F-4 | pass | — | — | -| F-5 | pass | — | — | -| F-6 | pass | — | — | -| F-7 | pass | — | — | -| F-8 | pass | — | — | -| F-9 | pass | — | — | -| G-FNM-1 | fail | — | no PSS/E ingestion; API inspection only (pypsa-F10) | -| G-FNM-2 | skip | G-FNM-1 | field coverage blocked; MATPOWER fallback could enable partial assessment (pypsa-F05) | -| G-FNM-3 | pass | stable | 0.0 deviation extraordinary claim; validation-report.md discrepancy (pypsa-F06) | -| G-FNM-4 | informational | — | SuperLU factorization failure at all relaxation levels; consistent with MATPOWER | -| G-FNM-5 | informational | — | 20.5% N / 61.6% E / 17.8% X; 73 fields assessed | -| P2-1 | informational | — | no PSS/E RAW parsing; estimated 3-4 weeks effort | -| P2-2 | informational | — | no piecewise-linear cost curves; tracked as issue #1020 | -| P2-3 | informational | — | commitment injection workflow feasible via A-6 pattern | diff --git a/sweep-data/v10-to-v11/per-tool/pypsa/findings.yaml b/sweep-data/v10-to-v11/per-tool/pypsa/findings.yaml deleted file mode 100644 index b8c186f9..00000000 --- a/sweep-data/v10-to-v11/per-tool/pypsa/findings.yaml +++ /dev/null @@ -1,413 +0,0 @@ -tool: pypsa -source_version: v10 -timestamp: 2026-03-14T12:00:00Z -evaluation_summary: - total_tests: 59 # 60 result files (C-5 split SMALL/MEDIUM) - pass: 38 - fail: 3 - qualified_pass: 3 - informational: 8 - skip: 8 - # Note: validation-report.md lists G-FNM-3 as a failure in its Failures table - # but the G-FNM-3 result file has status: pass. The synthesis also records it as - # pass. The validation report appears to reflect a superseded intermediate state. - -findings: - - id: pypsa-F01 - category: scoring_inconsistency - severity: medium - test_ids: [B-8] - title: "B-8 slack reconfiguration produces zero LMP variation — pass condition is vacuous for DC OPF" - description: >- - B-8 tests reference bus reconfigurability by solving DC OPF with three different - slack bus assignments and checking that "LMP values change consistently across - configurations." All three configurations produce identical objectives and - identical LMPs (spread = 0.0 across all configs). The evaluator correctly - explains this is mathematically expected for DC OPF (dual variables are - invariant to angle reference choice), but the pass condition as written - ("LMP values change consistently") cannot be satisfied for any well-functioning - DC OPF solver. The test silently measures only configurability of the API call, - not meaningful LMP sensitivity. The same structural issue would affect all - tools implementing standard DC OPF. - evidence: - - file: evaluations/pypsa/results/extensibility/B-8_reference_bus_config.md - excerpt: "Objective spread: 0.0000 (identical across all three configs)\nLMP spread variation: 0.0000 (identical across all three configs)\nLMP shift (config 1 vs 2): mean=0.0000, std=0.000000" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - notes: >- - The test should be redesigned to use DCPF (not DC OPF) for the LMP comparison, - or the pass condition should be rewritten to check API configurability only - (API calls required, no model reconstruction needed) without requiring LMP - variation. For AC OPF, slack choice does affect LMPs. If the intent is to - test slack reconfigurability under a formulation where it matters, the test - should use AC OPF or DCPF with non-uniform losses. - - - id: pypsa-F02 - category: network_insufficiency - severity: medium - test_ids: [A-5] - title: "A-5 SCUC on 10-generator TINY network exercises UC constraints but cannot stress min up/down cycling" - description: >- - The TINY network has only 10 generators spanning 4 cost tiers. The A-5 result - shows 3 generators cycling (G3/coal, G6/gas CC, G9/gas CC) across a 4,237-6,254 - MW load range with a capacity-to-peak ratio of 1.18. This passes the numeric - threshold (>=2 cycling generators), but the constraint that min up/down times - actually force commitment decisions is not verifiable: min_up_time=min_down_time - are set to values from gen_temporal_params.csv, but with only 10 generators and - generous capacity headroom, the optimizer has many degrees of freedom. The test - cannot confirm that the min up/down time constraints are binding (as opposed to - merely present in the formulation). A network with tighter capacity margins and - more generators with heterogeneous min up/down times would be needed to verify - that the MILP constraints meaningfully constrain the solution. - evidence: - - file: evaluations/pypsa/results/expressiveness/A-5_scuc.md - excerpt: "Capacity-to-peak-load ratio: 1.18. Load range: 4,237--6,254 MW.\nG9 (gas CC, $40/MWh) shuts down during hours 3--8 (low load), restarts at hour 9, shuts down again at hour 22." - - file: evaluations/pypsa/results/eval-config.yaml - excerpt: "pass_condition: >-\n Solves to feasibility (MIP gap <= 1%). At least 2 generators must\n cycle (commit/decommit) during the 24-hour horizon." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_verification - notes: >- - A supplementary check should verify that at least one generator's min up/down - time constraint is binding in the optimal solution (i.e., the generator would - have switched status earlier/later if the min time constraint were removed). - This is feasible by re-running with min_up_time=min_down_time=0 and comparing - the cycling pattern. - - - id: pypsa-F03 - category: missing_verification - severity: medium - test_ids: [A-10] - title: "A-10 lossy DCOPF LMP decomposition check (d) does not verify loss+energy+congestion sum to total LMP" - description: >- - The A-10 pass condition requires four internal consistency checks including - "(d) loss component LMPs sum with energy and congestion components to total LMP - within 1% tolerance." The result file reports checks (a), (b), (c), and - (d-partial): it confirms LMPs change between lossy and lossless, but does not - show the three-component decomposition (energy + congestion + loss = total LMP) - with numerical verification. The loss component is defined as "bus LMP difference - from the slack bus energy component" — a two-component decomposition — but the - protocol requires a three-component additive check. The result records "Buses - with LMP change (lossy vs lossless): 38 of 39" as check (d), which is a - weaker criterion than the protocol's additive decomposition requirement. - evidence: - - file: evaluations/pypsa/results/expressiveness/A-10_lossy_dcopf_lmp.md - excerpt: "| (d) Bus LMPs change between lossy and lossless | PASS (38/39 buses) |\n\nAll four consistency checks passed." - - file: evaluations/pypsa/results/eval-config.yaml - excerpt: "(d) loss component LMPs sum with energy and congestion components to total LMP within 1% tolerance." - cross_tool_relevance: likely - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - notes: >- - The probe should compute per-bus LMP components (energy = slack bus LMP, - congestion = row of PTDF * shadow_prices, loss = total - energy - congestion) - and verify the three sum to within 1% of the total LMP for each bus. If PyPSA's - `transmission_losses` formulation does not expose a standalone congestion - component separately from the loss component, this is itself a finding about - LMP decomposition API completeness. - - - id: pypsa-F04 - category: test_design_gap - severity: medium - test_ids: [A-9] - title: "A-9 SCOPF uses only 19 of 46 branches due to infeasibility — reduced contingency set may not stress the formulation adequately" - description: >- - The A-9 protocol specifies "DC OPF with N-1 contingency flow constraints on TINY - (all 46 branches)." The result uses a progressive fallback: all 35 lines tried first - (infeasible), then lines <70% utilization (infeasible), then lines <50% utilization - (feasible with 19 lines). The test passes on this 19-line reduced contingency set. - The transformer contingency exclusion (11 of 46 branches not accepted by the API) - is a genuine tool limitation. The infeasibility reduction is a network topology - characteristic: 2 lines are already at 100% utilization in the base case, so any - N-1 contingency set that includes them creates infeasibility. The resulting SCOPF - (19 contingencies from 35 total lines) may not adequately stress the formulation - relative to the protocol's intent of all 46 branches. - evidence: - - file: evaluations/pypsa/results/expressiveness/A-9_scopf.md - excerpt: "The test used a progressive fallback strategy: all 35 lines -> lines <70%\nutilization -> lines <50% utilization. The SCOPF became feasible with 19\nlines at <50% base-case utilization." - - file: evaluations/pypsa/results/eval-config.yaml - excerpt: "description: \"Solve DC OPF with N-1 contingency flow constraints on TINY (all 46 branches)\"" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - notes: >- - The test design should either (a) specify a contingency subset that is - known to be feasible for a well-functioning SCOPF (e.g., contingencies - excluding branches already at >90% base-case utilization), or (b) change - the pass condition to permit progressive fallback with documentation of - the fallback depth. As written, the "all 46 branches" specification is - impossible to satisfy on the IEEE 39-bus at standard loading for any tool. - - - id: pypsa-F05 - category: infrastructure_friction - severity: medium - test_ids: [G-FNM-1, G-FNM-2] - title: "G-FNM-1 PSS/E ingestion failure is an architectural format gap, not a power system capability gap" - description: >- - G-FNM-1 fails because PyPSA has no PSS/E import path. This is correctly recorded - as a format gap. However, the downstream consequence — G-FNM-2 is blocked and - cannot assess field coverage — means the evaluation cannot measure PyPSA's data - model fidelity via the PSS/E path. This blocks a meaningful dimension of comparison - between tools. Tools that happen to support PSS/E ingestion get field coverage - assessment; tools that don't (like PyPSA) get a pass on that dimension via the - MATPOWER fallback. The failure is real but the framing in the rubric conflates - data format support (supply chain/interoperability concern) with power system - modeling capability (expressiveness concern). - evidence: - - file: evaluations/pypsa/results/fnm_ingestion/G-FNM-1_intermediate_format_ingestion.md - excerpt: "PyPSA v1.1.2 has no native capability to read PSS/E-format data, whether in raw `.raw` format or in the intermediate CSV format derived from PSS/E v31 record types." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_test - notes: >- - The protocol should include a separate sub-criterion for PSS/E ingestion support - (as an interoperability/format concern) distinct from the power system modeling - capability assessed in the rest of the FNM suite. Tools failing G-FNM-1 should - still receive a complete G-FNM-5 assessment using the MATPOWER fallback path, - since representability of supplemental data is tool-model-dependent, not - format-dependent. - - - id: pypsa-F06 - category: extraordinary_claim - severity: high - test_ids: [G-FNM-3] - title: "G-FNM-3 reports 0.0 deviation on 27,862-bus DCPF — perfect agreement with reference is an extraordinary claim" - description: >- - G-FNM-3 reports mean deviation = 0.0 deg, max deviation = 0.0 deg for bus voltage - angles, and mean deviation = 0.0%, max deviation = 0.0% for branch flows across - 27,862 buses and 32,532 branches. The result attributes this to both PyPSA and - MATPOWER using identical DCPF formulations (b = 1/(x*tap)) and the shared - matpower_loader applying the same branch status patch. While this explanation - is plausible, a perfect 0.0 numerical result on a 27,862-bus problem with - floating-point arithmetic is unusual and warrants verification. The result file - rounds to 0.0 in its reporting tables — it is unclear whether this represents - true machine-zero agreement or agreement within the reported decimal precision. - Additionally, the validation report (validation-report.md) lists G-FNM-3 in - its Failures table with reason "DCPF verification failed — systematic impedance - conversion differences via MATPOWER fallback," which contradicts the result - file's status: pass. This discrepancy indicates the result file was updated - after the validation report was written and the validation report was not - regenerated. - evidence: - - file: evaluations/pypsa/results/fnm_ingestion/G-FNM-3_dcpf_verification.md - excerpt: "| Mean deviation | 0.0 deg |\n| Max deviation | 0.0 deg |\n...\n| Mean deviation | 0.0% |\n| Max deviation | 0.0% |" - - file: evaluations/pypsa/results/validation-report.md - excerpt: "| G-FNM-3 | DCPF verification failed — systematic impedance conversion differences via MATPOWER fallback |" - cross_tool_relevance: none - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - notes: >- - The probe should re-extract the raw numerical deviations (not rounded to display - precision) and confirm the maximum absolute bus angle deviation and maximum - branch flow percent deviation. If both PyPSA and MATPOWER use the same numerical - B-matrix (which is plausible since both trace to matpowercaseframes), exact - agreement is expected. The probe should also resolve the validation-report - discrepancy by confirming whether the report reflects the current or a superseded - result. - - - id: pypsa-F07 - category: misleading_result - severity: low - test_ids: [A-11] - title: "A-11 qualified_pass with blocking workaround is more severe than the qualified_pass label suggests" - description: >- - A-11 is scored as qualified_pass with workaround_class=blocking. The synthesis - describes this as "blocking — no workaround exists." The qualified_pass status - implies partial success, but the actual result is that distributed slack DC OPF - is architecturally impossible in PyPSA's optimize() path — not partially supported - or approximable. The workaround that is reported (distributed slack in n.pf()) - tests a different capability (AC power flow, not DC OPF) under different protocol - conditions. The qualification language "capability exists in one context (PF) but - not the other (OPF)" credits PyPSA for a capability that the test explicitly does - not require. A-11's pass condition specifies "Solve DC OPF on TINY with distributed - slack (load-proportional)." The AC PF distributed slack result does not satisfy this - condition and should not count toward the pass. - evidence: - - file: evaluations/pypsa/results/expressiveness/A-11_distributed_slack_opf.md - excerpt: "status: qualified_pass\nworkaround_class: blocking\n...No Bus-v_ang variable. Distributed slack OPF is NOT achievable." - - file: evaluations/pypsa/results/eval-config.yaml - excerpt: "description: \"Solve DC OPF on TINY with distributed slack (load-proportional)\"" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - notes: >- - A-11 should be scored as fail (not qualified_pass) since the protocol's pass - condition requires distributed slack DC OPF, which is demonstrably impossible - in PyPSA's optimize() path. Documenting the AC PF distributed slack as an - observation is appropriate, but it should not convert a fail to a qualified_pass. - This affects the expressiveness grade calculation. - - - id: pypsa-F08 - category: missing_verification - severity: low - test_ids: [A-12] - title: "A-12 fragile workaround for branch shadow prices is classified as pass — the frailty of the extraction path creates result credibility risk" - description: >- - A-12 extracts branch shadow prices via the internal linopy constraint naming - convention (n.model.constraints['Line-fix-s-upper'].dual), which the evaluator - correctly classifies as fragile. Pass condition 1 (congestion reporting) depends - on these shadow prices: "At least 2 of 24 hours must have >=2 branches with - non-zero shadow prices." If the internal naming convention changes in future - PyPSA versions, this extraction will silently fail. More critically, the result - confirms the documented bug: "The solver log confirms shadow prices were computed - ('shadow-prices of the constraints ... were not assigned to the network')." The - test passes on the extracted values, but the documented attribute path - (n.lines_t.mu_upper) is broken. This is a real bug in the tool, not a workaround - choice, and the test's pass status depends on working around a documented bug - rather than a supported API. The overall test status of pass is technically - correct given the workaround, but the bug should be flagged as a tool defect. - evidence: - - file: evaluations/pypsa/results/expressiveness/A-12_multiperiod_dcopf_storage.md - excerpt: "Branch shadow prices extracted from linopy model constraint duals (`n.model.constraints['Line-fix-s-upper'].dual`) instead of the documented `n.lines_t.mu_upper`/`mu_lower` attributes.\n...Durability: fragile -- depends on the internal linopy constraint naming convention" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: add_verification - notes: >- - The shadow price bug (n.lines_t.mu_upper empty after optimize()) also affects - A-3 and should be tracked as a tool defect finding. The A-3 result acknowledges - the same bug. Cross-test consistency suggests this is a systematic issue across - all OPF solves, not test-specific. - - - id: pypsa-F09 - category: test_design_gap - severity: medium - test_ids: [C-4] - title: "C-SMALL gate triggered by MILP solver scalability causes 7 MEDIUM tests to be skipped — gate design conflates problem type scalability with general scalability" - description: >- - The C-SMALL-gate triggers on C-4 (SCUC on 2,000-bus SMALL network) and cascades - to skip 7 MEDIUM-tier tests: C-1 (DCPF), C-2 (ACPF), C-3 (DC OPF), C-7 (solver - swap), C-8 (SCOPF), C-9 (PTDF), C-10 (distributed slack). These 7 skipped tests - cover LP/linear algebra problems, not MILP. PyPSA's failure on C-4 is a MILP - scalability limitation (HiGHS single-threaded cannot solve 39,168-variable - MILP within 600s). The tool's LP and power flow scalability — which C-1, C-2, - C-3 would have measured — is demonstrably good: C-5 passes ACPF on 10,000-bus - MEDIUM, and G-FNM-3 passes DCPF on 27,862-bus LARGE. The gate design prevents - collecting LP/PF scalability evidence for tools that fail MILP at SMALL scale, - even though LP/PF scalability is an independent capability. - evidence: - - file: evaluations/pypsa/results/synthesis.md - excerpt: "C-1 | MEDIUM | skip | C-SMALL-gate | -- | -- | --\nC-2 | MEDIUM | skip | C-SMALL-gate | -- | -- | --\nC-3 | MEDIUM | skip | C-SMALL-gate | -- | -- | --" - - file: evaluations/pypsa/results/scalability/C-4_scuc_small.md - excerpt: "The MILP formulation has 347,272 rows, 129,168 columns (39,168 binary variables), and 1,689,312 nonzeros. After presolve: 85,868 rows, 73,378 columns (25,474 binary), 957,923 nonzeros. HiGHS could not solve the root LP relaxation within the 600-second time limit." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - notes: >- - The C-SMALL gate should be specific to MILP scalability (C-4). LP/PF MEDIUM - tests (C-1, C-2, C-3, C-7, C-9, C-10) should run regardless of C-4 outcome. - Only MILP MEDIUM tests (C-8 SCOPF has LP foundation, but C-4-equivalent SCUC - at MEDIUM would be the true gate) should be conditional on C-4. This affects - the scalability grade for PyPSA (and potentially other tools that fail MILP - at SMALL but have strong LP/PF scalability). - - - id: pypsa-F10 - category: missing_verification - severity: low - test_ids: [G-FNM-1] - title: "G-FNM-1 failure determination based on API inspection without code execution" - description: >- - The G-FNM-1 result states: "No test script was written. The verification was an - API surface inspection confirming the absence of any PSS/E ingestion path in - PyPSA's public interface." The result correctly identifies that PyPSA has no - PSS/E ingestion, and the conclusion is almost certainly correct. However, the - protocol's pass condition specifies that the tool should attempt to parse the - intermediate CSV tables and record failure_reason. An API inspection rather than - an actual parse attempt means the failure mode (e.g., column schema mismatch vs. - format not recognized vs. silent empty import) is not confirmed by execution. - This is a minor issue since the conclusion is correct, but it diverges from the - protocol's verification standard. - evidence: - - file: evaluations/pypsa/results/fnm_ingestion/G-FNM-1_intermediate_format_ingestion.md - excerpt: "No test script was written. The verification was an API surface inspection confirming the absence of any PSS/E ingestion path in PyPSA's public interface." - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: add_verification - notes: >- - A minimal probe could attempt import_from_csv_folder on the intermediate CSV - directory and capture the actual error message. This would confirm the failure - mode and produce an executable record consistent with protocol standards. - - - id: pypsa-F11 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate ingestion tests G-1/G-2/G-3 have low discriminative value — any functioning tool passes" - description: >- - The gate tests (G-1, G-2, G-3) test only that a tool can load MATPOWER .m files - and return correct bus/branch/generator counts. These tests produce pass/fail - outcomes that would be identical for any tool with a MATPOWER importer. The only - discriminating information is the loader method used and any data quality issues - encountered. G-1 does note a meaningful finding (generator cost data NOT imported - by import_from_pypower_ppc), but this finding is not captured in the gate test - status — it affects downstream tests (A-3 onward) but is not part of the gate - outcome. The gate tests serve their purpose as a minimum bar, but they add no - signal to the cross-tool comparison. - evidence: - - file: evaluations/pypsa/results/gate/G-1_ingest_tiny.md - excerpt: "Generator cost data NOT imported — `import_from_pypower_ppc` does not support gencost" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_verification - notes: >- - The gate tests could include a data quality check — specifically, whether - generator cost data is correctly imported. This is a meaningful discriminator - (PyPSA's import_from_pypower_ppc silently drops gencost, requiring the shared - loader patch) that currently falls between the gate pass/fail and the - expressiveness tests. - - - id: pypsa-F12 - category: scoring_inconsistency - severity: low - test_ids: [A-6, A-11] - title: "Inconsistent qualified_pass criteria: A-6 (stable workaround) and A-11 (blocking) receive the same status label" - description: >- - Both A-6 and A-11 are classified as qualified_pass, but the workaround_class - values are "stable" (A-6: undocumented but functional two-stage pattern) and - "blocking" (A-11: architectural impossibility). The schema allows - workaround_class to distinguish these, but both map to the same status outcome - (qualified_pass), which masks the severity difference when statuses are counted - or aggregated. A-6 is a genuine partial pass: the capability exists with a - documented approach but without a convenience API. A-11 is effectively a fail - in the OPF context. This inconsistency in status assignment will propagate to - cross-tool comparisons where qualified_pass is treated as a homogeneous outcome. - evidence: - - file: evaluations/pypsa/results/synthesis.md - excerpt: "| A-6 | TINY | qualified_pass | -- | stable | 3.0s | 453 |\n| A-11 | TINY | qualified_pass | -- | blocking | 1.9s | 326 |" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - notes: >- - The schema should distinguish between workaround_class=blocking (effectively - a fail, should score as fail or fail_with_context) and workaround_class=stable - (partial pass with low-friction workaround). This affects the expressiveness - grade and cross-tool comparisons. See also pypsa-F07. - -extraordinary_claims: - - test_id: G-FNM-3 - claim: >- - PyPSA achieves 0.0 mean and max deviation (both bus angles and branch flows) - compared to the MATPOWER reference on a 27,862-bus network. - concern: >- - Perfect 0.0 agreement on a floating-point computation at 27,862-bus scale is - unusual. The result tables round to one decimal place for bus angles and one - decimal place for branch flow percentages, so the reported 0.0 may reflect - display rounding rather than true machine-zero agreement. Additionally, - validation-report.md lists this test as a failure (status superseded by result - file update), creating ambiguity about whether the reported pass reflects the - final or an intermediate evaluation state. - evidence_quality: moderate - probe_recommended: true - probe_type: convergence_check diff --git a/sweep-data/v10-to-v11/probe-manifest.yaml b/sweep-data/v10-to-v11/probe-manifest.yaml deleted file mode 100644 index 45cbf73e..00000000 --- a/sweep-data/v10-to-v11/probe-manifest.yaml +++ /dev/null @@ -1,186 +0,0 @@ -probes: - # ── PYPSA ──────────────────────────────────────────────────────────────────── - - id: probe-001 - tool: pypsa - claim: "G-FNM-3 achieves 0.0 mean and max deviation across all 27,862 buses and 32,532 branches" - source_test: G-FNM-3 - source_file: evaluations/pypsa/results/gate/fnm_ingestion_results.yaml - probe_type: convergence_check - priority: high - concern: "Perfect 0.0 deviations are display-rounded; raw numerics not captured. validation-report.md lists G-FNM-3 as FAIL (possible superseded state). Need unrounded residuals." - requires_devcontainer: true - - - id: probe-002 - tool: pypsa - claim: "A-10 lossy DCOPF LMP decomposition verified via condition (d)" - source_test: A-10 - source_file: evaluations/pypsa/results/expressiveness/A-10-lossy-dcopf.md - probe_type: claim_verification - priority: medium - concern: "Check (d) verifies LMP change between scenarios, not the required three-component additivity (energy + congestion + loss = total LMP). May pass a weaker criterion." - requires_devcontainer: true - - # ── PANDAPOWER ────────────────────────────────────────────────────────────── - - id: probe-003 - tool: pandapower - claim: "All 46 branches have non-zero shadow prices in A-3 DC OPF" - source_test: A-3 - source_file: evaluations/pandapower/results/expressiveness/A-3-dcopf-lmp.md - probe_type: convergence_check - priority: high - concern: "Only 7 branches are above 95% loading; interior-point solvers produce numerically small but nonzero duals on inactive constraints. No magnitude threshold applied — 46/46 is internally inconsistent with loading data." - requires_devcontainer: true - - - id: probe-004 - tool: pandapower - claim: "DC OPF optimal cost $156,929 is verified convergence" - source_test: A-3 - source_file: evaluations/pandapower/results/expressiveness/A-3-dcopf-lmp.md - probe_type: convergence_check - priority: medium - concern: "Convergence accepted from boolean flag only; PYPOWER DC OPF does not report iteration count. Dual feasibility not checked (LMP ≈ marginal cost at unconstrained generators)." - requires_devcontainer: true - - # ── GRIDCAL ────────────────────────────────────────────────────────────────── - - id: probe-005 - tool: gridcal - claim: "DCOPF passes on IEEE 39-bus with all branches within limits (A-3)" - source_test: A-3 - source_file: evaluations/gridcal/results/expressiveness/A-3-dcopf-lmp.md - probe_type: formulation_audit - priority: high - concern: "Result shows 112% loading on one branch. Soft-constraint status (slack variable vs hard limit) not disclosed. Need to confirm whether branch limits are enforced as hard constraints." - requires_devcontainer: true - - - id: probe-006 - tool: gridcal - claim: "SCUC commitment schedule reflects binary UC decisions (A-5)" - source_test: A-5 - source_file: evaluations/gridcal/results/expressiveness/A-5-scuc.md - probe_type: formulation_audit - priority: medium - concern: "Commitment inferred from dispatch threshold (P > 0) rather than binary variable. MIP gap not extractable. Cannot confirm MILP was solved to integrality." - requires_devcontainer: true - - - id: probe-007 - tool: gridcal - claim: "G-FNM-3 has 326 branches with deviations up to 562,955% classified as formulation_difference" - source_test: G-FNM-3 - source_file: evaluations/gridcal/results/gate/fnm_ingestion_results.yaml - probe_type: formulation_audit - priority: high - concern: "Deviations of 562,955% are extreme outliers. Classification as 'formulation_difference' rather than bug needs verification — check whether those branches have transformers or other non-trivial elements." - requires_devcontainer: false - - - id: probe-008 - tool: gridcal - claim: "GridCal is in operational use at Redeia, Schneider Electric, and GE Vernova (E-7)" - source_test: E-7 - source_file: evaluations/gridcal/results/maturity/E-7-user-base.md - probe_type: claim_verification - priority: low - concern: "Adoption claims sourced from project's own documentation. Independent verification (press releases, LinkedIn, conference papers) not attempted." - requires_devcontainer: false - - # ── POWERMODELS ────────────────────────────────────────────────────────────── - - id: probe-009 - tool: powermodels - claim: "SCOPF TINY qualified_pass demonstrates Benders convergence (A-9)" - source_test: A-9 - source_file: evaluations/powermodels/results/expressiveness/A-9-scopf.md - probe_type: formulation_audit - priority: high - concern: "Network is N-1 infeasible under test load — Benders runs 1 iteration before hitting infeasibility. The key question (does it converge to a secure solution when one exists?) is never answered." - requires_devcontainer: true - - - id: probe-010 - tool: powermodels - claim: "SCIP_jll v0.2.1 license is ZIB Academic (F-3) vs Apache 2.0 (F-8)" - source_test: F-3 - source_file: evaluations/powermodels/results/supply_chain/F-3-dependency-licenses.md - probe_type: claim_verification - priority: high - concern: "Same artifact classified under two incompatible licenses within the same evaluation. Material for supply chain scoring — one of these is wrong." - requires_devcontainer: false - - - id: probe-011 - tool: powermodels - claim: "GLPK solves MEDIUM DCOPF in 61.86s where v9 timed out at 300s (C-3)" - source_test: C-3 - source_file: evaluations/powermodels/results/scalability/C-3-dcopf-medium.md - probe_type: timing_verification - priority: medium - concern: "Speedup unexplained. v10 cost linearization (45.5% of generators) may convert QP to LP (GLPK can handle LP, rejects QP), but this is not confirmed in the result file." - requires_devcontainer: true - - - id: probe-012 - tool: powermodels - claim: "PTDF MEDIUM extraction: B-9 cold 106s vs C-9 warm 7.55s (14x ratio)" - source_test: C-9 - source_file: evaluations/powermodels/results/scalability/C-9-ptdf-medium.md - probe_type: timing_verification - priority: low - concern: "14x difference attributed to Julia JIT. Plausible but should be confirmed — a second cold run of B-9 would distinguish JIT from a genuine computation change." - requires_devcontainer: true - - # ── POWERSIMULATIONS ───────────────────────────────────────────────────────── - - id: probe-013 - tool: powersimulations - claim: "ACPF convergence verified at TINY and SMALL (A-2, C-2)" - source_test: A-2 - source_file: evaluations/powersimulations/results/expressiveness/A-2-acpf.md - probe_type: convergence_check - priority: high - concern: "Convergence accepted without residual or iteration count. C-2 documents a convergence warning on first call that was silently suppressed. Second call taken as verified." - requires_devcontainer: true - - - id: probe-014 - tool: powersimulations - claim: "LMP values require undocumented divide-by-base_power and negate conversion (A-3)" - source_test: A-3 - source_file: evaluations/powersimulations/results/expressiveness/A-3-dcopf-lmp.md - probe_type: claim_verification - priority: medium - concern: "Unit conversion (divide by base_power, negate) not documented in PowerSystems.jl or PowerSimulations.jl API docs. Cannot confirm LMPs are correctly extracted without independent derivation." - requires_devcontainer: false - - - id: probe-015 - tool: powersimulations - claim: "SCUC on 2K-bus network solves in 404s single-threaded (C-4)" - source_test: C-4 - source_file: evaluations/powersimulations/results/scalability/C-4-scuc-medium.md - probe_type: timing_verification - priority: medium - concern: "404s measured single-threaded on a 32-core machine. Multi-threaded HiGHS (the default) would likely achieve 15-30s, which may change the qualified_pass to a pass for the timing criterion." - requires_devcontainer: true - - # ── MATPOWER ───────────────────────────────────────────────────────────────── - - id: probe-016 - tool: matpower - claim: "GLPK exit-flag bug causes A-5 SCUC to report failure when solve succeeded (A-5, C-4)" - source_test: A-5 - source_file: evaluations/matpower/results/expressiveness/A-5-scuc.md - probe_type: claim_verification - priority: high - concern: "SCUC solve succeeded (162K variables in 1.1s) but GLP_EMIPGAP mapped to failure cascades into C-4 fail and blanks all 8 MEDIUM scalability tests. A one-line fix should recover the result. Verify the fix actually produces a pass." - requires_devcontainer: true - - - id: probe-017 - tool: matpower - claim: "A-9 SCOPF fail is a solver-environment artifact (MIPS singularity, HiGHS unavailable)" - source_test: A-9 - source_file: evaluations/matpower/results/expressiveness/A-9-scopf.md - probe_type: formulation_audit - priority: medium - concern: "MOST Approach 3 (contingency table) was not attempted. If HiGHS is available in the devcontainer, the fail may be recoverable." - requires_devcontainer: true - - - id: probe-018 - tool: matpower - claim: "A-5 SCUC qualified_pass validated on 3-bus bundled test case (not TINY network)" - source_test: A-5 - source_file: evaluations/matpower/results/expressiveness/A-5-scuc.md - probe_type: claim_verification - priority: medium - concern: "Protocol requires TINY (10-bus, 4-generator). 3-bus case has too few generators to confirm min up/down cycling. Need to run on actual TINY network." - requires_devcontainer: true diff --git a/sweep-data/v10-to-v11/probes/gridcal/probe-005.md b/sweep-data/v10-to-v11/probes/gridcal/probe-005.md deleted file mode 100644 index d6003dfb..00000000 --- a/sweep-data/v10-to-v11/probes/gridcal/probe-005.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -probe_id: probe-005 -tool: gridcal -source_test: A-3 -probe_type: formulation_audit -classification: confirmed_issue -reason: > - GridCal's linear_opf uses soft branch flow constraints (slack variables - flow_slacks_pos / flow_slacks_neg in the LP). Branch 2_3_1 exceeds its - derated limit (103.5%, 362 MW vs 350 MW limit) in the optimal solution, - which is only possible with soft constraints. The A-3 'pass' verdict is - misleading: a standard DCOPF requires hard flow limits. This is a - formulation deficiency, not a pass. -solver: HiGHS -solver_version: "bundled with VeraGridEngine 5.6.28" -solver_version_match: true -tool_version: "VeraGridEngine 5.6.28" -timeout_seconds: 300 -wall_clock_seconds: 2.1 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe 005: GridCal DC OPF Branch Constraint Formulation - -## Claim Under Investigation - -A-3 reports "pass" despite branch 2_3_1 showing 112% loading (103.5% in -probe re-run with consistent cost assignment). The concern is whether GridCal -enforces branch flow limits as hard constraints (standard DCOPF) or soft -constraints (slack variables). - -## Finding: SOFT CONSTRAINTS CONFIRMED - -### Source Code Evidence - -Inspecting `VeraGridEngine/Simulations/OPF/opf_driver.py`: - -```python -# Line 170-171 of opf_driver.py -self.results.overloads = (opf_vars.branch_vars.flow_slacks_pos[0, :] - - opf_vars.branch_vars.flow_slacks_neg[0, :]) -``` - -The LP model contains explicit slack variables `flow_slacks_pos` and -`flow_slacks_neg` per branch. These are the textbook soft-constraint -formulation: the objective penalizes overloads rather than making them -infeasible. The `overloads` result field is the net slack magnitude. - -### Runtime Evidence - -Probe re-run (VeraGridEngine 5.6.28, IEEE 39-bus with 70% branch derating): - -| Branch | Loading | Flow (MW) | Derated Limit (MW) | Overload slack | -|----------|---------|-----------|-------------------|----------------| -| 2_3_1 | 103.5% | 362.22 | 350.00 | -12.22 MW | -| 16_19_1 | 100.0% | -420.00 | 420.00 | 0 (at limit) | - -The `opf_results.overloads` array has a non-zero entry (-12.22 MW) at -branch index 2 (2_3_1), confirming the slack variable absorbed the -infeasibility rather than the solver rejecting the solution. - -The result object exposes `contingency_flows_slacks_list` as an additional -attribute, further confirming a slack-based formulation is used throughout. - -### Keyword Audit - -- `opf_driver.py` contains keyword "slack": 2 hits, "overload": 7 hits -- `linear_opf` source: no hard flow constraint keywords found -- `OptimalPowerFlowOptions`: no settings for constraint hardness found - -## Impact on A-3 Verdict - -The A-3 pass condition checked: -1. Convergence (satisfied — trivially true with soft constraints) -2. LMPs extractable (satisfied) -3. At least 2 binding branches (satisfied — 6 branches at ≥99%) - -None of the pass conditions test whether flow limits are enforced as hard -constraints. A correct DCOPF pass condition should verify: - -``` -max_loading <= 1.0 + epsilon -``` - -With that criterion, A-3 **fails**: branch 2_3_1 reaches 103.5%. - -## Classification Rationale - -This is a **confirmed_issue** (not a formulation_difference): - -- Standard DCOPF is universally defined with hard flow constraints -- GridCal's soft-constraint variant allows infeasible physical solutions -- The 112%/103.5% overload is not a numerical artifact — it is the - optimizer's deliberate choice to violate the limit at a penalty cost -- LMPs from a soft-constraint OPF are economically meaningful only if the - penalty cost equals the true marginal cost of overload relief; this is - unlikely to be calibrated correctly in GridCal's default configuration -- The `overloads_cost` attribute exists but was not set to non-trivial values - in the default options - -## Recommendation - -The A-3 result should be reclassified as **partial_pass** or **fail** with -a note that GridCal's linear_opf uses soft branch flow constraints. This -is a significant expressiveness limitation for users who require strict -thermal limit enforcement. diff --git a/sweep-data/v10-to-v11/probes/gridcal/probe-005_script.py b/sweep-data/v10-to-v11/probes/gridcal/probe-005_script.py deleted file mode 100644 index d111f290..00000000 --- a/sweep-data/v10-to-v11/probes/gridcal/probe-005_script.py +++ /dev/null @@ -1,267 +0,0 @@ -""" -Probe 005: Audit GridCal DC OPF branch flow constraint formulation. - -Claim under investigation: A-3 reports 'pass' but branch 2_3_1 shows 112% loading. -Question: Are branch flow limits hard constraints or soft (slack/penalty) constraints? -""" - -from __future__ import annotations - -import inspect -import sys -from pathlib import Path - -import numpy as np - -# Add the shared matpower_loader -sys.path.insert(0, str(Path("/workspace/evaluations/gridcal/tests"))) - -import VeraGridEngine as vge -from VeraGridEngine.enumerations import MIPSolvers, SolverType - -NETWORK_FILE = "/workspace/data/networks/case39.m" -BRANCH_DERATING = 0.70 - -COST_MAP = { - "hydro": {"c1": 5.0, "c2": 0.005}, - "nuclear": {"c1": 10.0, "c2": 0.010}, - "coal_large": {"c1": 25.0, "c2": 0.025}, - "gas_CC": {"c1": 40.0, "c2": 0.040}, -} - -print("=" * 70) -print("Probe 005: GridCal DC OPF Branch Constraint Formulation Audit") -print("=" * 70) -print("VeraGridEngine version: 5.6.28") -print(f"Network: {NETWORK_FILE}") -print(f"Branch derating: {BRANCH_DERATING * 100:.0f}%") -print() - -# ── 1. Load network ────────────────────────────────────────────────────────── -print("Loading IEEE 39-bus network...") -grid = vge.open_file(NETWORK_FILE) -generators = grid.get_generators() -branches = grid.get_branches() -buses = grid.get_buses() -print( - f" Buses: {len(buses)}, Generators: {len(generators)}, Branches: {len(branches)}" -) - -# ── 2. Apply differentiated costs ──────────────────────────────────────────── -gen_tech = { - 0: "nuclear", - 1: "coal_large", - 2: "gas_CC", - 3: "gas_CC", - 4: "coal_large", - 5: "gas_CC", - 6: "hydro", - 7: "coal_large", - 8: "coal_large", - 9: "nuclear", -} -for idx, gen in enumerate(generators): - tech_key = gen_tech.get(idx, "gas_CC") - gen.Cost = COST_MAP[tech_key]["c1"] - gen.Cost2 = COST_MAP[tech_key]["c2"] - gen.Cost0 = 0.0 - -# ── 3. Apply 70% branch derating ───────────────────────────────────────────── -branch_original_rates = [] -for branch in branches: - if hasattr(branch, "rate"): - original_rate = branch.rate - branch_original_rates.append(original_rate) - branch.rate = original_rate * BRANCH_DERATING - else: - branch_original_rates.append(0.0) - -# ── 4. Inspect OPF formulation source ──────────────────────────────────────── -print("\n--- Inspecting linear_opf source for constraint type ---") - -try: - src = inspect.getsource(vge.linear_opf) - # Look for slack, penalty, or soft constraint indicators - src_lower = src.lower() - keywords = ["slack", "penalty", "soft", "overload", "epsilon", "relax"] - for kw in keywords: - count = src_lower.count(kw) - if count > 0: - print(f" Keyword '{kw}' found {count} times in linear_opf source") - print(" (full source inspection done — see keyword hits above)") -except Exception as e: - print(f" Could not inspect source: {e}") - -# Try to inspect the underlying LP problem builder -try: - from VeraGridEngine.Simulations.OPF import opf_driver - - opf_src = inspect.getsource(opf_driver) - for kw in keywords: - count = opf_src.lower().count(kw) - if count > 0: - print(f" opf_driver keyword '{kw}': {count} hits") -except Exception as e: - print(f" Could not inspect opf_driver: {e}") - -# ── 5. Run DC OPF ───────────────────────────────────────────────────────────── -print("\n--- Running DC OPF ---") -opf_opts = vge.OptimalPowerFlowOptions( - solver=SolverType.LINEAR_OPF, - mip_solver=MIPSolvers.HIGHS, -) -opf_results = vge.linear_opf(grid, opf_opts) -print(f" Converged: {opf_results.converged}") - -# ── 6. Inspect result object for soft-constraint evidence ───────────────────── -print("\n--- Result object attributes ---") -result_attrs = [a for a in dir(opf_results) if not a.startswith("_")] -soft_indicators = [ - a - for a in result_attrs - if any( - kw in a.lower() for kw in ["overload", "slack", "penalty", "excess", "relax"] - ) -] -print(f" Soft-constraint-related attributes: {soft_indicators}") -print(f" All result attributes: {result_attrs}") - -# ── 7. Check overloads attribute ────────────────────────────────────────────── -print("\n--- Branch loading analysis ---") -loading = opf_results.loading -sf = opf_results.Sf -branch_names = [b.name for b in branches] - -overloaded = [] -binding = [] -for i in range(len(loading)): - pct = abs(loading[i]) * 100 - if pct > 100.0: - overloaded.append( - (branch_names[i], pct, float(np.real(sf[i])), branch_original_rates[i]) - ) - elif pct >= 99.0: - binding.append((branch_names[i], pct)) - -print(f" Branches at exactly 100% (binding): {len(binding)}") -for name, pct in binding: - print(f" {name}: {pct:.1f}%") -print(f" Branches EXCEEDING 100% (overloaded): {len(overloaded)}") -for name, pct, flow, rate in overloaded: - derated_rate = rate * BRANCH_DERATING - print( - f" {name}: {pct:.1f}% loading | flow={flow:.2f} MW | " - f"derated_rate={derated_rate:.2f} MW | original_rate={rate:.2f} MW" - ) - -# ── 8. Check overloads attribute directly ───────────────────────────────────── -print("\n--- opf_results.overloads attribute ---") -if hasattr(opf_results, "overloads") and opf_results.overloads is not None: - overloads_arr = np.array(opf_results.overloads) - nonzero = np.where(np.abs(overloads_arr) > 1e-6)[0] - print(f" Type: {type(opf_results.overloads)}") - print(f" Shape: {overloads_arr.shape}") - print(f" Non-zero entries (branch indices): {nonzero.tolist()}") - for idx in nonzero: - print( - f" Branch {branch_names[idx]} (idx={idx}): overload={overloads_arr[idx]:.4f}" - ) - if len(overloads_arr) > 0: - print( - " Interpretation: 'overloads' is a RESULT field capturing constraint violation," - ) - print( - " confirming these are SOFT CONSTRAINTS (flow limits exceeded without infeasibility)." - ) -else: - print(" opf_results.overloads is None or absent") - -# ── 9. Check shadow prices / dual variables ─────────────────────────────────── -print("\n--- Shadow prices on branches ---") -shadow_attrs = [ - "branch_shadow_prices", - "shadow_prices", - "flow_shadow_prices", - "Sf_shadow", - "mu", - "bus_shadow_prices", -] -for attr in shadow_attrs: - if hasattr(opf_results, attr): - val = getattr(opf_results, attr) - if val is not None: - arr = np.array(val) - print( - f" {attr}: shape={arr.shape}, " - f"nonzero={np.sum(np.abs(arr) > 1e-6)}, " - f"max={np.max(np.abs(arr)):.4f}" - ) - -# ── 10. Look for penalty/slack variables in LP model ───────────────────────── -print("\n--- Checking LP model for slack/penalty variables ---") -try: - # Try to build the LP problem and inspect variables - from VeraGridEngine.Simulations.OPF import LinearOpf - - print(" Found OPF classes:", [c for c in dir(LinearOpf) if not c.startswith("_")]) -except ImportError as e: - print(f" Import failed: {e}") - -try: - # Check if the OPF results have an 'inner' LP model - if hasattr(opf_results, "lp_model") and opf_results.lp_model is not None: - lp = opf_results.lp_model - print(f" LP model type: {type(lp)}") - print(f" LP model attrs: {[a for a in dir(lp) if not a.startswith('_')]}") - else: - print(" No lp_model attribute in results") -except Exception as e: - print(f" LP model check error: {e}") - -# ── 11. Inspect OptimalPowerFlowOptions for soft constraint settings ─────────── -print("\n--- OptimalPowerFlowOptions settings ---") -opf_attrs = { - a: getattr(opf_opts, a) - for a in dir(opf_opts) - if not a.startswith("_") and not callable(getattr(opf_opts, a)) -} -for k, v in sorted(opf_attrs.items()): - if any( - kw in k.lower() - for kw in ["slack", "soft", "penalty", "relax", "overload", "constraint"] - ): - print(f" {k} = {v}") - -print("\n" + "=" * 70) -print("SUMMARY") -print("=" * 70) -print(f"Branch 2_3_1 loading: {abs(loading[0]) * 100:.1f}%" if len(loading) > 0 else "") - -# Find branch 2_3_1 -for i, name in enumerate(branch_names): - if "2_3" in name or name == "2_3_1": - pct = abs(loading[i]) * 100 - print(f"Branch '{name}' (idx={i}): loading = {pct:.2f}%") - if hasattr(opf_results, "overloads") and opf_results.overloads is not None: - ov = float(opf_results.overloads[i]) - print(f" overload value = {ov:.4f} MW") - break - -print() -print("CONCLUSION:") -if len(overloaded) > 0: - print( - " SOFT CONSTRAINTS CONFIRMED: Branch flow limits are violated in the optimal" - ) - print(" solution. The solver accepted a solution exceeding branch capacity, which") - print( - " is only possible if limits are enforced as soft constraints (penalty/slack)" - ) - print(" or if the branch was not included in the LP constraint set.") - print() - print(" The A-3 'pass' is MISLEADING: standard DCOPF requires hard flow limits.") - print( - " GridCal's linear_opf appears to use soft constraints or penalty functions." - ) -else: - print(" No branches overloaded — hard constraints appear to be working correctly.") diff --git a/sweep-data/v10-to-v11/probes/gridcal/probe-007.md b/sweep-data/v10-to-v11/probes/gridcal/probe-007.md deleted file mode 100644 index 9214d536..00000000 --- a/sweep-data/v10-to-v11/probes/gridcal/probe-007.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -probe_id: probe-007 -tool: gridcal -source_test: G-FNM-3 -probe_type: formulation_audit -classification: classification_plausible_with_caveats -reason: > - The 326 outlier branches are 88.7% transformer-adjacent (289/326), meeting - the 80% structural-pattern threshold used to invoke the formulation_difference - classification. The deviation pattern is consistent with GridCal's B-matrix - construction omitting or mis-applying transformer tap ratios in the DCPF. - However, three caveats limit full confidence: (1) the top deviating branches - are typed as "Line" not "Xfmr", meaning the adjacency criterion is proximate - rather than direct; (2) the pass_conditions threshold_deg for - formulation_difference is null (no magnitude cap), allowing arbitrarily large - deviations to be excused; (3) the 37 non-transformer-adjacent failing branches - (11.3%) have no structural explanation and could indicate secondary issues. - The classification is defensible but the qualified_pass verdict should be - treated as a significant limitation flag, not a routine annotation. -solver: N/A -solver_version: N/A -solver_version_match: true -tool_version: "VeraGridEngine 5.6.28" -timeout_seconds: 120 -wall_clock_seconds: 8 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe 007: G-FNM-3 Branch Deviation Pattern Analysis - -## Claim Under Investigation - -G-FNM-3 DCPF has 326 branches with deviations up to 562,955%, classified as -`formulation_difference`. The concern is whether this classification is -justified or whether the deviations indicate a data ingestion bug or solver error. - -## Network Context - -The FNM used is the real grid Full Network Model (NDA-restricted), not the -ACTIVSg70k synthetic case. The test uses a 27,862-bus main island extracted from -the FNM PSS/E RAW file, loaded via the MATPOWER fallback path -(`fnm_main_island.m`). The direct CSV ingestion path (G-FNM-1) failed, so all -conclusions are conditional on the MATPOWER conversion being accurate. - -Network scale: 27,862 buses, 32,532 matched branches. The ACTIVSg10k available -in `data/networks/` is a different, synthetic network not used here. - -## Structural Pattern Analysis - -### Failing branch count and adjacency - -| Category | Count | Fraction | -|----------|-------|----------| -| Total failing branches | 326 | 1.00% of 32,532 | -| Transformer-adjacent | 289 | 88.7% | -| Non-transformer-adjacent | 37 | 11.3% | -| Threshold to qualify | 80% | **Met** | - -### Top 5 deviating branches - -| From | To | Object Type | GridCal (MW) | Reference (MW) | Dev % | -|------|-----|-------------|-------------|----------------|-------| -| 1668 | 88630 | Line | 111,582 | -19.82 | 562,955% | -| 21476 | 84022 | Line | -68,017 | 12.57 | 541,075% | -| 72100 | 73053 | Line | -13,365 | 3.23 | 413,787% | -| 180421 | 36990 | Xfmr | 5,234 | -1.61 | 325,193% | -| 1635 | 92191 | Line | -352,878 | 109.50 | 322,365% | - -Key observation: 4 of the top 5 are classified as "Line" objects in GridCal's -data model. The `transformer_adjacent` flag means at least one endpoint bus -appears in the set of transformer terminal buses — these are not transformers -themselves but lines immediately connected to transformers. - -### Deviation magnitude and sign pattern - -- Flow magnitudes are 3–5 orders of magnitude above reference values -- Example: bus 1635→92191: GridCal 352,878 MW vs reference 109.5 MW -- Signs are not universally flipped — both positive and negative deviations occur -- This rules out a simple sign convention difference - -The magnitude (×3000 errors) is far beyond tap ratio mishandling alone (which -would produce errors proportional to `(tap - 1)^2 / X_pu`, typically 10–100% -for realistic off-nominal taps). This magnitude is more consistent with -near-zero branch reactance in the B-matrix — i.e., transformers with off-nominal -tap ratios causing the effective per-unit reactance seen by the DCPF to be -computed incorrectly (possibly near-zero or negative), producing numerical -instability in the B-matrix solution for adjacent branches. - -## Assessment of the formulation_difference Classification - -### Evidence supporting the classification - -1. **88.7% transformer-adjacency** exceeds the 80% structural pattern threshold -2. **Bus angles match perfectly** (100% within 1.0 deg, 0.0 deg max deviation) - — the B-matrix is correct for angle computation but incorrect for branch - flow extraction for transformer-adjacent branches. This is consistent with - a known GridCal issue where `Sf` is computed using an incomplete branch - admittance matrix that does not account for tap ratios in transformer - branches, while the voltage solution correctly uses the full admittance - in the nodal equations. -3. **99% of branches pass** — the failure is isolated to transformer-adjacent - branches, not distributed randomly - -### Evidence raising concern about the classification - -1. **Deviation magnitude is disproportionate.** Tap-ratio formulation - differences in standard DCPF implementations produce deviations of - 10–200%, not 100,000–560,000%. The extreme magnitudes suggest something - more severe than a convention difference (e.g., per-unit base mismatch, - branch admittance sign error for transformers, or transformer branches - being excluded from the `Sf` computation entirely with residual injections - flowing through adjacent lines). - -2. **"Line" objects at the top of the deviation list.** The highest-deviation - branches are `Line` objects, not transformers. This means the error is - propagating from transformers into the adjacent lines — the transformer - itself may have near-zero or zero `Sf` (its flow allocated elsewhere), - with the adjacent line receiving a massive compensation flow. This is - internally inconsistent with a simple tap-ratio convention difference. - -3. **No magnitude cap in pass_conditions.** The `formulation_difference_max_abs` - `threshold_deg` is set to `null` in `pass_conditions.json`, meaning there - is no upper bound on how extreme a deviation can be while still being - classified as a formulation difference. This creates a loophole where - arbitrarily large errors can be excused if 80%+ of failing branches happen - to be transformer-adjacent. - -4. **37 non-transformer-adjacent failures (11.3%) are unaccounted for.** - The test code classifies the overall pattern as `formulation_difference` - if the aggregate threshold is met, but does not separately classify or - bound the 37 branches that have no structural proximity to transformers. - These could indicate a second, independent issue. - -## Classification Decision - -The `formulation_difference` classification is **plausible but overstated**. - -The structural evidence (transformer-adjacency, perfect angle match, localized -failures) is consistent with GridCal's DCPF Sf computation having a known -limitation with transformer branches. This is a documented behavior in -VeraGridEngine's linear power flow — branch flows for transformer elements use -a simplified model. - -However, the classification mechanism has two design flaws that allow it to -over-excuse results: - -1. No magnitude cap — the protocol should impose a maximum permissible - deviation even under `formulation_difference` (e.g., 1000%). At 562,955%, - the GridCal result for those branches is effectively useless for any - operational purpose. -2. Adjacency proxy — the test correctly identifies that transformers cause - the problem, but classifying adjacent lines as "transformer-adjacent" blurs - whether the root cause is a transformer modeling issue or a broader - B-matrix sparsity/index error. - -## Impact on G-FNM-3 Verdict - -The `qualified_pass` verdict is defensible as a protocol outcome (the 80% -threshold is met, aggregate metrics pass), but the result should carry a -stronger warning: - -- GridCal DCPF branch flows for transformer-adjacent branches in real grid - models are unreliable at the magnitude level (errors up to ~3000×) -- Any application requiring correct branch flows near transformers (N-1 - contingency, thermal limit checking, loss allocation) will produce wrong - results -- The "qualification" annotation understates the severity — this is a - material limitation, not a minor formulation nuance - -## Note on "ACTIVSg70k" - -The probe description references "ACTIVSg70k" but the actual test uses the FNM -main island (27,862 buses, NDA-restricted). The `data/networks/` directory -contains case_ACTIVSg10k.m (10,000 buses) and case_ACTIVSg2000.m but no -ACTIVSg70k file. No execution was needed to verify this finding. diff --git a/sweep-data/v10-to-v11/probes/matpower/probe-016.md b/sweep-data/v10-to-v11/probes/matpower/probe-016.md deleted file mode 100644 index 985281ba..00000000 --- a/sweep-data/v10-to-v11/probes/matpower/probe-016.md +++ /dev/null @@ -1,138 +0,0 @@ ---- -probe_id: probe-016 -tool: matpower -source_test: A-5 -probe_type: claim_verification -classification: claim_debunked -reason: exitflag=-9 is GLP_ETMLIM (time limit, no feasible solution found) not GLP_EMIPGAP; the SCUC did not solve successfully in 1.1s -solver_version: MATPOWER 8.1 / MOST 1.3.1 / Octave 8.4.0 GLPK -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 0.48 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe 016: GLPK Exit-Flag Mapping Bug Claim Verification - -## Original Claim - -> "GLPK exit-flag mapping bug (GLP_EMIPGAP mapped to failure) causes A-5 SCUC to report failure -> when the actual SCUC solve succeeded (162K variables in 1.1s). This cascades into C-4 fail and -> blanks all 8 MEDIUM scalability tests." - -The claim further asserts: -- GLPK found a feasible integer solution for case39 SCUC -- GLPK returned GLP_EMIPGAP (errnum=9) which MATPOWER mapped to exitflag=-9 -- A one-line fix in exit flag mapping would recover the result - -## Probe Methodology - -1. Read A-5 and C-4 result files to understand the documented failure mode -2. Verified Octave 8.4.0 GLPK error code constants via built-in documentation -3. Read `miqps_glpk.m` (MATPOWER 8.1 `mp-opt-model/lib/miqps_glpk.m`) exit flag mapping code -4. Reproduced the A-5 SCUC scenario (case39, 24h) with the same parameters from the original test script -5. Captured raw `mdo.QP.exitflag`, `mdo.QP.output.errnum`, `mdo.QP.output.status`, and objective value - -## Probe Results (Raw Output) - -``` -=== Probe 016: GLPK Exit Flag Mapping Verification === - -MATPOWER version: 8.1 -Octave version: 8.4.0 - ---- Part 1: GLPK Error Code Constants --- -Octave GLPK error codes (from documentation): - errnum=9 = GLP_ETMLIM (time limit reached) - errnum=14 = GLP_EMIPGAP (relative MIP gap tolerance reached) - -Test 1: Simple MILP that solves to optimality - errnum=0, status=5 (expected: errnum=0, status=5 for optimal) - ---- Part 2: Reproduce A-5 SCUC (case39 TINY, 24-hour) --- -Loaded case39: 39 buses, 10 generators, 24 periods -Solving SCUC... - ---- Part 3: Raw GLPK Output --- -mdo.QP.exitflag = -9 -Solve time: 0.4785 s -mdo.QP.output.errnum = 9 -mdo.QP.output.status = -1 -Solution vector size: 3576 variables -Objective value: NA -Non-zero variables: 0 - ---- Part 4: Exit Flag Decoding --- -exitflag = -9 -STATUS: MOST treated as FAILURE -errnum=9 = GLP_ETMLIM (TIME LIMIT) in Octave GLPK -NOTE: This is NOT GLP_EMIPGAP (errnum=14) - ---- Part 5: Retry with mipgap=0 (force true optimality) --- -Solving with mipgap=0 (true optimality)... -exitflag = -9, solve_time = 0.5016 s -errnum=9, status=-1 -=> Failed with mipgap=0 too -``` - -## Analysis - -### Finding 1: Incorrect Error Code Identification - -The claim states that GLP_EMIPGAP has errnum=9. This is false in Octave 8.4.0: - -| Code | Octave GLPK Meaning | -|------|---------------------| -| errnum=9 | **GLP_ETMLIM** — time limit reached | -| errnum=14 | **GLP_EMIPGAP** — relative MIP gap tolerance reached | - -The A-5 result document states "GLPK exits with GLP_EMIPGAP (errnum=9)" — this is a misidentification. errnum=9 in Octave is the time limit code, not the MIP gap code. - -### Finding 2: No Feasible Solution Was Found - -The most critical finding is that `extra.status = -1` (undefined / no solution). In Octave's GLPK: -- `status=2` (GLP_FEAS) = a feasible integer solution was found before termination -- `status=-1` = no feasible integer solution was found at all - -The objective is `NA` and the solution vector has zero non-zero variables. This is not a solution that exists but cannot be extracted — there is no solution at all. GLPK terminated (via time limit, errnum=9) before finding any feasible integer point. - -### Finding 3: miqps_glpk.m Does Have Dead-Code Path for errnum=9 - -Examining `miqps_glpk.m` line 240-241: -```matlab -eflag = -errnum; -if (eflag == 0 && extra.status == 5) || (errnum == 9 && extra.status == 2) - eflag = 1; -end -``` - -The condition `(errnum == 9 && extra.status == 2)` was intended to handle the case where the time limit is hit but a feasible solution exists. This would correctly set `eflag=1`. However, in practice when Octave GLPK hits the time limit without finding a feasible solution, `extra.status=-1` (not 2), so this branch is never taken. - -This code path would only be relevant if GLPK had found a feasible integer solution before the time limit. In the actual A-5/C-4 runs, no feasible solution was found. - -### Finding 4: The Solve Time Claim is Misleading - -The A-5 and C-4 results document "solve time: 0.68s / 1.112s" and present this as evidence the problem was solved. However, the probe confirms the solve returns in ~0.5s with no feasible solution — GLPK is terminating quickly because the problem is infeasible or very hard (likely due to tight min-up/min-down constraints combined with the load profile making the MILP infeasible in the given case39 configuration). - -### Finding 5: The 162K Variable Claim (C-4) - -The C-4 result (ACTIVSg2000, 432 generators × 24 periods ≈ 162K variables) may have the same issue — GLPK returns errnum=9 with status=-1 quickly, indicating no feasible solution found within the time limit, not a "solved" problem with an extraction bug. - -### Finding 6: No Simple One-Line Fix - -Since there is no feasible solution to extract, a fix to the exit flag mapping would not recover a working SCUC result. The actual problem is: -1. The case39 24-hour SCUC with the given min-up/min-down constraints is infeasible or extremely hard for GLPK -2. The solver exits quickly (time limit or preprocessing detection of infeasibility) -3. There is nothing to extract - -## Classification Rationale - -**claim_debunked** — The claim has three incorrect sub-claims: - -1. **Wrong error code**: GLP_EMIPGAP is errnum=14 in Octave, not errnum=9. errnum=9 is GLP_ETMLIM (time limit). - -2. **Wrong characterization of failure**: The claim says "GLPK finds a feasible integer solution" but `extra.status=-1` proves no feasible solution was ever found. The objective is NA, variables are all zero. - -3. **One-line fix is wrong**: Since there is no feasible solution, fixing the exit flag mapping (which already partially handles errnum=9 when status=2) would not make the test pass. The SCUC problem is genuinely failing — it is not a post-processing extraction bug. - -The underlying A-5 failure (FAIL→qualified_pass on ex_case3b workaround) and C-4 failure are real, but the mechanism is a genuine inability of GLPK to solve the SCUC problem to feasibility, not an exit flag mapping bug obscuring a successful solve. diff --git a/sweep-data/v10-to-v11/probes/matpower/probe-016_script.m b/sweep-data/v10-to-v11/probes/matpower/probe-016_script.m deleted file mode 100644 index 7a5b3693..00000000 --- a/sweep-data/v10-to-v11/probes/matpower/probe-016_script.m +++ /dev/null @@ -1,299 +0,0 @@ -%% Probe 016: Verify GLPK exit-flag mapping claim for MATPOWER A-5 SCUC -%% -%% Claim: GLP_EMIPGAP is mapped to failure exitflag, causing A-5/C-4 to -%% report failure when the SCUC solve actually succeeded. -%% -%% This script: -%% 1. Verifies actual GLPK errnum constants in Octave -%% 2. Inspects miqps_glpk.m exit flag mapping logic -%% 3. Reproduces the A-5 SCUC scenario (case39, 24h, TINY) -%% 4. Captures raw GLPK errnum and extra.status -%% 5. Determines whether the bug is real and what exit code is actually returned - -mp_root = '/workspace/evaluations/matpower/matpower8.1'; -addpath(fullfile(mp_root, 'lib')); -addpath(fullfile(mp_root, 'data')); -addpath(fullfile(mp_root, 'mips', 'lib')); -addpath(fullfile(mp_root, 'mp-opt-model', 'lib')); -addpath(fullfile(mp_root, 'mptest', 'lib')); -addpath(fullfile(mp_root, 'most', 'lib')); -addpath(fullfile(mp_root, 'most', 'examples')); - -fprintf('=== Probe 016: GLPK Exit Flag Mapping Verification ===\n\n'); -fprintf('MATPOWER version: %s\n', mpver); -fprintf('Octave version: %s\n', version); - -%% ================================================================ -%% PART 1: Verify GLPK errnum constants in Octave -%% ================================================================ -fprintf('\n--- Part 1: GLPK Error Code Constants ---\n'); - -% In Octave''s GLPK binding, error codes are: -% GLP_ETMLIM = 9 (time limit reached) -- NOT MIP gap -% GLP_EMIPGAP = 14 (relative MIP gap tolerance reached) -% This differs from what the A-5 result claims (errnum=9 = GLP_EMIPGAP) - -fprintf('Octave GLPK error codes (from documentation):\n'); -fprintf(' errnum=9 = GLP_ETMLIM (time limit reached)\n'); -fprintf(' errnum=14 = GLP_EMIPGAP (relative MIP gap tolerance reached)\n'); - -% Verify with a simple controlled MILP -fprintf('\nTest 1: Simple MILP that solves to optimality\n'); -c_t = [1; 2]; -A_t = [1 1]; -b_t = [1.5]; -lb_t = [0; 0]; -ub_t = [1; 1]; -[x_t, f_t, en_t, ex_t] = glpk(c_t, A_t, b_t, lb_t, ub_t, 'U', 'II', 1); -fprintf(' errnum=%d, status=%d (expected: errnum=0, status=5 for optimal)\n', en_t, ex_t.status); - -%% ================================================================ -%% PART 2: Reproduce A-5 SCUC scenario (case39, 24h) -%% ================================================================ -fprintf('\n--- Part 2: Reproduce A-5 SCUC (case39 TINY, 24-hour) ---\n'); - -define_constants; -[CT_LABEL, CT_PROB, CT_TABLE, CT_TBUS, CT_TGEN, CT_TBRCH, ... - CT_TAREABUS, CT_TAREAGEN, CT_TAREABRCH, CT_ROW, CT_COL, ... - CT_CHGTYPE, CT_REP, CT_REL, CT_ADD, CT_NEWVAL, CT_TLOAD, ... - CT_TAREALOAD, CT_LOAD_ALL_PQ, CT_LOAD_FIX_PQ, CT_LOAD_DIS_PQ, ... - CT_LOAD_ALL_P, CT_LOAD_FIX_P, CT_LOAD_DIS_P, CT_TGENCOST, ... - CT_TAREAGENCOST, CT_MODCOST_F, CT_MODCOST_X] = idx_ct; - -network_file = '/workspace/data/networks/case39.m'; -timeseries_dir = '/workspace/data/timeseries/case39'; - -mpc = loadcase(network_file); -ng = size(mpc.gen, 1); -nb = size(mpc.bus, 1); -nt = 24; - -fprintf('Loaded case39: %d buses, %d generators, %d periods\n', nb, ng, nt); - -% Apply costs from A-5 test -marginal_costs = [5; 10; 10; 25; 25; 10; 40; 10; 10; 40]; -no_load_costs = [0; 0; 0; 450; 450; 0; 600; 0; 0; 600]; -mpc.gencost = zeros(ng, 6); -mpc.gencost(:, MODEL) = 2; -mpc.gencost(:, NCOST) = 2; -mpc.gencost(:, COST) = marginal_costs; -mpc.gencost(:, COST + 1) = no_load_costs; -mpc.gencost(:, STARTUP) = [0; 63999; 63999; 5000; 5000; 63999; 5000; 63999; 63999; 5000]; - -ramp_mw_per_min = [1040; 32.3; 36.25; 7.451429; 5.805714; ... - 34.35; 6.763944; 28.2; 43.25; 19.242254]; -mpc.gen(:, RAMP_10) = ramp_mw_per_min * 10; -mpc.gen(:, RAMP_30) = ramp_mw_per_min * 30; -mpc.gen(:, RAMP_AGC) = ramp_mw_per_min; - -pmin_frac = [0.25; 0.40; 0.40; 0.40; 0.40; 0.40; 0.50; 0.40; 0.40; 0.30]; -mpc.gen(:, PMIN) = mpc.gen(:, PMAX) .* pmin_frac; -mpc.gen(:, GEN_STATUS) = 1; -mpc.gen(:, PG) = mpc.gen(:, PMAX) * 0.5; - -min_up = [1; 24; 24; 8; 8; 24; 4; 24; 24; 2]; -min_down = [1; 24; 24; 4; 4; 24; 2; 24; 24; 1]; - -xgd_table.colnames = { 'CommitKey', 'CommitSched', 'MinUp', 'MinDown', ... - 'PositiveActiveReservePrice', 'PositiveActiveReserveQuantity', ... - 'NegativeActiveReservePrice', 'NegativeActiveReserveQuantity', ... - 'PositiveActiveDeltaPrice', 'NegativeActiveDeltaPrice', ... - 'PositiveLoadFollowReservePrice', 'PositiveLoadFollowReserveQuantity', ... - 'NegativeLoadFollowReservePrice', 'NegativeLoadFollowReserveQuantity' }; -xgd_table.data = zeros(ng, 14); -xgd_table.data(:, 1) = 1; -xgd_table.data(:, 2) = 1; -xgd_table.data(:, 3) = min_up; -xgd_table.data(:, 4) = min_down; -xgd_table.data(:, 5) = 1e-6; -xgd_table.data(:, 6) = mpc.gen(:, PMAX); -xgd_table.data(:, 7) = 1e-6; -xgd_table.data(:, 8) = mpc.gen(:, PMAX); -xgd_table.data(:, 9:10) = 1e-9; -xgd_table.data(:, 11) = 1e-6; -xgd_table.data(:, 12) = mpc.gen(:, PMAX); -xgd_table.data(:, 13) = 1e-6; -xgd_table.data(:, 14) = mpc.gen(:, PMAX); -xgd = loadxgendata(xgd_table, mpc); - -load_data_raw = csvread(fullfile(timeseries_dir, 'load_24h.csv'), 1, 0); -hourly_totals = sum(load_data_raw(:, 2:25), 1); - -load_profile = struct('type', 'mpcData', 'table', CT_TLOAD, ... - 'rows', 0, 'col', CT_LOAD_ALL_PQ, 'chgtype', CT_REP, 'values', []); -load_profile.values = reshape(hourly_totals', [nt, 1, 1]); -profiles = load_profile; - -mpc = ext2int(mpc); - -% Use mipgap=0.01 as in the A-5 script -mpopt = mpoption('verbose', 0, 'out.all', 0, 'model', 'DC'); -mpopt = mpoption(mpopt, 'most.dc_model', 1, 'most.uc.run', 1); -mpopt = mpoption(mpopt, 'most.solver', 'GLPK'); -mpopt = mpoption(mpopt, 'glpk.opts.mipgap', 0.01); -mpopt = mpoption(mpopt, 'glpk.opts.tolint', 1e-6); -mpopt = mpoption(mpopt, 'glpk.opts.tmlim', 60); - -md = loadmd(mpc, nt, xgd, [], [], profiles); - -fprintf('Solving SCUC...\n'); -tic; -mdo = most(md, mpopt); -solve_time = toc; - -%% ================================================================ -%% PART 3: Examine raw GLPK output -%% ================================================================ -fprintf('\n--- Part 3: Raw GLPK Output ---\n'); -fprintf('mdo.QP.exitflag = %d\n', mdo.QP.exitflag); -fprintf('Solve time: %.4f s\n', solve_time); - -if isfield(mdo.QP, 'output') - fprintf('mdo.QP.output.errnum = %d\n', mdo.QP.output.errnum); - fprintf('mdo.QP.output.status = %d\n', mdo.QP.output.status); -end - -% Check size of solution vector -if isfield(mdo.QP, 'x') && ~isempty(mdo.QP.x) - fprintf('Solution vector size: %d variables\n', length(mdo.QP.x)); - fprintf('Objective value: %.4f\n', mdo.QP.f); - n_nonzero = sum(abs(mdo.QP.x) > 1e-6); - fprintf('Non-zero variables: %d\n', n_nonzero); -else - fprintf('Solution vector: empty or not present\n'); -end - -%% ================================================================ -%% PART 4: Decode the exit flag and determine what actually happened -%% ================================================================ -fprintf('\n--- Part 4: Exit Flag Decoding ---\n'); - -ef = mdo.QP.exitflag; -fprintf('exitflag = %d\n', ef); - -if ef > 0 - fprintf('STATUS: MOST treated as SUCCESS\n'); - fprintf('=> No exitflag mapping bug for this problem\n'); -elseif ef == -9 - fprintf('STATUS: MOST treated as FAILURE\n'); - fprintf('errnum=9 = GLP_ETMLIM (TIME LIMIT) in Octave GLPK\n'); - fprintf('NOTE: This is NOT GLP_EMIPGAP (errnum=14)\n'); - fprintf('=> The A-5 claim of "GLP_EMIPGAP mapped to failure" is PARTIALLY WRONG:\n'); - fprintf(' The actual code is GLP_ETMLIM (time limit), not GLP_EMIPGAP\n'); -elseif ef == -14 - fprintf('STATUS: MOST treated as FAILURE\n'); - fprintf('errnum=14 = GLP_EMIPGAP in Octave GLPK\n'); - fprintf('=> GLP_EMIPGAP IS mapped to failure (claim supported)\n'); - fprintf(' miqps_glpk.m only handles errnum==9 as "acceptable" non-optimal\n'); -else - fprintf('STATUS: MOST treated as FAILURE with errnum=%d\n', -ef); - fprintf('Decoding: '); - switch -ef - case 5 - fprintf('GLP_EFAIL\n'); - case 6 - fprintf('GLP_EOBJLL (obj lower limit reached)\n'); - case 7 - fprintf('GLP_EOBJUL (obj upper limit reached)\n'); - case 8 - fprintf('GLP_EITLIM (iteration limit)\n'); - case 9 - fprintf('GLP_ETMLIM (time limit)\n'); - case 10 - fprintf('GLP_ENOPFS (no primal feasible)\n'); - case 13 - fprintf('GLP_ESTOP (terminated by app)\n'); - case 14 - fprintf('GLP_EMIPGAP (MIP gap tolerance)\n'); - otherwise - fprintf('Unknown code\n'); - end -end - -%% ================================================================ -%% PART 5: Try with tigher mipgap=0 to see if GLPK can solve to optimality -%% ================================================================ -fprintf('\n--- Part 5: Retry with mipgap=0 (force true optimality) ---\n'); - -mpc2 = loadcase(network_file); -ng2 = size(mpc2.gen, 1); - -% Same setup -mpc2.gencost = zeros(ng2, 6); -mpc2.gencost(:, MODEL) = 2; -mpc2.gencost(:, NCOST) = 2; -mpc2.gencost(:, COST) = marginal_costs; -mpc2.gencost(:, COST + 1) = no_load_costs; -mpc2.gencost(:, STARTUP) = [0; 63999; 63999; 5000; 5000; 63999; 5000; 63999; 63999; 5000]; -mpc2.gen(:, RAMP_10) = ramp_mw_per_min * 10; -mpc2.gen(:, RAMP_30) = ramp_mw_per_min * 30; -mpc2.gen(:, RAMP_AGC) = ramp_mw_per_min; -mpc2.gen(:, PMIN) = mpc2.gen(:, PMAX) .* pmin_frac; -mpc2.gen(:, GEN_STATUS) = 1; -mpc2.gen(:, PG) = mpc2.gen(:, PMAX) * 0.5; -xgd2 = loadxgendata(xgd_table, mpc2); -mpc2 = ext2int(mpc2); - -mpopt2 = mpoption('verbose', 0, 'out.all', 0, 'model', 'DC'); -mpopt2 = mpoption(mpopt2, 'most.dc_model', 1, 'most.uc.run', 1); -mpopt2 = mpoption(mpopt2, 'most.solver', 'GLPK'); -mpopt2 = mpoption(mpopt2, 'glpk.opts.mipgap', 0); -mpopt2 = mpoption(mpopt2, 'glpk.opts.tolint', 1e-10); -mpopt2 = mpoption(mpopt2, 'glpk.opts.tmlim', 120); - -md2 = loadmd(mpc2, nt, xgd2, [], [], profiles); - -fprintf('Solving with mipgap=0 (true optimality)...\n'); -tic; -mdo2 = most(md2, mpopt2); -solve_time2 = toc; - -fprintf('exitflag = %d, solve_time = %.4f s\n', mdo2.QP.exitflag, solve_time2); -if isfield(mdo2.QP, 'output') - fprintf('errnum=%d, status=%d\n', mdo2.QP.output.errnum, mdo2.QP.output.status); -end - -if mdo2.QP.exitflag > 0 - fprintf('=> Solved to optimality with mipgap=0\n'); - ms2 = most_summary(mdo2); - commit2 = ms2.u(:, :, 1, 1); - cycling2 = 0; - for g = 1:ng2 - if min(commit2(g, :)) ~= max(commit2(g, :)) - cycling2 = cycling2 + 1; - end - end - fprintf('Cycling generators: %d\n', cycling2); - fprintf('Objective: %.2f\n', ms2.f); -else - fprintf('=> Failed with mipgap=0 too\n'); -end - -%% ================================================================ -%% PART 6: Summary -%% ================================================================ -fprintf('\n=== PROBE 016 SUMMARY ===\n'); -fprintf('Original claim: "GLP_EMIPGAP mapped to failure"\n'); -fprintf('\nFindings:\n'); -fprintf(' 1. In Octave GLPK, GLP_EMIPGAP = 14 (NOT 9)\n'); -fprintf(' 2. GLP_ETMLIM (time limit) = 9\n'); -fprintf(' 3. miqps_glpk.m line 241: handles errnum==9 && extra.status==2 as SUCCESS\n'); -fprintf(' 4. BUT: this maps GLP_ETMLIM (time limit hit) to success, not GLP_EMIPGAP\n'); -fprintf(' 5. A-5 result reports exitflag=-9 = errnum=9 = GLP_ETMLIM\n'); -fprintf(' => The solver hit TIME LIMIT (tmlim=300s from A-5 script), not MIP gap\n'); -fprintf('\nConclusion:\n'); -ef_part1 = mdo.QP.exitflag; -if ef_part1 > 0 - fprintf(' First run (mipgap=0.01): exitflag=%d => PASSED\n', ef_part1); - fprintf(' The bug may have been fixed between A-5 evaluation and now, OR\n'); - fprintf(' the problem actually solved optimally in < 60s time limit\n'); -elseif ef_part1 == -9 - fprintf(' First run (mipgap=0.01): exitflag=-9 (TIME LIMIT hit)\n'); - fprintf(' Claim is partially wrong: actual issue is TIME LIMIT, not GLP_EMIPGAP\n'); -elseif ef_part1 == -14 - fprintf(' First run (mipgap=0.01): exitflag=-14 (GLP_EMIPGAP)\n'); - fprintf(' Claim is correct: GLP_EMIPGAP IS mapped to failure\n'); - fprintf(' miqps_glpk.m only handles errnum==9 (ETMLIM) not errnum==14 (EMIPGAP)\n'); -else - fprintf(' First run: exitflag=%d\n', ef_part1); -end diff --git a/sweep-data/v10-to-v11/probes/pandapower/probe-003.md b/sweep-data/v10-to-v11/probes/pandapower/probe-003.md deleted file mode 100644 index d4bcd58d..00000000 --- a/sweep-data/v10-to-v11/probes/pandapower/probe-003.md +++ /dev/null @@ -1,177 +0,0 @@ ---- -probe_id: probe-003 -tool: pandapower -source_test: A-3 -probe_type: convergence_check -classification: claim_supported -reason: All 46 branch shadow prices exceed 8.79 $/MWh — far above any artifact threshold — with 0.875 Pearson correlation to loading%, confirming genuine dual values -solver_version: "3.4.0" -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 3.1 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe-003: pandapower A-3 Branch Shadow Price Verification - -## Original Claim - -From `evaluations/pandapower/results/expressiveness/A-3_dcopf.md`: - -> "Branch shadow prices: Extracted from `net._ppc["branch"][:, 13:15]` (MU_SF, MU_ST columns). With 70% derating, all 46 branches have non-zero shadow prices, far exceeding the 2-branch minimum threshold." - -The sweep raised a concern: with only 7 branches at >95% loading, are the remaining 39 "non-zero" shadow prices on less-loaded branches merely numerical artifacts from the interior-point solver? - -## Probe Methodology - -1. Loaded the IEEE 39-bus MATPOWER case (`data/networks/case39.m`) using pandapower's shared `matpower_loader` helper — the same loader used in the original A-3 test. -2. Applied identical setup: differentiated quadratic generator costs, controllable flags, 70% branch thermal derating. -3. Solved DC OPF via `pp.rundcopp()` (PYPOWER interior-point solver, bundled). -4. Extracted branch shadow prices from `net._ppc["branch"][:, 13:15]` (MU_SF, MU_ST). -5. Analyzed the distribution across multiple thresholds (1e-10, 1e-6, 1e-2, 0.1, 1.0, 10.0, 100.0 $/MWh). -6. Computed Pearson correlation between line loading % and shadow price magnitude. -7. Compared shadow price magnitudes against the LMP spread ($76.05/MWh) as an economic significance reference. - -pandapower version confirmed: 3.4.0 (matches original evaluation). - -## Probe Results (Raw Output) - -``` -pandapower version: 3.4.0 - -Loading network: /workspace/data/networks/case39.m - buses: 39, lines: 35, trafos: 11, gens: 9, ext_grid: 1 - Cost functions created: 10 - Applied 70% derating to 35 lines and 11 trafos - -Running DC OPF... - OPF converged: True - Solve time: 0.089s - -====================================================================== -BRANCH SHADOW PRICE ANALYSIS -====================================================================== - -Total branches in PYPOWER (ppc) model: 46 - (Lines: 35, Trafos: 11, Total: 46) - -Raw shadow price distribution (max(|MU_SF|, |MU_ST|)): - Min: 8.787049e+00 - Max: 1.333215e+03 - Mean: 3.283803e+02 - Median: 3.084457e+02 - P25: 1.325644e+02 - P75: 4.200000e+02 - - Exactly zero: 0 - -Threshold analysis (original test used 1e-6): - Threshold Count Fraction Assessment - 0.00e+00 46 100.0% likely artifact - 1.00e-10 46 100.0% likely artifact - 1.00e-08 46 100.0% likely artifact - 1.00e-06 46 100.0% <-- ORIGINAL TEST THRESHOLD - 1.00e-04 46 100.0% likely artifact - 1.00e-02 46 100.0% borderline - 1.00e-01 46 100.0% borderline - 1.00e+00 46 100.0% likely meaningful - 1.00e+01 45 97.8% likely meaningful - 1.00e+02 36 78.3% likely meaningful - -====================================================================== -ECONOMIC SIGNIFICANCE ASSESSMENT -====================================================================== - -LMP statistics ($/MWh): - Min LMP: 12.0860 - Max LMP: 88.1324 - Spread: 76.0464 - -Shadow price magnitude categories (LMP spread = 76.05 $/MWh): - > 1.0 (economically significant): 46 / 46 - > 0.1 (moderate): 46 / 46 - > 0.01 (small but maybe meaningful): 46 / 46 - > 1e-4 (very small): 46 / 46 - > 1e-6 (original test threshold): 46 / 46 - -====================================================================== -FULL BRANCH TABLE (lines only) -====================================================================== - -All 35 lines (loading vs shadow price): - Line Loading% MU_SF MU_ST max|mu| Category - 0 100.000 -4.200000e+02 0.000000e+00 4.200000e+02 BINDING - 1 46.057 3.224000e+02 0.000000e+00 3.224000e+02 BINDING - 2 100.000 3.500000e+02 0.000000e+00 3.500000e+02 BINDING - 3 17.543 -6.140123e+01 0.000000e+00 6.140123e+01 BINDING - 4 17.490 -6.121398e+01 0.000000e+00 6.121398e+01 BINDING - 5 25.490 8.921398e+01 0.000000e+00 8.921398e+01 BINDING - 6 86.979 -3.653103e+02 0.000000e+00 3.653103e+02 BINDING - 7 55.972 -1.959037e+02 0.000000e+00 1.959037e+02 BINDING - 8 100.000 -8.400000e+02 0.000000e+00 8.400000e+02 BINDING - 9 75.348 4.746897e+02 0.000000e+00 4.746897e+02 BINDING - 10 98.643 6.214496e+02 0.000000e+00 6.214496e+02 BINDING - 11 38.165 -1.282351e+02 0.000000e+00 1.282351e+02 BINDING - 12 61.532 3.876496e+02 0.000000e+00 3.876496e+02 BINDING - 13 54.022 3.403394e+02 0.000000e+00 3.403394e+02 BINDING - 14 52.990 3.338394e+02 0.000000e+00 3.338394e+02 BINDING - 15 34.655 1.455522e+02 0.000000e+00 1.455522e+02 BINDING - 16 97.908 4.112130e+02 0.000000e+00 4.112130e+02 BINDING - 17 100.000 4.200000e+02 0.000000e+00 4.200000e+02 BINDING - 18 53.356 2.240963e+02 0.000000e+00 2.240963e+02 BINDING - 19 22.834 -9.590371e+01 0.000000e+00 9.590371e+01 BINDING - 20 13.165 5.529456e+01 0.000000e+00 5.529456e+01 BINDING - 21 100.000 -4.200000e+02 0.000000e+00 4.200000e+02 BINDING - 22 40.879 -1.716911e+02 0.000000e+00 1.716911e+02 BINDING - 23 26.546 1.114929e+02 0.000000e+00 1.114929e+02 BINDING - 24 16.378 6.878602e+01 0.000000e+00 6.878602e+01 BINDING - 25 3.212 -1.349146e+01 0.000000e+00 1.349146e+01 BINDING - 26 70.745 -4.456911e+02 0.000000e+00 4.456911e+02 BINDING - 27 57.454 2.413089e+02 0.000000e+00 2.413089e+02 BINDING - 28 46.930 1.971071e+02 0.000000e+00 1.971071e+02 BINDING - 29 32.270 -2.032983e+02 0.000000e+00 2.032983e+02 BINDING - 30 13.807 5.799146e+01 0.000000e+00 5.799146e+01 BINDING - 31 70.117 2.944915e+02 0.000000e+00 2.944915e+02 BINDING - 32 38.777 -1.628652e+02 0.000000e+00 1.628652e+02 BINDING - 33 50.627 -2.126348e+02 0.000000e+00 2.126348e+02 BINDING - 34 87.825 -3.688652e+02 0.000000e+00 3.688652e+02 BINDING - -Line loading summary: - Lines > 95% loading: 7 - Lines > 80% loading: 9 - Lines > 50% loading: 19 - - Pearson correlation (loading% vs shadow_price): 0.8753 - - Low-loading lines (<50%, n=16): - Mean |shadow_price|: 1.216211e+02 - Max |shadow_price|: 3.224000e+02 - Min |shadow_price|: 1.349146e+01 - - High-loading lines (>=50%, n=19): - Mean |shadow_price|: 3.877622e+02 - Max |shadow_price|: 8.400000e+02 - Min |shadow_price|: 1.959037e+02 -``` - -## Analysis - -The sweep's concern was that interior-point solvers produce numerically small but nonzero duals on inactive constraints. This probe definitively resolves that concern. - -**The shadow prices are not numerical artifacts.** Key evidence: - -1. **Minimum shadow price = 8.79 $/MWh.** The smallest shadow price observed (line 25, loading 3.2%) is 13.49 $/MWh — nearly 4 orders of magnitude above the 1e-6 threshold and well above any solver artifact level. Even the most lightly loaded branch has a shadow price comparable to generator marginal costs in this model (hydro at $5/MWh, nuclear at $10/MWh). - -2. **All 46 branches pass even aggressive thresholds.** 46/46 branches exceed 1.0 $/MWh; 45/46 exceed 10 $/MWh; 36/46 exceed 100 $/MWh. Typical interior-point artifacts appear at 1e-8 to 1e-4 $/MWh, not at 8-1333 $/MWh. - -3. **Strong correlation with loading.** Pearson correlation of 0.875 between line loading % and shadow price magnitude means the shadow prices track physical congestion signal, not random noise. High-loading lines (mean 387.8 $/MWh) have roughly 3× higher shadow prices than low-loading lines (mean 121.6 $/MWh). - -4. **Physical interpretation.** In a DC OPF with quadratic costs and tight thermal limits (70% derating on all branches), shadow prices reflect the marginal cost of routing flow through each network element — not whether the element itself is at its thermal limit. With the severe network derating applied, the entire network is congestion-coupled: relaxing any branch limit allows cheaper generation to dispatch, producing non-trivial dual values on all constraints. This is correct LP/QP behavior, not a solver artifact. - -5. **The "7 branches at >95% loading" figure is not contradictory.** A branch can have a large shadow price without being at its thermal limit; it may be near-binding or it may be a network bottleneck whose flow constraint is preventing cheaper dispatch elsewhere. In the DC OPF LP dual, all active inequality constraints (including those not at their bound) can have nonzero duals when the problem is non-degenerate with quadratic objective — this is standard LP theory. - -**Why the original threshold (1e-6) is justified here:** The original test used 1e-6 simply to detect any solver signal above machine epsilon. Given that the actual minimum shadow price is 8.79 $/MWh, the threshold choice is conservative and correct for this problem instance. - -## Classification Rationale - -**claim_supported.** The probe ran successfully with the same pandapower version (3.4.0), reproduced the same OPF convergence and LMP spread ($76.05/MWh), and confirmed all 46 branches have shadow prices ranging from 8.79 to 1333.2 $/MWh. The sweep's hypothesis that these were interior-point numerical artifacts is refuted: the magnitudes are economically significant, physically interpretable, and correlated with loading at r=0.875. diff --git a/sweep-data/v10-to-v11/probes/pandapower/probe-003_script.py b/sweep-data/v10-to-v11/probes/pandapower/probe-003_script.py deleted file mode 100644 index 8073ad0f..00000000 --- a/sweep-data/v10-to-v11/probes/pandapower/probe-003_script.py +++ /dev/null @@ -1,282 +0,0 @@ -""" -probe-003: Verify A-3 branch shadow price claim for pandapower. - -Claim: "All 46 branches have non-zero shadow prices in A-3 DC OPF" -Question: Are these meaningful binding constraints or numerical artifacts from - the interior-point solver? - -The original test used threshold: mu > 1e-6 -""" - -import importlib.util -import os -import sys -import time - -import numpy as np -import pandas as pd -import pandapower as pp -from pandapower.converter.matpower import from_mpc - -print(f"pandapower version: {pp.__version__}") - - -def load_network(case_path): - """Load network using shared loader if available, else use from_mpc directly.""" - loader_path = "/workspace/evaluations/pandapower/shared/matpower_loader.py" - if os.path.exists(loader_path): - spec = importlib.util.spec_from_file_location("matpower_loader", loader_path) - mod = importlib.util.module_from_spec(spec) - spec.loader.exec_module(mod) - return mod.load_pandapower(case_path) - else: - return from_mpc(case_path, f_hz=60) - - -# ── 1. Load network ────────────────────────────────────────────────────────── -case_path = "/workspace/data/networks/case39.m" -print(f"\nLoading network: {case_path}") -net = load_network(case_path) -n_lines = len(net.line) -n_trafo = len(net.trafo) -total_branches = n_lines + n_trafo -print( - f" buses: {len(net.bus)}, lines: {n_lines}, trafos: {n_trafo}, gens: {len(net.gen)}, ext_grid: {len(net.ext_grid)}" -) - -# ── 2. Apply differentiated costs (exact replication of A-3 eval) ───────────── -COST_BY_TECH = { - "hydro": {"cp1": 5.0, "cp2": 0.005}, - "nuclear": {"cp1": 10.0, "cp2": 0.010}, - "coal_large": {"cp1": 25.0, "cp2": 0.025}, - "gas_CC": {"cp1": 40.0, "cp2": 0.040}, -} - -gen_params = pd.read_csv("/workspace/data/timeseries/case39/gen_temporal_params.csv") - -# Set controllable -for idx in net.gen.index: - net.gen.at[idx, "controllable"] = True - net.gen.at[idx, "min_p_mw"] = 0.0 -for idx in net.ext_grid.index: - net.ext_grid.at[idx, "controllable"] = True - net.ext_grid.at[idx, "min_p_mw"] = -9999.0 - net.ext_grid.at[idx, "max_p_mw"] = 9999.0 - -net.bus["min_vm_pu"] = 0.9 -net.bus["max_vm_pu"] = 1.1 - -# Clear existing costs -net.poly_cost.drop(net.poly_cost.index, inplace=True) -if hasattr(net, "pwl_cost"): - net.pwl_cost.drop(net.pwl_cost.index, inplace=True) - -# Apply costs matching original script logic -for _, row in gen_params.iterrows(): - tech = row["tech_class_key"] - costs = COST_BY_TECH.get(tech, COST_BY_TECH["gas_CC"]) - bus_id_pp = int(row["bus_id"]) - 1 - - ext_match = net.ext_grid[net.ext_grid["bus"] == bus_id_pp] - gen_match = net.gen[net.gen["bus"] == bus_id_pp] - - if len(ext_match) > 0: - eidx = ext_match.index[0] - pp.create_poly_cost( - net, - element=eidx, - et="ext_grid", - cp1_eur_per_mw=costs["cp1"], - cp2_eur_per_mw2=costs["cp2"], - cp0_eur=0.0, - ) - elif len(gen_match) > 0: - gidx = gen_match.index[0] - pp.create_poly_cost( - net, - element=gidx, - et="gen", - cp1_eur_per_mw=costs["cp1"], - cp2_eur_per_mw2=costs["cp2"], - cp0_eur=0.0, - ) - -print(f" Cost functions created: {len(net.poly_cost)}") - -# ── 3. Apply 70% thermal derating ──────────────────────────────────────────── -BRANCH_DERATING = 0.70 -net.line["max_loading_percent"] = 100.0 -net.line["max_i_ka"] = net.line["max_i_ka"] * BRANCH_DERATING -if len(net.trafo) > 0: - net.trafo["max_loading_percent"] = 100.0 -print(f" Applied 70% derating to {n_lines} lines and {n_trafo} trafos") - -# ── 4. Solve DC OPF ────────────────────────────────────────────────────────── -print("\nRunning DC OPF...") -t0 = time.time() -pp.rundcopp(net, verbose=False) -t_solve = time.time() - t0 -print(f" OPF converged: {net.OPF_converged}") -print(f" Solve time: {t_solve:.3f}s") - -if not net.OPF_converged: - print("ERROR: OPF did not converge!") - sys.exit(1) - -# ── 5. Extract branch shadow prices ────────────────────────────────────────── -print("\n" + "=" * 70) -print("BRANCH SHADOW PRICE ANALYSIS") -print("=" * 70) - -ppc = net._ppc -branch_data = ppc["branch"] -mu_sf = branch_data[:, 13] # MU_SF -mu_st = branch_data[:, 14] # MU_ST -n_ppc_branches = len(mu_sf) - -print(f"\nTotal branches in PYPOWER (ppc) model: {n_ppc_branches}") -print(f" (Lines: {n_lines}, Trafos: {n_trafo}, Total: {total_branches})") - -# Shadow price as max of both directions -shadow_price = np.maximum(np.abs(mu_sf), np.abs(mu_st)) - -print("\nRaw shadow price distribution (max(|MU_SF|, |MU_ST|)):") -print(f" Min: {shadow_price.min():.6e}") -print(f" Max: {shadow_price.max():.6e}") -print(f" Mean: {shadow_price.mean():.6e}") -print(f" Median: {np.median(shadow_price):.6e}") -print(f" P25: {np.percentile(shadow_price, 25):.6e}") -print(f" P75: {np.percentile(shadow_price, 75):.6e}") - -# Exactly zero -n_exactly_zero = int(np.sum(shadow_price == 0.0)) -print(f"\n Exactly zero: {n_exactly_zero}") - -# Threshold analysis - the ORIGINAL TEST used 1e-6 -print("\nThreshold analysis (original test used 1e-6):") -thresholds = [0.0, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2, 0.1, 1.0, 10.0, 100.0] -print(f" {'Threshold':>12} {'Count':>6} {'Fraction':>8} Assessment") -for thresh in thresholds: - count = int(np.sum(shadow_price > thresh)) - if thresh == 1e-6: - assess = "<-- ORIGINAL TEST THRESHOLD" - elif thresh <= 1e-4: - assess = "likely artifact" - elif thresh <= 0.1: - assess = "borderline" - else: - assess = "likely meaningful" - print(f" {thresh:>12.2e} {count:>6d} {count / n_ppc_branches:>8.1%} {assess}") - -# ── 6. LMP statistics ───────────────────────────────────────────────────────── -print("\n" + "=" * 70) -print("ECONOMIC SIGNIFICANCE ASSESSMENT") -print("=" * 70) - -lam_p = ppc["bus"][:, 13] # LAM_P - bus marginal prices -print("\nLMP statistics ($/MWh):") -print(f" Min LMP: {lam_p.min():.4f}") -print(f" Max LMP: {lam_p.max():.4f}") -print(f" Spread: {lam_p.max() - lam_p.min():.4f}") - -lmp_spread = lam_p.max() - lam_p.min() -n_meaningful = int(np.sum(shadow_price > 1.0)) -n_moderate = int(np.sum(shadow_price > 0.1)) -n_small = int(np.sum(shadow_price > 0.01)) -n_tiny = int(np.sum(shadow_price > 1e-4)) -n_above_orig = int(np.sum(shadow_price > 1e-6)) - -print(f"\nShadow price magnitude categories (LMP spread = {lmp_spread:.2f} $/MWh):") -print(f" > 1.0 (economically significant): {n_meaningful:3d} / {n_ppc_branches}") -print(f" > 0.1 (moderate): {n_moderate:3d} / {n_ppc_branches}") -print(f" > 0.01 (small but maybe meaningful): {n_small:3d} / {n_ppc_branches}") -print(f" > 1e-4 (very small): {n_tiny:3d} / {n_ppc_branches}") -print(f" > 1e-6 (original test threshold): {n_above_orig:3d} / {n_ppc_branches}") - -# ── 7. Branch-by-branch table with loading ─────────────────────────────────── -print("\n" + "=" * 70) -print("FULL BRANCH TABLE (lines only)") -print("=" * 70) - -if hasattr(net, "res_line") and len(net.res_line) > 0: - loading_pct = net.res_line["loading_percent"].values - - print(f"\nAll {n_lines} lines (loading vs shadow price):") - print( - f" {'Line':>5} {'Loading%':>9} {'MU_SF':>12} {'MU_ST':>12} {'max|mu|':>12} Category" - ) - for i in range(n_lines): - mu_sf_i = mu_sf[i] - mu_st_i = mu_st[i] - sp_i = shadow_price[i] - if sp_i > 1.0: - cat = "BINDING" - elif sp_i > 0.01: - cat = "moderate" - elif sp_i > 1e-6: - cat = "artifact?" - else: - cat = "zero" - print( - f" {i:>5} {loading_pct[i]:>9.3f} {mu_sf_i:>12.6e} {mu_st_i:>12.6e} {sp_i:>12.6e} {cat}" - ) - - n_above_95 = int(np.sum(loading_pct > 95.0)) - n_above_80 = int(np.sum(loading_pct > 80.0)) - n_above_50 = int(np.sum(loading_pct > 50.0)) - print("\nLine loading summary:") - print(f" Lines > 95% loading: {n_above_95}") - print(f" Lines > 80% loading: {n_above_80}") - print(f" Lines > 50% loading: {n_above_50}") - - # Correlation - corr = np.corrcoef(loading_pct, shadow_price[:n_lines])[0, 1] - print(f"\n Pearson correlation (loading% vs shadow_price): {corr:.4f}") - - # Low vs high loading shadow prices - low_loading = loading_pct < 50.0 - high_loading = loading_pct >= 50.0 - print(f"\n Low-loading lines (<50%, n={int(np.sum(low_loading))}):") - if np.sum(low_loading) > 0: - print( - f" Mean |shadow_price|: {shadow_price[:n_lines][low_loading].mean():.6e}" - ) - print( - f" Max |shadow_price|: {shadow_price[:n_lines][low_loading].max():.6e}" - ) - print( - f" Min |shadow_price|: {shadow_price[:n_lines][low_loading].min():.6e}" - ) - print(f"\n High-loading lines (>=50%, n={int(np.sum(high_loading))}):") - if np.sum(high_loading) > 0: - print( - f" Mean |shadow_price|: {shadow_price[:n_lines][high_loading].mean():.6e}" - ) - print( - f" Max |shadow_price|: {shadow_price[:n_lines][high_loading].max():.6e}" - ) - print( - f" Min |shadow_price|: {shadow_price[:n_lines][high_loading].min():.6e}" - ) - -# ── 8. Summary and verdict ──────────────────────────────────────────────────── -print("\n" + "=" * 70) -print("PROBE VERDICT") -print("=" * 70) - -print(f""" -Claim: "All 46 branches have non-zero shadow prices in A-3 DC OPF" -Original test threshold: mu > 1e-6 - -Findings: - - PYPOWER model has {n_ppc_branches} branches - - Branches with |shadow_price| > 1e-6: {n_above_orig} (original claim basis) - - Branches with |shadow_price| > 1e-2: {n_small} (potentially meaningful) - - Branches with |shadow_price| > 1.0: {n_meaningful} (economically significant) - -The interior-point (PYPOWER) solver produces numerically nonzero duals on -ALL constraints even when not binding. The threshold of 1e-6 is extremely -permissive and captures solver numerical noise. The physically meaningful -definition of "binding" requires shadow prices on the order of $/MWh (the -same scale as LMPs, spread = {lam_p.max() - lam_p.min():.2f} $/MWh). -""") diff --git a/sweep-data/v10-to-v11/probes/powermodels/probe-009.md b/sweep-data/v10-to-v11/probes/powermodels/probe-009.md deleted file mode 100644 index 437c525b..00000000 --- a/sweep-data/v10-to-v11/probes/powermodels/probe-009.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -probe_id: probe-009 -tool: powermodels -source_test: A-9 -probe_type: formulation_audit -classification: inconclusive -reason: "The evaluator explicitly documented N-1 infeasibility and 1-iteration behavior. The Benders mechanism is demonstrated but convergence to a secure solution was never tested because the network has no feasible N-1 secure dispatch. The key question — does iterative Benders converge when a feasible SCOPF solution exists? — remains unanswered." -solver_version: "Julia 1.10.7 / PowerModels 0.21.5 / HiGHS 1.x" -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 0 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe 009: SCOPF TINY — Benders Convergence Claim - -## Verdict: INCONCLUSIVE - -No code was run for this probe. The A-9 result file (`evaluations/powermodels/results/expressiveness/A-9_scopf_TINY.md`) already contains a thorough and transparent account of what happened. The probe is resolved by document analysis. - -## What the Result File Says - -The evaluator explicitly documents all relevant facts: - -1. **N-1 infeasibility is stated**: "The IEEE 39-bus system with the Modified Tiny load/generation profile is not fully N-1 secure at original branch ratings." The full SCOPF LP (all 4,140 N-1 constraints simultaneously) is **INFEASIBLE**. - -2. **1-iteration behavior is documented**: The `convergence_iterations: 1` frontmatter field and the table entry "Iterative iterations: 1 (hits infeasibility at iteration 2)" are explicit. - -3. **The evaluator's framing**: The result treats N-1 infeasibility as a "physical property of the network configuration and load profile, not a code limitation." The `qualified_pass` is awarded for demonstrating the API mechanism (PTDF/LODF computation, constraint injection via two-level API, re-solve), not for demonstrating multi-iteration Benders convergence. - -## Assessment of the Claim - -The sweep claim that "SCOPF TINY qualified_pass demonstrates Benders convergence" is **not supported by the result file itself**. The file does not claim Benders converged — it explicitly says the algorithm terminated at 1 iteration due to infeasibility of the augmented problem at iteration 2. - -What the result *does* demonstrate: -- The PowerModels two-level API (instantiate_model → var(pm,:p) → @constraint → optimize_model!) supports security constraint injection -- PTDF/LODF computation using `calc_basic_ptdf_matrix` is functional -- The base OPF is solved and N-1 violations are correctly identified by LODF screening -- The cost differential between unconstrained ($98,091/h) and security-constrained ($144,663/h) is plausible - -What the result **does not** demonstrate: -- Multi-iteration Benders convergence on a feasible SCOPF instance -- That the iterative algorithm would converge to an optimal secure dispatch if given a network where such a dispatch exists - -## Root Cause of the Test Design Gap - -The TINY network (IEEE 39-bus with Modified Tiny load profile) is N-1 infeasible at 100% ratings. The evaluator discovered this, tried 70% derating (which made individual contingencies infeasible), and settled on 100% ratings — but the full SCOPF remains infeasible. There is no tested network configuration for which the iterative Benders loop completes multiple iterations and converges. - -The `qualified_pass` grade is defensible as a mechanism verification (the API works, the formulation is correct), but the specific claim of "demonstrates Benders convergence" overstates what was shown. - -## Classification Rationale - -**inconclusive**: The evaluator correctly documented the network's N-1 infeasibility and the 1-iteration behavior. The documentation is transparent and accurate. However, the test design did not produce a scenario where Benders convergence could be observed. The claim that the result "demonstrates Benders convergence" is an overstatement of what was proven, but the underlying capability is plausible and the API mechanism is real. A definitive pass or fail requires a feasible SCOPF test network. - -## No Code Run - -The A-9 result file is sufficiently detailed. Running additional code would require: -1. Finding or constructing a network that is N-1 feasible with the TINY load profile (non-trivial) -2. Verifying multi-iteration convergence on that network - -This is beyond the scope of a formulation audit probe and would constitute a new evaluation test. diff --git a/sweep-data/v10-to-v11/probes/powermodels/probe-010.md b/sweep-data/v10-to-v11/probes/powermodels/probe-010.md deleted file mode 100644 index fa8f2d2d..00000000 --- a/sweep-data/v10-to-v11/probes/powermodels/probe-010.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -probe_id: probe-010 -tool: powermodels -source_test: F-3 -probe_type: claim_verification -classification: claim_supported -reason: "F-3 correctly classifies SCIP_jll v0.2.1 (wrapping SCIP 8.0.0) as ZIB Academic License. F-8 is incorrect: it states SCIP 8.0 is Apache 2.0, but the Apache 2.0 switch happened at SCIP 8.0.3 (December 2022). SCIP 8.0.0 (April 2022, wrapped by SCIP_jll v0.2.1+0) remains under ZIB Academic. F-8's supply chain score upgrade is unwarranted for the pinned version." -solver_version: "SCIP_jll v0.2.1" -solver_version_match: true -timeout_seconds: 120 -wall_clock_seconds: 0 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe 010: SCIP_jll v0.2.1 License — ZIB Academic vs Apache 2.0 - -## Verdict: CLAIM SUPPORTED (F-3 is correct; F-8 contains a material error) - -## Evidence Gathered - -### 1. Manifest.toml confirms SCIP_jll v0.2.1+0 - -From `evaluations/powermodels/Manifest.toml`: -```toml -[[deps.SCIP_jll]] -deps = ["Artifacts", "Bzip2_jll", ...] -git-tree-sha1 = "4a23f926d711535640963aea90a3f5d931ae52c7" -uuid = "e5ac4fe4-a920-5659-9bf8-f9f73e9e79ce" -version = "0.2.1+0" -``` - -### 2. SCIP_jll v0.2.1 wraps SCIP 8.0.0 - -The `SCIP-v0.2.1+0` tag on the JuliaBinaryWrappers/SCIP_jll.jl GitHub repository contains a README identifying the source tarball as: -``` -https://scipopt.org/download/release/scipoptsuite-8.0.0.tgz -``` - -This is confirmed by the commit timeline: SCIP_jll v0.2.1+0 was built on **April 28, 2022**, which is contemporaneous with the SCIP 8.0.0 release (November 2021). The next SCIP_jll version using the new versioning scheme (800.0.300+0) was built on **December 14, 2022** and wraps SCIP 8.0.3. - -### 3. SCIP 8.0.0 is under ZIB Academic License — not Apache 2.0 - -From the official SCIP website (scipopt.org): - -> "Since version **8.0.3**, SCIP is licensed under the Apache 2.0 License. Releases up to and including Version **8.0.2** remain under the ZIB Academic License." - -The license switch happened at **SCIP 8.0.3** (December 2022), not at SCIP 8.0.0. - -### 4. SCIP_jll versioning scheme change - -The JuliaBinaryWrappers project changed the versioning scheme for SCIP_jll around the 8.0.3 release: - -| SCIP_jll version | SCIP version | Date | License | -|-----------------|--------------|------|---------| -| 0.1.0+0 | pre-8.x | Aug 2020 | ZIB Academic | -| 0.2.0+0 | 8.0.0 (beta) | Dec 2021 | ZIB Academic | -| **0.2.1+0** | **8.0.0** | **Apr 2022** | **ZIB Academic** | -| 800.0.300+0 | 8.0.3 | Dec 2022 | Apache 2.0 | -| 800.0.400+0 | 8.0.4 | Sep 2023 | Apache 2.0 | -| 800.100.0+0 | 8.1.0 | Dec 2023 | Apache 2.0 | -| 900.0.0+1 | 9.0.0 | Apr 2024 | Apache 2.0 | - -## Discrepancy Analysis - -### F-3 Assessment (correct) - -F-3 states: "SCIP binary (SCIP 8.0.0 via SCIP_jll v0.2.1) uses the ZIB Academic License, which restricts use to non-commercial academic institutions." - -This is **accurate**. F-3 even notes correctly: "SCIP 9.x (released 2024) moved to Apache 2.0, but the version pinned in this manifest is SCIP 8.0.0 and remains under ZIB Academic." - -### F-8 Assessment (incorrect) - -F-8 states: "SCIP v8.0 (November 2021) switched from the ZIB Academic License to Apache 2.0, making it fully permissive for commercial use. The SCIP_jll v0.2.1 in this manifest wraps SCIP 8.0, confirmed by `SCIP.SCIPversion()` returning `8.0` in the devcontainer." - -This contains two errors: -1. **Wrong version for license switch**: The switch to Apache 2.0 happened at 8.0.3, not 8.0 (8.0.0). The official SCIP documentation is unambiguous on this point. -2. **`SCIPversion()` returns "8.0" but this means SCIP 8.0.0**: The version string "8.0" corresponds to 8.0.0 (pre-Apache switch), not to 8.0.3+ (post-Apache switch). F-8 treats "8.0" as confirming Apache 2.0, but 8.0.0 is still ZIB Academic. - -F-8's conclusion that SCIP_jll v0.2.1 carries Apache 2.0 is **factually wrong**. - -## Implications for Supply Chain Scoring - -F-8 was used to upgrade the F-8 score from `qualified_pass` (v9) to `pass`, citing the "correction" that SCIP 8.0 is Apache 2.0. This upgrade is not warranted: - -- The pinned version (SCIP_jll v0.2.1 = SCIP 8.0.0) is ZIB Academic, non-commercial only -- F-3's `qualified_pass` (commercial deployments must exclude SCIP) remains the correct assessment -- F-8's `pass` status overstates the permissiveness of the pinned solver stack - -The correct supply chain picture: SCIP in this manifest is **not** commercially usable. To use SCIP commercially, the manifest would need to be updated to SCIP_jll 800.0.300+0 or newer (wrapping SCIP 8.0.3+). - -## SCIP_jll Package vs. Binary License (Secondary Question) - -The SCIP_jll Julia wrapper package itself carries a permissive license (the JLL wrapper code is MIT). However, SCIP_jll bundles the SCIP binary directly as a Julia artifact — it is not downloaded separately at runtime. The binary's license (ZIB Academic for SCIP 8.0.0) governs the artifact as distributed. The distinction between wrapper license and bundled binary license is exactly the kind of nuance that makes this a material discrepancy: the wrapper is MIT, the binary is ZIB Academic, and the binary is what runs. - -## Summary - -| Source | SCIP_jll version | SCIP version | License claimed | Correct? | -|--------|-----------------|--------------|----------------|---------| -| F-3 | v0.2.1 | 8.0.0 | ZIB Academic | **Yes** | -| F-8 | v0.2.1 | 8.0.0 | Apache 2.0 | **No** | -| Ground truth | v0.2.1 | 8.0.0 | ZIB Academic | — | - -F-3's claim is supported. F-8 contains a material factual error that inflated the supply chain score. diff --git a/sweep-data/v10-to-v11/probes/powersimulations/probe-013.md b/sweep-data/v10-to-v11/probes/powersimulations/probe-013.md deleted file mode 100644 index da1937dd..00000000 --- a/sweep-data/v10-to-v11/probes/powersimulations/probe-013.md +++ /dev/null @@ -1,217 +0,0 @@ ---- -probe_id: probe-013 -tool: powersimulations -source_test: A-2 -probe_type: convergence_check -classification: claim_debunked -reason: Convergence IS logged at @info level (suppressed in evaluation) and iteration count IS available — the claim that "no convergence diagnostics are accessible" is false; additionally the C-2 first-call warning scenario cannot be reproduced on TINY and the solution is physically valid. -solver_version: PowerFlows 0.9.0 / PowerSimulations 0.30.2 -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 25 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe 013: ACPF Convergence Diagnostic Check - -## Original Claim - -A-2 (qualified pass): "ACPF convergence verified at TINY and SMALL" — qualified because -PowerFlows.jl v0.9.0 "does not expose Newton-Raphson iteration count or convergence -residual in its public return value." Convergence was inferred only from non-trivial -voltage profiles (100% of buses differ from 1.0 pu flat start). - -C-2 (pass): First ACPF call at 10K-bus logged `"NewtonRaphsonACPowerFlow solver failed -to converge"` but returned results; second call "converged cleanly." C-2 accepted the -second-call result as valid convergence. - -## Probe Methodology - -1. Ran `solve_powerflow(ACPowerFlow(), sys)` on IEEE 39-bus with full `Logging.Debug` - output captured to a buffer. -2. Inspected the return type and all fields of `Dict{String, DataFrame}` for hidden - convergence metadata. -3. Used `names(PowerFlows, all=true)` to enumerate all symbols in the PowerFlows module, - filtering for convergence-related names. -4. Examined `PowerFlowData` struct fields (including the `:converged` vector) and - PowerFlows source at `/opt/julia-depot/packages/PowerFlows/Ilrf1/src/`. -5. Computed post-hoc power balance: true system losses = gen − load (MW). -6. Attempted to reproduce the C-2 first-call convergence warning on the TINY network. -7. Compared first vs. second call Vm profiles to detect diverged-then-retried behavior. - -## Probe Results - -### Section 0: Package Versions - -``` -PowerFlows = 0.9.0 -PowerSystems = 4.6.2 -PowerSimulations = 0.30.2 -``` - -### Section 1: Convergence IS logged — at @info level - -With `Logging.Debug` (or even `Logging.Info`) enabled, both calls produce: - -``` -[ Info: The NewtonRaphsonACPowerFlow solver converged after 1 iterations. -[ Info: PowerFlow solve converged, the results are exported in DataFrames -[ Info: Voltages are exported in pu. Powers are exported in MW/MVAr. -``` - -**The original evaluation set `global_logger(ConsoleLogger(stderr, Logging.Error))`, -which suppressed @info messages.** The iteration count (1) was logged but invisible. - -The PowerFlows source confirms (powerflow_method.jl line 314): - -```julia -@info("The $T solver converged after $i iterations.") -``` - -And on failure (line 320): - -```julia -@error("The $T solver failed to converge.") -``` - -Crucially, on failure the `solve_powerflow` (non-mutating) function returns `missing`, -not a `Dict`. So a returned `Dict{String, DataFrame}` is a guaranteed convergence -indicator at the API level. - -### Section 2: API fields — iteration count not in return value, but convergence is binary-safe - -The returned `Dict` has only two keys: `"bus_results"` and `"flow_results"`. No -convergence metadata columns exist. However: - -- The return type itself is the convergence signal: `missing` = did not converge, - `Dict` = converged. -- The `PowerFlowData` struct (internal) has a `:converged::Vector{Bool}` field. -- `PowerFlows.get_converged(pfd::PowerFlowData)` is a public function. -- `PowerFlows.DEFAULT_NR_MAX_ITER = 30`, `WARN_LARGE_RESIDUAL = 10`, `MAX_INIT_RESIDUAL = 10.0`. - -The original evaluation's conclusion that "convergence residual is not accessible" is -correct in the strict sense (the residual value is not returned), but the evaluation -missed that the iteration count is accessible via logging and that convergence is -structurally guaranteed by the return type. - -### Section 3: Post-hoc power balance (IEEE 39-bus, TINY) - -All power values are in MW (not pu × base_MVA, despite base_power = 100 MVA): - -| Metric | Value | -|--------|-------| -| Total generation | 6,297.9 MW | -| Total load | 6,254.2 MW | -| True system losses (gen − load) | 43.6 MW (0.69%) | -| Max |P_gen − P_load − P_net| per bus | 0.0 MW (exact) | -| Max branch |P_from + P_to − P_losses| | 0.0 MW (exact) | - -The system-level active power balance is consistent: 43.6 MW of losses on a ~6,300 MW -system (0.7%) is physically realistic for the IEEE 39-bus case. Bus-level and branch-level -internal consistency checks both pass with machine-precision zero error. - -Note: The `P_losses` column uses the convention `P_losses = P_from_to + P_to_from`, where -`P_from_to` and `P_to_from` are signed with opposite polarity. The column sum is −69 MW -due to bidirectional sign assignment across branches — **this is not a solver error**. The -correct system loss figure is `sum(P_gen) − sum(P_load) = 43.6 MW`. - -### Section 4: PowerFlows internals confirm NR with 1-iteration convergence - -``` -ACPowerFlow() type: ACPowerFlow{NewtonRaphsonACPowerFlow} -ACPowerFlow fields: (:check_reactive_power_limits, :exporter, :calculate_loss_factors) -DEFAULT_NR_MAX_ITER = 30 -WARN_LARGE_RESIDUAL = 10 -MAX_INIT_RESIDUAL = 10.0 -``` - -IEEE 39-bus from a flat start (Vm=1.0, Va=0.0 for load buses; PV buses initialized from -matpower data) converges in exactly 1 NR iteration. This is expected: case39 is -well-conditioned and the MATPOWER initial values include generator setpoints that are -near the AC solution. - -### Section 5: C-2 first-call convergence warning NOT reproducible on TINY - -``` -First-call log output: - [ Info: The NewtonRaphsonACPowerFlow solver converged after 1 iterations. - [ Info: PowerFlow solve converged, the results are exported in DataFrames - [ Info: Voltages are exported in pu. Powers are exported in MW/MVAr. - -Second-call log output: (identical) - -First vs Second call Vm comparison: - Max |Vm1 - Vm2|: 0.0 pu - Mean |Vm1 - Vm2|: 0.0 pu - -> Results IDENTICAL: both calls converged to same solution -``` - -The "convergence warning" described in C-2 (10K-bus MEDIUM network) is specific to the -ACTIVSg 10K network at large scale. On TINY (39-bus), there is no first-call warning — -both calls converge identically in 1 iteration. The C-2 behavior (warning then clean -convergence) appears to be a scale-dependent NR initialization effect. - -The probe could not independently verify the C-2 first-call failure (10K-bus network -takes significant time), but the key risk — that the MEDIUM result was accepted from a -non-converged first call — is mitigated by the structural API guarantee: `solve_powerflow` -returns `missing` on non-convergence. The C-2 result correctly shows `Converged: Yes`. - -## Analysis - -### What the original evaluation got wrong - -1. **Iteration count IS available** — at `@info` log level. The evaluation suppressed - info-level logging (`Logging.Error`), making it invisible. The claim "NR iteration - count not accessible" is incorrect: it is accessible by enabling Julia's standard - logging at Info level. - -2. **Convergence is structurally guaranteed** — `solve_powerflow` returns `missing` on - non-convergence. The evaluation treated convergence as inferred-only, but the return - type is a binary convergence indicator stronger than just checking voltage profiles. - -3. **The "no residual exposed" claim is correct** — the tolerance (default 1e-9) and - final residual value are internal. A user cannot retrieve the numerical convergence - residual from the public API without accessing PowerFlowData internals. - -### What the original evaluation got right - -1. The voltage profile proxy (100% of buses differ from flat start) correctly identifies - a genuine AC solution on case39. -2. The A-2 qualified_pass status is defensible — the 1-iteration convergence and - physically consistent solution confirm genuine NR convergence. -3. Physical outputs (V, θ, P, Q, losses) are present and physically reasonable. - -### Classification rationale - -The extraordinary claim being probed is "ACPF convergence verified at TINY and SMALL." -The probe finds: - -- **On TINY (39-bus):** Convergence IS genuine. NR converges in 1 iteration with a - physically valid solution (43.6 MW losses = 0.69%). The evidence of convergence is - stronger than the evaluation claimed — both the log message and the non-`missing` - return value confirm it. - -- **On the methodology claim:** The evaluation's statement that "iteration count and - residual are not accessible" is **partially false**. Iteration count IS logged at - `@info`. The residual value is not accessible via public API (true), but the convergence - boolean is structurally guaranteed. - -- **On C-2's convergence warning:** The first-call warning scenario is scale-dependent - (10K network only). The C-2 result is structurally valid (non-`missing` return - guarantees convergence), but the pass was accepted without awareness that the API - guarantees convergence by return type — the evaluation inferred from voltage profiles - when a stronger guarantee was already in place. - -Classification: **claim_debunked** — not because the convergence is fake (it is real), -but because the claim that "no convergence diagnostics are accessible" is falsified. The -evaluation's convergence proxy methodology was unnecessary and weaker than what the API -actually provides. The qualified_pass annotation in A-2 overstates the limitation. - -## Classification Rationale - -`claim_debunked`: The core methodology claim — that PowerFlows.jl provides no -convergence diagnostics, requiring indirect inference from voltage profiles — is -incorrect. Iteration count is emitted at `@info` level, and the API structurally -guarantees convergence (returns `missing` on failure). The actual convergence on TINY -is genuine and physically consistent, but the evaluation framework used an unnecessarily -weak proxy for a determination that the API provides more directly. diff --git a/sweep-data/v10-to-v11/probes/powersimulations/probe-013_script.jl b/sweep-data/v10-to-v11/probes/powersimulations/probe-013_script.jl deleted file mode 100644 index 0d9acb08..00000000 --- a/sweep-data/v10-to-v11/probes/powersimulations/probe-013_script.jl +++ /dev/null @@ -1,267 +0,0 @@ -#= -Probe 013: ACPF Convergence Check for PowerSimulations / PowerFlows.jl -Probe Type: convergence_check -Source Test: A-2 (TINY / IEEE 39-bus) + C-2 context (MEDIUM / ACTIVSg 10k) - -Objectives: - 1. Verify whether PowerFlows.jl exposes any convergence diagnostics - (iteration count, residual, termination status) beyond a boolean. - 2. Reproduce the A-2 ACPF on IEEE 39-bus with ALL logging enabled to - capture any internal convergence messages. - 3. Check PowerFlows.jl source/internals for accessible convergence fields. - 4. Compute an independent post-hoc residual from the returned bus solution - to quantify convergence quality numerically. - 5. Reproduce the C-2 first-call convergence warning scenario on TINY to - confirm whether "solver failed to converge" warning on first call is - reproducible and whether the second call truly converges. -=# - -using PowerSystems -using PowerFlows -using DataFrames -using Logging -using Dates - -println("=" ^ 70) -println("Probe 013: ACPF Convergence Diagnostic Check") -println("Timestamp: ", Dates.now()) -println("=" ^ 70) - -# ── Section 0: PowerFlows version ────────────────────────────────────────── -println("\n[0] Package versions:") -using Pkg: Pkg -for (name, ver) in - [("PowerFlows", nothing), ("PowerSystems", nothing), ("PowerSimulations", nothing)] - try - pkg_info = Pkg.dependencies() - for (uuid, dep) in pkg_info - if dep.name == name - println(" $name = $(dep.version)") - end - end - catch e - println(" $name = (error: $e)") - end -end - -# ── Section 1: Inspect PowerFlows.jl return type for hidden fields ────────── -println("\n[1] Inspecting PowerFlows.jl ACPowerFlow result type:") -println(" Loading IEEE 39-bus (TINY)...") -sys_tiny = System("/workspace/data/networks/case39.m") - -# Enable ALL logging to capture any internal convergence messages -global_logger(ConsoleLogger(stderr, Logging.Debug)) - -println(" --- First ACPF call (with Debug logging, warm-up) ---") -pf1 = solve_powerflow(ACPowerFlow(), sys_tiny) - -println(" --- Second ACPF call (timed) ---") -t0 = time() -pf2 = solve_powerflow(ACPowerFlow(), sys_tiny) -elapsed = time() - t0 - -# Restore quieter logging for output clarity -global_logger(ConsoleLogger(stderr, Logging.Warn)) - -println("\n Return type: ", typeof(pf2)) -println(" Return keys: ", collect(keys(pf2))) - -# Inspect bus_results DataFrame schema -bus_df = pf2["bus_results"] -println(" bus_results columns: ", names(bus_df)) -println(" bus_results nrow: ", nrow(bus_df)) - -flow_df = pf2["flow_results"] -println(" flow_results columns: ", names(flow_df)) - -# ── Section 2: Check if result object has hidden convergence fields ────────── -println("\n[2] Probing for hidden convergence fields in returned Dict:") -for key in keys(pf2) - val = pf2[key] - println(" key='$key' type=$(typeof(val))") - if val isa DataFrame - println(" columns: ", names(val)) - # Check for any convergence-related column names - for col in names(val) - if occursin(r"(conv|iter|resid|status|tol)"i, col) - println(" *** CONVERGENCE COLUMN FOUND: $col ***") - println(" values: ", val[!, col]) - end - end - elseif val isa Dict - println(" sub-keys: ", collect(keys(val))) - end -end - -# ── Section 3: Independent post-hoc residual computation ──────────────────── -println("\n[3] Post-hoc power mismatch residual (independent verification):") -println(" Computing |P_gen - P_load - P_losses| per bus...") - -# Bus-level power balance check -# P_net = P_gen - P_load should equal sum of P flowing out of bus -# Use the available columns -vm_vals = bus_df[!, "Vm"] -p_gen = bus_df[!, "P_gen"] -p_load = bus_df[!, "P_load"] -p_net = bus_df[!, "P_net"] -q_gen = bus_df[!, "Q_gen"] -q_load = bus_df[!, "Q_load"] -q_net = bus_df[!, "Q_net"] - -# P_net should equal P_gen - P_load (signed injection) -p_inj_check = p_gen .- p_load .- p_net -println(" Max |P_gen - P_load - P_net| (should be ~0): ", maximum(abs.(p_inj_check)), " pu") -println(" Mean |P_gen - P_load - P_net|: ", sum(abs.(p_inj_check)) / length(p_inj_check), " pu") - -# System-level power balance -base_mva = get_base_power(sys_tiny) -total_gen_mw = sum(p_gen) * base_mva -total_load_mw = sum(p_load) * base_mva -p_losses_mw = sum(flow_df[!, "P_losses"]) * base_mva -println(" System balance check:") -println(" Total generation: $(round(total_gen_mw, digits=2)) MW") -println(" Total load: $(round(total_load_mw, digits=2)) MW") -println(" Branch P losses: $(round(p_losses_mw, digits=2)) MW") -println( - " Imbalance (gen - load - losses): $(round(total_gen_mw - total_load_mw - p_losses_mw, digits=4)) MW", -) - -# Voltage profile stats -println(" Voltage profile (convergence proxy):") -println(" Vm min: $(round(minimum(vm_vals), digits=6)) pu") -println(" Vm max: $(round(maximum(vm_vals), digits=6)) pu") -println( - " Buses with |Vm - 1.0| > 1e-4: $(count(v -> abs(v - 1.0) > 1e-4, vm_vals)) / $(length(vm_vals))", -) -println( - " Buses with |Vm - 1.0| > 0.01: $(count(v -> abs(v - 1.0) > 0.01, vm_vals)) / $(length(vm_vals))", -) - -# Branch-level check: are flows physically consistent with bus voltages? -# For each branch: P_from + P_to = P_losses (active power balance) -p_from = flow_df[!, "P_from_to"] -p_to = flow_df[!, "P_to_from"] -p_losses_branch = flow_df[!, "P_losses"] -p_branch_balance = p_from .+ p_to .- p_losses_branch -println(" Branch power balance (P_from + P_to - P_losses, should be ~0):") -println(" Max: $(maximum(abs.(p_branch_balance))) pu") -println(" Mean: $(sum(abs.(p_branch_balance)) / length(p_branch_balance)) pu") - -# ── Section 4: Attempt to access internal solver state ────────────────────── -println("\n[4] Attempting to access PowerFlows.jl internal NR solver state:") -try - # Check if ACPowerFlow() has any configurable tolerance/iteration fields - acpf = ACPowerFlow() - println(" ACPowerFlow() type: ", typeof(acpf)) - println(" ACPowerFlow fields: ", fieldnames(typeof(acpf))) - for field in fieldnames(typeof(acpf)) - println(" $field = ", getfield(acpf, field)) - end -catch e - println(" Error: $e") -end - -# Try to find the NR solver type and its fields -try - # PowerFlows may have internal types we can inspect - println(" PowerFlows module names:") - pf_names = names(PowerFlows; all=true) - convergence_names = filter( - n -> occursin(r"(conv|newton|raphson|iter|resid|nlsolve)"i, string(n)), pf_names - ) - println(" Convergence-related names in PowerFlows: ", convergence_names) -catch e - println(" Could not inspect PowerFlows internals: $e") -end - -# ── Section 5: Reproduce first-call convergence warning scenario ───────────── -println("\n[5] Reproducing first-call convergence warning (C-2 scenario):") -println(" Loading fresh System to get uninitialized state...") -sys_fresh = System("/workspace/data/networks/case39.m") - -# Capture all log output during first call -buf = IOBuffer() -with_logger(ConsoleLogger(buf, Logging.Debug)) do - global pf_fresh1 = solve_powerflow(ACPowerFlow(), sys_fresh) -end -log_output = String(take!(buf)) - -println(" First-call log output length: $(length(log_output)) chars") -if length(log_output) > 0 - println(" --- First-call log output ---") - println(log_output) - println(" --- End of log output ---") - # Check for convergence warning - if occursin(r"(fail|warn|conv|not conv)"i, log_output) - println(" *** CONVERGENCE WARNING DETECTED IN FIRST CALL ***") - else - println(" No convergence warning found in first-call log.") - end -else - println(" No log output captured (solver may not use Julia logging system).") -end - -# Second call on same system -buf2 = IOBuffer() -with_logger(ConsoleLogger(buf2, Logging.Debug)) do - global pf_fresh2 = solve_powerflow(ACPowerFlow(), sys_fresh) -end -log_output2 = String(take!(buf2)) -println(" Second-call log output length: $(length(log_output2)) chars") -if length(log_output2) > 0 - println(" --- Second-call log output ---") - println(log_output2) - println(" --- End of log output ---") -end - -# Compare first and second call results -if pf_fresh1 !== nothing && pf_fresh2 !== nothing - bus1 = pf_fresh1["bus_results"] - bus2 = pf_fresh2["bus_results"] - vm1 = bus1[!, "Vm"] - vm2 = bus2[!, "Vm"] - vm_diff = abs.(vm1 .- vm2) - println(" First vs Second call Vm comparison:") - println(" Max |Vm1 - Vm2|: $(maximum(vm_diff)) pu") - println(" Mean |Vm1 - Vm2|: $(sum(vm_diff)/length(vm_diff)) pu") - if maximum(vm_diff) < 1e-6 - println(" -> Results IDENTICAL: both calls converged to same solution") - elseif maximum(vm_diff) < 1e-3 - println(" -> Results NEARLY identical: minor numerical difference") - else - println(" -> Results DIFFER SIGNIFICANTLY: first call may not have converged!") - for (i, d) in enumerate(vm_diff) - if d > 1e-3 - println( - " Bus $(bus1[i, "bus_number"]): Vm1=$(round(vm1[i], digits=6)), Vm2=$(round(vm2[i], digits=6)), diff=$(round(d, digits=6))", - ) - end - end - end -end - -# ── Section 6: Timing summary ──────────────────────────────────────────────── -println("\n[6] Timing:") -println(" A-2 ACPF solve time (second call): $(round(elapsed, digits=6)) s") - -# ── Section 7: Summary ─────────────────────────────────────────────────────── -println("\n[7] Convergence Diagnostic Summary:") -println(" API exposes iteration count: NO") -println(" API exposes convergence residual: NO") -println(" API exposes termination status: NO (only implicit - returns result or nothing)") -println( - " Post-hoc system power imbalance: $(round(abs(total_gen_mw - total_load_mw - p_losses_mw), digits=4)) MW", -) -println( - " Post-hoc branch balance max error: $(round(maximum(abs.(p_branch_balance)) * base_mva, digits=6)) MW", -) -println( - " Voltage profile: min=$(round(minimum(vm_vals), digits=4)) pu, max=$(round(maximum(vm_vals), digits=4)) pu", -) -println( - " Buses with non-trivial Vm: $(count(v -> abs(v - 1.0) > 1e-4, vm_vals)) / $(length(vm_vals))" -) - -println("\n" * "=" ^ 70) -println("Probe 013 complete.") -println("=" ^ 70) diff --git a/sweep-data/v10-to-v11/probes/pypsa/probe-001.md b/sweep-data/v10-to-v11/probes/pypsa/probe-001.md deleted file mode 100644 index f64d19f6..00000000 --- a/sweep-data/v10-to-v11/probes/pypsa/probe-001.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -probe_id: probe-001 -tool: pypsa -source_test: G-FNM-3 -probe_type: convergence_check -classification: claim_debunked -reason: Deviations are non-zero at float64 precision (max bus angle 1.07e-8 deg, max branch flow 5.76e-7 %), but round to 0.0 at 6 decimal places — the reported "0.0" values are display artifacts of the rounding in the result file, not exact zeros -solver_version: pypsa-1.1.2 -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 18 -timestamp: "2026-03-14T00:00:00Z" ---- - -# Probe-001: G-FNM-3 DCPF Zero-Deviation Claim - -## Original Claim - -The G-FNM-3 result file (`evaluations/pypsa/results/fnm_ingestion/G-FNM-3_dcpf_verification.md`) states: - -> "100% of buses and 100% of branches pass all tolerance thresholds. Zero deviations from the MATPOWER reference solution." - -The numeric tables show: - -| Metric | Value | -|--------|-------| -| Mean deviation (bus angles) | 0.0 deg | -| Max deviation (bus angles) | 0.0 deg | -| Mean deviation (branch flows) | 0.0% | -| Max deviation (branch flows) | 0.0% | - -This covers 27,862 buses and 32,532 branches on the FNM main island (ACTIVSg70k-derived network). - -## Validation Report Discrepancy - -The `evaluations/pypsa/results/validation-report.md` lists G-FNM-3 as a **FAIL**: - -> "G-FNM-3 | DCPF verification failed — systematic impedance conversion differences via MATPOWER fallback" - -This contradicts the individual result file's PASS status. The validation report appears to have been generated from an **older version of the test** (before the shared `matpower_loader.load_pypsa()` was introduced with the branch-status patch). The result file itself was updated to PASS but the validation report was not regenerated. The validation report is therefore stale relative to the result file. - -## Probe Methodology - -1. Read the original test script (`test_g_fnm_3_dcpf_verification.py`) to reproduce the exact methodology. -2. Wrote an independent probe script using the same shared `matpower_loader.load_pypsa()` utility, same input files, and same reference data. -3. Computed all deviations at full float64 precision using `np.float64` arrays and `:.18e` formatting. -4. Checked deviation distributions across multiple thresholds (exact 0.0, 1e-12, 1e-9, 1e-6). -5. Verified PyPSA version inside the devcontainer matches the version reported in the original test (1.1.2). -6. Ran inside devcontainer: `/devcontainer/dc-exec -C /workspace/evaluations/pypsa timeout 300 uv run python probe-001_script.py` - -**Input files used:** -- Network: `/workspace/data/fnm/reference/cleaned/fnm_main_island.m` -- Reference buses: `/workspace/data/fnm/reference/dcpf/buses_dcpf.csv` -- Reference branches: `/workspace/data/fnm/reference/dcpf/branches_dcpf.csv` - -## Probe Results (Raw Output) - -``` -PyPSA version: 1.1.2 -NumPy version: 2.3.5 -Pandas version: 2.3.3 - -Excluded buses: 2445 -Reference buses: 27862 -Reference branches: 32532 - -Loading MATPOWER case via shared matpower_loader.load_pypsa()... - Load time: 0.73s - Buses: 27862 - Lines: 23125 - Transformers: 9481 - Generators: 5741 - MATPOWER total branches: 32606, active: 32532 - -Running DCPF (net.lpf())... - Solve time: 17.16s - -====================================================================== -BUS VOLTAGE ANGLE COMPARISON (float64 precision) -====================================================================== -Buses compared (non-excluded): 27862 - Max deviation (deg): 1.073351540981093422e-08 - Mean deviation (deg): 3.316293917875641802e-09 - Min deviation (deg): 0.000000000000000000e+00 - Std deviation (deg): 2.369635338727692015e-09 - P50 deviation (deg): 2.923016495515184943e-09 - P95 deviation (deg): 7.940541024709091620e-09 - P99 deviation (deg): 9.055444820660341462e-09 - Buses with dev > 1e-6 deg: 0 - Buses with dev > 1e-9 deg: 22427 - Buses with dev > 1e-12 deg: 27851 - Buses with dev > 0.0 (exact): 27858 - No outlier buses with dev > 1e-6 deg. - - Distribution of nonzero deviations (27858 buses): - Max: 1.073351540981093422e-08 - Min: 2.344791028008330613e-13 - Mean: 3.316770089017558105e-09 - -====================================================================== -BRANCH FLOW COMPARISON (float64 precision) -====================================================================== -Branches compared: 32532 - Max deviation (%): 5.757743807042548304e-07 - Mean deviation (%): 1.175349767552511295e-08 - Min deviation (%): 0.000000000000000000e+00 - P95 deviation (%): 4.589095774355148793e-08 - P99 deviation (%): 2.370491861436153432e-07 - Branches with dev > 1e-6 %: 0 - Branches with dev > 1e-9 %: 15646 - Branches with dev > 0.0 (exact): 26892 - - Lines: 23056 compared - Max dev (%): 5.757743807042548304e-07 - Mean dev (%): 1.244668512309947464e-08 - - Transformers: 9476 compared - Max dev (%): 4.956535848421594892e-07 - Mean dev (%): 1.006690736407783267e-08 - -====================================================================== -SUMMARY -====================================================================== -Bus angle max dev: 1.073351540981093422e-08 deg -Bus angle mean dev: 3.316293917875641802e-09 deg -Branch flow max dev: 5.757743807042548304e-07 % -Branch flow mean dev: 1.175349767552511295e-08 % - -Claim: 0.0 mean and max deviation across all buses and branches -Claim supported (exact float64 zero): False -Claim supported (rounded to 0.0 at 6 decimal places): True - -Total wall clock: 17.9s (load: 0.7s, solve: 17.2s) -``` - -## Analysis - -### The "0.0" Values Are Rounding Artifacts - -The original test script computes deviations correctly as float64, but then formats results with `round(..., 6)` (6 decimal places) before storing them in the result dictionary. For example: - -```python -"max_deviation_deg": round(float(np.max(va_deviations)), 6) -"mean_deviation_deg": round(float(np.mean(va_deviations)), 6) -``` - -The actual maximum bus angle deviation is **1.07e-8 degrees** — this rounds to 0.000000 at 6 decimal places. Similarly, the maximum branch flow deviation is **5.76e-7 percent** — also rounds to 0.000000. So the reported "0.0" values in the result file are correct as 6-decimal display values, but they misrepresent the actual float64 deviations. - -### Magnitude of Actual Deviations - -| Metric | Probe Result | Claimed | -|--------|-------------|---------| -| Bus angle max deviation | 1.07e-8 deg | 0.0 deg | -| Bus angle mean deviation | 3.32e-9 deg | 0.0 deg | -| Branch flow max deviation | 5.76e-7 % | 0.0% | -| Branch flow mean deviation | 1.18e-8 % | 0.0% | - -The deviations are sub-nanodegree for bus angles and sub-micron-percent for branch flows. These are **numerical floating-point rounding errors intrinsic to the DCPF linear algebra**, not physically meaningful discrepancies. All 27,862 buses and 32,532 branches pass the 1.0-degree / 10% tolerance thresholds by an enormous margin. The PASS grade is fully warranted. - -### The Nature of the Deviations - -These are consistent with floating-point round-trip errors in the B-matrix assembly and sparse linear solve — expected when comparing two independently computed solutions that use equivalent but not identical code paths. The deviations (~1e-8 to 1e-7 magnitude) are at the limit of float64 precision for values of this scale, not evidence of a formulation difference. - -### Validation Report vs. Result File Discrepancy - -The validation report lists G-FNM-3 as FAIL with reason "systematic impedance conversion differences via MATPOWER fallback." This reflects the state **before** the branch-status patch was introduced. The individual result file (`G-FNM-3_dcpf_verification.md`) was updated to PASS after the shared loader fixed the bug, but the validation report was not regenerated. The probe independently confirms the result file's PASS conclusion is accurate. - -## Classification Rationale - -Classification: **claim_debunked** — the deviations are not exactly 0.0 at float64 precision. - -However, this is a **weak debunking**: the deviations are at the level of floating-point noise (~1e-8 to 1e-7), far below any physically meaningful threshold. The PASS grade and the engineering conclusion that "PyPSA matches MATPOWER exactly" are both correct. The claim of "0.0" is a rounding artifact from the 6-decimal-place display format used in the result file, not an error in the test methodology or an inflated performance claim. - -The validation report discrepancy (FAIL vs PASS) is a stale artifact — the probe confirms the result file's PASS assessment is accurate. diff --git a/sweep-data/v10-to-v11/probes/pypsa/probe-001_script.py b/sweep-data/v10-to-v11/probes/pypsa/probe-001_script.py deleted file mode 100644 index d97821da..00000000 --- a/sweep-data/v10-to-v11/probes/pypsa/probe-001_script.py +++ /dev/null @@ -1,316 +0,0 @@ -""" -Probe-001: Verify G-FNM-3 claim that PyPSA achieves 0.0 mean and max deviation -(both bus angles and branch flows) across all 27,862 buses and 32,532 branches -in G-FNM-3 DCPF on the ACTIVSg70k (FNM) network. - -This is an independent re-execution of the test using the same shared matpower_loader -that the original test used. Deviations are reported at full float64 precision. -""" - -from __future__ import annotations - -import sys -import time -from pathlib import Path - -import numpy as np - -# Add shared loader to path (same as original test) -sys.path.insert(0, "/workspace/evaluations/shared") - -CLEANED_M = Path("/workspace/data/fnm/reference/cleaned/fnm_main_island.m") -REF_BUSES = Path("/workspace/data/fnm/reference/dcpf/buses_dcpf.csv") -REF_BRANCHES = Path("/workspace/data/fnm/reference/dcpf/branches_dcpf.csv") -PASS_CONDITIONS = Path("/workspace/data/fnm/reference/pass_conditions.json") -EXCLUDED_BUSES = Path("/workspace/data/fnm/reference/excluded_buses.json") - - -def main() -> None: - import json - - import pandas as pd - import pypsa - from matpower_loader import load_pypsa - from matpowercaseframes import CaseFrames - - print(f"PyPSA version: {pypsa.__version__}") - print(f"NumPy version: {np.__version__}") - print(f"Pandas version: {pd.__version__}") - print() - - # Load pass conditions - with open(PASS_CONDITIONS) as f: - pass_conds = json.load(f) - dcpf_conds = pass_conds["dcpf"] - P_BASE_FLOOR = dcpf_conds["aggregate"]["branch_flow"]["p_base_floor_mw"] - - # Load excluded buses - with open(EXCLUDED_BUSES) as f: - excl_data = json.load(f) - excluded_bus_set = { - int(b["bus_number"]) for b in excl_data.get("excluded_buses", []) - } - print(f"Excluded buses: {len(excluded_bus_set)}") - - # Load reference data - ref_buses_df = pd.read_csv(REF_BUSES) - ref_branches_df = pd.read_csv(REF_BRANCHES) - print(f"Reference buses: {len(ref_buses_df)}") - print(f"Reference branches: {len(ref_branches_df)}") - print() - - # Load MATPOWER case via shared loader (same as original test) - print("Loading MATPOWER case via shared matpower_loader.load_pypsa()...") - t0 = time.perf_counter() - net = load_pypsa(str(CLEANED_M), overwrite_zero_s_nom=100000.0) - net.set_snapshots([0]) - t_load = time.perf_counter() - t0 - print(f" Load time: {t_load:.2f}s") - print(f" Buses: {len(net.buses)}") - print(f" Lines: {len(net.lines)}") - print(f" Transformers: {len(net.transformers)}") - print(f" Generators: {len(net.generators)}") - - # Also load raw CaseFrames for branch analysis - cf = CaseFrames(str(CLEANED_M)) - bus_array = cf.bus.values - branch_array = cf.branch.values - branch_status = branch_array[:, 10].astype(int) - n_active = int((branch_status == 1).sum()) - print(f" MATPOWER total branches: {len(branch_array)}, active: {n_active}") - print() - - # Run DCPF - print("Running DCPF (net.lpf())...") - t1 = time.perf_counter() - net.lpf() - t_solve = time.perf_counter() - t1 - print(f" Solve time: {t_solve:.2f}s") - print() - - # Extract results - if ( - hasattr(net, "buses_t") - and "v_ang" in net.buses_t - and len(net.buses_t.v_ang) > 0 - ): - pypsa_va_rad = net.buses_t.v_ang.iloc[0] - else: - pypsa_va_rad = net.buses.v_ang - pypsa_va_deg_series = np.degrees(pypsa_va_rad) - - if hasattr(net, "lines_t") and "p0" in net.lines_t and len(net.lines_t.p0) > 0: - pypsa_line_p0 = net.lines_t.p0.iloc[0] - else: - pypsa_line_p0 = net.lines.get("p0", pd.Series(dtype=float)) - - if ( - hasattr(net, "transformers_t") - and "p0" in net.transformers_t - and len(net.transformers_t.p0) > 0 - ): - pypsa_xfmr_p0 = net.transformers_t.p0.iloc[0] - else: - pypsa_xfmr_p0 = net.transformers.get("p0", pd.Series(dtype=float)) - - # ── Bus voltage angle comparison ───────────────────────────────────────── - print("=" * 70) - print("BUS VOLTAGE ANGLE COMPARISON (float64 precision)") - print("=" * 70) - - ref_bus_va = { - int(row["bus_number"]): float(row["va_deg"]) - for _, row in ref_buses_df.iterrows() - } - - va_deviations = [] - va_bus_numbers = [] - for bus_name in pypsa_va_deg_series.index: - try: - bus_num = int(bus_name) - except (ValueError, TypeError): - continue - if bus_num in excluded_bus_set: - continue - if bus_num in ref_bus_va: - dev = abs(float(pypsa_va_deg_series[bus_name]) - ref_bus_va[bus_num]) - va_deviations.append(dev) - va_bus_numbers.append(bus_num) - - va_deviations = np.array(va_deviations, dtype=np.float64) - n_buses_compared = len(va_deviations) - - print(f"Buses compared (non-excluded): {n_buses_compared}") - print(f" Max deviation (deg): {np.max(va_deviations):.18e}") - print(f" Mean deviation (deg): {np.mean(va_deviations):.18e}") - print(f" Min deviation (deg): {np.min(va_deviations):.18e}") - print(f" Std deviation (deg): {np.std(va_deviations):.18e}") - print(f" P50 deviation (deg): {np.median(va_deviations):.18e}") - print(f" P95 deviation (deg): {np.percentile(va_deviations, 95):.18e}") - print(f" P99 deviation (deg): {np.percentile(va_deviations, 99):.18e}") - - # Count buses exceeding various thresholds - n_above_1e6 = int(np.sum(va_deviations > 1e-6)) - n_above_1e9 = int(np.sum(va_deviations > 1e-9)) - n_above_1e12 = int(np.sum(va_deviations > 1e-12)) - n_nonzero = int(np.sum(va_deviations > 0.0)) - print(f" Buses with dev > 1e-6 deg: {n_above_1e6}") - print(f" Buses with dev > 1e-9 deg: {n_above_1e9}") - print(f" Buses with dev > 1e-12 deg: {n_above_1e12}") - print(f" Buses with dev > 0.0 (exact): {n_nonzero}") - - # Outlier buses (dev > 1e-6) - if n_above_1e6 > 0: - outlier_indices = np.where(va_deviations > 1e-6)[0] - print( - f"\n Outlier buses (dev > 1e-6 deg): top {min(20, len(outlier_indices))}" - ) - sorted_idx = outlier_indices[np.argsort(va_deviations[outlier_indices])[::-1]] - for i in sorted_idx[:20]: - print(f" Bus {va_bus_numbers[i]}: dev = {va_deviations[i]:.18e} deg") - else: - print(" No outlier buses with dev > 1e-6 deg.") - - # Tally of nonzero deviation buses (any floating-point deviation) - if n_nonzero > 0: - nonzero_devs = va_deviations[va_deviations > 0.0] - print(f"\n Distribution of nonzero deviations ({len(nonzero_devs)} buses):") - print(f" Max: {nonzero_devs.max():.18e}") - print(f" Min: {nonzero_devs.min():.18e}") - print(f" Mean: {nonzero_devs.mean():.18e}") - - # ── Branch flow comparison ──────────────────────────────────────────────── - print() - print("=" * 70) - print("BRANCH FLOW COMPARISON (float64 precision)") - print("=" * 70) - - # Classify branches as line vs transformer (same logic as original test) - bus_v_nom = dict(zip(bus_array[:, 0].astype(int), bus_array[:, 9])) - n_branches_mat = branch_array.shape[0] - is_xfmr = np.zeros(n_branches_mat, dtype=bool) - for i in range(n_branches_mat): - fbus = int(branch_array[i, 0]) - tbus = int(branch_array[i, 1]) - tap = branch_array[i, 8] - shift = branch_array[i, 9] - v0 = bus_v_nom.get(fbus, 0) - v1 = bus_v_nom.get(tbus, 0) - is_xfmr[i] = (v0 != v1) or (tap != 0.0 and tap != 1.0) or (shift != 0.0) - - line_counter = 0 - xfmr_counter = 0 - branch_pypsa_name = [] - for i in range(n_branches_mat): - if is_xfmr[i]: - branch_pypsa_name.append(("transformer", f"T{xfmr_counter}")) - xfmr_counter += 1 - else: - branch_pypsa_name.append(("line", f"L{line_counter}")) - line_counter += 1 - - line_p0_dict = dict(zip(pypsa_line_p0.index, pypsa_line_p0.values)) - xfmr_p0_dict = dict(zip(pypsa_xfmr_p0.index, pypsa_xfmr_p0.values)) - - p_deviations_pct = [] - p_dev_lines = [] - p_dev_xfmrs = [] - p_abs_deviations = [] - active_row = 0 - - for mat_row in range(n_branches_mat): - if branch_status[mat_row] == 0: - continue - if active_row >= len(ref_branches_df): - break - - comp_type, comp_name = branch_pypsa_name[mat_row] - ref_p = float(ref_branches_df.iloc[active_row]["pf_mw"]) - - if comp_type == "line": - pypsa_p = line_p0_dict.get(comp_name, float("nan")) - else: - pypsa_p = xfmr_p0_dict.get(comp_name, float("nan")) - - if not np.isnan(pypsa_p): - abs_dev = abs(float(pypsa_p) - ref_p) - denom = max(abs(ref_p), P_BASE_FLOOR) - dev_pct = (abs_dev / denom) * 100.0 - p_deviations_pct.append(dev_pct) - p_abs_deviations.append(abs_dev) - if comp_type == "line": - p_dev_lines.append(dev_pct) - else: - p_dev_xfmrs.append(dev_pct) - - active_row += 1 - - p_deviations_pct = np.array(p_deviations_pct, dtype=np.float64) - p_abs_deviations = np.array(p_abs_deviations, dtype=np.float64) - n_branches_compared = len(p_deviations_pct) - - print(f"Branches compared: {n_branches_compared}") - print(f" Max deviation (%): {np.max(p_deviations_pct):.18e}") - print(f" Mean deviation (%): {np.mean(p_deviations_pct):.18e}") - print(f" Min deviation (%): {np.min(p_deviations_pct):.18e}") - print(f" P95 deviation (%): {np.percentile(p_deviations_pct, 95):.18e}") - print(f" P99 deviation (%): {np.percentile(p_deviations_pct, 99):.18e}") - - n_above_1e6 = int(np.sum(p_deviations_pct > 1e-6)) - n_above_1e9 = int(np.sum(p_deviations_pct > 1e-9)) - n_nonzero_br = int(np.sum(p_deviations_pct > 0.0)) - print(f" Branches with dev > 1e-6 %: {n_above_1e6}") - print(f" Branches with dev > 1e-9 %: {n_above_1e9}") - print(f" Branches with dev > 0.0 (exact): {n_nonzero_br}") - - print(f"\n Lines: {len(p_dev_lines)} compared") - p_dev_lines_arr = np.array(p_dev_lines, dtype=np.float64) - if len(p_dev_lines_arr) > 0: - print(f" Max dev (%): {np.max(p_dev_lines_arr):.18e}") - print(f" Mean dev (%): {np.mean(p_dev_lines_arr):.18e}") - - print(f"\n Transformers: {len(p_dev_xfmrs)} compared") - p_dev_xfmrs_arr = np.array(p_dev_xfmrs, dtype=np.float64) - if len(p_dev_xfmrs_arr) > 0: - print(f" Max dev (%): {np.max(p_dev_xfmrs_arr):.18e}") - print(f" Mean dev (%): {np.mean(p_dev_xfmrs_arr):.18e}") - - # Summary - print() - print("=" * 70) - print("SUMMARY") - print("=" * 70) - bus_max = float(np.max(va_deviations)) if len(va_deviations) > 0 else float("nan") - bus_mean = float(np.mean(va_deviations)) if len(va_deviations) > 0 else float("nan") - br_max = ( - float(np.max(p_deviations_pct)) if len(p_deviations_pct) > 0 else float("nan") - ) - br_mean = ( - float(np.mean(p_deviations_pct)) if len(p_deviations_pct) > 0 else float("nan") - ) - - print(f"Bus angle max dev: {bus_max:.18e} deg") - print(f"Bus angle mean dev: {bus_mean:.18e} deg") - print(f"Branch flow max dev: {br_max:.18e} %") - print(f"Branch flow mean dev: {br_mean:.18e} %") - print() - - # Assess claim - CLAIM_EXACT_ZERO = ( - bus_max == 0.0 and bus_mean == 0.0 and br_max == 0.0 and br_mean == 0.0 - ) - print("Claim: 0.0 mean and max deviation across all buses and branches") - print(f"Claim supported (exact float64 zero): {CLAIM_EXACT_ZERO}") - - # If not exact zero, check whether within display rounding (1e-6 threshold) - CLAIM_ROUNDED_ZERO = bus_max < 1e-6 and br_max < 1e-6 - print(f"Claim supported (rounded to 0.0 at 6 decimal places): {CLAIM_ROUNDED_ZERO}") - - total_time = t_load + t_solve - print( - f"\nTotal wall clock: {total_time:.1f}s (load: {t_load:.1f}s, solve: {t_solve:.1f}s)" - ) - - -if __name__ == "__main__": - main() diff --git a/sweep-data/v10-to-v11/tool-paths.yaml b/sweep-data/v10-to-v11/tool-paths.yaml deleted file mode 100644 index bd2ab16b..00000000 --- a/sweep-data/v10-to-v11/tool-paths.yaml +++ /dev/null @@ -1,60 +0,0 @@ -tools: - pypsa: - worktree: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11 - results_dir: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/pypsa/results - synthesis: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/pypsa/results/synthesis.md - config: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/pypsa/results/eval-config.yaml - result_count: 98 - protocol_version: v10 - status: available - note: results on main branch (merged from eval/pypsa-v10 worktree) - - pandapower: - worktree: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11 - results_dir: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/pandapower/results - synthesis: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/pandapower/results/synthesis.md - config: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/pandapower/results/eval-config.yaml - result_count: 94 - protocol_version: v10 - status: available - note: results on main branch (merged from eval/pandapower-v10 worktree) - - gridcal: - worktree: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11 - results_dir: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/gridcal/results - synthesis: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/gridcal/results/synthesis.md - config: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/gridcal/results/eval-config.yaml - result_count: 94 - protocol_version: v10 - status: available - note: results on main branch (merged from eval/gridcal-v10 worktree) - - powermodels: - worktree: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11 - results_dir: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/powermodels/results - synthesis: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/powermodels/results/synthesis.md - config: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/powermodels/results/eval-config.yaml - result_count: 120 - protocol_version: v10 - status: available - note: results on main branch (merged from eval/powermodels-v10 worktree) - - powersimulations: - worktree: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11 - results_dir: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/powersimulations/results - synthesis: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/powersimulations/results/synthesis.md - config: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/powersimulations/results/eval-config.yaml - result_count: 67 - protocol_version: v10 - status: available - note: results on main branch (merged from eval/powersimulations-v10 worktree) - - matpower: - worktree: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11 - results_dir: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/matpower/results - synthesis: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/matpower/results/synthesis.md - config: /home/joe/code/zge-workspace/grc-tech-evaluation/.claude/worktrees/sweep/v10-to-v11/evaluations/matpower/results/eval-config.yaml - result_count: 86 - protocol_version: v10 - status: available - note: results on main branch (merged from eval/matpower-v10 worktree) diff --git a/sweep-data/v10-to-v11/validation-report.md b/sweep-data/v10-to-v11/validation-report.md deleted file mode 100644 index dbd68b65..00000000 --- a/sweep-data/v10-to-v11/validation-report.md +++ /dev/null @@ -1,68 +0,0 @@ -# Sweep Validation Report — v10 → v11 - -**Date:** 2026-03-15 - -## Checks Passed: 14 / 14 - -### Findings Report (`sweep-reports/v10-to-v11.md`) - -| Check | Result | -|-------|--------| -| File exists | ✓ (514 lines) | -| Executive Summary present | ✓ | -| Cross-Tool Comparison Matrices present | ✓ | -| Low-Signal Tests section present | ✓ | -| Spot-Check Probe Results present | ✓ | -| Proposed Changes section present | ✓ | -| Test-ID Mapping Table present | ✓ | -| GitHub Issue Triage section present | ✓ | -| Deferred Items section present | ✓ | -| Methodology section present | ✓ | -| All PC-01–PC-17 referenced in report | ✓ | - -### Protocol / Rubric - -| Check | Result | -|-------|--------| -| Protocol stamped v11 | ✓ (version history table entry 2026-03-15) | -| Rubric stamped v11 | ✓ (version history table entry 2026-03-15) | -| PC-01 (C-SMALL gate decoupling) reflected in protocol | ✓ (Suite C gating structure section, line 327) | -| PC-07 (five-tier outcomes) reflected in rubric | ✓ (partial_pass, constrained_pass present) | -| PC-10 (DCOPF hard constraints) in protocol | ✓ (A-3 pass condition: max_loading ≤ 1.0 + 1e-4) | -| All proposed changes referenced in protocol changelog | ✓ | -| Protocol and rubric internally consistent | ✓ | - -### Skill Files - -| File | Lines | Status | -|------|-------|--------| -| references/cross-tool-watchpoints.md | 503 | ✓ 5 new watchpoints added | -| references/result-template.md | 183 | ✓ New frontmatter fields added | -| references/workaround-classification.md | 166 | ✓ Five-tier outcome system documented | -| references/convergence-protocol.md | 113 | ✓ Four-tier evidence hierarchy added | -| prompts/code-evaluator-prompt.md | 550 | ✓ Verification guardrails added | -| prompts/config-generator-prompt.md | 378 | ✓ v11 version, gate_minimum_bar, C-suite gate logic | -| prompts/synthesis-prompt.md | 194 | ✓ Pass rate exclusion rules added | -| prompts/audit-evaluator-prompt.md | 250 | ✓ JLL binary license audit added | -| SKILL.md | 550 | ✓ Outcome tiers, gate logic, cascade logic updated | - -### Cross-Reference Consistency - -| Check | Result | -|-------|--------| -| All PC-01–PC-17 appear in findings report | ✓ | -| All PC-01–PC-17 appear in aggregation/proposed-changes.yaml | ✓ | -| No in-scope GitHub issues missing from report | ✓ (zero issues found) | -| Deferred items have rationale | ✓ (6 deferred items with rationale) | - -## No Issues Found - -Validation completed with 0 failures. All outputs are complete and internally consistent. - -## Output Summary - -- **Findings report:** `sweep-reports/v10-to-v11.md` (514 lines) -- **Updated protocol:** `evaluation_guides/Phase1_Test_Protocol.md` (483 lines, v11) -- **Updated rubric:** `evaluation_guides/Phase1_Evaluation_Rubric.md` (471 lines, v11) -- **Updated skill:** `.claude/skills/evaluate-tool/` (9 files modified) -- **Sweep data:** `sweep-data/v10-to-v11/` (per-tool findings, probes, aggregation) diff --git a/sweep-data/v4-to-v5/.progress.yaml b/sweep-data/v4-to-v5/.progress.yaml deleted file mode 100644 index 7ee555aa..00000000 --- a/sweep-data/v4-to-v5/.progress.yaml +++ /dev/null @@ -1,10 +0,0 @@ -source_version: v4 -target_version: v5 -tools_available: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] -completed_states: [INIT, SWEEP, PROBE, AGGREGATE, GENERATE, VALIDATE] -current_state: DONE -timestamp: "2026-03-09T12:00:00Z" -probes_run: 18 -probes_supported: 13 -probes_debunked: 4 -probes_inconclusive: 0 diff --git a/sweep-data/v4-to-v5/aggregation/comparison-matrices.md b/sweep-data/v4-to-v5/aggregation/comparison-matrices.md deleted file mode 100644 index eaa6ae80..00000000 --- a/sweep-data/v4-to-v5/aggregation/comparison-matrices.md +++ /dev/null @@ -1,182 +0,0 @@ -# Cross-Tool Comparison Matrices - -## Phase 1 Tool Selection | v4-to-v5 Aggregation - ---- - -## Test Outcome Matrix - -Legend: P = pass, F = fail, QP = qualified_pass, I = informational, -- = not attempted/blocked - -### Gate Tests - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | -|---------|-------|------------|---------|-------------|----------|----------| -| G-1 (TINY) | P | P | P | P | P | P | -| G-2 (SMALL) | P | P | P | P | P | P | -| G-3 (MEDIUM) | P | P | P | P | P | P | - -**Signal:** None. Unanimous pass. All tools handle MATPOWER .m format. - -### Suite A: Problem Expressiveness - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| A-1 TINY | P | P | P | P | P | P | 1 | Low | -| A-1 MED | P* | P | P | P | P | P | 2 | Low | -| A-2 TINY | P | P | P | P | P | P | 1 | Low | -| A-2 MED | P** | P | P | F | F | P | 3 | High | -| A-3 TINY | P | QP | P | P | QP | P | 2 | Low | -| A-3 MED | F | QP | P | P | -- | P | 3 | High | -| A-4 TINY | P | P | P | P | QP | P | 2 | Low | -| A-4 MED | F | P | P | QP | -- | P | 3 | High | -| A-5 TINY | P | F | F | QP | QP | P | 3 | High | -| A-5 SMALL | F | F | F | F | -- | P | 2 | Medium | -| A-6 TINY | P | F | F | QP | QP | P | 3 | High | -| A-6 SMALL | F | F | F | F | -- | P | 2 | Medium | -| A-7 TINY | P | P | QP | QP | P | P | 2 | Low | -| A-7 MED | F | P | QP | F | -- | P | 3 | High | -| A-8 TINY | F | F | F | F | F | P | 2 | Medium | -| A-8 SMALL | F | F | F | F | -- | -- | 1 | Low | -| A-9 TINY | P | F | F | QP | QP | P | 3 | High | -| A-9 SMALL | P | F | F | QP | -- | P | 3 | High | -| A-10 TINY | P | F | QP | QP | F | QP | 4 | High | -| A-10 SMALL | P | F | QP | QP | F | QP | 4 | High | -| A-11 TINY | P | F | F | QP | QP | QP | 4 | High | -| A-11 SMALL | P | F | F | QP | -- | QP | 3 | High | - -Notes: -- *P\** = A-1 MEDIUM PyPSA: all flows NaN due to singular matrix; pass is misleading (probe-001 related) -- *P\*\** = A-2 MEDIUM PyPSA: non-convergence warning; debunked by probe-001 (0 NR iterations, 83% flat start) - -### Suite B: Extensibility - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| B-1 TINY | P | QP | F | P | P | P | 3 | High | -| B-1 MED | F | QP | F | P | -- | P | 3 | High | -| B-2 TINY | P | P | P | QP | QP | P | 2 | Low | -| B-2 MED | P | P | P | QP | -- | P | 2 | Low | -| B-3 TINY | P | P | P | P | P | P | 1 | Low | -| B-3 MED | F | P | P | P | -- | P | 2 | Medium | -| B-4 TINY | P | QP | QP | P | P | P | 2 | Low | -| B-4 SMALL | P | QP | QP | P | P | P | 2 | Low | -| B-5 TINY | P | P | P | P | P | P | 1 | Low | -| B-5 MED | P | P | P | P | -- | P | 1 | Low | -| B-6 | I | P | P | P | P | I | 2 | Low | -| B-7 TINY | P | P | P | P | QP | P | 2 | Low | -| B-7 MED | P | P | P | P | -- | P | 1 | Low | -| B-8 TINY | P | QP | P | P | P | QP | 2 | Low | -| B-8 SMALL | P | QP | P | P | -- | QP | 2 | Low | -| B-9 TINY | P | QP | P | P | P | P | 2 | Low | -| B-9 MED | F | QP | QP | P | -- | P | 3 | High | - -### Suite C: Scalability - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| C-1 MED | P | P | P | P | P | P | 1 | Low | -| C-2 MED | P | P | P | F | F | P | 2 | Medium | -| C-3 MED | P | QP | P | P | QP* | P | 3 | Medium | -| C-4 SMALL | F | F | F | F | QP* | F | 2 | Medium | -| C-5 MED | QP | P | P | F | QP* | P | 3 | Medium | -| C-6 SMALL | P | QP | F | P | QP* | F | 3 | Medium | -| C-7 MED | P | F | P | QP | P | P | 2 | Medium | -| C-8 MED | F | F | F | F | F | F | 1 | Low | -| C-9 MED | QP | P | QP | P | P | P | 2 | Low | -| C-10 MED | P | F | F | QP | F | QP | 3 | Medium | - -Notes: -- *QP\** = PowerSimulations C-3/C-4/C-5/C-6: estimated timings only, no actual measurement (probe-020 confirms) - -### Suite D: Workforce Accessibility - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| D-1 | P | P | QP | QP | QP | I | 3 | Medium | -| D-2 | QP | QP | I | QP | I | I | 2 | Low | -| D-3 | P | P | QP | QP | I | I | 3 | Medium | -| D-4 | QP | QP | F | QP | QP | I | 3 | Medium | -| D-5 | I | I | I | I | I | I | 1 | Low | - -### Suite E: Maturity & Sustainability - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| E-1 | I | P | P | P | I | I | 2 | Low | -| E-2 | I | P | P | QP | I | I | 2 | Low | -| E-3 | I | P | QP | F | I | I | 3 | Medium | -| E-4 | I | P | F | I | I | I | 2 | Low | -| E-5 | I | P | QP | QP | I | I | 2 | Low | -| E-6 | I | P | QP | P | I | I | 2 | Low | -| E-7 | I | P | QP | F | I | I | 3 | Medium | - -### Suite F: Supply Chain - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| F-1 | P | P | QP | P | P | P | 2 | Low | -| F-2 | QP | P | P | I | I | P | 3 | Low | -| F-3 | QP | P | P | QP | I | P | 2 | Low | -| F-4 | P | P | P | P | P | P | 1 | Low | -| F-5 | P | P | P | P | P | P | 1 | Low | -| F-6 | P | P | P | P | P | I | 2 | Low | -| F-7 | P | P | F | P | P | P | 2 | Low | -| F-8 | P | P | QP | P | P | P | 2 | Low | -| F-9 | QP | QP | QP | P | I | I | 3 | Low | - -### Phase 2 Readiness - -| Test ID | PyPSA | pandapower | GridCal | PowerModels | PowerSim | MATPOWER | Spread | Signal | -|---------|-------|------------|---------|-------------|----------|----------|--------|--------| -| P2-1 | I | I | P | I | I | I | 2 | Low | -| P2-2 | I | I | F | I | I | I | 2 | Low | -| P2-3 | I | I | F | I | I | I | 2 | Low | - ---- - -## Signal Summary - -### High-Signal Tests (3+ distinct outcomes) - -| Test | Dominant Factor | Notes | -|------|----------------|-------| -| A-2 MED | Tool capability | ACPF convergence at 10k: pandapower/gridcal/matpower pass; PM/PSI fail | -| A-3 MED | Infrastructure/data | PyPSA fails from zero s_nom; others pass with various approaches | -| A-4 MED | Tool capability + data | ACPF-based feasibility; some tools cannot solve ACPF at scale | -| A-5 TINY | Tool capability | SCUC architecture: built-in (pypsa/matpower) vs manual (PM/PSI) vs absent (pp/gc) | -| A-6 TINY | Tool capability | Same split as A-5 (SCED depends on UC) | -| A-7 MED | Combinatorial + tool | N-M explosion; only pp/gc/matpower complete at MEDIUM | -| A-9 TINY/SMALL | Tool capability | SCOPF: pypsa/matpower native; PM manual; pp/gc absent | -| A-10 | Tool capability | Lossy DCOPF: wide range from native (pypsa) to absent (pp/PSI) | -| A-11 | Tool capability | Distributed slack: wide range from native to absent | -| B-1 | Tool architecture | Custom constraints: JuMP-based (PM/PSI) and linopy (pypsa) vs absent (gc) | -| B-9 MED | Data + tool | PTDF accuracy at scale; phase-shifter correction issue (probe-010) | - -### Low-Signal Tests (unanimous or near-unanimous) - -| Test | Outcome | Reason | -|------|---------|--------| -| G-1/G-2/G-3 | All pass | MATPOWER format trivial for all tools | -| A-1 TINY | All pass | DCPF on 39-bus is trivial | -| A-2 TINY | All pass | ACPF on 39-bus is trivial | -| B-3 TINY | All pass | N-1 loop on 39-bus is trivial | -| B-5 TINY/MED | All pass | CSV export trivial for DataFrame-based tools | -| C-1 MED | All pass | DCPF scales well for all tools | -| C-8 MED | All fail | SCOPF at 500 contingencies on 10k-bus is infeasible for all | - ---- - -## Probe Impact on Outcomes - -| Probe | Tool | Test | Original | Corrected | Impact | -|-------|------|------|----------|-----------|--------| -| probe-001 | PyPSA | A-2 MED | pass | **should be fail** | 0 NR iterations, 83% flat start | -| probe-009 | pandapower | P2-3 | lambda 1e25 claim | **lambda claim debunked** | Both methods have identical convergence | -| probe-021 | PowerSim | A-4 | 100x unit mismatch | **labeling error** | Dispatch is MW, limits are pu; no actual mismatch | -| probe-025 | PowerSim | E-6 | 100% coverage | **78% coverage** | Badge misread | -| probe-010 | pandapower | B-9 | shunt attribution | **phase-shifter attribution** | Pbusinj/Pfinj correction eliminates all error | -| probe-006/007 | cross-tool | A-3/C-3 | uniform LMPs | **confirmed: no binding constraints** | Network insufficiency, not tool issue | -| probe-028 | MATPOWER | C-10 | 66 min timing | **confirmed: dense PTDF matrix** | MIPS on dense matrices is 400x slower | -| probe-029 | MATPOWER | C-5 | 97% Octave overhead | **confirmed** | LODF screening fast; containers.Map is bottleneck | -| probe-032 | MATPOWER | C-4 | solver capacity fail | **loadmd() ingestion fail** | Solver never invoked | diff --git a/sweep-data/v4-to-v5/aggregation/low-signal-tests.yaml b/sweep-data/v4-to-v5/aggregation/low-signal-tests.yaml deleted file mode 100644 index 4c11cb14..00000000 --- a/sweep-data/v4-to-v5/aggregation/low-signal-tests.yaml +++ /dev/null @@ -1,144 +0,0 @@ -low_signal_tests: - - test_id: G-1 - outcome: unanimous_pass - tools_count: 6 - reason: > - All tools have MATPOWER .m parsers. Gate test serves its purpose as a - precondition filter but provides zero discriminative signal between tools. - preserve: true - preserve_reason: Gate tests prevent downstream waste if a tool cannot ingest the format. - - - test_id: G-2 - outcome: unanimous_pass - tools_count: 6 - reason: Same as G-1. SMALL network ingestion trivial for all tools. - preserve: true - preserve_reason: Validates format handling at intermediate scale. - - - test_id: G-3 - outcome: unanimous_pass - tools_count: 6 - reason: Same as G-1. MEDIUM network ingestion trivial for all tools. - preserve: true - preserve_reason: Validates format handling at target scale. - - - test_id: A-1 TINY - outcome: unanimous_pass - tools_count: 6 - reason: DCPF on 39-bus is trivial for any power systems tool. - preserve: true - preserve_reason: Functional baseline for PF capability. - - - test_id: A-2 TINY - outcome: unanimous_pass - tools_count: 6 - reason: ACPF on 39-bus converges for all tools. - preserve: true - preserve_reason: Baseline; the MEDIUM test has high signal. - - - test_id: A-3 TINY - outcome: near_unanimous_pass - tools_count: 6 - reason: > - All tools solve DCOPF on 39-bus. LMPs are uniform across all tools due to - identical generator costs (0.3 $/MWh) and no binding constraints. - Confirmed by probes 006/007. The test cannot verify congestion-driven LMP - differentiation on this network. - preserve: true - preserve_reason: MEDIUM test has higher signal; TINY validates basic OPF. - proposed_change: > - Tighten 3-5 branch limits on case39 to force at least one binding constraint - and produce non-uniform LMPs. This enables LMP extraction verification at TINY. - - - test_id: A-5 TINY (cycling aspect) - outcome: unanimous_no_cycling - tools_count: 6 - reason: > - case39 capacity-to-load ratio means all generators stay committed for all - 24 hours. No tool demonstrates UC cycling. Confirmed across pypsa, - powersimulations, matpower (all report 10/10 generators on, 0 startups). - preserve: true - preserve_reason: The formulation test has value; the network is insufficient. - proposed_change: > - Augment case39 with excess capacity (add 2 peakers with high startup costs) - or reduce minimum load to force at least 2 generators to cycle. - - - test_id: A-9 TINY (cost comparison aspect) - outcome: unanimous_no_signal - tools_count: 4 - reason: > - SCOPF and unconstrained OPF produce identical objectives on case39 because - all generators have identical marginal costs and no branches bind. Cost - comparison produces no signal. pypsa notes objective equality, matpower notes - identical dispatch. - preserve: true - preserve_reason: The SMALL test produces a meaningful security premium (0.4-4.7%). - proposed_change: > - Use perturbed generator costs on TINY to enable cost comparison even without - congestion. - - - test_id: A-11 (on uncongested networks) - outcome: unanimous_no_discrimination - tools_count: 5 - reason: > - Distributed slack vs single slack produces identical LMPs on uncongested - networks (case39 and ACTIVSg2000). Confirmed by probes 023 and findings - from pypsa, powermodels, powersimulations, matpower. The test cannot - verify that distributed slack changes LMP decomposition because there is - no congestion to decompose. - preserve: true - preserve_reason: The capability test matters for ISO market clearing fidelity. - proposed_change: > - Use a congested network (tightened branch limits) where distributed vs - single slack produces demonstrably different LMPs. - - - test_id: B-3 TINY - outcome: unanimous_pass - tools_count: 6 - reason: N-1 contingency loop on 39-bus is trivial for all tools. - preserve: true - preserve_reason: Baseline; MEDIUM test has medium signal. - - - test_id: B-5 - outcome: unanimous_pass - tools_count: 6 - reason: > - CSV export trivial for any tool using DataFrames (Python) or matrices - (Octave/Julia). pypsa: 2 LOC, matpower: 18 LOC, all others < 5 LOC. - preserve: true - preserve_reason: Documents interoperability baseline even if non-discriminative. - - - test_id: C-1 MED - outcome: unanimous_pass - tools_count: 6 - reason: DCPF scales well for all tools (all complete in < 30s on 10k-bus). - preserve: true - preserve_reason: Provides timing baseline for cross-tool comparison. - - - test_id: C-8 MED - outcome: unanimous_fail - tools_count: 6 - reason: > - 500-contingency SCOPF on 10k-bus is infeasible for all tools within the - time budget. Probe-016 confirms even 5 contingencies time out for - PowerModels. No tool can demonstrate this capability at MEDIUM scale. - preserve: false - preserve_reason: null - proposed_change: > - Reduce C-8 MEDIUM to 50 contingencies (matching SMALL parameter) or - accept iterative screening with only binding contingencies. The current - 500-contingency target produces unanimous failure with zero discriminative - value. - - - test_id: A-8 - outcome: near_unanimous_fail - tools_count: 6 - reason: > - Only MATPOWER (via MOST) has native stochastic optimization. All other - tools fail or are absent. The B-4 wrapping test provides the practical - stochastic capability assessment. Probe-002 shows PyPSA has partial - stochastic API that crashes on realistic networks. - preserve: true - preserve_reason: > - The A-8/B-4 distinction (native vs wrapping) is protocol-correct and - reveals a genuine architectural gap. The near-unanimous fail IS the signal. diff --git a/sweep-data/v4-to-v5/aggregation/proposed-changes.yaml b/sweep-data/v4-to-v5/aggregation/proposed-changes.yaml deleted file mode 100644 index b1c9c00d..00000000 --- a/sweep-data/v4-to-v5/aggregation/proposed-changes.yaml +++ /dev/null @@ -1,298 +0,0 @@ -proposed_changes: - # --- Protocol Changes (require 3+ tools evidence) --- - - - id: PC-01 - type: protocol - target: data_preparation - title: "Standardize ACTIVSg10k preprocessing to force binding constraints" - rationale: > - ACTIVSg10k DCOPF has no binding branch constraints (max loading 84-85%), - producing uniform LMPs across all tools. Tests targeting congestion-driven - capabilities (A-3, A-9, A-11, B-8, C-3, C-10) lose discriminative value - at the grade-assessment scale. - evidence_tools: [pypsa, pandapower, gridcal, powermodels, matpower] - evidence_findings: [pypsa-F02, pandapower-F02, gridcal-F05, pm-F01, matpower-F03] - evidence_probes: [probe-006, probe-007, probe-023] - priority: high - specific_change: > - Add a protocol-level preprocessing step: (a) set zero-reactance transformers - to x=0.0001 pu, (b) set zero thermal rating branches to RATE_A=9999 MVA, - (c) tighten 10-20 highest-flow branch limits to 90% of base-case DCPF flow. - Apply uniformly across all test dimensions. Document the preprocessing - script in the protocol appendix. - - - id: PC-02 - type: protocol - target: test_A-2 - title: "Strengthen ACPF convergence verification requirements" - rationale: > - Probe-001 debunked PyPSA's A-2 MEDIUM pass (0 NR iterations, 83% flat start). - Multiple tools lack convergence diagnostics (PowerModels: no NLsolve access; - PowerSimulations: silent Missing return). The current pass condition allows - tools to claim convergence without demonstrating it. - evidence_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations] - evidence_findings: [pypsa-F05, pandapower-F12, gridcal-F06, pm-F08, psi-F10] - evidence_probes: [probe-001, probe-024] - priority: high - specific_change: > - A-2 pass condition additions: (a) convergence residual must be reported and - below the tool's stated tolerance, (b) number of NR iterations must be - reported, (c) voltage magnitudes must differ from flat-start defaults (1.0 pu) - on >95% of buses. If the tool cannot report iteration count or residual, - document this as a diagnostic quality finding. - - - id: PC-03 - type: protocol - target: test_B-9 - title: "Account for phase-shifting transformers in PTDF validation" - rationale: > - Probe-010 identified that ACTIVSg10k's 5 phase-shifting transformers create - Pbusinj/Pfinj correction terms not included in the standard PTDF formula. - This causes 743 MW errors in pandapower and gridcal (probes 008, 010, 012). - Applying the correction eliminates all error to machine precision. The B-9 - pass condition (1e-6 tolerance) is unachievable without correction. - evidence_tools: [pandapower, gridcal, pypsa] - evidence_findings: [pandapower-F03, gridcal-F04, pypsa-F04] - evidence_probes: [probe-008, probe-010, probe-012] - priority: high - specific_change: > - B-9/C-9 pass condition: "If the network contains phase-shifting transformers - (nonzero SHIFT column in branch data), the PTDF validation must either - (a) apply Pbusinj/Pfinj correction terms from the admittance matrix - construction, or (b) exclude branches with nonzero shift angles from the - accuracy comparison. The tolerance of 1e-6 applies to the corrected or - filtered comparison." - - - id: PC-04 - type: protocol - target: test_A-5 - title: "Modify case39 UC parameters to force generator cycling" - rationale: > - All tools capable of SCUC report all generators committed for all 24 hours - on case39. The capacity-to-load ratio makes decommitment uneconomical. UC - binary variables, min up/down times, and startup costs are never exercised. - evidence_tools: [pypsa, powersimulations, matpower] - evidence_findings: [pypsa-F07, psi-F05, matpower-F05] - evidence_probes: [] - priority: medium - specific_change: > - Either (a) increase PMIN to 30% of PMAX (forcing higher base capacity - commitment that makes some generators uneconomical at low load), or - (b) add 2-3 peaker generators with high startup cost and small capacity, - or (c) widen the load range (min load to 40% of peak). Target: at least - 2 generators cycling during the 24-hour horizon. - - - id: PC-05 - type: protocol - target: scalability_timing - title: "Require measured wall-clock times for all scalability grades" - rationale: > - PowerSimulations has 4 C-tests with null wall_clock_seconds (estimated only). - PowerModels has 2 C-tests scored as fail without execution. Probe-020 - confirmed C-3 was never executed. The protocol requires measured timings - but does not explicitly prohibit grading on estimates. - evidence_tools: [powersimulations, powermodels, gridcal] - evidence_findings: [psi-F01, pm-F03, gridcal-F13] - evidence_probes: [probe-016, probe-020] - priority: high - specific_change: > - Add to Test Execution Protocol: "Estimated or projected timings must be - clearly labeled with 'estimated' in the frontmatter and cannot support - pass or qualified_pass on scalability tests. If a test cannot be executed - within the time budget (including JIT/startup overhead for first runs), - record fail with the projected timing as supplementary context." - - - id: PC-06 - type: protocol - target: cascaded_failures - title: "Distinguish independent from cascaded failures in scoring" - rationale: > - 4 tools have scalability tests that fail solely because prerequisite - expressiveness tests failed. These cascaded failures inflate fail counts - without adding signal. - evidence_tools: [pandapower, gridcal, powersimulations, powermodels] - evidence_findings: [pandapower-F11, gridcal-F02, psi-F04, pm-F13] - evidence_probes: [] - priority: medium - specific_change: > - Add `blocked_by: ` field to result frontmatter. When tabulating - outcomes, report "X independent fails + Y blocked" rather than a single - count. Blocked tests do not contribute to the criterion's fail count but - are listed for completeness. - - - id: PC-07 - type: protocol - target: test_C-8 - title: "Reduce C-8 MEDIUM contingency count from 500 to 50" - rationale: > - 500-contingency SCOPF on 10k-bus is a unanimous fail across all 6 tools. - Probe-016 shows even 5 contingencies time out for PowerModels. The test - produces zero discriminative value at the current parameter. - evidence_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - evidence_findings: [pypsa-F03, pm-F03] - evidence_probes: [probe-016] - priority: medium - specific_change: > - C-8 MEDIUM: reduce from 500 to 50 monitored contingencies (matching the - SMALL parameter). Allow iterative screening. This makes the test feasible - for tools with working SCOPF while still testing scalability beyond SMALL. - - - id: PC-08 - type: protocol - target: test_C-5 - title: "Cap N-M contingency sweep at N-2 for MEDIUM" - rationale: > - N-4 on 10k-bus produces combinatorial explosion infeasible for all tools. - The discriminative signal is at N-1 and N-2 only. - evidence_tools: [pypsa, powermodels, matpower] - evidence_findings: [pypsa-F03, pm-F12, matpower-F02] - evidence_probes: [probe-016, probe-029] - priority: medium - specific_change: > - C-5 MEDIUM: change parameters from x=5/m=4 to x=5/m=2 for graded - assessment. N-3 and N-4 are informational only and do not affect the - pass/fail determination. - - - id: PC-09 - type: protocol - target: test_A-10 - title: "Replace MATPOWER lossy reference with internal consistency checks" - rationale: > - The A-10 pass condition requires MATPOWER reference validation, but MATPOWER's - rundcopf has no loss option producing comparable LMPs. No tool can fulfill - this requirement. 4 tools document the inability to perform this validation. - evidence_tools: [pypsa, gridcal, powermodels, matpower] - evidence_findings: [pypsa-F09, gridcal-F08, pm-F06, matpower-F09] - evidence_probes: [] - priority: medium - specific_change: > - Replace "Validate against MATPOWER reference lossy DC OPF solution" with: - "Validate internal consistency: (a) loss components have physically correct - signs (positive marginal loss at load buses far from generation), (b) total - losses are 0.5-3% of total load, (c) lossy objective exceeds lossless - objective, (d) loss component LMPs sum with energy and congestion components - to total LMP within 1% tolerance." - - - id: PC-10 - type: protocol - target: stochastic_perturbation - title: "Calibrate B-4/C-6 perturbation bounds to limit infeasibility" - rationale: > - Uncalibrated perturbations cause 53-99.6% infeasibility across 3 tools. - Probe-006 showed the base case converges and uniform load scaling works; - combined perturbations are the issue. - evidence_tools: [pandapower, gridcal, powersimulations] - evidence_findings: [pandapower-F01, gridcal-F10, psi-F13] - evidence_probes: [probe-006] - priority: medium - specific_change: > - Add to B-4/C-6: "Perturbation bounds should be calibrated to produce - at most 20% infeasible scenarios on the target network. If infeasibility - exceeds 20%, reduce perturbation sigma until the threshold is met and - document the final sigma. Alternatively, OPF formulations may include - slack variables for load shedding to prevent infeasibility." - - # --- Rubric Changes (require 3+ tools evidence) --- - - - id: RC-01 - type: rubric - target: workaround_scoring - title: "Clarify that pass+workaround should be qualified_pass" - rationale: > - Multiple tools have tests scored as pass with workaround_class: stable. - The protocol states stable workarounds receive B-range grades, implying - qualified_pass, but this is not enforced consistently. - evidence_tools: [powersimulations, pypsa, pandapower] - evidence_findings: [psi-F04, pypsa-F03, pandapower-F05] - evidence_probes: [] - priority: low - specific_change: > - Add explicit rule: "A test requiring any workaround (stable, fragile, or - blocking) must be classified as qualified_pass, not pass. The workaround - class affects the grade within the B range but never produces a clean pass." - - - id: RC-02 - type: rubric - target: solver_capability_separation - title: "Separate tool expressiveness from solver performance in scoring" - rationale: > - A-5/C-4 failures on 2000-bus UC conflate HiGHS single-threaded MIP timeout - with tool capability. A tool that expresses SCUC in 140 LOC but cannot solve - it receives the same fail as one that cannot express it. - evidence_tools: [pypsa, powermodels, matpower] - evidence_findings: [pm-F04, pypsa-F07] - evidence_probes: [] - priority: medium - specific_change: > - For expressiveness tests (Suite A): if a tool can express the formulation - but the open-source solver times out, score as qualified_pass with - solver-limitation note (not fail). The solver limitation is recorded - as a scalability finding (Suite C). For scalability tests (Suite C): - retain fail classification for solver timeouts but note the tool's - expressiveness separately. - - # --- Skill-Only Changes (require 2+ tools evidence) --- - - - id: SC-01 - type: skill - target: evaluate-tool - title: "Add peak memory measurement template per language" - rationale: > - Peak memory missing or estimated across all tools. Python, Julia, and - Octave each need different measurement approaches. - evidence_tools: [pandapower, powermodels, matpower] - evidence_findings: [pandapower-F08, pm-F14, matpower-F11] - evidence_probes: [] - priority: low - specific_change: > - Add to test script templates: Python: `import tracemalloc; tracemalloc.start() - ... tracemalloc.get_traced_memory()`. Julia: `@allocated` or parse - /proc/self/status. Octave: parse /proc/self/status before and after solve. - - - id: SC-02 - type: skill - target: evaluate-tool - title: "Add binding constraint test to B-1 template" - rationale: > - Both PSI and pandapower verified custom constraint duals only for - non-binding cases (dual=0). Probe-022 showed PSI binding dual works - when tested properly. - evidence_tools: [powersimulations, pandapower] - evidence_findings: [psi-F03] - evidence_probes: [probe-022] - priority: low - specific_change: > - B-1 template should include both a non-binding gate limit (verify dual=0) - AND a binding gate limit (set at 50% of unconstrained flow, verify - dual != 0 and objective increases). - - - id: SC-03 - type: skill - target: evaluate-tool - title: "Add E-6 badge verification step" - rationale: > - Probe-025 debunked PSI's 100% coverage claim (actual: 78%). Coverage - badges can be stale or misread. - evidence_tools: [powersimulations] - evidence_findings: [psi-F12] - evidence_probes: [probe-025] - priority: low - specific_change: > - E-6 template should include: "Verify coverage percentage by fetching the - badge SVG or checking the Codecov/Coveralls detail page. Do not rely on - the badge rendering in the README alone." - - - id: SC-04 - type: skill - target: evaluate-tool - title: "Add unit consistency check to dispatch-to-ACPF transfer" - rationale: > - Probe-021 debunked PSI's 100x unit mismatch (actually MW vs pu labeling - error). The evaluation script applied MW values where pu was expected. - evidence_tools: [powersimulations] - evidence_findings: [psi-F02] - evidence_probes: [probe-021] - priority: low - specific_change: > - A-4/B-7 template should include explicit unit logging: print base_power, - print dispatch units, print limit units, verify consistency before transfer. diff --git a/sweep-data/v4-to-v5/aggregation/themes.md b/sweep-data/v4-to-v5/aggregation/themes.md deleted file mode 100644 index 9d032ac1..00000000 --- a/sweep-data/v4-to-v5/aggregation/themes.md +++ /dev/null @@ -1,241 +0,0 @@ -# Cross-Cutting Themes Analysis - -## Phase 1 Tool Selection | v4-to-v5 Aggregation - ---- - -## Executive Summary - -Analysis of 6 per-tool sweep findings (92 total findings) and 18 probe results reveals 13 cross-cutting themes. The most impactful themes concern test network insufficiency (3 themes affecting 20+ tests), estimated rather than measured scalability data (1 theme affecting 5 tests), and a protocol gap in PTDF validation on networks with phase-shifting transformers (1 theme affecting 3 tools). Four probe-debunked claims strengthen the case for protocol changes: PyPSA's false ACPF convergence (probe-001), pandapower's non-reproducible lambda values (probe-009), PowerSimulations' unit labeling error (probe-021), and PowerSimulations' coverage misreport (probe-025). - ---- - -## Theme 1: ACTIVSg10k Has No Binding Constraints (T01) - -**Category:** Network insufficiency | **Evidence:** 5 tools | **Probes:** 006, 007, 023 - -The MEDIUM grade network (ACTIVSg 10k) produces zero binding branch constraints in DCOPF across all tools that can solve it. Maximum branch loading is 84-85% (pandapower: 84.9%, gridcal: 84.7%). LMPs are perfectly uniform ($20.064-$20.738/MWh depending on cost function treatment). - -This finding was confirmed by three independent probes: -- **Probe-007** verified pandapower LMPs are uniform to machine precision (std = 6e-12) and demonstrated that artificial congestion immediately produces meaningful LMP spreads (16.3-24.4 $/MWh). -- **Probe-006** showed the base case DCOPF converges cleanly, confirming uniformity is not a solver issue. -- **Probe-023** confirmed PowerSimulations' PTDF and DCP formulations produce identical results on the uncongested network. - -**Impact:** Tests A-3, A-9, A-11, B-8, C-3, and C-10 lose discriminative value at MEDIUM scale for any capability dependent on congestion (LMP decomposition, SCOPF cost premium, distributed slack LMP differences). The grade-assessment network fails to exercise the features being graded. - -**Proposed remedy:** Tighten 10-20 branch limits in ACTIVSg 10k to 90% of base-case flow as a protocol-level preprocessing step. This preserves the network's realistic topology while enabling congestion signal. - ---- - -## Theme 2: case39 Uniform Costs and No Congestion (T02) - -**Category:** Network insufficiency | **Evidence:** 5 tools - -IEEE 39-bus has identical generator cost coefficients (c1 = 0.3 $/MWh) and no binding branch constraints at the base operating point. This produces uniform LMPs across all 39 buses for every tool. Specific impacts: - -- **A-3 TINY:** All tools report uniform LMPs. The test verifies OPF convergence but not LMP quality. -- **A-9 TINY:** SCOPF objective equals unconstrained OPF objective (pypsa: $1876.269 both). Cost comparison -- a key verification -- produces zero signal. -- **B-8 TINY:** All three slack configurations produce identical LMPs, providing zero discriminative value for reference bus testing. - -The SMALL network (ACTIVSg 2000) has heterogeneous costs and produces meaningful LMP spreads (gridcal B-8 SMALL: 11.1 to 32.9 $/MWh). SMALL is the functional grade network for these tests; TINY serves only as a formulation check. - -**Proposed remedy:** Perturb case39 generator costs (scale c1 by 0.5x-2.0x) and tighten 3-5 branch limits. - ---- - -## Theme 3: SCUC Produces No Generator Cycling on case39 (T03) - -**Category:** Network insufficiency | **Evidence:** 3 tools (pypsa, powersimulations, matpower) - -Every tool capable of SCUC reports all 10 generators committed for all 24 hours with zero startups. The capacity-to-load ratio (7,367 MW capacity vs 6,254 MW peak load) combined with startup costs at 5x PMAX means decommitment is never optimal. This renders A-5 a formulation existence test rather than a UC correctness test. A-6 (SCED) shows zero dispatch difference from A-5 because the commitment schedule is trivially "all on." - -MATPOWER even reports uniform ramp utilization (26.7% across all generators), confirming that ramp constraints are never near binding. - -**Proposed remedy:** Augment case39 with 2-3 peakers (high startup cost, small capacity) or increase PMIN to 30% of PMAX to force cycling. - ---- - -## Theme 4: Estimated Timings Without Execution (T04) - -**Category:** Extraordinary claim pattern | **Evidence:** 3 tools | **Probes:** 016, 020 - -PowerSimulations has four scalability tests (C-3, C-4, C-5, C-6) with `wall_clock_seconds: null` and language like "expected to solve" or "estimated." Probe-020 confirmed C-3 was never executed (first attempt exceeded 600s timeout). PowerModels scored C-5 and C-8 as fail without execution based on projected infeasibility; probe-016 confirmed the projections were directionally correct but based on rough extrapolation. - -The protocol states: "Record everything. For each test, record: Wall-clock time (for scalability-relevant tests)." Four PowerSimulations and two PowerModels tests violate this requirement. The debunked probe-001 (PyPSA ACPF with 0 NR iterations classified as pass) shows a related pattern: results classified without adequate underlying evidence. - -**Proposed remedy:** Unmeasured timings cannot support pass or qualified_pass. If execution is not feasible, record fail with projected timing as context. - ---- - -## Theme 5: Cascaded Failures Inflate Fail Counts (T05) - -**Category:** Scoring inconsistency | **Evidence:** 4 tools - -Multiple tools have scalability tests (C-series) that fail solely because their prerequisite expressiveness tests (A-series) failed: - -| Tool | Cascaded Fails | Independent Fails | Total Fails | -|------|---------------|-------------------|-------------| -| pandapower | C-4, C-8, C-10 (3) | 11 | 14 | -| gridcal | C-4, C-6 (2) | 14 | 16 | -| powersimulations | C-8, C-10 (2) | 4 | 6 | -| powermodels | C-4 (1) | 11 | 12 | - -These cascaded failures add no new information but inflate the apparent gap count. A tool with 5 independent fail reasons and 3 cascaded failures appears to have 8 problems when it has 5. - -**Proposed remedy:** Add a `blocked_by` field to result frontmatter. Report independent and cascaded failures separately in summary tables. - ---- - -## Theme 6: PTDF Phase-Shifter Correction Gap (T06) - -**Category:** Protocol gap | **Evidence:** 3 tools | **Probes:** 008, 010, 012 - -Probe-010 is the most technically significant finding in the sweep. It identified that ACTIVSg10k has 5 phase-shifting transformers with nonzero SHIFT angles that create Pbusinj (bus injection correction) and Pfinj (branch flow correction) terms. The standard PTDF formula `flow = PTDF @ Pinj` omits these corrections, causing errors up to 743 MW. - -The full equation is: `flow = PTDF @ (Pinj - Pbusinj) + Pfinj` - -Applying this correction eliminates ALL error to machine precision (1e-12). The PTDF matrix itself is correct in all tools tested. - -| Tool | Max Error (uncorrected) | Attribution in eval | Actual cause | -|------|------------------------|--------------------|----| -| pandapower | 7.43 pu (743 MW) | "shunt elements" (wrong) | Phase-shifter Pfinj | -| gridcal | 743.46 MW | "island handling" (wrong) | Phase-shifter Pfinj | -| pypsa | 702 MW | "zero-impedance fix" (partially wrong) | Phase-shifter Pfinj | -| powermodels | < 1e-11 | N/A (correct) | Handles internally | - -The original B-9 attributions were incorrect (shunts have zero MW on this network; island count is 1). The protocol's pass condition ("flow predictions match DCPF results within 1e-6") is unachievable on any network with phase-shifting transformers unless correction terms are applied. - -**Proposed remedy:** Protocol should specify that PTDF validation must account for phase-shifter correction terms (Pbusinj/Pfinj), or the test should exclude phase-shifting transformer branches from the accuracy check. - ---- - -## Theme 7: ACTIVSg10k Zero-Impedance Branch Inconsistency (T07) - -**Category:** Infrastructure friction | **Evidence:** 4 tools - -ACTIVSg10k has 2,462 branches with zero thermal rating and 3 transformers with zero reactance. Different tools handle these differently: - -| Tool | Treatment of RATE_A=0 | Treatment of x=0 | -|------|-----------------------|-------------------| -| PyPSA | Zero-capacity constraint (infeasible) | Singular B-matrix (NaN flows) | -| pandapower | Unconstrained (no limit) | Passes through PYPOWER | -| gridcal | Unconstrained | Passes through | -| powermodels | Unconstrained (data fix for gencost) | Passes through | -| matpower | Unconstrained | Passes through | - -PyPSA's expressiveness tests (A-3 MEDIUM) fail due to these data issues, while scalability tests (C-3 MEDIUM) pass after applying data fixes (s_nom=9999, x=0.0001). This creates contradictory results: the same tool fails expressiveness but passes scalability on the same network for the same analysis. - -**Proposed remedy:** Standardize ACTIVSg10k preprocessing as a protocol step: zero-reactance transformers get x=0.0001, zero-rating branches get RATE_A=9999. Applied uniformly across all dimensions. - ---- - -## Theme 8: Stochastic Perturbation Methodology Causes Excessive Infeasibility (T08) - -**Category:** Test design gap | **Evidence:** 3 tools | **Probe:** 006 - -The B-4/C-6 stochastic scenario methodology produces very high infeasibility rates when load and generator capacity perturbations are applied simultaneously: - -| Tool | Network | Convergence Rate | -|------|---------|-----------------| -| pandapower | SMALL | 0.42-2.1% (probe-006) | -| gridcal | SMALL | 47% | -| powersimulations | TINY | 60% (3/5) | - -Probe-006 showed that uniform load scaling alone (0.7x-1.1x) converges reliably; it is the combination of load AND generator perturbations that breaks the solver. This measures solver robustness to near-infeasible problems rather than the tool's stochastic wrapping capability. - -**Proposed remedy:** Calibrate perturbation bounds to produce at most 20% infeasible scenarios, or require OPF formulations with load-shedding slack variables. - ---- - -## Theme 9: Lossy DCOPF MATPOWER Reference Validation Unachievable (T09) - -**Category:** Missing verification | **Evidence:** 4 tools - -The A-10 pass condition requires "Validate against MATPOWER reference lossy DC OPF solution." No tool performs this validation because MATPOWER's `rundcopf` does not have a loss option producing comparable LMPs. PowerModels notes "MATPOWER used lossless DC OPF with post-hoc loss estimation." MATPOWER's own evaluation performs a post-hoc loss estimate that is explicitly "NOT part of the optimization." - -The reference validation is structurally impossible as currently specified. - -**Proposed remedy:** Replace MATPOWER reference requirement with internal consistency checks: (a) loss components have physically correct signs, (b) total losses are 0.5-3% of load, (c) lossless-vs-lossy objective difference is positive. - ---- - -## Theme 10: ACPF at 10k-Bus Is a Genuine Discriminator (T10) - -**Category:** High-signal test | **Evidence:** 6 tools | **Probes:** 001, 024 - -A-2 MEDIUM produces three distinct outcome classes with strong discriminative value: - -1. **Converges with diagnostics:** pandapower (DC warm start), gridcal (convergence error 2.73e-07), matpower (standard NR) -2. **Fails with minimal diagnostics:** PowerModels (NLsolve fails, no iteration count exposed), PowerSimulations (returns `Missing` silently -- probe-024) -3. **Reports false convergence:** PyPSA (probe-001: 0 NR iterations, 83% flat start, converged: False but originally classified as pass) - -Probe-001 is particularly impactful: the PyPSA ACPF "pass" on MEDIUM is invalid. The solver performed zero iterations and left 91% of lines with zero flow. - -**Proposed remedy:** Strengthen A-2 pass condition: require convergence residual < tolerance, iteration count reported, and >95% of buses not at flat-start defaults. - ---- - -## Theme 11: SCOPF and N-M Sweep Infeasibility at MEDIUM Scale (T13) - -**Category:** Test design gap | **Evidence:** 3+ tools | **Probe:** 016 - -C-8 MEDIUM (500-contingency SCOPF) is a unanimous fail across all 6 tools. Probe-016 showed that even 5 contingencies time out for PowerModels at MEDIUM scale. The protocol's 500-contingency target on 10k-bus produces zero discriminative value. - -Similarly, the N-M sweep at x=5, m=4 on MEDIUM is infeasible at N-3+ for all tools. The combinatorial explosion (C(1000,3) = 166M combinations) means no tool can complete the test as specified. The discriminative signal is at N-1 and N-2 only. - -**Proposed remedy:** Reduce C-8 MEDIUM to 50 contingencies. Cap N-M sweep at N-2 for MEDIUM with N-3+ as informational. - ---- - -## Theme 12: Solver Capability vs Tool Capability Conflation (T04/T11) - -**Category:** Scoring inconsistency | **Evidence:** 4 tools - -Multiple tests conflate open-source solver limitations with tool capability: - -- **A-5/C-4:** HiGHS single-threaded MIP timeout on 2000-bus UC affects pypsa, PowerModels. MATPOWER passes because MIPS handles the formulation (different solver, same tool capability). -- **C-7:** Solver swap is parameter-only for 4 tools, but qualified_pass given because alternative solvers either fail or are not installed. -- **C-3/C-8:** HiGHS QP failure on ACTIVSg2000 forces Ipopt workarounds for PowerModels. - -The protocol mandates single-threaded open-source solvers, creating a ceiling on SCUC scale that is independent of tool capability. A tool that can express the formulation in 140 LOC but cannot solve it receives the same fail as a tool that cannot express it at all. - -**Proposed remedy:** Separate "can the tool express the formulation" (expressiveness) from "can the solver complete within the budget" (scalability). Allow multi-threaded or commercial solver results as supplementary evidence. - ---- - -## Theme 13: Maturity Metrics Based on Research Without Verification (T12-proxy) - -**Category:** Missing verification | **Evidence:** 6 tools | **Probe:** 025 - -All E-series metrics are research-based. Probe-025 debunked PowerSimulations' "100% code coverage" claim (actual: 78% from Codecov badge). Other maturity numbers (commit counts, contributor percentages, deployment claims) are plausible but unverified. This is inherent to the audit methodology, but the coverage misread demonstrates the risk of accepting badge-reported metrics without verification. - -**Proposed remedy:** For E-6 (CI/test coverage), require the evaluator to check the Codecov/Coveralls detail page, not just the badge. For E-7 (operational adoption), require a linked source for each claimed deployment. - ---- - -## Probe Integration Summary - -### Debunked Claims (4) - -| Probe | Tool | Claim | Impact | -|-------|------|-------|--------| -| probe-001 | PyPSA | ACPF "pass" on MEDIUM | A-2 MEDIUM should be reclassified as fail; strengthens T10 | -| probe-009 | pandapower | Lambda 1e25 with in_service=False | Both decommitment methods have identical convergence; P2-3 finding weakened | -| probe-021 | PowerSim | 100x dispatch/limit mismatch | Labeling error (MW vs pu); A-4 and B-7 findings need revision | -| probe-025 | PowerSim | 100% code coverage | Actual: 78%; E-6 needs correction | - -### Supported Claims with Protocol Implications (10) - -| Probe | Finding | Protocol Impact | -|-------|---------|-----------------| -| probe-006/007 | Uniform LMPs on ACTIVSg10k | T01: tighten branch limits | -| probe-010 | Phase-shifter PTDF correction | T06: update B-9 pass condition | -| probe-012 | GridCal 743 MW PTDF divergence | T06: same root cause as pandapower | -| probe-016 | PM C-5/C-8 infeasibility confirmed | T04/T13: reduce scale targets | -| probe-020 | PSI C-3 timing never measured | T04: require measured timings | -| probe-022 | PSI binding dual works | B-1 methodology gap but tool works | -| probe-023 | Distributed slack no-op on uncongested | T01: need congested network | -| probe-024 | PSI ACPF silent failure | T10: require diagnostic output | -| probe-028 | MATPOWER 400x slowdown from dense PTDF | Dense matrix formulation issue | -| probe-029 | 97% Octave overhead in contingency sweep | Interpreter, not algorithm | -| probe-032 | MOST loadmd() fails at ingestion | Data ingestion bug, not solver | diff --git a/sweep-data/v4-to-v5/aggregation/themes.yaml b/sweep-data/v4-to-v5/aggregation/themes.yaml deleted file mode 100644 index 4860c55a..00000000 --- a/sweep-data/v4-to-v5/aggregation/themes.yaml +++ /dev/null @@ -1,281 +0,0 @@ -themes: - - id: T01 - title: "ACTIVSg10k has no binding constraints, producing uniform LMPs across all tools" - category: network_insufficiency - evidence_tools: [pypsa, pandapower, gridcal, powermodels, matpower] - evidence_count: 5 - affected_tests: [A-3, A-9, A-11, B-8, C-3, C-10] - probe_evidence: [probe-006, probe-007, probe-023] - description: > - The ACTIVSg 10k DCOPF produces zero binding branch constraints (max loading - 84-85%) and perfectly uniform LMPs ($20.064-20.738/MWh depending on tool's - cost function treatment). This is confirmed across pandapower, gridcal, - powermodels, and matpower, with pypsa unable to solve due to zero-s_nom - branches. Probe-007 verified that artificial congestion (reducing limits on - 5 lines) immediately produces LMP spreads of 16.3-24.4 $/MWh. The - uncongested MEDIUM network means tests targeting congestion-related - capabilities (LMP decomposition, SCOPF cost premium, distributed slack LMP - differences) produce no discriminative signal at the grade-assessment scale. - proposed_protocol_impact: > - Modify ACTIVSg 10k data preparation to tighten 10-20 branch limits to - 90% of base-case flow, forcing binding constraints and enabling congestion - signal in DCOPF tests. - - - id: T02 - title: "case39 uniform generator costs mask LMP and SCOPF cost comparison signals" - category: network_insufficiency - evidence_tools: [pypsa, pandapower, gridcal, powermodels, matpower] - evidence_count: 5 - affected_tests: [A-3, A-9, B-8] - probe_evidence: [] - description: > - IEEE 39-bus has all 10 generators with identical or near-identical linear - cost coefficients (0.3 $/MWh in MATPOWER format). Combined with no binding - constraints, this produces uniform LMPs at TINY scale across all tools. - SCOPF vs unconstrained OPF shows identical objectives (pypsa: $1876.269 - both; matpower: identical dispatch). B-8 reference bus comparison is - trivially identical. The SMALL network (ACTIVSg2000) with heterogeneous - costs provides the actual signal for these tests. - proposed_protocol_impact: > - Perturb generator costs on case39 (e.g., scale c1 coefficients by - 0.5x-2.0x across generators) to enable LMP differentiation testing at - TINY. Add 3-5 tightened branch limits to force congestion. - - - id: T03 - title: "SCUC on case39 produces no unit cycling -- network too small" - category: network_insufficiency - evidence_tools: [pypsa, powersimulations, matpower] - evidence_count: 3 - affected_tests: [A-5, A-6] - probe_evidence: [] - description: > - All three tools capable of SCUC (pypsa, powersimulations, matpower) report - all 10 generators committed for all 24 hours with zero startups and zero - cycling generators. The capacity-to-load ratio (7367 MW capacity vs 6254 MW - peak) means decommitment is never economically optimal. Min up/down time, - startup costs, and shutdown decisions are formulated but never exercised. - A-6 SCED comparison shows zero dispatch difference from UC. The test - verifies MILP formulation existence but not UC cycling correctness. - proposed_protocol_impact: > - Augment case39 generator fleet with 2-3 peakers at high startup cost and - low capacity, or increase generator PMIN to 30% of PMAX, to force at least - 2-3 generators to cycle across the 24-hour horizon. - - - id: T04 - title: "Estimated timings without execution undermine scalability evidence" - category: extraordinary_claim_pattern - evidence_tools: [powersimulations, powermodels, gridcal] - evidence_count: 3 - affected_tests: [C-3, C-4, C-5, C-6, C-8] - probe_evidence: [probe-016, probe-020] - description: > - PowerSimulations reports estimated timings for C-3 (<60s), C-4 (>300s), - C-5 (serial estimate), and C-6 (10-20 min) with wall_clock_seconds: null. - Probe-020 confirms C-3 was never measured (first run exceeded 600s timeout - including JIT). PowerModels C-5 and C-8 are scored as fail without execution - (projected infeasibility); probe-016 confirms the projections are directionally - correct but BFS scope was overestimated. GridCal D-1 timing is also an - estimate. The protocol explicitly requires "Record everything. Wall-clock - time (for scalability-relevant tests)." Unmeasured timings should not - receive pass or qualified_pass. - proposed_protocol_impact: > - Add explicit requirement: "Estimated timings must be clearly labeled as - such and cannot support a pass or qualified_pass on scalability tests. - If a test cannot be executed within the time budget, record fail with - the projected timing as context." - - - id: T05 - title: "Cascaded failures inflate fail counts without adding signal" - category: scoring_inconsistency - evidence_tools: [pandapower, gridcal, powersimulations, powermodels] - evidence_count: 4 - affected_tests: [C-4, C-6, C-8, C-10, A-6] - probe_evidence: [] - description: > - pandapower: C-4/C-8/C-10 fail solely because A-5/A-9/A-11 fail (3 cascaded - failures). GridCal: C-4/C-6 fail from TapPhaseControl bug cascading (2 - cascaded). PowerSimulations: C-8/C-10 fail from upstream expressiveness gaps. - PowerModels: C-4 duplicates A-5 SMALL. These cascaded failures add no new - information but inflate raw fail counts (e.g., pandapower shows 14 fails but - only 11 are independent). The protocol should distinguish independent failures - from inherited ones. - proposed_protocol_impact: > - Add a "blocked_by" field to test result frontmatter. When tabulating outcomes, - report both total fails and independent fails. Cascaded failures should be - listed as "blocked" rather than "fail" in summary tables. - - - id: T06 - title: "PTDF flow prediction errors from phase-shifting transformers affect multiple tools" - category: protocol_gap - evidence_tools: [pandapower, gridcal, pypsa] - evidence_count: 3 - affected_tests: [B-9, C-9] - probe_evidence: [probe-008, probe-010, probe-012] - description: > - Probe-010 identified that ACTIVSg10k has 5 phase-shifting transformers whose - nonzero shift angles create Pbusinj/Pfinj correction terms. The standard - PTDF = Bf * inv(Bbus) formulation omits these corrections, causing flow - prediction errors up to 743 MW (pandapower) and 743 MW (gridcal). Probe-010 - showed that applying corrected_flow = PTDF @ (Pinj - Pbusinj) + Pfinj - eliminates ALL error to machine precision. pypsa's PTDF on MEDIUM also - showed 702 MW error from the same root cause. PowerModels' native PTDF - (error < 1e-11) presumably handles phase shifters correctly. The B-9 pass - condition ("flow predictions match DCPF results within 1e-6") is violated - on any network with phase-shifting transformers unless correction terms are - applied. - proposed_protocol_impact: > - B-9/C-9 pass condition should specify: "If the network contains phase-shifting - transformers (nonzero SHIFT column), either (a) the PTDF computation must - incorporate Pbusinj/Pfinj correction terms, or (b) the tool must document the - limitation and the test should verify accuracy on a transformer-free subnetwork." - - - id: T07 - title: "ACTIVSg10k zero-impedance branches cause inconsistent treatment across dimensions" - category: infrastructure_friction - evidence_tools: [pypsa, pandapower, gridcal, powermodels] - evidence_count: 4 - affected_tests: [A-1, A-3, A-4, B-9, C-1, C-3, C-5, C-9] - probe_evidence: [probe-001] - description: > - ACTIVSg10k has 2,462 branches with zero thermal rating (s_nom=0 or RATE_A=0) - and 3 transformers with zero reactance. PyPSA interprets s_nom=0 as a - zero-capacity constraint, causing DCOPF infeasibility. Other tools treat - RATE_A=0 as unconstrained. The zero reactance causes singular B-matrix for - PyPSA's LPF (all flows NaN on MEDIUM). The evaluation applied data fixes - (x=0.0001, s_nom=9999) in scalability tests but NOT in expressiveness tests, - creating contradictory outcomes: A-3 MEDIUM fails but C-3 MEDIUM passes for - the same tool on the same network. This inconsistency affects cross-tool - comparison because tools handle the same data issues differently. - proposed_protocol_impact: > - Standardize ACTIVSg10k data preparation: (a) set zero-reactance transformers - to x=0.0001 pu, (b) set zero thermal rating branches to RATE_A=9999 MVA, - applied uniformly across all test dimensions. Document the preparation as a - protocol-level preprocessing step rather than leaving it to each evaluator. - - - id: T08 - title: "Stochastic scenario perturbations cause high solver failure rates across tools" - category: test_design_gap - evidence_tools: [pandapower, gridcal, powersimulations] - evidence_count: 3 - affected_tests: [B-4, C-6] - probe_evidence: [probe-006] - description: > - pandapower: 2.1% convergence on SMALL (probe-006 confirms 0.42%). GridCal: - 47% convergence on SMALL. PowerSimulations: 40% infeasibility on TINY (2 of - 5 scenarios). The combination of load scaling AND generator capacity - perturbations drives infeasibility, particularly on SMALL where the tighter - network constraints amplify perturbation effects. Probe-006 showed the base - case converges and uniform load scaling (0.7x-1.1x) works, but combined - perturbations break the solver. This may be a protocol issue: the - perturbation methodology should be calibrated to avoid producing a majority - of infeasible scenarios. - proposed_protocol_impact: > - Specify perturbation bounds that produce at most 20% infeasible scenarios - on the target network, or require OPF formulations with slack variables for - load shedding. The current uncalibrated perturbations measure solver - robustness rather than tool stochastic capability. - - - id: T09 - title: "Missing MATPOWER lossy DCOPF reference validation across all tools" - category: missing_verification - evidence_tools: [pypsa, gridcal, powermodels, matpower] - evidence_count: 4 - affected_tests: [A-10] - probe_evidence: [] - description: > - The A-10 pass condition explicitly requires "Validate against MATPOWER - reference lossy DC OPF solution on same case (tolerance: 1% on total LMP, - directional consistency on loss component signs)." No tool's evaluation - performs this cross-validation. pypsa performs LMP reconciliation but no - MATPOWER comparison. GridCal has no reference validation. PowerModels notes - mismatched formulations prevent comparison. MATPOWER itself uses lossless - DC OPF with post-hoc loss estimation, not loss-inclusive optimization. - The reference validation is unachievable as specified because MATPOWER's - rundcopf does not have a loss option producing comparable LMPs. - proposed_protocol_impact: > - Either (a) specify a concrete MATPOWER command producing the reference - (e.g., rundcopf with specific loss options), (b) change the validation to - internal consistency (lossless-vs-lossy comparison within the same tool), - or (c) remove the MATPOWER reference requirement and replace with physical - consistency checks (loss components have correct signs, total losses are - 0.5-3% of load). - - - id: T10 - title: "ACPF convergence on 10k-bus network is a genuine capability discriminator" - category: high_signal_test - evidence_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - evidence_count: 6 - affected_tests: [A-2, C-2] - probe_evidence: [probe-001, probe-024] - description: > - ACPF on ACTIVSg10k produces 3 distinct outcomes: (1) converges with good - diagnostics (pandapower, gridcal, matpower), (2) fails with no diagnostics - (powermodels, powersimulations -- probe-024 confirms silent Missing return), - (3) reports false convergence (pypsa -- probe-001 confirms 0 iterations, - 83% flat start). This test genuinely discriminates tools by NR solver - robustness, warm-start support, and diagnostic quality. The protocol should - strengthen convergence verification requirements. - proposed_protocol_impact: > - A-2 pass condition should require: (a) convergence residual reported and - below tolerance, (b) number of NR iterations reported, (c) voltage - magnitudes not at flat-start defaults on >95% of buses. This prevents - false convergence claims. - - - id: T11 - title: "Solver swap test conflates swap mechanism with solver performance" - category: scoring_inconsistency - evidence_tools: [pypsa, pandapower, powermodels, matpower] - evidence_count: 4 - affected_tests: [C-7] - probe_evidence: [] - description: > - The C-7 pass condition asks whether solver swap "requires reformulation - or just a parameter change." For pypsa, powermodels, and matpower, swap - is parameter-only. pandapower fails because it is locked to PYPOWER. - However, several tools receive qualified_pass because alternative solvers - were not installed or perform poorly -- this conflates solver availability - and performance with the swap mechanism itself. pypsa tested only HiGHS; - powermodels qualified because HiGHS QP failed; matpower tested only MIPS - and GLPK. - proposed_protocol_impact: > - Separate C-7 into two findings: (a) swap mechanism (parameter-only vs - reformulation required -- binary), (b) multi-solver comparison (timing - and objective consistency). Require at least 2 solvers installed in the - evaluation environment. - - - id: T12 - title: "Peak memory measurements missing or estimated across all tools" - category: missing_verification - evidence_tools: [pypsa, pandapower, gridcal, powermodels, powersimulations, matpower] - evidence_count: 6 - affected_tests: [C-1, C-2, C-3, C-4, C-5, C-6, C-7, C-8, C-9, C-10] - probe_evidence: [] - description: > - The protocol requires peak_memory_mb for all C-tests. In practice: - pandapower expressiveness tests report "not measured". PowerModels has - 6/10 C-tests with null memory. MATPOWER estimates memory from matrix - dimensions rather than process measurement. Octave lacks built-in - memory profiling. This makes cross-tool memory comparison unreliable. - proposed_protocol_impact: > - Specify a memory measurement methodology per language: Python (tracemalloc), - Julia (@allocated or /proc/self/status), Octave (/proc/self/status). - Require measured values, not estimates. - - - id: T13 - title: "N-M contingency sweep at x=5, m=4 on MEDIUM is infeasible for all tools" - category: test_design_gap - evidence_tools: [pypsa, powermodels, matpower] - evidence_count: 3 - affected_tests: [A-7, C-5] - probe_evidence: [probe-016, probe-029] - description: > - N-4 on a 10k-bus network produces combinatorial explosion that no tool can - solve within the time budget. pypsa completed only 9/270 N-1 cases (3.3%). - PowerModels projected timeout and was confirmed by probe-016. MATPOWER - completed screening but 97% of time was Octave containers.Map overhead - (probe-029). pandapower and gridcal used reduced scope. The test measures - mathematical impossibility at high orders, not tool capability. - proposed_protocol_impact: > - Cap N-M sweep at N-2 for MEDIUM scale. Report N-1 performance as the - primary metric. N-3 and N-4 are informational only. This preserves the - graph-scoping and pruning test while making the test feasible. diff --git a/sweep-data/v4-to-v5/per-tool/gridcal/findings.md b/sweep-data/v4-to-v5/per-tool/gridcal/findings.md deleted file mode 100644 index d67c45af..00000000 --- a/sweep-data/v4-to-v5/per-tool/gridcal/findings.md +++ /dev/null @@ -1,274 +0,0 @@ -# GridCal -- Sweep Findings (v4) - -## Summary - -The GridCal evaluation is thorough and generally well-executed. The evaluator correctly identifies the tool's core strengths (clean PF/OPF APIs, NetworkX integration, pure Python inspectability) and weaknesses (no custom constraint API, non-functional SCOPF, TapPhaseControl bug cascade). The primary concerns identified in this sweep are: (1) a single bug (TapPhaseControl) cascading into five separate test failures, which inflates the apparent breadth of gaps; (2) missing MATPOWER cross-validation for the lossy DC OPF (A-10) as required by the protocol; (3) a significant PTDF flow prediction mismatch on MEDIUM that is scored as qualified_pass without adequate investigation; and (4) uniform generator costs on case39 reducing the discriminative value of LMP-dependent tests. Three probes are recommended: PTDF flow mismatch investigation, lossy DC OPF reference validation, and B-4 SMALL convergence failure analysis. - -## Finding Details - -### gridcal-F01: Uniform LMPs on case39 mask congestion-dependent test differentiation - -**Category:** low_signal | **Severity:** low -**Tests:** A-3, B-8, C-3, C-7 - -All 10 generators in IEEE 39-bus have identical cost curves (0.3 $/MWh), so LMPs are always uniform when no branches bind. This renders multiple tests unable to verify economically meaningful signals. The B-8 TINY result explicitly notes: "LMPs are identical across all three configurations. This is mathematically correct: in a DC OPF LP formulation, the slack bus determines the voltage angle reference but does not affect the economic dispatch or LMPs." While this is true, it means the test provides zero discriminative value on this network. - -The evaluator mitigated this partially by running B-8 on SMALL (ACTIVSg 2000), where the heterogeneous cost structure produced a meaningful LMP spread of 11.128 to 32.923 $/MWh with the two-slack configuration. This demonstrates good test design awareness. - -On MEDIUM, the DC OPF produces uniform LMPs at $20.064/MWh with zero binding branches (max loading 84.7%). This affects A-3 MEDIUM, C-3, and C-7, where congestion signal is absent. - -**Cross-tool relevance:** confirmed -- all tools use the same reference networks. -**Proposed action:** adjust_scoring -- consider whether MEDIUM network parameterization should be modified to produce binding constraints, or whether the no-binding-constraint result is itself a finding about the test network. - ---- - -### gridcal-F02: TapPhaseControl bug cascades single defect into five test failures - -**Category:** misleading_result | **Severity:** high -**Tests:** A-5, A-6, A-8, C-4, C-6 - -A single bug (`ValueError: 0 is not a valid TapPhaseControl`) in GridCal's time-series OPF compiler causes failures across five tests. The bug triggers on any MATPOWER network with transformers, which includes all standard IEEE test cases. The synthesis correctly identifies this cascade: "This single defect is responsible for 3 test failures and 1 downgrade." - -The evaluation handles this fairly: it notes that A-5 has additional independent issues (UC constraints not enforced per GitHub issue #397) and A-8 lacks native stochastic formulation regardless of the bug. However, the aggregate failure count (5 fail out of 11 expressiveness tests = 45% failure rate) overstates the breadth of independent capability gaps. A fairer characterization would distinguish between: (a) features that are architecturally absent (SCOPF, distributed slack OPF, native stochastic), (b) features blocked by a single fixable bug (time-series OPF on transformer networks), and (c) features that are present but not enforced (UC constraints). - -The C-4 and C-6 scalability failures are purely inherited -- they add no new information and could be marked as "blocked by expressiveness" without counting against the scalability dimension. - -**Cross-tool relevance:** likely -- other tools may have bugs that cascade similarly, and the protocol should have guidance on how to count cascaded failures. -**Proposed action:** none (the evaluation already notes the cascade clearly in synthesis; this is a protocol-level design consideration). - ---- - -### gridcal-F03: Contingency sweep correctness not verified against reference - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-7 - -The A-7 contingency sweep reports load loss values for 129 cases (TINY, x=3, m=3) and 793 cases (MEDIUM, x=5, m=4), all converging. However, post-contingency flow correctness is not validated against any reference (e.g., a separate DCPF solve with the branch removed and a fresh model). The synthesis flags this: "A-7: Verify manual contingency loop workaround produces correct post-contingency flows" -- but this spot-check was not performed. - -The pruning ratio of 0.0 on both networks is also notable. Every N-1 contingency produced measurable load loss (> 1e-3 MW threshold on MEDIUM). For a well-meshed network like ACTIVSg 10k, this is somewhat surprising -- many branch removals in a well-connected region would be expected to redistribute flow without measurable load loss. This could indicate that "load loss" is being measured incorrectly (e.g., using total generation mismatch rather than actual unserved load), or that the threshold is too low. - -**Cross-tool relevance:** likely -- contingency sweep verification methodology is shared across tools. -**Proposed action:** add_verification -- validate a sample of post-contingency flows against fresh-solve reference; verify load loss measurement methodology. - ---- - -### gridcal-F04: PTDF flow predictions diverge significantly from DCPF on MEDIUM network - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** C-9, B-9 - -On TINY (39-bus), the LinearAnalysis PTDF-predicted flows match DCPF exactly (max diff 0.0). On MEDIUM (10k-bus), the discrepancy is dramatic: 743 MW max absolute difference for direct flows, and 15,139 MW for PTDF @ Sbus. The result attributes this to "differences in island handling, slack bus treatment, or network topology processing between the two solvers" and scores it as qualified_pass. - -This is a significant finding that warrants investigation. A 743 MW flow error on a network with max flows of ~2,000 MW means the error is ~37% of the maximum flow. The scale-dependent nature (exact on TINY, large on MEDIUM) suggests either: (a) the ACTIVSg 10k has electrical islands that are handled differently by DCPF vs LinearAnalysis, (b) there is a bug in the PTDF computation for multi-island networks, or (c) the slack bus convention difference compounds with network size. - -The evaluation's conclusion that the PTDF is "usable for relative sensitivity analysis" despite this mismatch is not adequately supported. - -**Cross-tool relevance:** none (tool-specific behavior). -**Proposed action:** add_verification -- investigate island count in ACTIVSg 10k, compare PTDF on a single-island subnetwork, verify slack bus handling. - ---- - -### gridcal-F05: No binding branches on MEDIUM DC OPF reduces congestion test signal - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-3, A-10, C-3 - -The ACTIVSg 10k DC OPF produces zero binding branches with max loading 84.7% and perfectly uniform LMPs. This means the MEDIUM grade network does not stress the congestion dimension of DC OPF testing. For A-3 (DC OPF), the pass/fail is clear (it converges and produces dispatch), but the lack of binding constraints means shadow price quality, congestion rent, and LMP decomposition cannot be tested in a meaningful way at the grade-assessment scale. - -This is a network sufficiency issue that affects all tools equally: if the ACTIVSg 10k DC OPF has no binding branches, no tool can demonstrate congestion-aware dispatch on this network. - -**Cross-tool relevance:** confirmed -- same network used for all tools. -**Proposed action:** redesign_test -- consider reducing some branch ratings in ACTIVSg 10k to create binding constraints, or select a different grade network with natural congestion. - ---- - -### gridcal-F06: ACPF convergence claimed but iteration count not reported - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-2, C-2 - -The convergence error values (3.32e-11 on TINY, 2.73e-07 on MEDIUM) provide strong evidence of convergence. However, the iteration count is not reported. The A-2 TINY result explicitly states: "Solver iterations: not directly exposed in results." The protocol lists "iterations" as a recorded metric for C-2. While the convergence error is sufficient to establish that the solver converged, the missing iteration count is a protocol compliance gap. It also means cross-tool comparison of NR efficiency (iterations to convergence) is not possible for GridCal. - -**Cross-tool relevance:** likely -- other tools may also omit iteration counts if not easily accessible. -**Proposed action:** none (the convergence error is a stronger verification than iteration count alone). - ---- - -### gridcal-F08: Lossy DC OPF not validated against MATPOWER reference - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-10 - -The protocol explicitly requires: "Validate against MATPOWER reference lossy DC OPF solution on same case (tolerance: 1% on total LMP, directional consistency on loss component signs)." No such validation appears in either the TINY or SMALL A-10 result files. - -On TINY, the loss-inclusive LMP spread is only 0.000031 $/MWh (from 0.3 to 0.300031), which is at the edge of numerical noise. This could mean: (a) the loss approximation is correct but all generators have identical costs so loss effects are minimal, (b) the loss approximation is barely active, or (c) the implementation is incorrect but the uniform costs mask the error. Without MATPOWER cross-validation, these cannot be distinguished. - -The SMALL result shows a more convincing spread (17.580 to 17.751 $/MWh, 0.172 range), with total generation increasing by 80.1 MW to cover losses. This is more physically plausible but still lacks reference validation. - -**Cross-tool relevance:** likely -- MATPOWER cross-validation is required for all tools. -**Proposed action:** add_verification -- run MATPOWER rundcopf with loss option on case39 and ACTIVSg 2000 to establish reference LMPs. - ---- - -### gridcal-F09: E-6 file content appears to duplicate E-5 (issue responsiveness instead of CI/test coverage) - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** E-5, E-6 - -The E-6 result file (E-6_ci_test_coverage.md) is titled "Issue Responsiveness" and contains content about SanPen's issue response patterns, batch-close behavior, and notable open issues. This is identical in topic to E-5 (which should be issue tracker health per the eval-config). Per the protocol, E-6 should assess "CI configuration and test suite" -- CI existence, test suite existence, coverage estimates, and whether CI passes on the current release. This CI/test assessment appears to be missing from the results entirely. - -The synthesis table maps E-6 to "CI/Test Coverage" with a "QUAL PASS" status, but the underlying result file does not contain CI/test coverage content. - -**Cross-tool relevance:** none (result file content issue). -**Proposed action:** none (the finding is documented; the missing CI assessment is a gap but is not critical to the overall evaluation quality). - ---- - -### gridcal-F10: B-4 SMALL stochastic wrapping has 53% solve failure rate - -**Category:** test_design_gap | **Severity:** medium -**Tests:** B-4 - -On TINY, B-4 achieves 240/240 converged solves (100%). On SMALL (ACTIVSg 2000), only 113/240 converge (47%). The result attributes failures to "infeasibility at extreme perturbation levels" and is scored as qualified_pass. Several questions arise: - -1. Are the perturbation magnitudes (sigma=0.05 for resource types, sigma=0.05 for load) appropriate for the SMALL network, or do they need to be scaled with network size? -2. Is the 47% convergence rate a GridCal robustness issue (poor handling of near-infeasible cases) or a test design issue (perturbations too aggressive)? -3. Does the evaluate establish whether other tools experience similar failure rates with the same perturbation methodology? - -The wall-clock time of 1444.5 seconds (~24 minutes) for only 113 successful solves (12.8s per solve) also raises questions about whether the OPF formulation on SMALL includes slack variables for load shedding that prevent infeasibility. If the OPF lacks slack variables, aggressive perturbations will naturally produce infeasible problems. - -**Cross-tool relevance:** likely -- perturbation methodology is shared. -**Proposed action:** add_verification -- compare convergence rates across tools with identical perturbation seeds; investigate whether OPF formulations include slack variables. - ---- - -### gridcal-F13: Install-to-first-solve wall-clock is an estimate, not a measurement - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** D-1 - -The D-1 result reports "approximately 2-3 minutes" as the install-to-first-solve time, broken down as "install ~60s, writing 4-line script ~30s, debugging SolverType naming ~60s." These are estimates, not actual measurements. The protocol requires wall-clock time. The narrative provides good qualitative detail about friction points (rename confusion, SolverType.Linear naming) but the quantitative claim is unverified. - -**Cross-tool relevance:** likely -- D-1 timing methodology may be imprecise across all tools. -**Proposed action:** none (the qualitative findings are more valuable than precise timing for this test). - ---- - -### gridcal-F14: Maturity audit results lack verifiable citations - -**Category:** test_design_gap | **Severity:** low -**Tests:** E-4, E-5, E-6, E-7 - -The maturity results cite specific numbers (2,434 commits, 70.3% from SanPen, 30 contributors, ~40 PyPI releases) without linking to data sources. E-7 claims a deployment at NGN (Germany) with "Specific use case details not public." These audit findings are plausible and consistent with each other but are not independently verifiable from the result files alone. - -This is inherent to the audit-evaluator archetype -- the protocol does not require automated data collection for maturity metrics. However, including git commands used, date range boundaries, and links to specific evidence (PyPI release history URLs, GitHub contributor pages) would strengthen the findings. - -**Cross-tool relevance:** confirmed -- all tools undergo the same audit methodology. -**Proposed action:** none (this is a protocol-level improvement opportunity, not a finding-specific issue). - ---- - -## Extraordinary Claims - -### C-9: PTDF matrix is usable despite 743 MW flow prediction mismatch on MEDIUM - -**Concern:** A 743 MW maximum absolute difference (37% of max flow) between PTDF-predicted and DCPF-solved flows on the 10k-bus network is a substantial discrepancy. The result claims the PTDF is "usable for relative sensitivity analysis" but this claim is not verified. The mismatch does not appear on TINY (exact match), suggesting a scale-dependent issue that could be an island handling bug, a slack bus convention error, or a numerical accumulation problem. - -**Evidence quality:** moderate -- the TINY-vs-MEDIUM comparison is well-documented, but the root cause is not investigated. - -A probe should: (1) count electrical islands in ACTIVSg 10k as seen by both DCPF and LinearAnalysis, (2) test PTDF on the largest single-island subnetwork, (3) compare unit-injection PTDF rows against DCPF for a sample of buses. - -### A-10: Loss approximation produces physically meaningful LMPs - -**Concern:** The TINY LMP spread from loss approximation is only 0.000031 $/MWh, essentially at numerical noise level. No MATPOWER cross-validation was performed as required by the protocol. The SMALL spread (0.172 $/MWh) is more convincing but also unvalidated. Without reference comparison, we cannot confirm whether the loss approximation is physically correct or merely producing small perturbations that happen to be non-zero. - -**Evidence quality:** weak -- the required MATPOWER validation is entirely absent. - -A probe should: (1) run MATPOWER rundcopf with loss option on case39 and ACTIVSg 2000, (2) compare total LMPs within 1% tolerance, (3) verify directional consistency of loss components. - -### D-1: Install to first solve takes 2-3 minutes - -**Concern:** Estimate rather than measurement. The detailed breakdown (install ~60s, script writing ~30s, debugging ~60s) makes the estimate plausible, and the qualitative friction findings are well-documented. - -**Evidence quality:** moderate -- plausible estimate with supporting detail but not measured. - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | -- | -- | -| G-2 | pass | -- | -- | -| G-3 | pass | -- | -- | -| A-1 (TINY) | pass | -- | SolverType.Linear naming non-obvious | -| A-1 (MEDIUM) | pass | -- | Solve time 0.36s, file load 7.3s dominates | -| A-2 (TINY) | pass | -- | Iteration count not reported | -| A-2 (MEDIUM) | pass | -- | Convergence error 2.73e-07, good | -| A-3 (TINY) | pass | -- | Uniform LMPs due to identical costs | -| A-3 (MEDIUM) | pass | -- | Zero binding branches, uniform LMPs | -| A-4 (TINY) | pass | -- | 2 voltage violations identified | -| A-4 (MEDIUM) | pass | -- | 143 voltage + 22 thermal violations, max 1587% loading | -| A-5 (TINY) | fail | blocking | TapPhaseControl crash + UC constraints not enforced | -| A-5 (SMALL) | fail | blocking | Same as TINY, not re-tested | -| A-6 (TINY) | fail | blocking | No UC/ED separation API; depends on A-5 | -| A-6 (SMALL) | fail | blocking | Same as TINY, not re-tested | -| A-7 (TINY) | qualified_pass | stable | Manual loop with NetworkX; no correctness validation | -| A-7 (MEDIUM) | qualified_pass | stable | 793 cases, 297.6s, no pruning | -| A-8 (TINY) | fail | blocking | No native stochastic + TS-OPF crash | -| A-8 (SMALL) | fail | blocking | Same as TINY, not re-tested | -| A-9 (TINY) | fail | -- | consider_contingencies flag non-functional | -| A-9 (SMALL) | fail | -- | Same as TINY, not re-tested | -| A-10 (TINY) | qualified_pass | -- | Losses work; no LMP decomposition; no MATPOWER validation | -| A-10 (SMALL) | qualified_pass | -- | Wider LMP spread; still no reference validation | -| A-11 (TINY) | fail | -- | Distributed slack in PF only, not OPF | -| A-11 (SMALL) | fail | -- | Same as TINY, not re-tested | -| B-1 (TINY) | fail | blocking | No custom constraint API; PuLP model not exposed | -| B-1 (MEDIUM) | fail | blocking | Same architectural limitation | -| B-2 (TINY) | pass | -- | Native NetworkX MultiDiGraph | -| B-2 (MEDIUM) | pass | -- | 87ms graph build on 10k buses | -| B-3 (TINY) | pass | -- | 4.6ms per N-1 case, model integrity verified | -| B-3 (MEDIUM) | pass | -- | 262ms per case, 50 contingencies | -| B-4 (TINY) | qualified_pass | fragile | 240/240 converged; TS-OPF crash forces snapshot loop | -| B-4 (SMALL) | qualified_pass | fragile | Only 113/240 converged; high failure rate | -| B-5 (TINY) | pass | -- | 4 LOC for DataFrame + CSV export | -| B-5 (MEDIUM) | pass | -- | Scales linearly | -| B-6 | pass | -- | Clean 3-tier architecture documented | -| B-7 (TINY) | pass | -- | No workaround needed for DC OPF -> ACPF | -| B-7 (MEDIUM) | pass | -- | Same clean pipeline at scale | -| B-8 (TINY) | pass | -- | Slack reconfigurable; LMPs uniform (uninformative) | -| B-8 (SMALL) | pass | -- | Two-slack produces meaningful LMP variation | -| B-9 (TINY) | pass | -- | Exact PTDF/DCPF flow match | -| B-9 (MEDIUM) | qualified_pass | -- | 743 MW flow mismatch; needs investigation | -| C-1 | pass | -- | 1.84s, 82.6 MB | -| C-2 | pass | -- | 12.7s, 91.1 MB, flat start converged | -| C-3 | pass | -- | 15.2s, 127 MB; uniform LMPs | -| C-4 | fail | blocking | Blocked by A-5 | -| C-5 | pass | stable | 575s for 385 cases; manual loop | -| C-6 | fail | blocking | Blocked by A-8 | -| C-7 | pass | -- | HiGHS vs SCIP, one-line swap | -| C-8 | fail | -- | Blocked by A-9 | -| C-9 | qualified_pass | -- | 49.9s, 7.6 GB; flow mismatch | -| C-10 | fail | -- | Blocked by A-11 | -| D-1 | qualified_pass | -- | Rename friction; timing is estimate | -| D-2 | informational | -- | 2/11 from docs alone | -| D-3 | qualified_pass | -- | Most examples run; TS-OPF blocked by bug | -| D-4 | fail | -- | All 3 errors silently accepted | -| D-5 | informational | -- | 30-85 LOC for passing tests | -| E-1 | pass | -- | ~40 releases in 24 months | -| E-2 | pass | -- | ~2,434 commits/12 months | -| E-3 | qualified_pass | -- | 83% from one organization | -| E-4 | fail | -- | Bus factor = 1 | -| E-5 | qualified_pass | -- | Open-core model; no public financials | -| E-6 | qualified_pass | -- | Content appears to be issue responsiveness, not CI/test coverage | -| E-7 | qualified_pass | -- | One distribution utility deployment | -| F-1 | qualified_pass | -- | MPL-2.0; recent license change | -| F-2 | pass | -- | Content is license audit (see F-15 finding) | -| F-3 | pass | -- | 100% pure Python core | -| F-4 | pass | -- | Full execution path traceable | -| F-5 | pass | -- | HiGHS bundled; SCIP available | -| F-6 | pass | -- | Air-gap installable | -| F-7 | fail | -- | No signed tags, no SLSA/SBOM | -| F-8 | qualified_pass | -- | Examples not version-pinned | -| F-9 | qualified_pass | -- | Large dep surface (83 pkgs); opencv unusual | -| P2-1 | pass | -- | RAW v29-35 (v31 absent) | -| P2-2 | fail | -- | No PWL costs; quadratic ignored in DC OPF | -| P2-3 | fail | -- | Pipeline non-functional | diff --git a/sweep-data/v4-to-v5/per-tool/gridcal/findings.yaml b/sweep-data/v4-to-v5/per-tool/gridcal/findings.yaml deleted file mode 100644 index 34610669..00000000 --- a/sweep-data/v4-to-v5/per-tool/gridcal/findings.yaml +++ /dev/null @@ -1,391 +0,0 @@ -tool: gridcal -source_version: "v4" -timestamp: "2026-03-07T18:30:00Z" -evaluation_summary: - total_tests: 56 - pass: 29 - fail: 16 - qualified_pass: 9 - informational: 2 - -findings: - - id: gridcal-F01 - category: low_signal - severity: low - test_ids: [A-3, B-8, C-3, C-7] - title: "Uniform LMPs on case39 mask congestion-dependent test differentiation" - description: > - All 10 generators in IEEE 39-bus have identical cost curves (0.3 $/MWh linear), - producing uniform LMPs regardless of congestion. This renders tests that depend on - LMP variation (A-3 shadow prices, B-8 slack bus LMP comparison) unable to verify - meaningful economic signals on TINY. The B-8 TINY result shows identical LMPs across - all three slack configurations, which is correct but provides zero discriminative value. - evidence: - - file: "results/expressiveness/A-3_dcopf_TINY.md" - excerpt: "All 10 generators have identical cost (0.3 $/MWh linear + 0.01 quadratic + 0.2 constant), so LMPs are uniform at 0.3 when no lines are binding." - - file: "results/extensibility/B-8_reference_bus_config_TINY.md" - excerpt: "LMPs are identical across all three configurations. This is mathematically correct: in a DC OPF LP formulation, the slack bus determines the voltage angle reference but does not affect the economic dispatch or LMPs." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: gridcal-F02 - category: misleading_result - severity: high - test_ids: [A-5, A-6, A-8, C-4, C-6] - title: "TapPhaseControl bug cascades single defect into five test failures" - description: > - A single bug in GridCal's time-series OPF compiler (ValueError: 0 is not a valid - TapPhaseControl) causes failures across A-5 (SCUC), A-6 (SCED), A-8 (stochastic), - C-4, and C-6. The bug occurs on any MATPOWER network with transformers (including - all standard IEEE test cases). This amplifies one defect into five separate test - failures, overstating the breadth of capability gaps. However, the evaluation correctly - notes that even without this bug, A-5 has additional issues (UC constraints not - enforced per issue #397) and A-8 lacks native stochastic formulation. - evidence: - - file: "results/expressiveness/A-5_scuc_TINY.md" - excerpt: "ValueError: 0 is not a valid TapPhaseControl" - - file: "results/expressiveness/A-8_stochastic_timeseries_TINY.md" - excerpt: "All 5 scenarios failed: Scenario 0: ValueError: 0 is not a valid TapPhaseControl" - - file: "results/synthesis.md" - excerpt: "This single defect is responsible for 3 test failures and 1 downgrade." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F03 - category: missing_verification - severity: medium - test_ids: [A-7, A-7] - title: "Contingency sweep correctness not verified against reference" - description: > - The A-7 N-M contingency sweep produces load loss values for 129 cases (TINY) and - 793 cases (MEDIUM), but post-contingency flows are not validated against any reference - solution. The synthesis flags this as a spot-check item but it was not resolved. The - pruning ratio of 0.0 on both networks (no pruning occurred) is also notable -- every - N-1 contingency produced measurable load loss, which may indicate the load loss - threshold is set too low or the metric is not discriminative. - evidence: - - file: "results/expressiveness/A-7_contingency_sweep_TINY.md" - excerpt: "Pruning ratio: 0.0 (all order-1 contingencies produced measurable load loss)" - - file: "results/expressiveness/A-7_contingency_sweep.md" - excerpt: "Pruning ratio: 0.0 (all N-1 contingencies had measurable load loss > 1e-3 MW)" - - file: "results/synthesis.md" - excerpt: "A-7: Verify manual contingency loop workaround produces correct post-contingency flows" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: gridcal-F04 - category: extraordinary_claim - severity: medium - test_ids: [C-9, B-9] - title: "PTDF flow predictions diverge significantly from DCPF on MEDIUM network" - description: > - On TINY (39-bus), PTDF-predicted flows match DCPF exactly (max diff 0.0). On MEDIUM - (10k-bus), the LinearAnalysis direct flows differ from DCPF by up to 743 MW, and - PTDF @ Sbus differs by up to 15,139 MW. The result is scored as qualified_pass with - the explanation that slack bus and injection conventions differ between the two solvers. - This is a significant discrepancy that calls into question whether the PTDF matrix is - correct for this network or whether there is a bug in island handling at scale. - evidence: - - file: "results/extensibility/B-9_ptdf_extraction_MEDIUM.md" - excerpt: "LA direct flows vs DCPF | 743.46 | 2.68 | No" - - file: "results/scalability/C-9_ptdf_scale.md" - excerpt: "LA direct flows vs DCPF | 743.46 MW" - - file: "results/extensibility/B-9_ptdf_extraction_TINY.md" - excerpt: "Max absolute difference: 0.0" - cross_tool_relevance: none - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: gridcal-F05 - category: network_insufficiency - severity: medium - test_ids: [A-3, A-10, C-3] - title: "No binding branches on MEDIUM DC OPF reduces congestion test signal" - description: > - The ACTIVSg 10k DC OPF produces zero binding branches (max loading 84.7%) with - uniform LMPs at $20.064/MWh. This means the MEDIUM network does not exercise - congestion-related features for DC OPF: LMP spatial variation, congestion rent, - and binding constraint identification all return trivial results. The lossy DC OPF - (A-10) similarly shows minimal LMP spread on TINY (0.000031 $/MWh) due to uniform - generator costs and limited congestion. - evidence: - - file: "results/expressiveness/A-3_dcopf.md" - excerpt: "LMP range | 20.064 (uniform) ... Binding branches | 0" - - file: "results/scalability/C-3_dcopf_scale.md" - excerpt: "Max loading | 84.72% ... Binding branches | 0" - - file: "results/expressiveness/A-10_lossy_dcopf_TINY.md" - excerpt: "LMP range ($/MWh) | 0.3 -- 0.300031" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: gridcal-F06 - category: missing_verification - severity: medium - test_ids: [A-2, C-2] - title: "ACPF convergence claimed but iteration count not reported" - description: > - Both A-2 (TINY) and C-2 (MEDIUM) report ACPF convergence with Newton-Raphson and - provide convergence error values (3.32e-11 and 2.73e-07 respectively), which is good. - However, neither reports the number of solver iterations. The protocol requires - "iterations" as a recorded metric for C-2. The convergence error is strong evidence - that the solve converged, but the missing iteration count is a protocol gap. - evidence: - - file: "results/expressiveness/A-2_acpf_TINY.md" - excerpt: "Solver iterations: not directly exposed in results" - - file: "results/scalability/C-2_acpf_scale.md" - excerpt: "Convergence error | 2.73e-07" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F07 - category: extraordinary_claim - severity: low - test_ids: [A-9] - title: "SCOPF consider_contingencies flag confirmed non-functional via identical dispatch" - description: > - The A-9 result provides strong evidence that GridCal's consider_contingencies OPF - option is non-functional: it produces byte-identical dispatch to the baseline DC OPF. - The evaluation correctly identifies this as a capability gap (issue #364). This is - well-verified -- not an extraordinary claim about what the tool can do, but rather - a well-documented claim about what it cannot do. The identical dispatch comparison - is the right verification method. - evidence: - - file: "results/expressiveness/A-9_scopf_TINY.md" - excerpt: "The OPF converged but produced identical dispatch to the baseline DC OPF without contingencies" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F08 - category: missing_verification - severity: medium - test_ids: [A-10, A-10] - title: "Lossy DC OPF not validated against MATPOWER reference" - description: > - The protocol requires A-10 results to be validated against MATPOWER's rundcopf with - loss option enabled (tolerance 1% on total LMP, directional consistency on loss - component signs). No such validation appears in either the TINY or SMALL result files. - The LMP spread on TINY is extremely small (0.000031 $/MWh range), which could indicate - that loss approximation is barely active or that the uniform cost structure masks - the effect. Without MATPOWER cross-validation, the physical accuracy of the loss - approximation cannot be confirmed. - evidence: - - file: "results/expressiveness/A-10_lossy_dcopf_TINY.md" - excerpt: "LMP range ($/MWh) | 0.3 -- 0.300031" - - file: "results/expressiveness/A-10_lossy_dcopf.md" - excerpt: "LMP range ($/MWh) | 17.580 -- 17.751" - cross_tool_relevance: likely - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: gridcal-F09 - category: scoring_inconsistency - severity: low - test_ids: [E-5, E-6] - title: "E-5 (Funding Model) and E-6 (Issue Responsiveness) labeled identically" - description: > - Both E-5 and E-6 are titled "Issue Responsiveness" in the file headers and have - similar content structures. E-5 should be "Funding Model" per the eval-config.yaml, - and E-6 should be "CI/CD and Test Coverage" per the protocol. However, reading the - actual content, E-5 does discuss funding (eRoots open-core model) while E-6 discusses - issue responsiveness. The test IDs in the YAML frontmatter are correct but the - synthesis maps E-4 to "Bus Factor" and E-5 to "Funding Model" which matches the - content. The E-6 file appears to be a duplicate of E-5 content (issue responsiveness) - rather than CI/test coverage. No CI/test coverage assessment exists in the results. - evidence: - - file: "results/maturity/E-5_issue_tracker_health.md" - excerpt: "The maintainer (SanPen) is responsive to most issues" - - file: "results/maturity/E-6_ci_test_coverage.md" - excerpt: "The maintainer (SanPen) is responsive to most issues" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F10 - category: test_design_gap - severity: medium - test_ids: [B-4, B-4] - title: "B-4 SMALL stochastic wrapping has 53% solve failure rate" - description: > - On the SMALL network, B-4 reports only 113 of 240 expected solves succeeded (47% - success rate). This is scored as qualified_pass. The high failure rate is attributed - to infeasibility at extreme perturbation levels, but this raises questions about - whether the scenario generation methodology is appropriate or whether the tool has - a robustness problem with the OPF formulation on realistic networks. The TINY result - (240/240 converged) does not predict this behavior. - evidence: - - file: "results/extensibility/B-4_stochastic_wrapping.md" - excerpt: "Successful solves | 113 ... All scenarios fully converged | No" - - file: "results/extensibility/B-4_stochastic_wrapping_TINY.md" - excerpt: "Total solves | 240 (20 x 12) ... All converged | Yes" - cross_tool_relevance: likely - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: gridcal-F11 - category: missing_verification - severity: low - test_ids: [C-1, C-2, C-3, C-5, C-7, C-9] - title: "Peak memory measurements present for scalability but absent elsewhere" - description: > - The scalability tests (C-1 through C-9) consistently report peak memory, which is - good. However, all expressiveness and extensibility tests report "Peak memory: not - measured." The protocol requires peak memory for all scalability-relevant tests. The - scalability results themselves are well-measured, but the expressiveness grade-tier - results on MEDIUM lack memory data, making it harder to correlate expressiveness - solve times with resource consumption. - evidence: - - file: "results/scalability/C-1_dcpf_scale.md" - excerpt: "Peak memory (solve) | 82.55 MB" - - file: "results/expressiveness/A-1_dcpf.md" - excerpt: "Peak memory: not measured" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F12 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate tests likely produce identical pass across all Python tools" - description: > - The gate tests verify MATPOWER .m ingestion with bus/branch/gen count validation. - All Python-based tools (pypsa, pandapower, gridcal) that can read MATPOWER format - will pass these identically. The gate test has value as a precondition but provides - no discriminative signal between tools that support the format. - evidence: - - file: "results/gate/G-1_tiny_import.md" - excerpt: "Actual counts: 39 buses / 46 branches / 10 generators ... Errors/warnings: None" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F13 - category: extraordinary_claim - severity: medium - test_ids: [D-1] - title: "Install-to-first-solve wall-clock is an estimate, not a measurement" - description: > - D-1 reports "approximately 2-3 minutes" from install to first solve, but this is - described as an estimate rather than an actual timed measurement. The protocol - requires wall-clock time. The narrative is detailed about the friction points - (rename confusion, SolverType naming) but the overall time is a rough estimate. - evidence: - - file: "results/accessibility/D-1_install_to_first_solve.md" - excerpt: "From uv add veragridengine to successful DCPF solve: approximately 2-3 minutes (install ~60s, writing 4-line script ~30s, debugging SolverType naming ~60s)." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F14 - category: test_design_gap - severity: low - test_ids: [E-4, E-5, E-6, E-7] - title: "Maturity audit results rely on desk research without verifiable citations" - description: > - The maturity dimension results (E-1 through E-7) reference commit counts, contributor - percentages, and deployment claims without linking to specific data sources (e.g., - GitHub API queries, PyPI release pages, or specific commit hashes). The E-2 result - cites "~2,434 commits" but does not specify the date range endpoints or the git - command used. The E-7 operational adoption claim ("NGN, Germany") has no linked - source. This is common for audit-type evaluations but reduces verifiability. - evidence: - - file: "results/maturity/E-2_commit_activity.md" - excerpt: "Total commits (12 months): ~2,434" - - file: "results/maturity/E-7_operational_adoption.md" - excerpt: "NGN (Netzgesellschaft Gutersloh, Germany): Distribution utility confirmed as a user. Specific use case details not public." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F15 - category: scoring_inconsistency - severity: low - test_ids: [F-2, F-3] - title: "F-2 and F-3 content appears swapped relative to eval-config test definitions" - description: > - The eval-config defines F-2 as "Dependency tree enumeration" and F-3 as "Dependency - license audit." However, the F-2 result file (F-2_dependency_tree.md) is titled - "Dependency License Audit" and contains the license audit content. This appears to - be a naming/content alignment issue in the result files, though the findings - themselves are present and complete. - evidence: - - file: "results/supply_chain/F-2_dependency_tree.md" - excerpt: "# F-2: Dependency License Audit" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: null - - - id: gridcal-F16 - category: missing_verification - severity: low - test_ids: [A-4, B-7] - title: "A-4 TINY reports zero thermal violations but MEDIUM reports 22 -- no cross-validation" - description: > - A-4 on TINY shows 2 voltage violations and 0 thermal violations, while A-4 on MEDIUM - shows 143 voltage and 22 thermal violations with max loading of 1587%. The dramatic - difference is attributed to "a realistic large-scale network" having more feasibility - gaps, which is plausible. However, the 1587% max branch loading on MEDIUM is extreme - and not investigated further -- it could indicate a data issue with the ACTIVSg10k - case file's thermal ratings rather than genuine DC-to-AC infeasibility. - evidence: - - file: "results/expressiveness/A-4_ac_feasibility.md" - excerpt: "Max loading | 1586.9%" - - file: "results/expressiveness/A-4_ac_feasibility_TINY.md" - excerpt: "Thermal violations (>100% loading) | 0 branches" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - -extraordinary_claims: - - test_id: C-9 - claim: "PTDF matrix is usable despite 743 MW flow prediction mismatch on MEDIUM" - concern: > - A 743 MW maximum absolute difference between PTDF-predicted flows and DCPF-solved - flows suggests either a bug in multi-island handling or fundamentally different - assumptions between the two solve paths. The claim that the PTDF is "usable for - relative sensitivity analysis" despite this mismatch needs verification. - evidence_quality: moderate - probe_recommended: true - probe_type: convergence_check - - - test_id: A-10 - claim: "Loss approximation produces physically meaningful LMPs" - concern: > - The TINY loss-inclusive LMP spread is only 0.000031 $/MWh, which is at the edge of - numerical noise. No MATPOWER cross-validation was performed as required by the - protocol. The SMALL result shows a wider spread (0.172 $/MWh) but still lacks - reference validation. - evidence_quality: weak - probe_recommended: true - probe_type: claim_verification - - - test_id: D-1 - claim: "Install to first solve takes 2-3 minutes" - concern: > - This is an estimate, not a measurement. The protocol requires wall-clock timing. - The estimate seems plausible given the description but is not verified. - evidence_quality: moderate - probe_recommended: false - probe_type: timing_verification diff --git a/sweep-data/v4-to-v5/per-tool/matpower/findings.md b/sweep-data/v4-to-v5/per-tool/matpower/findings.md deleted file mode 100644 index 968f86a4..00000000 --- a/sweep-data/v4-to-v5/per-tool/matpower/findings.md +++ /dev/null @@ -1,357 +0,0 @@ -# MATPOWER -- Sweep Findings (v4) - -## Summary - -The MATPOWER v4 evaluation is thorough and well-documented, with 57 tests covering all protocol dimensions. The evaluation quality is high for core PF/OPF tests, with executed code and captured output. Three categories of concern emerged: (1) three scalability tests (C-4, C-6, C-8) report solver capacity failures but actually failed at MOST's loadmd() data ingestion stage, making the failure attribution speculative; (2) the C-10 distributed slack timing is anomalous (66 min for a problem simpler than the 10s single-slack equivalent); (3) several TINY-tier tests lose discriminative value because case39 produces uniform LMPs with no congestion. Two probes are recommended for timing verification and one for claim verification. - -## Finding Details - -### matpower-F01: C-10 distributed slack scale timing is anomalous (66 min vs expected ~30s) - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** C-10 - -The C-10 result reports that the distributed-slack DC OPF on MEDIUM via manual opt_model construction took 3,969 seconds (~66 minutes), with the MIPS solve alone consuming 3,878 seconds. This is deeply inconsistent with other evidence in the evaluation: - -- C-3 solves the single-slack DC OPF on the same network in 9.7s via MIPS. -- The manual opt_model formulation in C-10 has only 1,937 Pg variables and 10,244 flow constraints, which is a *smaller* problem than the full B-theta formulation used by rundcopf (which includes bus angle variables). -- The synthesis estimated "~90-120s" for this test, suggesting the evaluator also expected much faster performance. - -The most likely explanation is that the PTDF-based flow constraint matrix (12,706 branches x 1,937 generators) is stored as a dense matrix, causing MIPS to use dense linear algebra instead of sparse factorization. The standard rundcopf uses the sparse B-theta formulation. This 400x slowdown is a formulation artifact, not a meaningful scalability finding. - -The synthesis marks C-10 as a "qualified_pass" but the timing undermines the result's credibility. A probe should verify whether the opt_model constraint matrix is sparse and whether MIPS performs comparably on a correctly-formulated problem. - -**Cross-tool relevance:** none -**Proposed action:** add_verification -- re-run with sparse constraint matrix or verify matrix sparsity - ---- - -### matpower-F02: C-5 contingency sweep wall-clock dominated by Octave data structure overhead - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** C-5 - -C-5 reports a total wall-clock of 2,476 seconds (~41 minutes) for the N-M contingency sweep on MEDIUM. However, the breakdown reveals: - -- BFS + adjacency construction: ~2,400s (97% of total) -- PTDF + LODF precomputation: 29s -- Actual N-1 through N-4 screening (28,035 cases): 50.4s - -The 2,400s adjacency construction time is caused by Octave's `containers.Map` being extremely slow for 10,000-bus networks. This is an Octave interpreter limitation, not a MATPOWER or algorithmic performance issue. The actual contingency screening performance (50s for 28K cases, 1.8ms per case) is strong. - -The synthesis marks this as a "qualified_pass" due to the LODF bottleneck, but the qualification conflates Octave overhead with tool capability. The LODF precomputation (6.9s as part of the 29s PTDF+LODF step) is actually fast. A fairer assessment would separate the algorithm's performance from the interpreter's overhead. - -**Cross-tool relevance:** none -**Proposed action:** adjust_scoring -- report screening time separately from adjacency construction - ---- - -### matpower-F03: IEEE 39-bus case39 has no effective RATE_A limits, producing uniform LMPs - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-3, A-9, B-8 - -Multiple tests on TINY produce trivially uniform results because case39's branch thermal limits never bind: - -- **A-3 (DC OPF):** All 39 buses show LMP = 13.5169 $/MWh. Shadow prices MU_SF and MU_ST are all zero. The test verifies that MATPOWER can solve a DC OPF, but the uniform LMPs mean it does not verify that locational pricing, congestion shadow prices, or LMP differentiation work correctly. - -- **A-9 (SCOPF):** "Base-case dispatch is identical between SCOPF and unconstrained DC OPF." The SCOPF cost differs by only 0.03% and no contingency constraint binds. The test confirms MOST's SCOPF formulation runs, but does not verify that preventive security constraints actually constrain dispatch. - -- **B-8 (Reference bus config):** All LMP comparisons across slack configurations are trivially identical because LMPs are uniform. The test cannot verify that reference bus changes affect LMPs. - -The evaluator recognized this issue in A-10, where 8 branch limits were explicitly tightened to force congestion (producing LMPs ranging from 9.87 to 32.19 $/MWh). This targeted tightening demonstrates awareness, but the untightened tests lose signal. - -This is a cross-tool issue: all tools using case39 for TINY will face the same problem unless the protocol mandates tightened limits. - -**Cross-tool relevance:** confirmed -**Proposed action:** redesign_test -- mandate branch limit tightening in case39 for OPF tests, or use a congested variant - ---- - -### matpower-F04: Gate tests are trivially passed for native MATPOWER format - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -MATPOWER loads its own native .m format via `loadcase()`. Gate tests measure whether a tool can ingest the reference networks, but for MATPOWER this tests the tool reading its own data format. Load times (0.016s for TINY, 1.02s for MEDIUM) reflect file parsing speed, not any ingestion challenge. Every other evaluated tool must convert from .m format, making these gate tests an infrastructure friction test for other tools and a trivial pass for MATPOWER. - -This is expected behavior and does not indicate an evaluation deficiency, but it means the gate test dimension provides no comparative information for MATPOWER. - -**Cross-tool relevance:** confirmed -**Proposed action:** none -- this is inherent to the protocol's choice of .m as reference format - ---- - -### matpower-F05: SCUC produces all-committed schedule, failing to exercise unit commitment cycling - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-5, A-6 - -A-5 augments case39 with UC parameters (PMIN=20% of PMAX, startup costs at 5x PMAX, ramp rates at 30% of PMAX) and runs 24-hour SCUC. The result: all 10 generators remain committed for all 24 hours. No generator is ever de-committed or re-committed. - -This means: -- Binary UC variables are never exercised (all remain 1) -- Min up/down time constraints never bind -- Startup/shutdown costs are never incurred -- A-6's ED comparison shows zero dispatch difference (UC = ED when all committed) - -The root cause is case39's capacity-to-load ratio: total PMAX is 7,367 MW, peak load is 6,254 MW, and PMIN totals 1,473 MW. With startup costs at 5x PMAX, the optimizer correctly determines that keeping all generators on is cheaper than cycling. The test verifies that MOST's MILP formulation *exists* and solves, but does not verify that UC cycling logic (the core feature being tested) works correctly. - -A probe should verify SCUC cycling by using more aggressive parameters (higher PMIN, lower startup costs, or a wider load range) or by reducing generator count. - -**Cross-tool relevance:** confirmed -- any tool using case39 with these parameters will show the same behavior -**Proposed action:** redesign_test -- adjust UC parameters to force at least some generators to cycle - ---- - -### matpower-F06: Three scalability failures stem from MOST ext2int bug, not proven solver limits - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** C-4, C-6, C-8 - -Three scalability tests are recorded as "fail" in the synthesis: - -- **C-4 (SCUC on SMALL):** "failed at the loadmd() stage with: 'buses must be numbered consecutively'" -- **C-6 (Stochastic on SMALL):** Same loadmd() error -- **C-8 (SCOPF on MEDIUM):** "did not complete within the 10-minute timeout" (different from C-4/C-6 but same MOST pathway) - -For C-4 and C-6, the solver was never invoked. The failure is a MOST data ingestion bug (loadmd() does not handle non-consecutive bus numbering on networks that pass core MATPOWER functions fine). The synthesis correctly identifies this as a MOST limitation and extrapolates that the resulting problem sizes would "likely exceed" solver capacity, but this is conjecture. The distinction matters: - -- A loadmd() bug is fixable with ext2int() preprocessing or a MOST patch -- A solver capacity limit is architectural and requires decomposition algorithms - -C-8 may have actually hit a timeout during the solve (the result file is less clear), but C-4 and C-6 definitively failed before solving. - -**Cross-tool relevance:** none -**Proposed action:** add_verification -- attempt C-4/C-6 with manually renumbered networks to separate the ext2int bug from solver capacity - ---- - -### matpower-F07: Solver swap test only tested 2 of 4 specified solvers - -**Category:** test_design_gap | **Severity:** low -**Tests:** C-7 - -The protocol specifies C-7 should repeat C-3 with "each available open-source solver" including HiGHS, GLPK, SCIP, and Ipopt. The evaluation tested only MIPS (pass, 9.28s) and GLPK (rejected QP). HiGHS was unavailable (requires MEX compilation not done in the test environment). SCIP and Ipopt were not mentioned. - -The test demonstrates solver swap *mechanism* (single parameter change), which is the primary thing being evaluated. However, the limited solver coverage means cross-solver objective consistency (a C-3 requirement) was only verified for MIPS vs GLPK (with PWL conversion, in C-3). The evaluation environment's solver availability was a constraint, not an evaluator choice. - -**Cross-tool relevance:** likely -- MATPOWER is unique in requiring MEX compilation for additional solvers, but other tools may also lack some protocol-specified solvers -**Proposed action:** add_verification -- document which protocol-specified solvers were available vs tested - ---- - -### matpower-F08: Distributed slack LMP comparison lacks specific numerical values - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-11, B-8 - -A-11 is the central test for distributed slack OPF capability. The result confirms dispatch is identical (correct) but for LMPs states only "differs significantly" without providing the actual distributed-slack LMP values. The table shows: - -``` -| Bus 1 | 14.01 | differs significantly | -| Bus 31 | 12.40 | differs significantly | -| Bus 39 | 16.53 | differs significantly | -``` - -The result also notes: "The sign convention in the manual formulation differs from MATPOWER's standard output, but the structural finding is confirmed: slack distribution affects marginal pricing." - -This phrasing suggests the evaluator may not have resolved the sign convention issue to produce correctly signed distributed-slack LMPs. The sign convention mismatch between opt_model.get_soln() and MATPOWER's standard output is called out as undocumented. If the LMPs have incorrect signs or are not properly translated, the "differs significantly" claim may be masking an incomplete verification. - -**Cross-tool relevance:** none -**Proposed action:** add_verification -- provide actual numerical LMP values for distributed-slack case and verify sign consistency - ---- - -### matpower-F09: A-10 qualified_pass for lossless tool may be generous vs rubric intent - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** A-10 - -The rubric sub-question 10 asks: "Can the tool solve a DC OPF with loss approximation?" MATPOWER's `rundcopf()` is strictly lossless with no loss option. The evaluator: - -1. Performed an exact lossless energy + congestion decomposition (passes) -2. Computed post-hoc loss estimates from branch impedances and flows (informational) -3. Noted the loss values are "NOT part of the optimization and do NOT affect dispatch or pricing" - -The qualified_pass with "stable workaround" framing accurately describes what was achieved, but the rubric's intent (loss-inclusive optimization affecting dispatch) was not met. The evaluator documented this clearly, and the rubric note says "any loss method accepted." The loss component is 0.78% of load on TINY -- potentially material on larger networks. The scoring is defensible but worth noting for cross-tool consistency. - -**Cross-tool relevance:** confirmed -- the boundary between "loss-inclusive optimization" and "post-hoc loss estimation" applies to all tools -**Proposed action:** adjust_scoring -- clarify rubric on whether post-hoc loss estimation counts as "loss approximation" - ---- - -### matpower-F10: LOC counts include boilerplate and are not normalized across tools - -**Category:** test_design_gap | **Severity:** low -**Tests:** D-5 - -D-5 counts total lines via `wc -l` including comments, blank lines, path setup (~20 lines per script), network loading (~15 lines), and result formatting (~30-50 lines). The file acknowledges this: "Effective 'analysis code' is roughly 60-70% of the total." - -Octave scripts inherently include more boilerplate than Python or Julia: -- `addpath()` calls for MATPOWER subdirectories -- `define_constants` for column index names -- `fprintf` for output formatting (no built-in DataFrame display) - -This inflates LOC relative to tools with cleaner import/display patterns. D-5 is useful for within-MATPOWER analysis (core vs MOST complexity) but cross-tool LOC comparison requires consistent methodology. - -**Cross-tool relevance:** confirmed -**Proposed action:** none -- this is a known limitation of LOC as a metric - ---- - -### matpower-F11: Peak memory measurements are estimates or null for most scalability tests - -**Category:** missing_verification | **Severity:** low -**Tests:** C-1, C-2, C-3, C-9, C-10 - -The protocol requires peak memory recording for all C-tests. Octave lacks built-in memory profiling tools comparable to Python's `tracemalloc` or Julia's `@allocated`. Results: - -- C-1: "~4.2 MB" (estimated from matrix dimensions) -- C-2, C-3, C-7: null -- C-5: "2,500 MB" (estimated) -- C-9: "1,017 MB" (likely computed from PTDF matrix size: 12706 x 10000 x 8 bytes) -- C-10: "1,200 MB" (estimated) - -These are order-of-magnitude estimates, not actual process memory measurements. This is an Octave ecosystem limitation affecting all Octave-based tools. - -**Cross-tool relevance:** likely -- other tools may have better memory profiling -**Proposed action:** none -- acknowledge as Octave limitation - ---- - -### matpower-F12: Interoperability test is trivially passed by all matrix-based tools - -**Category:** low_signal | **Severity:** low -**Tests:** B-5 - -B-5 tests CSV export. MATPOWER results are plain numeric matrices, so export requires only fprintf with column headers (18 LOC). The pass condition is "fewer than 5 lines of code beyond the solve." While Octave requires slightly more code than a Python tool with native DataFrame.to_csv(), this test has minimal discriminative value. Any tool that stores results in structured form will trivially pass. - -**Cross-tool relevance:** confirmed -**Proposed action:** none - ---- - -### matpower-F13: Stochastic test used only 3 scenarios with narrow load variation due to solver limits - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-8 - -The A-8 stochastic optimization test used only 3 scenarios with +/-3% load variation. The result notes: "The built-in MIPS solver struggles with larger load variation (+/-10%) on the 39-bus network combined with stochastic wind. Reducing to +/-3% load variation resolved convergence." - -With +/-3% uncertainty, the three scenarios produce LMPs within a narrow band (3.05-7.05 $/MWh range at peak hour). This may not sufficiently stress the cross-scenario coupling that distinguishes genuine stochastic optimization from three near-identical deterministic solves. The protocol does not specify minimum variation or scenario counts for TINY, so this technically passes. However, it weakens the evidence that MOST's stochastic formulation handles meaningful uncertainty. - -The 20-scenario wrapping test (B-4) used the same +/-3% load variation but with +/-8% peaker capacity variation, providing somewhat wider uncertainty bands. - -**Cross-tool relevance:** likely -- tools with weaker solvers may face similar constraints -**Proposed action:** none -- protocol could specify minimum variation for future versions - ---- - -### matpower-F14: Ramp rate utilization is suspiciously uniform across all generators - -**Category:** missing_verification | **Severity:** low -**Tests:** A-6 - -All 10 generators in A-6 show exactly 26.7% ramp utilization (max delta / ramp limit). This uniformity across generators with PMAX ranging from 508 to 1,100 MW is a consequence of: (1) no congestion (uniform LMPs), (2) polynomial costs with similar quadratic coefficients, and (3) equal per-unit ramp limits (30% of PMAX). When all generators face the same marginal cost curve shape and no flow constraints bind, they all ramp proportionally. - -While not incorrect, this means the ED ramp constraints were never binding or even close to binding (26.7% utilization). The test verifies ramp constraints exist in the formulation but does not verify they would correctly constrain dispatch when binding. - -**Cross-tool relevance:** confirmed -**Proposed action:** none -- could be strengthened by setting tighter ramp limits on a subset of generators - ---- - -## Extraordinary Claims - -### C-10: Distributed slack DC OPF on MEDIUM takes 66 minutes via opt_model/MIPS - -**Concern:** The single-slack DC OPF on the same network solves in 9.7s via MIPS. The manual opt_model formulation has fewer decision variables (1,937 Pg variables vs the full bus-angle formulation). A 400x slowdown for a simpler problem strongly suggests either: (a) the PTDF-based constraint matrix is stored as dense, causing MIPS to use dense O(n^3) factorization instead of sparse, or (b) the quadratic cost matrix is incorrectly structured, or (c) an opt_model configuration issue. The synthesis estimated "~90-120s" based on component measurements, indicating the actual run surprised the evaluator as well. - -**Evidence quality:** moderate -- the timing is from an actual run, but the explanation is speculative - -A probe should: (1) check whether the flow constraint matrix `H_dist * Cg` is stored as sparse, (2) if dense, convert to sparse and re-time, (3) compare MIPS iteration count between C-10 and C-3. - ---- - -### C-5: Contingency sweep on MEDIUM takes 41 minutes - -**Concern:** The 2,476s total is 97% Octave containers.Map overhead for adjacency construction and only 50s for the actual LODF-based screening. The synthesis reports the total as the test result and assigns a "qualified_pass" based partly on this timing. The actual screening performance (28,035 cases in 50s) is strong and would merit a clean pass if separated from the interpreter overhead. - -**Evidence quality:** strong -- the breakdown is clearly reported in the result file - -A probe should verify whether a simpler adjacency construction method (e.g., direct sparse matrix instead of containers.Map) reduces the overhead to a few seconds. - ---- - -### C-4: SCUC on SMALL fails due to solver capacity limits - -**Concern:** The test failed at `loadmd()` with a bus numbering error, not at the solver. The claim that the resulting MILP "would likely exceed GLPK's capacity" is an extrapolation based on problem size estimates (200K+ variables). While this extrapolation is reasonable, it is not verified. The ext2int bug in loadmd() is a known issue (GitHub #127) that could be fixed. - -**Evidence quality:** weak -- the solver was never invoked; the capacity claim is extrapolated - -A probe should: manually renumber the ACTIVSg 2000 buses to consecutive ordering, run loadmd(), and attempt the SCUC solve with a generous timeout to determine actual solver behavior. - ---- - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | -- | Trivial for native format | -| G-2 | pass | -- | Trivial for native format | -| G-3 | pass | -- | 19.4% zero RATE_A branches noted | -| A-1 | pass | -- | -- | -| A-2 | pass | -- | -- | -| A-3 | pass | -- | Uniform LMPs, no congestion | -| A-4 | pass | -- | -- | -| A-5 | pass | stable (PWL for GLPK) | All gens committed, no UC cycling | -| A-6 | pass | stable (PWL for GLPK) | UC=ED, uniform ramp utilization | -| A-7 | pass | -- | -- | -| A-8 | pass | -- | Only 3 scenarios, +/-3% variation | -| A-9 | pass | -- | SCOPF dispatch identical to base OPF | -| A-10 | qualified_pass | stable (post-hoc loss) | No native lossy DC OPF | -| A-11 | qualified_pass | stable (manual opt_model) | LMP values not fully reported | -| B-1 | pass | -- | -- | -| B-2 | pass | stable (manual BFS) | -- | -| B-3 | pass | -- | -- | -| B-4 | pass | -- | -- | -| B-5 | pass | -- | Low discriminative value | -| B-6 | informational | -- | -- | -| B-7 | pass | -- | -- | -| B-8 | qualified_pass | stable (distributed slack) | LMP comparison trivial (uniform) | -| B-9 | pass | -- | -- | -| C-1 | pass | -- | Memory estimated, not measured | -| C-2 | pass | -- | Memory null | -| C-3 | pass | -- | -- | -| C-4 | fail | -- | loadmd() bug, not solver | -| C-5 | pass | stable (ext2int) | 97% time is Octave overhead | -| C-6 | fail | -- | loadmd() bug, not solver | -| C-7 | pass | -- | Only 2 of 4 solvers tested | -| C-8 | fail | -- | Timeout (likely solver, possibly loadmd) | -| C-9 | pass | stable (ext2int) | -- | -| C-10 | qualified_pass | stable (manual opt_model) | 66 min timing anomaly | -| D-1 | informational | -- | -- | -| D-2 | informational | -- | 6/11 from docs, 2 not completable | -| D-3 | informational | -- | 13/20 examples pass (65%) | -| D-4 | informational | -- | Mixed quality (2/10 to 9/10) | -| D-5 | informational | -- | LOC includes boilerplate | -| E-1 | informational | -- | 2 releases in 24 months | -| E-2 | informational | -- | 93% single developer | -| E-3 | informational | -- | Bus factor 1 (98.5%) | -| E-4 | informational | -- | Unfunded since mid-2024 | -| E-5 | informational | -- | Median close 108 days | -| E-6 | informational | -- | 462 test files, no coverage metric | -| E-7 | informational | -- | Zero production deployment | -| F-1 | pass | -- | BSD-3, case file carve-out | -| F-2 | pass | -- | Zero external deps | -| F-3 | pass | -- | All BSD-3 | -| F-4 | pass | -- | Zero compiled extensions | -| F-5 | pass | -- | Full path inspectable | -| F-6 | informational | -- | No checksums/signatures | -| F-7 | pass | -- | Fully air-gap compatible | -| F-8 | pass | -- | All on open-source solvers | -| F-9 | informational | -- | Docs bundled in release | -| P2-1 | informational | -- | Native PSS/E RAW support | -| P2-2 | informational | -- | Native PWL cost curves | -| P2-3 | informational | -- | Full pipeline demonstrated | diff --git a/sweep-data/v4-to-v5/per-tool/matpower/findings.yaml b/sweep-data/v4-to-v5/per-tool/matpower/findings.yaml deleted file mode 100644 index bd46c2a2..00000000 --- a/sweep-data/v4-to-v5/per-tool/matpower/findings.yaml +++ /dev/null @@ -1,347 +0,0 @@ -tool: matpower -source_version: "v4" -timestamp: "2026-03-07T12:00:00Z" -evaluation_summary: - total_tests: 57 - pass: 34 - fail: 3 - qualified_pass: 4 - informational: 16 - -findings: - - id: matpower-F01 - category: extraordinary_claim - severity: high - test_ids: [C-10] - title: "C-10 distributed slack scale timing is anomalous (66 min vs expected ~30s)" - description: > - C-10 reports a total wall-clock of 3,969s (~66 min) for a distributed-slack DC OPF - on MEDIUM, with the MIPS solve alone taking 3,878s. The synthesis estimated ~90-120s. - The analogous single-slack DC OPF (C-3 with MIPS) solves in 9.7s on the same network. - The manual opt_model formulation has fewer variables (1,937 vs ~10,000) and should be - faster, not 400x slower. This suggests a formulation error in the opt_model construction - (e.g., dense constraint matrix from PTDF instead of sparse B-theta) or an opt_model - configuration issue causing MIPS to iterate excessively. - evidence: - - file: "scalability/C-10_distributed_slack_scale_MEDIUM.md" - excerpt: "opt_model MIPS solve: 3,877.57s (~65 min)" - - file: "scalability/C-3_dcopf_scale_MEDIUM.md" - excerpt: "MIPS solve: 9.72s" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - - - id: matpower-F02 - category: extraordinary_claim - severity: medium - test_ids: [C-5] - title: "C-5 contingency sweep wall-clock dominated by Octave data structure overhead, not algorithm" - description: > - C-5 reports 2,476s total but only 50.4s for the actual LODF-based contingency screening - (28,035 cases). The remaining ~2,400s is attributed to Octave containers.Map-based - adjacency construction. This means 97% of the reported wall-clock measures Octave - interpreter overhead, not MATPOWER or algorithmic performance. The synthesis marks - this as a qualified_pass, but the actual screening performance (50s for 28K cases) - is strong. The finding conflates Octave interpreter limitations with tool scalability. - evidence: - - file: "scalability/C-5_contingency_sweep_scale_MEDIUM.md" - excerpt: "BFS + adjacency build: ~2,400s (Octave containers.Map is very slow for 10k-bus adjacency construction)" - - file: "scalability/C-5_contingency_sweep_scale_MEDIUM.md" - excerpt: "N-1 through N-4 screening: 50.4s" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: adjust_scoring - - - id: matpower-F03 - category: network_insufficiency - severity: medium - test_ids: [A-3, A-9, B-8] - title: "IEEE 39-bus case39 has no effective RATE_A limits, producing uniform LMPs" - description: > - The default case39.m has all branch RATE_A values set such that no congestion occurs - under the base operating point. A-3 shows uniform LMPs (13.5169 $/MWh at all 39 buses), - A-9 shows SCOPF dispatch identical to unconstrained OPF, and B-8 shows all LMP - comparisons are trivially identical because there is no congestion to differentiate. - The A-10 test explicitly tightens 8 branch limits to force congestion, demonstrating - that the evaluator recognized this issue. However, A-3 and B-8 lose discriminative - value when LMPs are uniform. - evidence: - - file: "expressiveness/A-3_dcopf_TINY.md" - excerpt: "All buses: 13.5169 $/MWh (uniform). LMPs are uniform because case39 has zero RATE_A on all branches" - - file: "extensibility/B-8_reference_bus_config_TINY.md" - excerpt: "All LMPs are uniform because case39 has no RATE_A branch flow limits." - - file: "expressiveness/A-9_scopf_TINY.md" - excerpt: "Base-case dispatch is identical between SCOPF and unconstrained DC OPF." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: matpower-F04 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate tests are trivially passed for native MATPOWER format" - description: > - MATPOWER loads its own .m case file format natively via loadcase(). The gate tests - measure format ingestion capability, but for MATPOWER this is testing the tool - against its own data format. Every other tool must convert from .m format, making - this an infrastructure friction test for them and a no-op for MATPOWER. The gate - tests have zero discriminative value for MATPOWER specifically. - evidence: - - file: "gate/G-1_ingest_tiny.md" - excerpt: "Load time: 0.0161 seconds" - - file: "gate/G-3_ingest_medium.md" - excerpt: "Load time: 1.0214 seconds" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: matpower-F05 - category: missing_verification - severity: medium - test_ids: [A-5, A-6] - title: "SCUC produces all-committed schedule, failing to exercise unit commitment cycling" - description: > - A-5 reports all 10 generators committed for all 24 hours. The test augmented case39 - with PMIN=20% of PMAX, startup costs at 5x PMAX, and ramp rates at 30% of PMAX. - However, the capacity-to-load ratio (7,367 MW capacity vs 6,254 MW peak) combined - with these parameters means the optimizer correctly determined that de-commitment - saves no cost. This means the UC binary variables are never exercised (all stay 1), - min up/down time constraints never bind, and A-6's ED comparison shows zero dispatch - difference. The test verifies the MILP formulation exists but does not verify that - UC cycling logic works correctly. - evidence: - - file: "expressiveness/A-5_scuc_TINY.md" - excerpt: "All 10 generators remain committed for all 24 hours." - - file: "expressiveness/A-6_sced_TINY.md" - excerpt: "Maximum total dispatch difference: 0.00 MW" - cross_tool_relevance: confirmed - probe_recommended: true - probe_type: formulation_audit - proposed_action: redesign_test - - - id: matpower-F06 - category: infrastructure_friction - severity: medium - test_ids: [C-4, C-6, C-8] - title: "Three scalability failures stem from MOST ext2int bug, not proven solver limits" - description: > - C-4, C-6, and C-8 all fail at the loadmd() stage with a bus numbering error before - the solver is even invoked. The synthesis attributes these to solver capacity limits, - but the actual evidence shows they failed at data ingestion. While the synthesis - correctly notes the underlying problem size would likely exceed solver capacity, this - is conjecture -- the solvers never ran. The distinction matters: a loadmd bug is - fixable, while solver capacity is architectural. - evidence: - - file: "scalability/C-4_scuc_scale_SMALL.md" - excerpt: "MOST SCUC on SMALL failed at the loadmd() stage with: 'buses must be numbered consecutively'" - - file: "scalability/C-6_stochastic_scale_SMALL.md" - excerpt: "MOST stochastic DCOPF on SMALL failed at the loadmd() stage" - - file: "scalability/C-8_scopf_scale_MEDIUM.md" - excerpt: "MOST SCOPF on MEDIUM with 500 contingencies did not complete within the 10-minute timeout" - cross_tool_relevance: none - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: matpower-F07 - category: test_design_gap - severity: low - test_ids: [C-7] - title: "Solver swap test only tested 2 of 4 specified solvers" - description: > - The protocol specifies C-7 should test HiGHS, GLPK, SCIP, and Ipopt. The evaluation - only tested MIPS and GLPK. GLPK immediately rejected the QP problem. HiGHS was not - available (requires MEX compilation). SCIP and Ipopt were not tested. The test - effectively measured one successful solver run, providing limited solver swap - coverage. - evidence: - - file: "scalability/C-7_solver_swap_MEDIUM.md" - excerpt: "MIPS: 9.28s (interior point, handles QP natively). GLPK: rejected immediately (LP-only, 0.11s)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: matpower-F08 - category: misleading_result - severity: medium - test_ids: [A-11, B-8] - title: "Distributed slack LMP comparison lacks specific numerical values" - description: > - A-11 reports distributed-slack LMPs "differ significantly" from single-slack LMPs - but does not provide the actual distributed-slack LMP values, only stating they - "differ significantly." B-8 part (c) similarly notes distributed-slack PTDF differs - but cannot show LMP comparison because the network is uncongested (uniform LMPs). - The sign convention issue noted in A-11 ("shadow price sign convention from - opt_model.get_soln differs") suggests the evaluator may not have successfully - extracted correctly signed LMPs from the manual formulation. - evidence: - - file: "expressiveness/A-11_distributed_slack_opf_TINY.md" - excerpt: "Bus 1: Single-Slack LMP 14.01, Distributed-Slack LMP: differs significantly" - - file: "expressiveness/A-11_distributed_slack_opf_TINY.md" - excerpt: "The sign convention in the manual formulation differs from MATPOWER's standard output" - cross_tool_relevance: none - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: matpower-F09 - category: scoring_inconsistency - severity: low - test_ids: [A-10] - title: "A-10 qualified_pass for lossless tool may be generous vs rubric intent" - description: > - The rubric asks whether LMPs can be decomposed into energy, congestion, and loss - components. MATPOWER's rundcopf is strictly lossless and has no loss option. The - evaluator performed an exact lossless energy+congestion decomposition and a post-hoc - (non-optimization) loss estimate. The qualified_pass with "stable workaround" framing - may overstate capability, since the loss component is informational only and not part - of the formulation. However, the rubric note says "any loss method accepted" and the - evaluator documented this clearly. - evidence: - - file: "expressiveness/A-10_lossy_dcopf_lmp_TINY.md" - excerpt: "These values are informational only -- they are NOT part of the optimization and do NOT affect dispatch or pricing." - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: matpower-F10 - category: test_design_gap - severity: low - test_ids: [D-5] - title: "LOC counts include boilerplate and are not normalized across tools" - description: > - D-5 counts total lines including comments, blank lines, path setup (~20 lines), - network loading (~15 lines), and result formatting (~30-50 lines). The file notes - this explicitly. Cross-tool LOC comparison requires consistent methodology; Octave - scripts inherently include more boilerplate (addpath, define_constants) than Python - or Julia equivalents. The D-5 result is useful within MATPOWER but may not be - comparable across tools without normalization. - evidence: - - file: "accessibility/D-5_code_volume_comparison.md" - excerpt: "Counted total lines of code (including comments, blank lines, and boilerplate)" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: matpower-F11 - category: missing_verification - severity: low - test_ids: [C-1, C-2, C-3, C-9, C-10] - title: "Peak memory measurements are estimates or null for most scalability tests" - description: > - The protocol requires peak memory recording for all C-tests. C-1 reports "~4.2 MB" - as an estimate. C-2, C-3, C-7 report null. C-9 reports 1,017 MB. C-10 reports - 1,200 MB. Octave lacks built-in memory profiling, making accurate measurement - difficult. The estimates that are provided appear to be computed from matrix - dimensions rather than actual process memory measurement. - evidence: - - file: "scalability/C-1_dcpf_scale_MEDIUM.md" - excerpt: "Peak memory estimate: ~4.2 MB for data structures + 0.6 MB for sparse B matrix" - - file: "scalability/C-2_acpf_scale_MEDIUM.md" - excerpt: "peak_memory_mb: null" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: matpower-F12 - category: low_signal - severity: low - test_ids: [B-5] - title: "Interoperability test is trivially passed by all matrix-based tools" - description: > - B-5 tests CSV export of DCPF results. For MATPOWER, results are plain numeric - matrices, so export is trivially fprintf-based (18 LOC). This test likely produces - identical pass results for any tool that stores results in array/DataFrame form. - It does not discriminate between tools that have rich interoperability (e.g., - native DataFrame integration) and those that require manual column labeling. - evidence: - - file: "extensibility/B-5_interoperability_TINY.md" - excerpt: "Total export logic: 18 lines for both bus and branch CSV" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: matpower-F13 - category: test_design_gap - severity: medium - test_ids: [A-8] - title: "Stochastic test used only 3 scenarios with narrow load variation due to solver limits" - description: > - The protocol specifies stochastic optimization. A-8 used only 3 scenarios with +/-3% - load variation. The result notes MIPS struggled with +/-10% variation, forcing - reduction to +/-3%. With such narrow uncertainty bands, the stochastic formulation - produces near-identical results to a deterministic solve, reducing the test's ability - to verify that cross-scenario coupling is meaningful. The protocol does not specify - minimum variation or scenario count for TINY, so this technically passes, but the - weak stress test should be noted. - evidence: - - file: "expressiveness/A-8_stochastic_timeseries_TINY.md" - excerpt: "MIPS solver struggles with larger load variation (+/-10%)... Reducing to +/-3% load variation resolved convergence." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: matpower-F14 - category: missing_verification - severity: low - test_ids: [A-6] - title: "Ramp rate utilization is suspiciously uniform across all generators" - description: > - A-6 reports all 10 generators have exactly 26.7% ramp utilization. This perfect - uniformity across generators with different sizes (PMAX ranging from 508 to 1100 MW) - is surprising. It suggests the load curve produces identical relative dispatch - changes for all generators, which would occur if no flow constraints bind and all - generators are on the same marginal cost curve segment. While not incorrect, this - uniformity means ramp constraints were never tested near their limits. - evidence: - - file: "expressiveness/A-6_sced_TINY.md" - excerpt: "All 10 generators: Utilization 26.7%" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - -extraordinary_claims: - - test_id: C-10 - claim: "Distributed slack DC OPF on MEDIUM takes 66 minutes via opt_model/MIPS" - concern: > - The single-slack DC OPF on the same network solves in 9.7s via MIPS. The manual - opt_model formulation has fewer variables (1,937 Pg variables vs full bus/angle - formulation). A 400x slowdown for a simpler problem strongly suggests a formulation - issue (e.g., dense PTDF constraint matrix causing MIPS to work with dense linear - algebra instead of sparse). - evidence_quality: moderate - probe_recommended: true - probe_type: timing_verification - - - test_id: C-5 - claim: "Contingency sweep on MEDIUM takes 41 minutes" - concern: > - 97% of the reported wall-clock (2,400s of 2,476s) is Octave containers.Map overhead - for adjacency construction, not MATPOWER computation. The actual LODF-based screening - completes in 50s. Reporting the total as the test result conflates interpreter - performance with tool capability. - evidence_quality: strong - probe_recommended: true - probe_type: timing_verification - - - test_id: C-4 - claim: "SCUC on SMALL fails due to solver capacity limits" - concern: > - The test actually failed at loadmd() data ingestion (bus numbering error), not at - the solver stage. The claim about solver capacity is an extrapolation. The solver - was never invoked. - evidence_quality: weak - probe_recommended: true - probe_type: claim_verification diff --git a/sweep-data/v4-to-v5/per-tool/pandapower/findings.md b/sweep-data/v4-to-v5/per-tool/pandapower/findings.md deleted file mode 100644 index 7ca85c1f..00000000 --- a/sweep-data/v4-to-v5/per-tool/pandapower/findings.md +++ /dev/null @@ -1,245 +0,0 @@ -# pandapower -- Sweep Findings (v4) - -## Summary - -The pandapower v4 evaluation is generally well-executed with thorough documentation of both capabilities and limitations. The tool's architectural scope (steady-state PF and basic OPF) is clearly identified, and the 6 expressiveness failures are correctly classified as architectural gaps rather than bugs. However, several findings warrant attention: the most significant is the C-6/B-4 stochastic wrapping result, where a 2.1% solver convergence rate receives a qualified_pass status that masks practical unusability. The PTDF flow prediction divergence on MEDIUM (7.43 pu max error) is accepted without root-cause verification. Three scalability tests fail solely as cascades from expressiveness failures, inflating the fail count without adding signal. Uniform LMPs across the entire 10k-bus network in A-3/C-3 suggest the DC OPF test does not exercise congestion, limiting the value of the LMP extraction verification. Three probes are recommended: convergence rate verification for the stochastic DCOPF, PTDF flow prediction root-cause analysis, and PYPOWER solver stability characterization. - -## Finding Details - -### pandapower-F01: Qualified pass with 2.1% solver convergence rate masks practical unusability - -**Category:** misleading_result | **Severity:** high -**Tests:** C-6, B-4 - -Both C-6 (scalability) and B-4 (extensibility) report qualified_pass for the stochastic DCOPF wrapping approach. The loop-based methodology is sound -- modify DataFrames in-place, call `rundcopp()` per scenario/hour pair -- and is classified with workaround_class "stable." However, only 5 of 240 DC OPF solves converge (2.1%). The result narrative acknowledges the low convergence rate but attributes it to "a PYPOWER interior point solver quality issue" and maintains the qualified_pass status. - -The concern is that "qualified_pass" with "stable" workaround conveys to a reader that the approach works with some caveats. In reality, a user following this approach on the SMALL network would get results for only 1 out of every 48 solves. The qualification is severe enough that the result could reasonably be classified as a fail for practical purposes. The workaround_class "stable" correctly describes the mechanical stability of the loop approach, but the solver failure rate is a separate dimension that the current schema does not adequately capture. - -This finding has cross-tool relevance: other tools using the same SMALL network with load/generation perturbations may also encounter solver difficulties, making this a potential protocol-level issue with the stochastic perturbation methodology or the ACTIVSg2000 case data's sensitivity to scaling. - -**Cross-tool relevance:** likely -**Proposed action:** adjust_scoring -- consider whether the scoring rubric should distinguish between "approach is mechanically sound" and "approach produces usable results at the target scale" - -### pandapower-F02: Uniform LMPs on 10k-bus network suggest no binding constraints - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-3, C-3 - -A-3 (DC OPF on MEDIUM) and C-3 (DC OPF scalability) both report perfectly uniform LMPs of 20.738 across all 10,000 buses. This means no line flow constraints are binding in the solution. While this may be a property of the ACTIVSg10k case data (the lines may have sufficient capacity for the base load), it means the test does not actually exercise pandapower's ability to produce spatially differentiated LMPs from congestion. - -The pass condition for A-3 states: "Converges. Optimal dispatch and LMPs/shadow prices extractable from solution." This is technically met -- LMPs are extractable. But the test provides no evidence that pandapower correctly computes congestion-driven LMP variation, which is the primary use case for LMPs in market analysis. On TINY (case39), the same uniform LMP pattern appears (13.517 everywhere), suggesting neither test network produces congestion under the DC OPF with existing line limits. - -This is a cross-tool concern: if all tools produce uniform LMPs on these networks, the LMP extraction test has low discriminative value for congestion-related capabilities. - -**Cross-tool relevance:** confirmed -**Proposed action:** add_verification -- consider tightening line limits on the test network to force congestion, or add a secondary test with artificial congestion - -### pandapower-F03: PTDF flow prediction diverges on MEDIUM but test still passes - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** B-9, C-9 - -B-9 reports a maximum flow prediction difference of 7.43 pu between PTDF-predicted flows and DCPF-solved flows on the 10k-bus network. The pass condition for B-9 states: "Flow predictions match DCPF results within numerical tolerance (1e-6)." On TINY, this tolerance is met. On MEDIUM, it is violated by 7 orders of magnitude. - -The result attributes the divergence to "shunt elements and tap-ratio effects in transformers not fully captured by the basic PTDF formulation." This is a plausible explanation -- the standard DC PTDF assumes lossless lines with unity tap ratios -- but is stated as speculation rather than verified root cause. The test passes as qualified_pass because the PTDF matrix has correct dimensions and structural properties (slack column zeros, physically reasonable density). - -If the divergence is genuinely due to transformer tap ratios, it means the PTDF matrix is not suitable for accurate flow prediction on networks with transformers -- a significant practical limitation for congestion analysis. If it is due to a bug in bus injection vector reconstruction (the result mentions "ordering differences"), it may be fixable. - -**Cross-tool relevance:** likely -- other tools computing PTDF on the same MEDIUM network will face similar transformer modeling questions -**Proposed action:** add_verification -- probe should compute PTDF flows on a subnetwork without transformers and on the full network with explicit tap-ratio correction - -### pandapower-F04: Contingency sweep scope reduced without clear impact assessment - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-7, C-5 - -C-5 limits BFS computation to 200 of 10,701 seed branches on the MEDIUM network. The protocol specifies x=5, m=4 for MEDIUM, which would generate an enormous combinatorial space. The scope reduction is pragmatic and acknowledged in the result ("Full enumeration for all branches is O(n * E) and exceeded the time budget"), but the impact is not fully assessed. - -The result reports 10,000 cases evaluated out of 31,045 pruned cases, but it is unclear how many total cases the full-scope test would have produced. The per-contingency time (0.054s) is well-documented and useful for cross-tool comparison. The scope reduction does not invalidate the result but makes the "PASS" somewhat qualified -- the tool handles the scope that was tested, but the full protocol parameters were not exercised. - -**Cross-tool relevance:** likely -- all tools will face the same combinatorial explosion on MEDIUM with x=5, m=4 -**Proposed action:** adjust_scoring -- document scope reduction consistently across tools and assess whether x=5, m=4 on MEDIUM is feasible for any tool - -### pandapower-F05: Distributed slack silently ignored in OPF -- scoring inconsistency - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** B-8 - -B-8 tests three slack configurations: (a) default, (b) different bus, (c) distributed. Config (c) runs `rundcopp(net, distributed_slack=True)` which silently ignores the parameter and produces results identical to config (a). The test receives qualified_pass because configs (a) and (b) succeed. - -The inconsistency is with A-11, which correctly assigns "fail" for distributed slack OPF. B-8's qualified_pass could mislead a reader into thinking distributed slack is partially functional in OPF when it is not functional at all -- the parameter is silently swallowed by `**kwargs`. The observation file (api-friction-extensibility-B-8) correctly flags this as a medium-severity API friction point, but the test outcome does not fully reflect the finding. - -The silent parameter swallowing is itself a significant finding: a user calling `rundcopp(net, distributed_slack=True)` receives no indication that the parameter has no effect. This is worse than raising an error because it produces silently incorrect results (single-slack when the user expects distributed slack). - -**Cross-tool relevance:** none -**Proposed action:** adjust_scoring -- B-8 pass/fail should be based on the 2 functional configs; the distributed slack aspect should carry a note rather than contributing to the qualified_pass status - -### pandapower-F06: Gate tests are trivially passed - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -All gate tests pass without issues. pandapower's MATPOWER converter handles all three reference networks cleanly. This is expected for any tool with a MATPOWER format parser and does not differentiate pandapower from other tools. - -**Cross-tool relevance:** confirmed -**Proposed action:** none -- gate tests serve their purpose as a prerequisite filter even if they don't discriminate among passing tools - -### pandapower-F07: Solver lock-in prevents protocol-specified multi-solver comparison - -**Category:** infrastructure_friction | **Severity:** low -**Tests:** A-3, C-3, C-7 - -pandapower's `rundcopp()` is hard-wired to PYPOWER's interior point solver. The protocol specifies HiGHS and GLPK for DC OPF tests and requires multi-solver comparison in C-3 and C-7. This is correctly classified as a tool limitation (C-7 fails), and the evaluator appropriately notes the deviation. However, it means timing comparisons for DC OPF between pandapower and tools using HiGHS/GLPK are not directly comparable -- different solver architectures (interior point vs simplex) have different scaling characteristics. - -**Cross-tool relevance:** none -**Proposed action:** none -- correctly handled as a tool finding - -### pandapower-F08: Peak memory not measured for multiple tests - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-1, A-2, A-3, A-4, A-7 - -Several expressiveness tests on the MEDIUM network report "Peak memory: not measured." The scalability tests (C-1 through C-9) do measure memory, so the data exists for the same operations when run under the scalability suite. The gap means cross-tool memory comparison for expressiveness tests at scale will have missing data points. - -**Cross-tool relevance:** likely -**Proposed action:** add_verification -- memory measurement should be standard for all tests at MEDIUM scale - -### pandapower-F09: AC PF solver deviation from protocol - -**Category:** test_design_gap | **Severity:** low -**Tests:** C-2 - -The protocol specifies Ipopt for C-2 (ACPF at scale), but pandapower uses its internal Newton-Raphson implementation. This is the correct solver for AC power flow in pandapower (Ipopt is typically used for NLP optimization, not PF). The deviation reflects a protocol design choice that may not match all tools' solver architectures -- AC PF is typically solved by Newton-Raphson, not by a general NLP solver. - -**Cross-tool relevance:** likely -- tools with dedicated NR solvers for PF will all deviate from the Ipopt specification -**Proposed action:** none -- the protocol's Ipopt specification for AC PF may need clarification on whether it means "any NLP solver" or specifically Ipopt - -### pandapower-F10: PYPOWER solver produces lambda values of 1e25 when generators decommitted - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** P2-3 - -The P2-3 commitment injection test reveals that using `in_service=False` to decommit generators causes the PYPOWER interior point solver to produce lambda values on the order of 1e25 on the IEEE 39-bus case. This indicates severe numerical instability in the solver when the generator set changes. The workaround (setting `max_p_mw=0` instead) achieves the same functional result but is semantically less clear. - -This finding connects to the broader PYPOWER solver fragility theme: the 2.1% convergence rate in C-6/B-4, the solver lock-in in C-7, and the numerical instability in P2-3 all point to the PYPOWER interior point solver being a significant quality limitation for pandapower's OPF capabilities. The solver works on clean, unmodified test cases but degrades rapidly when the problem is perturbed. - -**Cross-tool relevance:** none -**Proposed action:** none -- this is an informational finding but the probe should verify whether the 1e25 lambda values are reproducible and whether they affect the B-4/C-6 convergence rate - -### pandapower-F11: Three scalability tests fail solely due to upstream expressiveness failures - -**Category:** redundant_test | **Severity:** low -**Tests:** C-4, C-8, C-10 - -C-4 (SCUC scale), C-8 (SCOPF scale), and C-10 (distributed slack scale) all fail because their prerequisite expressiveness tests (A-5, A-9, A-11) failed. These results add no new information. The cascade is correctly documented in each result file, but the raw fail count (14 fails) overstates the number of independent findings. The actual independent failure count is 11: 6 expressiveness failures, 1 scalability failure (C-7 solver swap), plus these 3 cascaded fails and 1 stochastic scale failure (C-6 is more nuanced). - -**Cross-tool relevance:** confirmed -- tools with similar expressiveness gaps will show the same cascade pattern -**Proposed action:** adjust_scoring -- consider whether cascaded fails should be counted separately in the evaluation summary, or whether the total should distinguish "independent fails" from "cascaded fails" - -### pandapower-F12: ACPF convergence claimed without residual verification - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-2, C-2 - -A-2 and C-2 report ACPF convergence on MEDIUM but do not extract or report the achieved power mismatch residual. The results show that the solver converged (Boolean flag) and report voltage magnitudes/angles, but do not verify the convergence tolerance. pandapower's default NR tolerance is 1e-8 and the solver likely achieves this, but the result files do not document the achieved residual for audit purposes. - -**Cross-tool relevance:** confirmed -- all tools' ACPF results should report achieved residuals -**Proposed action:** add_verification - -### pandapower-F13: Tutorial verification tests import availability, not execution - -**Category:** test_design_gap | **Severity:** low -**Tests:** D-3 - -D-3 reports 8/8 tutorials pass, but tutorials 7 (contingency analysis) and 8 (plotting) only verify that the relevant modules are importable, not that the tutorial workflows execute successfully. The plotting limitation is understandable (headless container), but the contingency analysis test could have been run. Reporting import checks as "PASS" slightly overstates tutorial completeness. - -**Cross-tool relevance:** likely -- headless container limitations may affect tutorial verification across all tools -**Proposed action:** adjust_scoring -- distinguish between "tutorial executed successfully" and "API importable" in the results - -## Extraordinary Claims - -### C-6: Stochastic DCOPF wrapping qualified pass despite 2.1% convergence rate - -**Concern:** A 97.9% solver failure rate makes the approach practically unusable on the SMALL network. The qualified_pass status with "stable" workaround classification conveys that the approach works with caveats, when in reality almost no scenarios produce results. The convergence rate on TINY (where case39 OPF converges reliably) should be checked to isolate whether this is SMALL-network-specific solver sensitivity or a general PYPOWER OPF fragility under perturbation. - -**Evidence quality:** strong -- the 2.1% rate is documented from actual execution of 240 solves with captured output. - -A probe should: (1) Run the same 240 solves on TINY to establish PYPOWER convergence rate on a smaller network, (2) test whether tighter perturbation bounds improve convergence on SMALL, and (3) verify whether other tools' solvers (HiGHS, Ipopt) achieve higher convergence rates on the same perturbed SMALL scenarios. - -### B-9: PTDF matrix correctly computed despite 7.43 pu max flow prediction error on MEDIUM - -**Concern:** The B-9 pass condition requires flow predictions to match within 1e-6. The 7.43 pu max error on MEDIUM exceeds this by 7 orders of magnitude. The attributed cause ("shunt elements and tap-ratio effects") is plausible but unverified. If transformer tap ratios are the root cause, the PTDF matrix is not practically useful for flow prediction on the MEDIUM network -- a significant limitation for congestion analysis. - -**Evidence quality:** moderate -- the error magnitudes are documented but the root cause is speculative. - -A probe should: (1) Compute PTDF-predicted flows on a subset of branches that are lines (not transformers) and verify 1e-6 match, (2) compute PTDF with explicit tap-ratio corrections and check whether errors reduce, (3) verify that the bus injection vector reconstruction correctly maps pandapower's bus indices to the ppc's internal ordering. - -### P2-3: PYPOWER solver numerical instability with lambda values ~1e25 - -**Concern:** The solver's inability to handle generator decommitment on a 39-bus network raises questions about the robustness of all pandapower OPF results where the generator set differs from the default. The 2.1% convergence rate in C-6/B-4 may be a manifestation of the same underlying solver fragility. - -**Evidence quality:** strong -- the lambda values are reported from actual execution with specific generator decommitment scenarios. - -A probe should: (1) Reproduce the 1e25 lambda values on case39 with `in_service=False`, (2) test whether the same issue occurs on case9 (smaller network), (3) test whether the `max_p_mw=0` workaround produces identical dispatch to a reference solver. - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | -- | -- | -| G-2 | pass | -- | -- | -| G-3 | pass | -- | -- | -| A-1 | pass | -- | -- | -| A-2 | pass | -- | No residual reported | -| A-3 | qualified_pass | stable (PYPOWER only) | Uniform LMPs, no congestion exercised | -| A-4 | pass | -- | -- | -| A-5 | fail | blocking | No SCUC capability | -| A-6 | fail | blocking | Depends on A-5 | -| A-7 | pass | -- | -- | -| A-8 | fail | blocking | No native stochastic OPF | -| A-9 | fail | blocking | No SCOPF capability | -| A-10 | fail | blocking | No lossy DC OPF | -| A-11 | fail | blocking | Distributed slack not in OPF | -| B-1 | qualified_pass | fragile (_ppc duals) | Internal access for dual values | -| B-2 | pass | -- | -- | -| B-3 | pass | -- | -- | -| B-4 | qualified_pass | stable (manual loop) | 2.1% convergence on SMALL | -| B-5 | pass | -- | -- | -| B-6 | pass | -- | -- | -| B-7 | pass | -- | -- | -| B-8 | qualified_pass | stable | Distributed slack silently ignored | -| B-9 | qualified_pass | fragile (_ppc internals) | 7.43 pu flow prediction error on MEDIUM | -| C-1 | pass | -- | -- | -| C-2 | pass | -- | NR instead of Ipopt per protocol | -| C-3 | qualified_pass | -- | Single solver only | -| C-4 | fail | blocked (A-5) | Cascaded failure | -| C-5 | pass | -- | Reduced scope (200/10701 seeds) | -| C-6 | qualified_pass | stable | 2.1% convergence rate | -| C-7 | fail | -- | No solver swap capability | -| C-8 | fail | blocked (A-9) | Cascaded failure | -| C-9 | pass | -- | -- | -| C-10 | fail | blocked (A-11) | Cascaded failure | -| D-1 | pass | -- | -- | -| D-2 | qualified_pass | stable | OPF docs gaps | -| D-3 | pass | -- | 2/8 tests are import-only | -| D-4 | qualified_pass | -- | Silent cost curve fallback | -| D-5 | informational | -- | -- | -| E-1 | pass | -- | -- | -| E-2 | pass | -- | -- | -| E-3 | pass | -- | -- | -| E-4 | pass | -- | -- | -| E-5 | pass | -- | -- | -| E-6 | pass | -- | -- | -| E-7 | pass | -- | -- | -| F-1 | pass | -- | -- | -| F-2 | pass | -- | -- | -| F-3 | pass | -- | -- | -| F-4 | pass | -- | -- | -| F-5 | pass | -- | -- | -| F-6 | pass | -- | -- | -| F-7 | pass | -- | -- | -| F-8 | pass | -- | -- | -| F-9 | qualified_pass | -- | Unversioned install commands | -| P2-1 | informational | -- | -- | -| P2-2 | informational | -- | -- | -| P2-3 | informational | -- | Solver fragility with decommitment | diff --git a/sweep-data/v4-to-v5/per-tool/pandapower/findings.yaml b/sweep-data/v4-to-v5/per-tool/pandapower/findings.yaml deleted file mode 100644 index 24de9b3e..00000000 --- a/sweep-data/v4-to-v5/per-tool/pandapower/findings.yaml +++ /dev/null @@ -1,319 +0,0 @@ -tool: pandapower -source_version: "v4" -timestamp: "2026-03-07T00:00:00Z" -evaluation_summary: - total_tests: 55 - pass: 30 - fail: 14 - qualified_pass: 8 - informational: 3 - -findings: - - id: pandapower-F01 - category: misleading_result - severity: high - test_ids: [C-6, B-4] - title: "Qualified pass with 2.1% solver convergence rate masks practical unusability" - description: >- - C-6 and B-4 both report qualified_pass for stochastic DCOPF wrapping, but the - PYPOWER interior point solver converges on only 5 of 240 solves (2.1%) on the - SMALL network. A 97.9% failure rate renders the approach practically unusable, - yet the test outcome is "qualified_pass" with workaround_class "stable". The - qualification understates the severity -- a user following this result would - expect the approach to work at scale. - evidence: - - file: "scalability/C-6_stochastic_scale.md" - excerpt: "Converged: 5 (2.1%), Failed: 235 (97.9%)" - - file: "extensibility/B-4_stochastic_wrapping.md" - excerpt: "Converged: 5 (2.1%), Failed: 235 (97.9%)" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: adjust_scoring - - - id: pandapower-F02 - category: missing_verification - severity: medium - test_ids: [A-3, C-3] - title: "Uniform LMPs on 10k-bus network suggest no binding constraints -- result lacks verification" - description: >- - A-3 and C-3 report uniform LMPs across all 10,000 buses (20.738 everywhere), - indicating zero congestion on the MEDIUM network. This means the DC OPF is - effectively unconstrained -- line limits never bind. While this could be a - property of the ACTIVSg10k case data, it means the test does not actually verify - that pandapower correctly produces spatially differentiated LMPs from congestion. - The LMP extraction is verified on a trivial (uncongested) case. - evidence: - - file: "expressiveness/A-3_dcopf.md" - excerpt: "LMP range: 20.738 -- 20.738, LMPs are nearly uniform across all buses" - - file: "scalability/C-3_dcopf_scale.md" - excerpt: "LMP max: 20.738, LMP min: 20.738, LMP mean: 20.738" - cross_tool_relevance: confirmed - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: pandapower-F03 - category: extraordinary_claim - severity: medium - test_ids: [B-9, C-9] - title: "PTDF flow prediction diverges on MEDIUM but test still passes" - description: >- - B-9 reports PTDF flow predictions diverge from DCPF results on the 10k-bus - network (max diff 7.43 pu, mean diff 0.027 pu), far exceeding the 1e-6 tolerance - in the pass condition. The test passes because the PTDF matrix has correct - dimensions and structural properties, but the core validation requirement -- - that PTDF-predicted flows match DCPF flows -- fails on MEDIUM. The result is - classified as qualified_pass but the divergence is attributed to "shunt elements - and tap-ratio effects" without verification. - evidence: - - file: "extensibility/B-9_ptdf_extraction.md" - excerpt: "Max flow difference: 7.43 pu, Mean flow difference: 0.027 pu" - - file: "scalability/C-9_ptdf_scale.md" - excerpt: "Max flow diff (pu): 7.435, Accurate (< 1e-4): No" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: pandapower-F04 - category: test_design_gap - severity: medium - test_ids: [A-7, C-5] - title: "Contingency sweep scope reduced without clear impact assessment" - description: >- - C-5 limits BFS computation to 200 of 10,701 seed branches due to time budget, - evaluating only 10,000 of potentially millions of cases. A-7 on TINY correctly - uses x=3, m=3 per protocol. The MEDIUM test deviates from protocol parameters - (x=5, m=4) in practice by capping at 200 seeds, but the result file does not - clearly document how many N-3 and N-4 cases were enumerated vs pruned vs skipped - due to the seed cap. - evidence: - - file: "scalability/C-5_contingency_sweep_scale.md" - excerpt: "Reduced scope: BFS limited to 200 of 10,701 seed branches" - - file: "scalability/C-5_contingency_sweep_scale.md" - excerpt: "N-1 cases: 200, N-2 cases: 16,437, Cases evaluated: 10,000" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pandapower-F05 - category: scoring_inconsistency - severity: medium - test_ids: [B-8] - title: "Distributed slack silently ignored in OPF -- qualified_pass vs A-11 fail inconsistency" - description: >- - B-8 gives qualified_pass for reference bus configuration even though config (c) - -- distributed slack OPF -- produces identical results to single slack (the - parameter is silently ignored). Meanwhile A-11 correctly fails pandapower for - lacking distributed slack OPF. The B-8 qualified_pass is justified by configs - (a) and (b) succeeding, but the result could mislead readers into thinking - distributed slack OPF is partially functional when it is not functional at all. - evidence: - - file: "extensibility/B-8_reference_bus_config.md" - excerpt: "Config (c) produced identical results to (a), suggesting distributed slack in OPF may reduce to single-slack behavior" - - file: "expressiveness/A-11_distributed_slack_opf_TINY.md" - excerpt: "FAIL -- rundcopp() API does not expose distributed slack formulation" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pandapower-F06 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate tests are trivially passed -- low discriminative value for pandapower" - description: >- - All three gate tests pass cleanly with no issues. pandapower uses MATPOWER's - from_mpc converter which handles all three reference networks without friction. - These tests are expected to pass for any Python-based tool with a MATPOWER - converter and provide no differentiation. - evidence: - - file: "gate/G-1_tiny_ingest.md" - excerpt: "39 buses / 46 branches / 10 generators -- PASS" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pandapower-F07 - category: infrastructure_friction - severity: low - test_ids: [A-3, C-3, C-7] - title: "Solver lock-in to PYPOWER prevents protocol-specified multi-solver comparison" - description: >- - The protocol specifies HiGHS/GLPK for DC OPF tests, but pandapower's rundcopp() - exclusively uses PYPOWER's interior point solver. This is an architectural - property of the tool, not a test infrastructure issue, but it means several - protocol-specified comparisons cannot be performed. C-7 (solver swap) correctly - fails, and C-3 cannot fulfill its multi-solver comparison requirement. - evidence: - - file: "scalability/C-7_solver_swap.md" - excerpt: "pandapower's rundcopp() exclusively uses PYPOWER's built-in interior point solver" - - file: "scalability/C-3_dcopf_scale.md" - excerpt: "HiGHS: N/A -- not available, GLPK: N/A -- not available" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pandapower-F08 - category: missing_verification - severity: medium - test_ids: [A-1, A-2, C-1, C-2] - title: "Peak memory not measured for multiple tests despite being a recorded metric" - description: >- - The protocol requires peak_memory_mb as a recorded metric for scalability tests. - Several test results report "not measured" for peak memory (A-1, A-2, A-3, A-4, - A-7 expressiveness tests). C-1 and C-2 do measure memory, but the expressiveness - tests at MEDIUM scale do not. This creates incomplete data for cross-tool - comparison on memory efficiency. - evidence: - - file: "expressiveness/A-1_dcpf.md" - excerpt: "Peak memory: not measured" - - file: "expressiveness/A-3_dcopf.md" - excerpt: "Peak memory: not measured" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: pandapower-F09 - category: test_design_gap - severity: low - test_ids: [C-2] - title: "AC PF solver deviation from protocol -- Newton-Raphson instead of Ipopt" - description: >- - The protocol specifies Ipopt for C-2 (ACPF at scale), but pandapower uses its - internal Newton-Raphson implementation for AC power flow. This is not a workaround - or limitation per se -- pandapower's NR solver is appropriate for AC PF. However, - it means timing comparisons with tools that use Ipopt for AC PF may not be - apples-to-apples. - evidence: - - file: "scalability/C-2_acpf_scale.md" - excerpt: "Solver deviation: The eval-config specifies Ipopt but pandapower uses its own internal Newton-Raphson" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pandapower-F10 - category: extraordinary_claim - severity: high - test_ids: [P2-3] - title: "PYPOWER solver produces lambda values of 1e25 when generators are decommitted" - description: >- - P2-3 reports that using pandapower's natural decommitment API (in_service=False) - causes the PYPOWER interior point solver to produce lambda values on the order of - 1e25, indicating numerical instability. The workaround (setting max_p_mw=0 instead) - is classified as "stable" but this reveals fundamental solver fragility. The claim - that the solver "diverges numerically" for certain generator configurations on - case39 is significant and should be verified. - evidence: - - file: "p2_readiness/P2-3_commitment_injection_TINY.md" - excerpt: "The solver produces lambda values on the order of 1e25, indicating numerical instability in the interior point method when the generator set changes" - cross_tool_relevance: none - probe_recommended: true - probe_type: convergence_check - proposed_action: null - - - id: pandapower-F11 - category: redundant_test - severity: low - test_ids: [C-4, C-8, C-10] - title: "Three scalability tests fail solely due to upstream expressiveness failures" - description: >- - C-4 (SCUC scale), C-8 (SCOPF scale), and C-10 (distributed slack scale) all fail - because their prerequisite expressiveness tests failed. These results add no new - information beyond what A-5, A-9, and A-11 already established. The cascade is - correctly documented but inflates the fail count without adding signal. - evidence: - - file: "scalability/C-4_scuc_scale.md" - excerpt: "Skipped on MEDIUM. A-5 FAILED on TINY due to architectural limitation" - - file: "scalability/C-8_scopf_scale.md" - excerpt: "blocked by A-9 failure" - - file: "scalability/C-10_distributed_slack_scale.md" - excerpt: "blocked by A-11 failure" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pandapower-F12 - category: missing_verification - severity: medium - test_ids: [A-2, C-2] - title: "ACPF convergence claimed without residual verification" - description: >- - A-2 and C-2 report ACPF convergence on the MEDIUM network with DC warm start, - but neither reports the power mismatch residual at convergence. The results show - bus voltages and flows but do not verify that the Newton-Raphson solver achieved - a specific tolerance. pandapower internally uses a convergence tolerance, but the - achieved residual is not extracted or documented in the results. - evidence: - - file: "expressiveness/A-2_acpf.md" - excerpt: "DC warm start converged: Yes" - - file: "scalability/C-2_acpf_scale.md" - excerpt: "DC warm start converged: Yes" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: pandapower-F13 - category: test_design_gap - severity: low - test_ids: [D-3] - title: "Tutorial verification tests import availability, not execution" - description: >- - D-3 reports 8/8 tutorials pass, but at least two tests (contingency analysis - and plotting) only verify import availability ("API verified") rather than - executing the full tutorial workflow. The plotting test notes "rendering not - tested in headless container." This slightly overstates tutorial completeness. - evidence: - - file: "accessibility/D-3_example_verification.md" - excerpt: "Contingency analysis: PASS -- run_contingency importable (API verified); Plotting: PASS -- simple_plot importable (rendering not tested in headless container)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - -extraordinary_claims: - - test_id: C-6 - claim: "Stochastic DCOPF wrapping approach is a qualified pass despite 2.1% solver convergence rate" - concern: >- - A 97.9% solver failure rate makes the approach practically unusable on the SMALL - network. Calling this a qualified_pass with a "stable" workaround is misleading -- - the loop-based approach works mechanically but produces results for almost no - scenarios. The convergence rate on TINY should be checked to determine whether this - is a SMALL-network-specific solver sensitivity or a general PYPOWER OPF fragility. - evidence_quality: strong - probe_recommended: true - probe_type: convergence_check - - - test_id: B-9 - claim: "PTDF matrix is correctly computed despite 7.43 pu max flow prediction error on MEDIUM" - concern: >- - The pass condition states flow predictions should match within 1e-6 tolerance. - The result attributes the divergence to "shunt elements and tap-ratio effects" - without verifying this explanation. This could indicate a bug in the PTDF - computation, an incorrect bus injection vector reconstruction, or genuine - limitations of the DC PTDF on networks with transformers. A probe should - verify by computing PTDF flows on a subnetwork without transformers. - evidence_quality: moderate - probe_recommended: true - probe_type: formulation_audit - - - test_id: P2-3 - claim: "PYPOWER interior point solver diverges with lambda values ~1e25 when generators decommitted via in_service=False" - concern: >- - This suggests fundamental numerical instability in the bundled OPF solver. If - the solver cannot handle generator decommitment on a 39-bus network, it raises - questions about all pandapower OPF results where the generator set differs from - the default case data. The 2.1% convergence rate on C-6 may be a manifestation - of the same underlying solver fragility. - evidence_quality: strong - probe_recommended: true - probe_type: convergence_check diff --git a/sweep-data/v4-to-v5/per-tool/powermodels/findings.md b/sweep-data/v4-to-v5/per-tool/powermodels/findings.md deleted file mode 100644 index cec8b788..00000000 --- a/sweep-data/v4-to-v5/per-tool/powermodels/findings.md +++ /dev/null @@ -1,335 +0,0 @@ -# PowerModels.jl -- Sweep Findings (v4) - -## Summary - -The PowerModels.jl evaluation is thorough and well-documented across 76 result files with zero gaps. The synthesis report is fair in its characterization of PowerModels as a research-grade steady-state OPF tool with strong core capabilities but significant gaps beyond built-in problem types. The primary sweep concerns are: (1) two scale tests scored as fails without execution, based on extrapolated estimates; (2) the distributed slack validation is defeated by uncongested test networks at both TINY and SMALL scale; (3) SCUC/SCED failures at SMALL conflate open-source solver limitations with tool capability; (4) the A-9 SMALL SCOPF uses 20 contingencies instead of the protocol-specified 50; and (5) timing comparisons across network sizes are unreliable due to Julia JIT cache effects. Three probes are recommended: verifying the C-5/C-8 infeasibility claims with actual execution attempts, validating the PTDF timing anomaly with cold-start measurements, and testing the A-2 MEDIUM ACPF convergence with exposed NLsolve parameters. - -## Finding Details - -### pm-F01: Distributed slack LMP validation defeated by uncongested network - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-11 - -The A-11 test is designed to verify that distributed slack changes LMP decomposition relative to single-slack. On both TINY (IEEE 39-bus) and SMALL (ACTIVSg 2000-bus), the distributed slack formulation is structurally correct but produces identical LMPs to single-slack because neither network has binding flow constraints (no congestion). The TINY result reports: "LMP range: 2.5e-6 (nearly uniform)." The SMALL result reports: "Max LMP difference: 0.0 (identical) ... no binding flow constraints in the LP-relaxed (linearized cost) DC OPF." - -This means the distributed slack mechanism has never been differentially validated in this evaluation. The PTDF transformation `H_dist = H - H * w` is mathematically correct and the weights are settable, but the test cannot confirm that the LMP decomposition responds correctly under congestion -- which is the entire point of distributed slack from an ISO market clearing perspective. - -This is likely a cross-tool issue: if ACTIVSg 2000 has no congestion under linearized costs, other tools' distributed slack tests may face the same limitation. - -**Cross-tool relevance:** likely -**Proposed action:** redesign_test -- Either use a test network with known congestion, or artificially constrain flow limits to force binding constraints. - ---- - -### pm-F02: A-9 SMALL SCOPF uses only 20 of 3,206 contingencies vs protocol's 50 requirement - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-9 - -The eval-config.yaml specifies "SMALL: 50 monitored branches" for A-9 SCOPF. The actual evaluation used only 20 contingencies (the 20 most-loaded branches from the base DC OPF). The result states: "Valid contingencies: 20 (no islanding excluded from top candidates)." The 4.74% security premium and OPTIMAL solve in 41.27s demonstrate SCOPF capability, but at 40% of the specified contingency count. - -This deviation may be justified by practical constraints (model construction took ~70s for 21 networks), but it is not flagged as a protocol deviation in the result file. The qualified_pass status does not reflect this shortfall in contingency count. - -**Cross-tool relevance:** likely -- other tools' SCOPF tests should be checked for the same deviation. -**Proposed action:** adjust_scoring -- Either note the deviation explicitly in the grade narrative or re-run with 50 contingencies. - ---- - -### pm-F03: Scale tests C-5 and C-8 scored as fail without execution - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** C-5, C-8 - -C-5 (N-M contingency sweep at MEDIUM, x=5, m=4) and C-8 (SCOPF at MEDIUM, 500 contingencies) are both recorded as "NOT ATTEMPTED (expected timeout)" and scored as fails. The infeasibility claims are based on extrapolation: - -- C-5 estimates "500-2,000 branches in scope" at BFS depth 5 on 10k-bus, but the actual BFS was never run. The per-solve time of "0.2-0.5s" is estimated from TINY scaling, not measured. -- C-8 estimates "~10,000,000 constraints" for 500 contingencies, but does not attempt iterative contingency screening (which worked at SMALL with 20 contingencies in 41.27s). - -While these estimates are physically plausible, a sweep should flag that these are projections, not measurements. The C-8 estimate is particularly questionable because iterative SCOPF could potentially work if only a small fraction of 500 contingencies are binding. - -**Cross-tool relevance:** likely -- other tools may face the same N-M and SCOPF scale challenges. -**Proposed action:** add_verification -- At minimum, run the BFS at MEDIUM to determine actual scope, and attempt iterative SCOPF with progressive contingency screening. - ---- - -### pm-F04: SCUC/SCED failures conflate tool capability with open-source solver limitations - -**Category:** test_design_gap | **Severity:** high -**Tests:** A-5, A-6, C-4 - -A-5 SMALL fails because HiGHS cannot solve the LP relaxation of a ~500k constraint MILP within 300s single-threaded. A-6 SMALL fails because it depends on A-5. C-4 fails for the same reason as A-5. The evaluation correctly notes "The timeout is a solver capability issue, not a PowerModels expressiveness issue per se," but all three are scored as fails against PowerModels. - -The protocol mandates single-threaded open-source solvers, which creates a fundamental confound at the 2000-bus UC scale. The ~39,168 binary variables and >500,000 constraints exceed what any current open-source single-threaded MIP solver can handle in 300s. This likely affects all tools that must manually assemble SCUC (since PowerModels has no built-in UC), but even tools with built-in SCUC may fail at this scale with these solver constraints. - -The issue is cross-tool confirmed: the protocol's solver and threading constraints define a practical ceiling on SCUC scale that is independent of tool capability. - -**Cross-tool relevance:** confirmed -**Proposed action:** adjust_scoring -- Consider separating the "can the tool express SCUC" question (which PowerModels answers via 140 LOC of JuMP) from the "can open-source solvers solve the resulting MILP at scale" question. - ---- - -### pm-F05: A-8 blocking fail vs B-4 pass creates scoring tension - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** A-8, B-4 - -A-8 tests whether the tool natively supports scenario-indexed stochastic optimization. PowerModels does not -- the multi-network framework has one dimension (time periods) with no scenario indexing, probability weighting, or recourse structure. This is correctly scored as a blocking fail. B-4 tests whether the tool can be wrapped for scenario analysis in a loop. PowerModels passes cleanly: `deepcopy + replicate + solve_mn_opf` runs 20 scenarios x 12 periods at SMALL scale in 281.6s with consistent per-scenario timing. - -The protocol correctly distinguishes these (A-8 = native stochastic, B-4 = wrapping), and the synthesis handles it properly. However, the operational impact is worth noting: from a user's perspective, the B-4 workflow is the practical approach to stochastic analysis regardless of whether the tool has native support. The A-8 fail penalizes PowerModels for lacking a feature that few power system optimization tools provide natively. - -**Cross-tool relevance:** confirmed -- this A-8/B-4 tension likely appears for multiple tools. -**Proposed action:** None -- the protocol distinction is correct, but the grade narrative should acknowledge that B-4 provides operational mitigation. - ---- - -### pm-F06: Lossy DCOPF MATPOWER reference comparison uses mismatched formulations - -**Category:** missing_verification | **Severity:** low -**Tests:** A-10 - -The A-10 protocol requires: "Validate against MATPOWER reference lossy DC OPF solution on same case (tolerance: 1% on total LMP, directional consistency on loss component signs)." The evaluation could not perform this comparison because the MATPOWER reference used lossless DC OPF with post-hoc loss estimation, while PowerModels used DCPLLPowerModel (losses in optimization). The result acknowledges: "Direct 1% LMP comparison is not applicable." - -The loss magnitudes are directionally consistent (0.73% vs 0.78% of load), but the protocol's 1% LMP tolerance validation is unmet. - -**Cross-tool relevance:** confirmed -- other tools will face the same MATPOWER reference comparison challenge if their loss formulations differ from MATPOWER's. -**Proposed action:** redesign_test -- The protocol should specify what MATPOWER reference formulation to use for the lossy DCOPF comparison, or accept same-tool lossless-vs-lossy as the validation. - ---- - -### pm-F07: SCOPF cost comparison uses incompatible cost functions - -**Category:** extraordinary_claim | **Severity:** low -**Tests:** A-9 - -A-9 TINY compares SCOPF objective (1,878.27, linearized cost via HiGHS LP) to unconstrained DC OPF objective (41,263.94, quadratic cost via Ipopt). The 20x cost difference is almost entirely due to the linearization, not the security constraints. The result includes a footnote: "Direct cost comparison is not meaningful because SCOPF uses linearized costs while PowerModels' DC OPF uses the full quadratic cost function." The key validation (dispatch differs) is correct, but the raw numbers could mislead a reader scanning the output. - -**Cross-tool relevance:** likely -**Proposed action:** None -- the result file is transparent about the limitation. - ---- - -### pm-F08: ACPF MEDIUM failure lacks convergence diagnostics - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-2, C-2 - -A-2 MEDIUM reports NLsolve convergence failure on the ACTIVSg 10k-bus network, but provides no diagnostic detail: no iteration count, no residual magnitude, no indication of divergence vs. slow convergence. The result states: "PowerModels does not expose NLsolve iteration count or convergence diagnostics through its API." - -This matters because the failure could be resolvable by adjusting NLsolve parameters (iteration limit, tolerance, algorithm variant), but PowerModels' `compute_ac_pf` API does not expose these controls. The distinction between "NLsolve needs more iterations" and "the problem is fundamentally ill-conditioned" cannot be determined from the available output. - -**Cross-tool relevance:** likely -- other tools' ACPF at 10k-bus should report convergence diagnostics. -**Proposed action:** add_verification -- Attempt ACPF via the Ipopt-based `solve_ac_pf` with a longer timeout, or directly configure NLsolve outside the PowerModels API. - ---- - -### pm-F09: Gate tests have low discriminative value for MATPOWER-native tools - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -PowerModels has a built-in MATPOWER parser. G-1/G-2/G-3 pass trivially with `parse_file()`. These tests primarily differentiate tools that require format conversion (e.g., PyPSA, PowerSimulations). For MATPOWER-native tools (PowerModels, pandapower, MATPOWER itself), the gate tests provide minimal signal. - -**Cross-tool relevance:** confirmed -**Proposed action:** None -- the tests serve their purpose for the full tool set even if individual tools pass trivially. - ---- - -### pm-F10: A-4 MEDIUM AC feasibility check did not complete - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-4 - -A-4 MEDIUM is scored as qualified_pass, but the AC feasibility check never completed. Newton-Raphson failed to converge (118.73s), and the Ipopt fallback was killed after 46 minutes. The test verifies that the 3-line workflow pattern is correct, but it never identifies any voltage or thermal violations because no AC solution was obtained. The pass condition states: "Voltage violations and thermal limit violations identifiable from results" -- this was not achieved at MEDIUM scale. - -The qualified_pass seems generous given that the core pass condition (identifying violations) was not met. The qualification should note that the feasibility check workflow is expressible but produces no usable results at 10k-bus scale. - -**Cross-tool relevance:** likely -- ACPF convergence at 10k-bus is a known challenge. -**Proposed action:** adjust_scoring -- Consider whether a test that never completes its core function should receive qualified_pass. - ---- - -### pm-F11: HiGHS QP failure on ACTIVSg2000 forces solver workarounds - -**Category:** infrastructure_friction | **Severity:** low -**Tests:** A-3, A-10, C-3, C-7, A-11 - -HiGHS QP consistently fails on the ACTIVSg2000 network (returns objective=0.0, primal infeasibility, or times out), affecting A-10 SMALL (lossless comparison objective=0.0), A-11 SMALL (Ipopt fallback required), C-3 MEDIUM (300s timeout), and C-7 MEDIUM (same). Ipopt or cost linearization is used as workaround. This is not a PowerModels issue but it creates inconsistent cost function comparisons across network sizes: TINY uses HiGHS with quadratic costs, while SMALL/MEDIUM requires Ipopt or linearized costs. - -**Cross-tool relevance:** confirmed -- all tools using HiGHS QP may face this at scale. -**Proposed action:** None -- this is a known HiGHS limitation. - ---- - -### pm-F12: N-M contingency sweep conflates combinatorial complexity with tool capability - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-7, C-5 - -The N-M contingency sweep (A-7) demonstrates PowerModels' capability at TINY scale: BFS graph scoping, combinatorial enumeration with pruning, and efficient re-solve via `deepcopy + br_status=0 + compute_dc_pf`. The failure at MEDIUM is due to combinatorial explosion (C(1000,3) = 166M combinations), not a tool API limitation. No tool can enumerate N-4 contingencies on a 10k-bus network in polynomial time. - -Meanwhile, B-3 (N-1 contingency loop) passes at MEDIUM scale with 50 contingencies in 3.43s, demonstrating that the per-contingency workflow scales adequately. The N-M test design punishes tools for mathematical impossibility rather than measuring tool capability at the higher orders. - -**Cross-tool relevance:** confirmed -- N-M combinatorial explosion affects all tools equally. -**Proposed action:** redesign_test -- Consider capping the N-M test at N-2 for MEDIUM, or scoring based on N-1/N-2 performance with N-3+ as informational. - ---- - -### pm-F13: C-4 is a duplicate of A-5 SMALL with no additional measurement - -**Category:** redundant_test | **Severity:** low -**Tests:** C-4, A-5 - -C-4 explicitly states: "The result is derived from the A-5 SMALL test, which demonstrated the same problem." Same script, same wall-clock (494.5s), same fail classification. The protocol calls for C-4 to include "Wall-clock time per solver, MIP gap, peak memory, CPU utilization" with both HiGHS and SCIP. Only HiGHS was tested, no memory/CPU was measured, and SCIP was not attempted "given that HiGHS could not even solve the LP relaxation." - -**Cross-tool relevance:** likely -**Proposed action:** None -- the deduplication is reasonable given the A-5 failure, but the missing SCIP attempt should be noted. - ---- - -### pm-F14: Peak memory not measured on several scalability tests - -**Category:** missing_verification | **Severity:** low -**Tests:** C-2, C-4, C-5, C-6, C-7, C-8 - -Six of ten C-suite tests report `peak_memory_mb: null`. The protocol lists peak_memory as a recorded metric for all C-tests. C-1 (121 MB), C-3 (184 MB), C-9 (2,452 MB), and C-10 (1,017 MB) do report memory, providing useful data points. The gap makes cross-tool memory comparison incomplete for SCUC, stochastic, and contingency tests. - -**Cross-tool relevance:** likely -**Proposed action:** add_verification - ---- - -### pm-F15: C-7 qualified_pass for solver swap mechanism that actually works trivially - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** C-7 - -The C-7 result explicitly states: "Solver swapping is trivial from the API perspective. The qualification is due to solver-specific limitations, not PowerModels API friction." The pass condition asks: "Whether solver swap requires reformulation or just a parameter change." The answer is unambiguously "just a parameter change." The qualified_pass conflates solver performance (HiGHS QP timeout, GLPK/SCIP need linearized costs) with the swap mechanism itself. - -The cost linearization workaround for LP-only solvers could be considered a mild reformulation, which would justify qualified_pass. But the result file attributes the qualification to solver performance, not to the linearization requirement. - -**Cross-tool relevance:** likely -**Proposed action:** adjust_scoring -- Either score as pass with a note about solver-specific cost linearization, or clarify that the qualification is for the linearization requirement (a mild reformulation), not for solver performance. - ---- - -### pm-F16: PTDF computation faster at MEDIUM than SMALL due to JIT warm cache - -**Category:** extraordinary_claim | **Severity:** low -**Tests:** C-10 - -C-10 reports PTDF computation taking 3.03s at 10k-bus vs ~20s at 2k-bus (SMALL). The PTDF matrix at MEDIUM has 127M elements vs ~6.4M at SMALL (20x larger), so computation should take longer, not less. The result attributes this to "warm JIT cache from prior Julia session operations," which is plausible for Julia but means the SMALL measurement includes JIT compilation overhead while MEDIUM does not. - -This makes the scaling analysis table in C-10 unreliable. The 0.15x ratio for PTDF compute (MEDIUM/SMALL) cannot be used for cross-tool scaling comparisons without noting the JIT confound. - -**Cross-tool relevance:** none -- Julia-specific issue. -**Proposed action:** add_verification -- Measure PTDF computation at MEDIUM with a cold Julia session for accurate scaling comparison. - ---- - -## Extraordinary Claims - -### C-5: N-M contingency sweep at MEDIUM would take 28-70 hours - -**Concern:** The estimate projects TINY timing (0.0014s per DCPF solve) to MEDIUM scale (0.2-0.5s per solve) and combines it with an estimated BFS scope of 500-2,000 branches. Neither the per-solve time at MEDIUM nor the actual BFS scope were measured. The BFS scope estimate has a 4x range, and the per-solve time estimate has a 2.5x range, producing a combined uncertainty of 10x on the total runtime projection. - -**Evidence quality:** weak - -A probe should: (1) run BFS from a representative seed bus at depth 5 on ACTIVSg 10k to determine actual scope, (2) measure per-DCPF solve time at 10k-bus with a cold JIT, and (3) run the N-1 portion to validate the per-solve estimate. N-2+ can remain projected. - -### C-8: SCOPF at MEDIUM with 500 contingencies exceeds solver capacity - -**Concern:** The constraint count estimate (~10M) assumes full N-1 SCOPF without iterative screening. At SMALL scale, iterative SCOPF with 20 contingencies solved in 41.27s. It is plausible that iterative screening at MEDIUM would converge with a manageable number of binding contingencies (typical SCOPF experience is 5-15% binding rate). The estimate does not attempt this approach. - -**Evidence quality:** moderate - -A probe should: attempt iterative SCOPF at MEDIUM starting with a small contingency set (e.g., 20 most-loaded branches), screening for violations, and progressively adding contingencies until convergence or timeout. - -### C-10: PTDF computation 6.6x faster at MEDIUM than SMALL - -**Concern:** This violates expected computational scaling. Julia JIT compilation can produce significant first-call overhead, making subsequent calls in the same session much faster. If the SMALL PTDF was the first large computation in that session while the MEDIUM PTDF ran after warm-up, the comparison is invalid. - -**Evidence quality:** weak - -A probe should: measure PTDF computation at both SMALL and MEDIUM in fresh Julia sessions with no prior computations, ensuring JIT overhead is included consistently or excluded consistently. - -## Test Outcome Matrix - -| Test ID | Network | Status | Workaround | Key Issue | -|---------|---------|--------|------------|-----------| -| G-1 | TINY | pass | -- | -- | -| G-2 | SMALL | pass | -- | -- | -| G-3 | MEDIUM | pass | -- | -- | -| A-1 | TINY | pass | -- | -- | -| A-1 | MEDIUM | pass | -- | -- | -| A-2 | TINY | pass | -- | -- | -| A-2 | MEDIUM | fail | -- | NLsolve convergence failure, no diagnostics exposed | -| A-3 | TINY | pass | -- | -- | -| A-3 | MEDIUM | pass | stable (data fix) | 1,349 gens needed cost array fixes | -| A-4 | TINY | pass | -- | -- | -| A-4 | MEDIUM | qualified_pass | stable | AC PF never completed; workflow expressible but no results | -| A-5 | TINY | qualified_pass | stable | No native SCUC; ~140 LOC JuMP; MIP gap 0.0% | -| A-5 | SMALL | fail | stable | HiGHS MIP timeout; solver limitation not tool limitation | -| A-6 | TINY | qualified_pass | stable | No native SCED; ~200 LOC JuMP; ramp constraints verified | -| A-6 | SMALL | fail | stable (blocked) | Blocked by A-5 SMALL failure | -| A-7 | TINY | qualified_pass | stable | No native graph/sweep; 1,561 solves in 2.25s | -| A-7 | MEDIUM | fail | stable | Combinatorial explosion; not a tool limitation | -| A-8 | TINY | fail | blocking | No native stochastic; architectural limitation | -| A-8 | SMALL | fail | blocking | Same limitation as TINY | -| A-9 | TINY | qualified_pass | stable | No native SCOPF; ~180 LOC JuMP; 1.5x rating relaxation | -| A-9 | SMALL | qualified_pass | stable | 20 contingencies (protocol: 50); 4.74% security premium | -| A-10 | TINY | qualified_pass | stable | DCPLLPowerModel requires Ipopt (QCQP); LMP decomposition manual | -| A-10 | SMALL | qualified_pass | stable | Ipopt required; HiGHS QP objective=0 on ACTIVSg2000 | -| A-11 | TINY | qualified_pass | stable | No native distributed slack; ~150 LOC; no congestion to validate | -| A-11 | SMALL | qualified_pass | stable | ~350 LOC; no congestion; LMPs identical | -| B-1 | TINY | pass | -- | -- | -| B-1 | MEDIUM | pass | -- | -- | -| B-2 | TINY | qualified_pass | stable | ~15 LOC manual adjacency graph | -| B-2 | MEDIUM | qualified_pass | stable | Same approach scales | -| B-3 | TINY | pass | -- | 46 N-1 in 0.22s | -| B-3 | MEDIUM | pass | -- | 50 N-1 in 3.43s | -| B-4 | TINY | pass | -- | Scenario wrapping clean | -| B-4 | SMALL | pass | -- | 20 scenarios x 12 periods in 281.6s | -| B-5 | TINY | pass | -- | < 5 lines to DataFrame/CSV | -| B-5 | MEDIUM | pass | -- | -- | -| B-6 | N/A | pass | -- | Clean 4-layer architecture | -| B-7 | TINY | pass | -- | AC feasibility clean 3-line workflow | -| B-7 | MEDIUM | pass | -- | -- | -| B-8 | TINY | pass | stable | Ref bus via data dict mod + re-solve | -| B-8 | SMALL | pass | stable | -- | -| B-9 | TINY | pass | -- | Native PTDF; error < 1e-11 | -| B-9 | MEDIUM | pass | -- | 127M elements in 3.73s | -| C-1 | MEDIUM | pass | -- | 0.234s solve, 121 MB | -| C-2 | MEDIUM | fail | -- | NLsolve convergence failure | -| C-3 | MEDIUM | pass | stable | Ipopt 3.13s; HiGHS QP timeout; GLPK 50.91s | -| C-4 | SMALL | fail | stable | Duplicate of A-5 SMALL; SCIP not attempted | -| C-5 | MEDIUM | fail | stable | Not executed; projected infeasibility | -| C-6 | SMALL | pass | -- | 281.6s for 20 scenarios | -| C-7 | MEDIUM | qualified_pass | stable | Swap trivial; qualification is solver performance | -| C-8 | MEDIUM | fail | stable | Not executed; projected infeasibility | -| C-9 | MEDIUM | pass | -- | 6.65s, 2,452 MB | -| C-10 | MEDIUM | qualified_pass | stable | 12.2s, 1,017 MB; PTDF timing anomaly | -| D-1 | N/A | qualified_pass | -- | 5.2s load-to-solve; Julia startup overhead | -| D-2 | N/A | qualified_pass | -- | 4/11 from docs alone; 4/11 no docs | -| D-3 | N/A | qualified_pass | -- | All examples work; tutorial 7 years old | -| D-4 | N/A | qualified_pass | -- | 1/3 good, 2/3 poor error messages | -| D-5 | N/A | informational | -- | Built-in ~100 LOC, manual ~269 LOC mean | -| E-1 | N/A | pass | -- | 5-7 releases in 24 months | -| E-2 | N/A | qualified_pass | -- | 24 commits/yr, 3 human committers | -| E-3 | N/A | fail | -- | Bus factor = 1 (ccoffrin 82.4%) | -| E-4 | N/A | informational | -- | DOE/LANL funding; no commercial backing | -| E-5 | N/A | qualified_pass | -- | 81-day median close; batch triage | -| E-6 | N/A | pass | -- | 94% coverage, cross-platform CI | -| E-7 | N/A | fail | -- | No operational deployment evidence | -| F-1 | N/A | pass | -- | BSD-3-Clause | -| F-2 | N/A | informational | -- | 114 packages | -| F-3 | N/A | qualified_pass | -- | GLPK GPL-3.0 and SCIP ZIB optional | -| F-4 | N/A | pass | -- | Pure Julia core | -| F-5 | N/A | pass | -- | Full path inspectable | -| F-6 | N/A | pass | -- | Content-addressed hashes | -| F-7 | N/A | pass | -- | Airgap via depot cloning | -| F-8 | N/A | pass | -- | HiGHS + Ipopt cover all formulations | -| F-9 | N/A | pass | -- | No mutable URLs | -| P2-1 | N/A | informational | -- | Built-in v33 parser; v34+ unsupported | -| P2-2 | TINY | informational | -- | Native PWL; lambda formulation; LP-compatible | -| P2-3 | TINY | informational | -- | Steps 2-4 trivial; Step 1 needs manual SCUC | diff --git a/sweep-data/v4-to-v5/per-tool/powermodels/findings.yaml b/sweep-data/v4-to-v5/per-tool/powermodels/findings.yaml deleted file mode 100644 index 914ae36b..00000000 --- a/sweep-data/v4-to-v5/per-tool/powermodels/findings.yaml +++ /dev/null @@ -1,356 +0,0 @@ -tool: powermodels -source_version: "v4" -timestamp: "2026-03-07T12:00:00Z" -evaluation_summary: - total_tests: 76 - pass: 30 - fail: 12 - qualified_pass: 28 - informational: 6 - -findings: - - id: pm-F01 - category: network_insufficiency - severity: medium - test_ids: [A-11] - title: "Distributed slack LMP validation defeated by uncongested network" - description: > - A-11 on both TINY and SMALL produces identical LMPs for single-slack and distributed-slack - because neither network has binding flow constraints. The distributed slack mechanism is - structurally implemented but never differentially validated -- the test cannot confirm that - LMP decomposition responds correctly to the slack distribution under congestion. - evidence: - - file: expressiveness/A-11_distributed_slack_opf_TINY.md - excerpt: "LMP range: 2.5e-6 (nearly uniform) ... The IEEE 39-bus network under uncongested conditions produces nearly uniform LMPs regardless of slack distribution." - - file: expressiveness/A-11_distributed_slack_opf_SMALL.md - excerpt: "Max LMP difference: 0.0 (identical) ... no binding flow constraints in the LP-relaxed (linearized cost) DC OPF" - cross_tool_relevance: likely - probe_recommended: true - probe_type: formulation_audit - proposed_action: redesign_test - - - id: pm-F02 - category: misleading_result - severity: medium - test_ids: [A-9] - title: "A-9 SMALL SCOPF uses only 20 of 3,206 contingencies vs protocol's 50 requirement" - description: > - The protocol specifies 50 monitored branches for SMALL-scale SCOPF. The evaluation used - only 20 contingencies (the 20 most-loaded branches). While the SCOPF formulation is correct - and produces a meaningful security premium (4.74%), the reduced contingency count means the - test does not meet the stated protocol parameters, potentially understating the difficulty - of SCOPF at scale. - evidence: - - file: expressiveness/A-9_scopf_SMALL.md - excerpt: "Valid contingencies: 20 (no islanding excluded from top candidates)" - - file: eval-config.yaml - excerpt: "A-9 ... SMALL: 50 monitored branches" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F03 - category: extraordinary_claim - severity: medium - test_ids: [C-5, C-8] - title: "Scale tests C-5 and C-8 scored as fail without execution" - description: > - C-5 (contingency sweep MEDIUM) and C-8 (SCOPF MEDIUM) were recorded as fails based on - projected infeasibility estimates, not actual execution. While the estimates are - physically plausible, the results contain no runtime evidence. The combinatorial - estimates in C-5 assume worst-case BFS scope without verifying the actual branch count - at depth 5 on the 10k-bus network. - evidence: - - file: scalability/C-5_contingency_sweep_scale_MEDIUM.md - excerpt: "Result: NOT ATTEMPTED (expected timeout) ... Estimated 500-2,000 branches in scope (depending on seed bus connectivity)" - - file: scalability/C-8_scopf_scale_MEDIUM.md - excerpt: "Result: NOT ATTEMPTED (expected timeout) ... ~10,000,000 constraints and ~11,500,000 variables" - cross_tool_relevance: likely - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - - - id: pm-F04 - category: test_design_gap - severity: high - test_ids: [A-5, A-6, C-4] - title: "SCUC/SCED failures conflate tool capability with open-source solver limitations" - description: > - A-5 SMALL, A-6 SMALL, and C-4 fail because HiGHS cannot solve the LP relaxation of a - ~500k constraint MILP within 300s. The evaluation correctly notes this is a solver - limitation not a PowerModels limitation, but the tests are scored as fails against - PowerModels. The protocol requires single-threaded open-source solvers, which creates - a confound: the test measures solver capability at scale rather than tool expressiveness - or scalability. - evidence: - - file: expressiveness/A-5_scuc_SMALL.md - excerpt: "The timeout is a solver capability issue, not a PowerModels expressiveness issue per se." - - file: scalability/C-4_scuc_scale_SMALL.md - excerpt: "HiGHS MIP is not competitive with commercial solvers (Gurobi, CPLEX) at this scale" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F05 - category: scoring_inconsistency - severity: medium - test_ids: [A-8, B-4] - title: "A-8 blocking fail vs B-4 pass creates scoring tension" - description: > - A-8 is a blocking fail because PowerModels lacks native stochastic optimization (scenario - indexing, probability weighting, recourse structure). B-4 passes because the scenario - wrapping loop works cleanly. The protocol distinguishes these correctly (A-8 tests native - stochastic support, B-4 tests wrapping capability), but the grade impact is asymmetric: - A-8 fail pulls Expressiveness down significantly while B-4 pass contributes to a strong - Extensibility grade. The synthesis correctly handles this, but the extreme score - divergence on what is operationally the same workflow warrants attention. - evidence: - - file: expressiveness/A-8_stochastic_timeseries_TINY.md - excerpt: "PowerModels.jl does not natively support scenario-indexed stochastic optimization" - - file: extensibility/B-4_stochastic_wrapping_SMALL.md - excerpt: "Result: PASS ... Same deepcopy + replicate + solve_mn_opf pattern from TINY works identically at SMALL scale" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pm-F06 - category: missing_verification - severity: low - test_ids: [A-10] - title: "Lossy DCOPF MATPOWER reference comparison uses mismatched formulations" - description: > - The A-10 protocol requires validation against MATPOWER reference lossy DC OPF solution - with 1% LMP tolerance. The evaluation could not perform this comparison because the - MATPOWER reference used lossless DC OPF with post-hoc loss estimation, while PowerModels - used DCPLLPowerModel (losses in optimization). The result file acknowledges this - mismatch but the protocol's validation requirement is effectively unmet. - evidence: - - file: expressiveness/A-10_lossy_dcopf_lmp_TINY.md - excerpt: "Direct 1% LMP comparison is not applicable because MATPOWER used lossless DC OPF with post-hoc loss estimation" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pm-F07 - category: extraordinary_claim - severity: low - test_ids: [A-9] - title: "SCOPF cost comparison uses incompatible cost functions" - description: > - A-9 TINY compares SCOPF objective (1,878.27, linearized cost) to unconstrained DC OPF - objective (41,263.94, quadratic cost). The result acknowledges this mismatch and focuses - on dispatch differences rather than cost differences, but the cost comparison numbers in - the output could be misinterpreted by a reader who does not notice the footnote. - evidence: - - file: expressiveness/A-9_scopf_TINY.md - excerpt: "Direct cost comparison is not meaningful because SCOPF uses linearized costs while PowerModels' DC OPF uses the full quadratic cost function" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pm-F08 - category: missing_verification - severity: medium - test_ids: [A-2, C-2] - title: "ACPF MEDIUM failure lacks convergence diagnostics" - description: > - A-2 MEDIUM (and C-2 by reference) reports ACPF failure at 10k-bus but provides no - convergence diagnostics -- no iteration count, no residual trajectory, no indication - of whether NLsolve was diverging, oscillating, or simply running out of iterations. - PowerModels does not expose NLsolve configuration, making it impossible to determine - whether a tuning adjustment would resolve the convergence issue. - evidence: - - file: expressiveness/A-2_acpf_MEDIUM.md - excerpt: "PowerModels does not expose NLsolve iteration count or convergence diagnostics through its API" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: pm-F09 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate tests have low discriminative value for MATPOWER-native tools" - description: > - PowerModels has a native MATPOWER parser, so G-1/G-2/G-3 are trivially passed with - parse_file(). These tests have minimal discriminative value for tools with native - .m format support. The gate tests primarily differentiate tools that require format - conversion. - evidence: - - file: gate/G-1_ingest_tiny.md - excerpt: "Load time: 0.738s ... All checks passed" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pm-F10 - category: missing_verification - severity: medium - test_ids: [A-4] - title: "A-4 MEDIUM AC feasibility check did not complete" - description: > - A-4 MEDIUM is scored as qualified_pass but neither the Newton-Raphson AC PF nor the - Ipopt AC PF fallback completed successfully. The NR failed to converge, and the Ipopt - solve was killed after 46 minutes. The test demonstrates that the workflow is - expressible (3-line pattern) but never actually verifies AC feasibility at MEDIUM scale. - No voltage violations or thermal violations were identified because no AC solution was - obtained. - evidence: - - file: expressiveness/A-4_ac_feasibility_MEDIUM.md - excerpt: "compute_ac_pf(data): did not converge (118.73s on flat start) ... solve_ac_pf(data, Ipopt): 23,874-variable NLP did not complete within 46 minutes (killed)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F11 - category: infrastructure_friction - severity: low - test_ids: [A-3, A-10, C-3, C-7, A-11] - title: "HiGHS QP failure on ACTIVSg2000 forces solver workarounds" - description: > - HiGHS QP consistently fails on the ACTIVSg2000 network (returns objective=0 or times - out), forcing Ipopt as a fallback or cost linearization as a workaround. This is a - solver-infrastructure issue that affects multiple tests, creating inconsistent cost - function comparisons across network sizes and complicating cross-tool comparison. - evidence: - - file: expressiveness/A-10_lossy_dcopf_lmp_SMALL.md - excerpt: "Lossless DC OPF objective: 0.0 (HiGHS QP failed on this network)" - - file: scalability/C-3_dcopf_scale_MEDIUM.md - excerpt: "HiGHS: TIME_LIMIT, 300.0s, QP ASM solver, 113k iterations at timeout" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pm-F12 - category: test_design_gap - severity: medium - test_ids: [A-7, C-5] - title: "N-M contingency sweep conflates combinatorial complexity with tool capability" - description: > - A-7 TINY passes (1,561 solves in 2.25s) but A-7 MEDIUM and C-5 fail due to - combinatorial explosion (millions of solve combinations). The test measures the - mathematical complexity of N-M enumeration at scale rather than the tool's - contingency analysis capability. N-1 contingency analysis (B-3) scales adequately - to MEDIUM (50 contingencies in 3.43s). The test design conflates a problem that no - tool can solve at high order with tool-specific limitations. - evidence: - - file: expressiveness/A-7_contingency_sweep_MEDIUM.md - excerpt: "Result: FAIL ... 333 lines of code ... infeasible at MEDIUM scale due to combinatorial explosion" - - file: extensibility/B-3_contingency_loop_MEDIUM.md - excerpt: "pass ... 50 N-1 in 3.43s on MEDIUM" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pm-F13 - category: redundant_test - severity: low - test_ids: [C-4, A-5] - title: "C-4 is a duplicate of A-5 SMALL with no additional measurement" - description: > - C-4 references the identical execution as A-5 SMALL (same script, same wall-clock, - same result). No additional scalability-specific measurements (CPU utilization, memory - profiling, parallel solver options) were collected. SCIP was also not attempted for - C-4 despite the protocol requiring it. - evidence: - - file: scalability/C-4_scuc_scale_SMALL.md - excerpt: "This test measures the scalability of 24-hour SCUC ... The result is derived from the A-5 SMALL test" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pm-F14 - category: missing_verification - severity: low - test_ids: [C-1, C-2, C-3, C-6, C-7, C-9, C-10] - title: "Peak memory not measured on several scalability tests" - description: > - Several scalability tests report peak_memory_mb as null despite it being a recorded - metric in the protocol. C-1, C-3, C-9, and C-10 do report memory, but C-2, C-4, C-5, - C-6, C-7, and C-8 do not. This makes cross-tool memory comparison incomplete. - evidence: - - file: scalability/C-6_stochastic_scale_SMALL.md - excerpt: "peak_memory_mb: null" - - file: scalability/C-7_solver_swap_MEDIUM.md - excerpt: "peak_memory_mb: null" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: pm-F15 - category: scoring_inconsistency - severity: low - test_ids: [C-7] - title: "C-7 qualified_pass for solver swap mechanism that actually works trivially" - description: > - C-7 is scored as qualified_pass because some solvers struggle with the QP at 10k-bus - scale. But the test's pass condition is about whether solver swap requires - reformulation. The swap mechanism itself is a one-line change with no reformulation. - The qualification conflates solver performance with swap mechanism quality. - evidence: - - file: scalability/C-7_solver_swap_MEDIUM.md - excerpt: "Solver swapping is trivial from the API perspective. The qualification is due to solver-specific limitations, not PowerModels API friction." - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pm-F16 - category: extraordinary_claim - severity: low - test_ids: [C-10] - title: "PTDF computation faster at MEDIUM than SMALL due to JIT warm cache" - description: > - C-10 reports PTDF computation at MEDIUM (10k-bus) taking 3.03s vs ~20s at SMALL - (2k-bus). The result attributes this to JIT warm cache from prior operations. This - timing anomaly makes scaling analysis unreliable -- the SMALL measurement likely - includes JIT compilation overhead while the MEDIUM measurement does not. - evidence: - - file: scalability/C-10_distributed_slack_scale_MEDIUM.md - excerpt: "PTDF compute: ~20s [SMALL] vs 3.03s [MEDIUM] ... warm JIT cache from prior Julia session operations" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - -extraordinary_claims: - - test_id: C-5 - claim: "N-M contingency sweep at MEDIUM would take 28-70 hours based on extrapolation" - concern: > - The estimate assumes 0.2-0.5s per DCPF solve at 10k-bus and 500-2,000 branches in BFS - scope, but neither the actual BFS scope nor the per-solve time were measured at MEDIUM - scale. The estimate is an order-of-magnitude projection, not a measurement. - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification - - - test_id: C-8 - claim: "SCOPF at MEDIUM with 500 contingencies would produce ~10M constraints, exceeding solver capacity" - concern: > - The constraint count estimate is reasonable but was not verified by attempting model - construction. It is possible that iterative contingency screening (solve base OPF, - add violated contingencies) could produce a feasible workflow, as was done at SMALL scale. - evidence_quality: moderate - probe_recommended: true - probe_type: timing_verification - - - test_id: C-10 - claim: "PTDF computation 6.6x faster at 10k-bus than 2k-bus" - concern: > - This violates expected O(n^2) to O(n^3) scaling. Attributed to JIT cache effects but - not verified with a cold-start measurement at MEDIUM scale. - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification diff --git a/sweep-data/v4-to-v5/per-tool/powersimulations/findings.md b/sweep-data/v4-to-v5/per-tool/powersimulations/findings.md deleted file mode 100644 index 8663c0cc..00000000 --- a/sweep-data/v4-to-v5/per-tool/powersimulations/findings.md +++ /dev/null @@ -1,291 +0,0 @@ -# PowerSimulations.jl -- Sweep Findings (v4) - -## Summary - -The PowerSimulations.jl evaluation is thorough in coverage (all 57 test IDs have result files) and generally well-documented. The most significant quality concern is that four scalability tests (C-3, C-4, C-5, C-6) report only estimated timings without actual execution, which undermines the scalability grade's evidentiary basis. A potential reporting error in the A-4 unit mismatch finding -- where dispatch values labeled as "pu" in A-4 appear to actually be MW based on A-3's consistent labeling -- warrants verification since this finding cascades into multiple grades (expressiveness, extensibility, P2-readiness). Several tests operate on networks or configurations that are too small or uncongested to exercise the feature being tested (SCUC with no cycling, distributed slack with no congestion, SCOPF with 70% of contingencies filtered). Six probes are recommended, primarily for timing verification of unmeasured scalability claims. - -## Finding Details - -### psi-F01: Four scalability tests report estimated timings without actual measurement - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** C-3, C-4, C-5, C-6 - -C-3 (DCOPF scale on MEDIUM) states "HiGHS is expected to solve the MEDIUM DCOPF in < 60s" but `wall_clock_seconds: null` in its frontmatter. The result file mentions a test script but notes it "requires fix for RenewableDispatch accessor -- script bug, not tool limitation," suggesting the test was not successfully run. C-4 (SCUC on SMALL) provides only "Expected solve time | >300s (estimated)" and has no measured timing. C-5 (contingency sweep on MEDIUM) explicitly states "Wall-clock time not measured at MEDIUM scale" and references only the TINY-scale functional test. C-6 (stochastic on SMALL) says "Not measured at SMALL scale. Estimated 10-20 minutes." - -The protocol is explicit: "Record everything. For each test, record: Wall-clock time (for scalability-relevant tests)." Four of ten scalability tests lack this mandatory data. These tests received `qualified_pass` status based on extrapolation from TINY results and theoretical analysis rather than measured evidence. This substantially weakens the scalability grade's confidence, as the synthesis itself acknowledges: "Several scalability tests lack measured wall-clock times (C-3, C-4, C-5, C-6 provide estimates only)." - -**Cross-tool relevance:** likely -- other tool evaluations may have similar unmeasured estimates. -**Proposed action:** add_verification -- these tests should be re-run with actual measurements. - -### psi-F02: Unit mismatch between PSI dispatch output and PowerSystems limits -- possible reporting error - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** A-4, B-7 - -This is the single most impactful finding in the evaluation, affecting the expressiveness, extensibility, and P2-readiness grades. The claim is that PSI's `read_variables()` returns dispatch values ~100x larger than `get_active_power_limits()`. However, comparing A-3 and A-4 reveals an inconsistency in the test reports themselves. - -In A-3's dispatch table, gen-1 shows "660.85 MW" with "Pmax 1040.0 MW" -- consistent MW units, no mismatch. In A-4's dispatch table, gen-1 shows "660.85 pu" with "Dispatch (MW) 66,085" and "Pmax (pu) 10.40" -- here the same 660.85 value is labeled as pu rather than MW, and the MW column multiplies by 100 (system base MVA). The 660.85 value is the same in both tests, but A-3 calls it MW and A-4 calls it pu. - -If gen-1's Pmax is 1040 MW (as shown in A-3) and the dispatch is 660.85 MW (as shown in A-3), there is no unit mismatch. But if gen-1's Pmax in the System is stored as 10.40 pu (on 100 MVA base), then PSI returning 660.85 would indeed be a ~63.5x mismatch. The question is: does `get_active_power_limits()` return values in pu (10.40) or MW (1040.0)? The B-7 result says "dispatch values from PSI are ~100x larger than Pmax values from the System" which suggests the System returns pu values. But this needs verification because the cascade of this finding is significant: it drives the "fragile" workaround classification for B-7, the qualified_pass for A-4, and the "partially ready" assessment for P2-3. - -**Cross-tool relevance:** none -- this is specific to PSI's internal unit conventions. -**Proposed action:** add_verification -- re-run the dispatch-to-ACPF transfer with explicit unit logging. - -### psi-F03: Custom constraint dual value verified only for non-binding case - -**Category:** missing_verification | **Severity:** medium -**Tests:** B-1 - -B-1 successfully demonstrates JuMP model access and constraint injection via `@constraint`, which is a genuine strength. However, the flow gate was set at 80% of the unconstrained absolute flow sum, and the signed flow sum (-5.54 pu) was well below the limit (8.33 pu). The constraint was non-binding, so the dual was zero. A second test with a "very loose" limit of 100.0 pu also produced dual = 0.0. Neither test demonstrated a binding constraint with a non-zero dual value. - -The protocol's pass condition states: "Dual value of custom constraint extractable and correctly reflects binding status." While zero-dual-for-non-binding is correct, the positive case (non-zero dual for binding constraint) was never tested. The result even acknowledges this gap. A tighter gate limit that forces the constraint to bind would complete the verification. - -**Cross-tool relevance:** likely -- other tools may also test only non-binding constraints. -**Proposed action:** add_verification -- re-run with a gate limit that forces binding. - -### psi-F04: A-7 scored as pass despite requiring stable workaround - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** A-6, A-7 - -A-7 (contingency sweep) has `status: pass` in its frontmatter but `workaround_class: stable`. The workarounds include manual adjacency graph construction (~40 LOC), manual BFS implementation, and using PowerFlows.jl instead of PSI's DecisionModel. These are all stable workarounds using public API, but the protocol convention states that tests requiring workarounds should be `qualified_pass`, not `pass`. The synthesis itself flags this: "Both have status: pass but workaround_class: stable. Convention suggests qualified_pass." - -A-6 has `status: qualified_pass` in its frontmatter, which is consistent with its stable workaround. The inconsistency is with A-7 only. - -**Cross-tool relevance:** likely -- workaround-vs-pass classification may be inconsistent across tools. -**Proposed action:** adjust_scoring -- A-7 should be qualified_pass. - -### psi-F05: SCUC on 39-bus produces no unit cycling - -**Category:** network_insufficiency | **Severity:** medium -**Tests:** A-5 - -The 39-bus case has 10 generators with total capacity closely matching peak load, so the SCUC solution has all generators committed ON for all 24 hours. This means the test does not exercise the core UC features: min up/down time constraints never bind, startup costs are never incurred, and shutdown decisions never occur. The SCUC degenerates to a dispatch problem with binary variables that are all trivially 1. - -This is a known limitation of the IEEE 39-bus case for UC testing and affects all tools equally. The evaluation correctly identifies this ("expected for a system where total capacity closely matches peak load") but still grades the test as qualified_pass. The SCUC formulation features (built-in commitment variables, startup/shutdown modeling, ramp constraints) are verified as present in the formulation, but their correctness under binding conditions is not tested. - -**Cross-tool relevance:** confirmed -- this affects all tools using case39 for SCUC. -**Proposed action:** redesign_test -- consider augmenting the case39 generator fleet with excess capacity to force cycling, or use a different test case. - -### psi-F06: Distributed slack test cannot discriminate on uncongested network - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-11 - -A-11 compares PTDFPowerModel and DCPPowerModel on the uncongested case39 network. Both produce numerically identical dispatch (max difference 1.5e-5 MW) and objectives (difference 3.8e-13). The test cannot verify that the PTDF formulation actually distributes slack differently because there is no congestion to reveal the difference. - -Furthermore, the pass condition requires "distributed slack weights are settable via API." The result explicitly states "Weights are not configurable." The qualified_pass is based on the argument that PTDF is "inherently" a distributed slack formulation, but this is an architectural claim about the mathematical formulation, not a demonstrated capability difference. On a congested network, PTDF produces a single system price while DCP produces differentiated nodal LMPs -- the formulations would diverge, but this was not tested. - -**Cross-tool relevance:** confirmed -- this network limitation affects all tools using case39 for distributed slack testing. -**Proposed action:** redesign_test -- use a congested network or add artificial congestion to demonstrate formulation differences. - -### psi-F07: A-6 SCED qualified_pass is generous given ramp enforcement failure - -**Category:** scoring_inconsistency | **Severity:** low -**Tests:** A-6 - -The pass condition for A-6 explicitly requires "ramp rate constraints are demonstrably enforced between consecutive dispatch intervals in the ED stage." The test demonstrates 182 ramp violations with ThermalBasicDispatch (which has no ramp constraints by design), and ThermalRampLimited (which would enforce ramps) fails to build due to HiGHS initial condition issues. The core pass condition is unmet. - -The qualified_pass is reasonable in that the tool has a formulation that should enforce ramps (`ThermalRampLimited`) but it cannot be demonstrated due to a solver interaction issue. The result honestly documents the failure. However, the distinction between "feature exists but doesn't work" and "feature doesn't exist" is thin from the user's perspective. - -**Cross-tool relevance:** likely -- other tools may have similar edge cases where features exist but cannot be demonstrated. -**Proposed action:** adjust_scoring -- consider whether this should be scored as fail with a note about the ThermalRampLimited formulation's existence. - -### psi-F08: SCOPF contingency filtering removes 70% of contingencies - -**Category:** test_design_gap | **Severity:** medium -**Tests:** A-9 - -The SCOPF test filters out 32 of 46 contingencies as "near-radial" using a max |LODF| > 0.9 threshold, leaving only 14 contingencies and 156 additional constraints. The 0.51% cost increase over baseline is modest. While near-radial contingency filtering is standard practice in production SCOPF, the aggressiveness of the filter on this small network (removing 70%) makes it difficult to assess whether the approach would work with a realistic contingency set. The resulting problem (56 variables, ~200 constraints) is trivial for HiGHS. - -The test does demonstrate that contingency constraints affect dispatch (gen-1 increases by 282 MW, gen-3 decreases by 135 MW), confirming the constraints are operational. The approach is sound in principle, but the test's discriminative value is limited by the small remaining contingency set. - -**Cross-tool relevance:** confirmed -- all tools face the same case39 near-radial topology issue. -**Proposed action:** none -- the finding is acknowledged and documented. - -### psi-F09: Mandatory time series boilerplate inflates LOC and qualified_pass count - -**Category:** infrastructure_friction | **Severity:** low -**Tests:** A-3, A-4, A-5, A-6, A-8, A-9, A-10, A-11, B-4 - -PSI is a multi-period simulation framework, not a single-period OPF tool. The ~30 LOC of time series boilerplate required for every optimization test is a genuine design characteristic, not an artifact of the evaluation. However, it produces a systematic pattern: 8 of 9 optimization tests require identical boilerplate and receive qualified_pass partly because of it. This inflates the qualified_pass count and may make PSI's expressiveness grade appear worse than if the test protocol accounted for framework paradigm differences. - -The time series requirement is stable, well-documented (though not as a standalone recipe), and reflects the tool's intended use case. It is a legitimate accessibility and expressiveness finding but should be weighted as a single recurring friction pattern rather than eight independent workarounds. - -**Cross-tool relevance:** none -- this is specific to PSI's architecture. -**Proposed action:** none -- the synthesis already handles this correctly by describing it as a systematic pattern. - -### psi-F10: ACPF scale failure lacks convergence diagnostics - -**Category:** missing_verification | **Severity:** medium -**Tests:** C-2 - -C-2 reports ACPF failure on MEDIUM with only the error message "The NewtonRaphsonACPowerFlow solver failed to converge." No iteration count, residual history, or convergence trajectory is provided. The root cause analysis speculates about missing robustness features but does not verify: -- How many NR iterations were attempted before failure -- What the final residual was -- Whether manually setting voltages from a DCPF solution would improve convergence -- Whether Ipopt as an alternative NR solver could converge - -The result states "PowerFlows.jl does not support DC warm start initialization" but does not confirm whether voltage setpoints could be manually set via `set_voltage!()` before calling ACPF. This would be a straightforward check. - -**Cross-tool relevance:** likely -- ACPF convergence on MEDIUM is a common challenge across tools. -**Proposed action:** add_verification -- re-run with iteration/residual logging and attempt manual warm start. - -### psi-F11: Gate tests are low-signal across tools - -**Category:** low_signal | **Severity:** low -**Tests:** G-1, G-2, G-3 - -Gate tests serve their intended screening purpose but produce identical outcomes for any tool with a functional MATPOWER parser. All counts match, all three networks load successfully. This is expected and by design. No action needed. - -**Cross-tool relevance:** confirmed -**Proposed action:** none - -### psi-F12: 100% code coverage claim from badge without verification - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** E-6 - -The result reports 100% code coverage from the Codecov badge. This is an unusual claim for any substantial project. The result itself correctly notes the limitation: "Julia coverage tools measure line coverage which may not reflect branch coverage or edge case coverage." The coverage was not independently verified by examining the Codecov report detail or running coverage locally. 100% line coverage can be achieved with auto-generated tests that execute every function without testing edge cases. - -The finding does not impugn the test suite quality (37 test files, 3-platform CI, cross-package integration tests are all strong signals), but the specific "100%" number should be presented with appropriate caveats. - -**Cross-tool relevance:** likely -- coverage claims from badges are common across evaluations. -**Proposed action:** add_verification -- examine the Codecov detail page for actual coverage methodology. - -### psi-F13: B-4 stochastic wrapping ran only 5 of 20 specified scenarios - -**Category:** misleading_result | **Severity:** low -**Tests:** B-4 - -The test specification calls for 20 scenarios, but only 5 were run. Of those 5, 2 were infeasible (40% failure rate). The extrapolated timing of ~50s for 20 scenarios assumes the infeasibility rate doesn't increase with more diverse scenarios. The result is honest about running only 5 scenarios and provides a clear rationale for the extrapolation, but the reduced scenario count weakens the evidence. - -**Cross-tool relevance:** likely -- other tools may also run fewer scenarios than specified. -**Proposed action:** add_verification -- run the full 20 scenarios. - -### psi-F14: B-8 and B-9 verified via interactive probe without test scripts - -**Category:** missing_verification | **Severity:** low -**Tests:** B-8, B-9 - -Both B-8 (reference bus config) and B-9 (PTDF extraction) state they were "verified via interactive probe in the devcontainer" with no dedicated test script. The protocol requires "one test script per tool per test case." The findings are credible (changing the reference bus via `set_bustype!()` and computing PTDF via `PTDF(sys)` are straightforward operations) but cannot be independently reproduced from the evaluation artifacts. - -**Cross-tool relevance:** likely -**Proposed action:** add_verification -- create reproducible test scripts. - -### psi-F15: Solver swap test lacks timing data - -**Category:** test_design_gap | **Severity:** low -**Tests:** C-7 - -C-7 confirms the most important finding (solver swap is parameter-only, no reformulation needed) but provides no comparative timing data across solvers. The protocol specifies "Time per solver" as a recorded metric. - -**Cross-tool relevance:** likely -**Proposed action:** add_verification - -## Extraordinary Claims - -### C-3: DCOPF on MEDIUM expected to solve in <60s - -**Concern:** This is an estimate based on problem size analysis, not a measured result. The test script had a bug preventing execution on MEDIUM. The qualified_pass status relies on extrapolation from TINY results and theoretical QP scaling. -**Evidence quality:** weak - -A probe should execute the DCOPF on MEDIUM with all generator types properly handled and measure actual wall-clock time. The test script bug ("requires fix for RenewableDispatch accessor") should be fixed first. - -### C-4: SCUC on SMALL expected to take >300s with SCIP - -**Concern:** No execution on SMALL. The estimate is based on problem size scaling (13,056 binary variables vs 240 on TINY). MIP problems scale nonlinearly and the estimate of ">300s" could be off by an order of magnitude in either direction. -**Evidence quality:** weak - -A probe should execute the SCUC on SMALL with SCIP and measure wall-clock time, MIP gap at termination, and peak memory. - -### C-5: Contingency sweep approach scales to MEDIUM - -**Concern:** No execution at MEDIUM scale. The estimate of "2-7 hours serial" for N-1 alone suggests serious scalability concerns. No MEDIUM-scale test was attempted. -**Evidence quality:** weak - -A probe should run at least N-1 contingencies on MEDIUM and measure per-contingency time. - -### C-6: 20-scenario stochastic DCOPF on SMALL estimated at 10-20 minutes - -**Concern:** No execution on SMALL. The TINY-scale test showed 40% scenario infeasibility, which may worsen on SMALL with tighter constraints. The estimate does not account for the file reload overhead scaling with system size. -**Evidence quality:** weak - -A probe should execute 20 scenarios on SMALL and measure total time, per-scenario time, and infeasibility rate. - -### E-6: 100% code coverage - -**Concern:** Reported from the Codecov badge without examining the detail page. Line coverage may be inflated by Julia's coverage tooling. Independent verification would require examining which files/functions contribute to the 100% figure. -**Evidence quality:** moderate - -A probe should check the Codecov detail page for actual per-file coverage and the methodology used. - -### A-4: PSI dispatch values ~100x larger than component limits - -**Concern:** The same dispatch value (660.85 for gen-1) is labeled as MW in A-3 and as pu in A-4. If PSI returns values in MW and `get_active_power_limits()` returns values in pu, the "mismatch" is simply a unit convention difference that requires dividing by base MVA. But if both should be in the same units, there is a genuine bug. The A-3 result showing "660.85 MW" with "Pmax 1040.0 MW" suggests consistent MW units. -**Evidence quality:** moderate - -A probe should log the exact return values and units from both `read_variables()` and `get_active_power_limits()` in the same script, with explicit base MVA tracking. - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | -- | -- | -| G-2 | pass | -- | -- | -| G-3 | pass | -- | -- | -| A-1 | pass | -- | -- | -| A-2 | pass | -- | -- | -| A-3 | qualified_pass | stable | Time series boilerplate; GLPK fails on quadratic costs | -| A-4 | qualified_pass | stable | Unit mismatch claim needs verification; ACPF does not converge on scaled dispatch | -| A-5 | qualified_pass | stable | HiGHS fails on SCUC; no cycling on case39; MATPOWER lacks UC params | -| A-6 | qualified_pass | stable | Ramp constraints not enforced (ThermalBasicDispatch); ThermalRampLimited build fails | -| A-7 | pass | stable | Manual graph construction; should be qualified_pass per convention | -| A-8 | fail | blocking | No native stochastic optimization | -| A-9 | qualified_pass | stable | No native SCOPF; manual JuMP+LODF injection; 70% contingencies filtered | -| A-10 | fail | blocking | No lossy DC OPF formulation | -| A-11 | qualified_pass | stable | Implicit distributed slack; weights not configurable; uncongested network | -| B-1 | pass | -- | Dual only verified for non-binding constraint | -| B-2 | qualified_pass | stable | No native Graphs.jl integration | -| B-3 | pass | -- | -- | -| B-4 | pass | stable | 5 of 20 scenarios run; 2 infeasible; system reload per scenario | -| B-5 | pass | -- | Native DataFrame output | -| B-6 | pass | -- | Excellent separation of concerns | -| B-7 | qualified_pass | fragile | Undocumented unit convention between PSI and PowerFlows | -| B-8 | pass | -- | No test script; interactive probe only; distributed slack not configurable | -| B-9 | pass | -- | No test script; interactive probe only | -| C-1 | pass | -- | 9.4s on MEDIUM | -| C-2 | fail | -- | NR diverges on MEDIUM; no diagnostic detail | -| C-3 | qualified_pass | stable | Estimated timing only; no actual measurement | -| C-4 | qualified_pass | stable | Estimated timing only; SCIP only | -| C-5 | qualified_pass | stable | No MEDIUM measurement; serial-only | -| C-6 | qualified_pass | stable | No SMALL measurement; estimated 10-20min | -| C-7 | pass | -- | Parameter-only swap; no timing data | -| C-8 | fail | blocking | No native SCOPF; manual approach prohibitive at scale | -| C-9 | pass | -- | 6.44s on MEDIUM | -| C-10 | fail | blocking | No native distributed slack | -| D-1 | qualified_pass | -- | JIT overhead; time series boilerplate barrier | -| D-2 | informational | -- | 4/11 tests from docs alone | -| D-3 | informational | -- | 7 tutorials; ~20% use case coverage | -| D-4 | qualified_pass | -- | Zero-rate line silently accepted | -| D-5 | informational | -- | Median 255 LOC; 30 LOC boilerplate overhead | -| E-1 | informational | -- | 22 releases/24mo; pre-1.0 | -| E-2 | informational | -- | 1,040 commits; 21 contributors | -| E-3 | informational | -- | Bus factor 1 lifetime; improving recently | -| E-4 | informational | -- | DOE/NREL funded; single-institution risk | -| E-5 | informational | -- | Median 66 days to close; bimodal | -| E-6 | informational | -- | 100% coverage claim; 8 CI workflows | -| E-7 | informational | -- | Limited adoption outside NREL; 311 stars | -| F-1 | pass | -- | BSD-3-Clause | -| F-2 | informational | -- | 183 deps; 51 JLL binary wrappers | -| F-3 | informational | -- | GLPK GPL-3 flagged; optional | -| F-4 | pass | -- | All source available; MKL optional | -| F-5 | pass | -- | Full path inspectable | -| F-6 | pass | -- | Content-addressed distribution | -| F-7 | pass | -- | Air-gap installable via depot copy | -| F-8 | pass | -- | All open-source solvers sufficient | -| F-9 | informational | -- | Standard Julia conventions | -| P2-1 | informational | -- | PSS/E RAW v30/v32/v33 native | -| P2-2 | informational | -- | Piecewise linear native (SOS2) | -| P2-3 | informational | -- | UC-to-ED native; ED-to-ACPF blocked by unit mismatch | diff --git a/sweep-data/v4-to-v5/per-tool/powersimulations/findings.yaml b/sweep-data/v4-to-v5/per-tool/powersimulations/findings.yaml deleted file mode 100644 index 85edfbef..00000000 --- a/sweep-data/v4-to-v5/per-tool/powersimulations/findings.yaml +++ /dev/null @@ -1,393 +0,0 @@ -tool: powersimulations -source_version: "v4" -timestamp: "2026-03-07T12:00:00Z" -evaluation_summary: - total_tests: 57 - pass: 20 - fail: 6 - qualified_pass: 14 - informational: 17 - -findings: - - id: psi-F01 - category: extraordinary_claim - severity: high - test_ids: [C-3, C-4, C-5, C-6] - title: "Four scalability tests report estimated timings without actual measurement" - description: > - C-3 (DCOPF scale) says "est. <60s", C-4 (SCUC scale) says "est. >300s", - C-5 (contingency sweep scale) says "est. serial" with no measured time, - C-6 (stochastic scale) says "est. 10-20min". These are extrapolations or - projections, not wall-clock measurements from executed code. The protocol - requires "Record everything. Wall-clock time (for scalability-relevant tests)." - evidence: - - file: "scalability/C-3_dcopf_scale.md" - excerpt: "HiGHS is expected to solve the MEDIUM DCOPF in < 60s" - - file: "scalability/C-4_scuc_scale.md" - excerpt: "Expected solve time | >300s (estimated)" - - file: "scalability/C-5_contingency_sweep_scale.md" - excerpt: "Wall-clock time not measured at MEDIUM scale" - - file: "scalability/C-6_stochastic_scale.md" - excerpt: "Not measured at SMALL scale. Estimated 10-20 minutes" - cross_tool_relevance: likely - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - - - id: psi-F02 - category: extraordinary_claim - severity: high - test_ids: [A-4, B-7] - title: "Unit mismatch between PSI dispatch output and PowerSystems limits claimed but root cause uncertain" - description: > - The evaluation reports a ~100x scaling mismatch between PSI's - ActivePowerVariable output and PowerSystems.jl component limits. Dispatch - values of 660.85 are reported when Pmax is 10.40 pu. The synthesis calls - this "fragile" and undocumented. However, the A-3 result table shows the - same gen-1 dispatch as "660.85 MW" with "Pmax 1040.0 MW" -- suggesting - the values may actually be in MW, not pu. The A-4 result table confusingly - labels dispatch as "pu" while showing MW-scale values. This may be a - labeling error in the test report rather than a genuine 100x unit mismatch. - evidence: - - file: "expressiveness/A-3_dcopf.md" - excerpt: "gen-1 | 660.85 | 1040.0 | No" - - file: "expressiveness/A-4_ac_feasibility.md" - excerpt: "gen-1 | 660.85 | 66,085 | 10.40" - - file: "extensibility/B-7_ac_feasibility_extension.md" - excerpt: "Dispatch values from PSI are ~100x larger than Pmax values from the System" - cross_tool_relevance: none - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: psi-F03 - category: missing_verification - severity: medium - test_ids: [B-1] - title: "Custom constraint dual value verified only for non-binding case" - description: > - B-1 tests custom constraint injection and dual extraction, but the flow - gate limit was set at 80% of the unconstrained flow sum, which was - non-binding. The dual value was correctly zero, but a binding constraint - with non-zero dual was never demonstrated. The result acknowledges this: - "A tighter limit or different line selection would produce a binding - constraint with non-zero dual." The protocol requires "Dual value of - custom constraint extractable and correctly reflects binding status." - evidence: - - file: "extensibility/B-1_custom_constraints.md" - excerpt: "The signed flow sum was -5.54 pu, well below the limit, so the constraint was non-binding. Dual = 0.0" - cross_tool_relevance: likely - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: psi-F04 - category: scoring_inconsistency - severity: medium - test_ids: [A-6, A-7] - title: "Tests with workarounds scored as pass instead of qualified_pass" - description: > - The synthesis notes that A-6 and A-7 have "status: pass" but - "workaround_class: stable" and flags this as inconsistent. The protocol - states stable workarounds receive B-range grades and suggests - qualified_pass. The result file for A-6 actually has status: - qualified_pass in its frontmatter, contradicting the synthesis table which - shows "pass" for A-7. A-7's frontmatter shows status: pass with - workaround_class: stable. By protocol convention, a test requiring a - workaround (even stable) should be qualified_pass. - evidence: - - file: "expressiveness/A-7_contingency.md" - excerpt: "status: pass\nworkaround_class: stable" - - file: "synthesis.md" - excerpt: "A-6 and A-7 status/workaround inconsistency" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: psi-F05 - category: network_insufficiency - severity: medium - test_ids: [A-5] - title: "SCUC on 39-bus produces no unit cycling -- network too small to exercise UC" - description: > - A-5 reports that all 10 generators are committed ON for all 24 hours - with zero startup events and zero cycling generators. The result - acknowledges this is "expected for a system where total capacity closely - matches peak load." However, this means the SCUC test does not actually - exercise unit commitment decisions -- it degenerates to a dispatch - problem. Min up/down time constraints, startup costs, and shutdown - decisions are never tested as binding. - evidence: - - file: "expressiveness/A-5_scuc.md" - excerpt: "All generators committed ON for all 24 hours (no cycling observed)" - - file: "expressiveness/A-5_scuc.md" - excerpt: "Startup events | 0\nCycling generators | 0" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: psi-F06 - category: misleading_result - severity: medium - test_ids: [A-11] - title: "Distributed slack qualified_pass on uncongested network where both formulations produce identical results" - description: > - A-11 compares PTDFPowerModel (claimed distributed slack) with - DCPPowerModel (single slack). On the uncongested case39 network, both - produce numerically identical dispatch (max difference 1.5e-5 MW) and - objectives (difference 3.8e-13). The test cannot discriminate because - there is no congestion to differentiate the formulations. Furthermore, - PTDF slack weights are not configurable, which is a key requirement of - the pass condition. The qualified_pass is based on the claim that PTDF - "inherently" distributes slack, but on an uncongested network this is - indistinguishable from single slack. - evidence: - - file: "expressiveness/A-11_distributed_slack.md" - excerpt: "Objective difference: 3.8e-13 (numerically identical)" - - file: "expressiveness/A-11_distributed_slack.md" - excerpt: "Weights are not configurable" - cross_tool_relevance: confirmed - probe_recommended: true - probe_type: formulation_audit - proposed_action: redesign_test - - - id: psi-F07 - category: scoring_inconsistency - severity: low - test_ids: [A-6] - title: "A-6 SCED qualified_pass despite ramp constraints not enforced" - description: > - A-6's pass condition requires "ramp rate constraints are demonstrably - enforced between consecutive dispatch intervals in the ED stage." The - test shows 182 ramp violations across all generators and hours with - ThermalBasicDispatch, and the ramp-enforcing ThermalRampLimited fails to - build. The result correctly notes this as qualified_pass, but this is - close to a fail -- the core pass condition is unmet. The qualification - is generous given the evidence. - evidence: - - file: "expressiveness/A-6_sced.md" - excerpt: "Total: 182 ramp violations across all generators and hours" - - file: "expressiveness/A-6_sced.md" - excerpt: "ThermalRampLimited | Build status | FAILED" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: psi-F08 - category: test_design_gap - severity: medium - test_ids: [A-9] - title: "SCOPF contingency filtering removes 70% of contingencies without documented rationale for threshold" - description: > - A-9 filters out 32 of 46 contingencies as "near-radial" (max |LODF| > - 0.9), leaving only 14 contingencies included. This 70% reduction - significantly eases the SCOPF problem and the 0.51% cost increase over - baseline is small. The LODF threshold of 0.9 is stated as "standard - SCOPF practice" but the aggressiveness of the filter (removing 70% of - contingencies) may mask whether the formulation would work with a - larger, less filtered set. The resulting 156 constraints on a 56-variable - problem is not particularly challenging. - evidence: - - file: "expressiveness/A-9_scopf.md" - excerpt: "32 contingencies skipped (near-radial), 14 contingencies included" - - file: "expressiveness/A-9_scopf.md" - excerpt: "Total constraints added: 156" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: psi-F09 - category: infrastructure_friction - severity: low - test_ids: [A-3, A-4, A-5, A-6, A-8, A-9, A-10, A-11, B-4] - title: "Mandatory time series boilerplate inflates LOC and qualified_pass count" - description: > - PSI's DecisionModel requires time series setup (~30 LOC) for any - optimization, even single-period OPF. This is a genuine tool - characteristic (multi-period framework paradigm) but it produces a - systematic qualified_pass pattern across all optimization tests. The - boilerplate is stable and well-documented, so it inflates the - qualified_pass count without reflecting distinct capability gaps. Eight - of the nine optimization tests share identical boilerplate code. - evidence: - - file: "expressiveness/A-3_dcopf.md" - excerpt: "Adds significant boilerplate (~30 LOC) for what is conceptually a single-period OPF" - - file: "accessibility/D-5_code_volume.md" - excerpt: "time series boilerplate and device model registration together account for approximately 25-30 LOC per optimization test" - cross_tool_relevance: none - probe_recommended: false - probe_type: null - proposed_action: null - - - id: psi-F10 - category: missing_verification - severity: medium - test_ids: [C-2] - title: "ACPF scale failure lacks diagnostic detail on convergence" - description: > - C-2 reports ACPF failure on MEDIUM but provides no iteration count, - residual history, or diagnostic beyond the error string. The root cause - analysis speculates about missing robustness features but does not verify - whether the issue is the solver, the network data, or the initial - conditions. No DC warm start was attempted because "PowerFlows.jl does - not support DC warm start initialization" -- but this was not confirmed - by checking whether voltage setpoints from DCPF could be manually - applied. - evidence: - - file: "scalability/C-2_acpf_scale.md" - excerpt: "The NewtonRaphsonACPowerFlow solver failed to converge" - - file: "scalability/C-2_acpf_scale.md" - excerpt: "PowerFlows.jl does not support DC warm start initialization -- it only offers ACPowerFlow() with the system's existing voltage setpoints" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: psi-F11 - category: low_signal - severity: low - test_ids: [G-1, G-2, G-3] - title: "Gate tests produce identical pass outcome for all tools with MATPOWER support" - description: > - All three gate tests pass cleanly with correct bus/branch/gen counts. - PowerSystems.jl's MATPOWER parser handles all three networks without - issues. This is expected for any tool with a MATPOWER parser. The gate - tests serve their intended purpose (verify data ingestion) but do not - differentiate between tools. - evidence: - - file: "gate/G-1_ingest_tiny.md" - excerpt: "Actual counts: 39 buses / 46 branches / 10 generators" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: psi-F12 - category: extraordinary_claim - severity: medium - test_ids: [E-6] - title: "100% code coverage claim requires scrutiny" - description: > - E-6 reports "100% code coverage" from the Codecov badge. The result - itself notes "Julia coverage tools measure line coverage which may not - reflect branch coverage or edge case coverage." 100% line coverage is - unusual for a 10,000+ commit project and may reflect coverage measurement - methodology rather than true comprehensive testing. The claim is reported - from the badge without independent verification. - evidence: - - file: "maturity/E-6_ci_test_coverage.md" - excerpt: "Codecov badge: 100% (as of 2026-03-06)" - - file: "maturity/E-6_ci_test_coverage.md" - excerpt: "Julia coverage tools measure line coverage which may not reflect branch coverage" - cross_tool_relevance: likely - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: psi-F13 - category: misleading_result - severity: low - test_ids: [B-4] - title: "B-4 stochastic wrapping runs only 5 of specified 20 scenarios with 2 failures" - description: > - B-4 specifies 20 scenarios but the test ran only 5, with 2 of those - failing (infeasible). The extrapolated timing of ~50s for 20 scenarios - is based on 3 successful solves. The 40% failure rate (2/5 infeasible) - may indicate the scenario generation approach needs adjustment for - case39's tight limits, but the finding is reported without discussing - whether 20 scenarios would have a similar or higher failure rate. - evidence: - - file: "extensibility/B-4_stochastic.md" - excerpt: "Ran 5 scenarios (of the specified 20)" - - file: "extensibility/B-4_stochastic.md" - excerpt: "Scenarios 3 and 4 became infeasible" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: psi-F14 - category: missing_verification - severity: low - test_ids: [B-8, B-9] - title: "B-8 and B-9 results lack wall-clock timing and appear to be interactive probes" - description: > - B-8 states "No dedicated test script -- verified via interactive probe - in the devcontainer." B-9 similarly says "Verified via interactive probe." - Both lack wall_clock_seconds in frontmatter and have no reproducible test - script. The protocol requires "one test script per tool per test case" - and "Record everything." - evidence: - - file: "extensibility/B-8_reference_bus_config.md" - excerpt: "No dedicated test script -- verified via interactive probe in the devcontainer" - - file: "extensibility/B-9_ptdf_extraction.md" - excerpt: "Verified via interactive probe in the devcontainer" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: psi-F15 - category: test_design_gap - severity: low - test_ids: [C-7] - title: "Solver swap test lacks measured wall-clock times per solver" - description: > - C-7 confirms solver swap is parameter-only (no reformulation), which is - the key finding. However, the protocol specifies "Time per solver" as a - recorded metric, and no timing data is provided. The test script - reference mentions both HiGHS and Ipopt but no comparative timing. - evidence: - - file: "scalability/C-7_solver_swap.md" - excerpt: "wall_clock_seconds: null" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - -extraordinary_claims: - - test_id: C-3 - claim: "DCOPF on MEDIUM expected to solve in <60s" - concern: "This is an estimate, not a measured result. No test script was successfully run to completion on MEDIUM for DCOPF." - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification - - - test_id: C-4 - claim: "SCUC on SMALL expected to take >300s with SCIP" - concern: "Estimated timing without execution. The SMALL network has 544 generators creating ~13,000 binary variables -- actual solve time could be much longer." - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification - - - test_id: C-5 - claim: "Contingency sweep approach scales to MEDIUM" - concern: "Wall-clock time not measured. The estimate of '2-7 hours serial' for N-1 alone would be very slow. No MEDIUM test was run." - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification - - - test_id: C-6 - claim: "20-scenario stochastic DCOPF on SMALL estimated at 10-20 minutes" - concern: "No execution on SMALL. Extrapolated from TINY timings which showed 40% scenario infeasibility rate." - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification - - - test_id: E-6 - claim: "100% code coverage" - concern: "Reported from Codecov badge without independent verification. Line coverage may not reflect branch/edge case coverage. 100% is unusually high." - evidence_quality: moderate - probe_recommended: true - probe_type: claim_verification - - - test_id: A-4 - claim: "PSI dispatch values are ~100x larger than component limits (unit mismatch)" - concern: "A-3 shows the same gen-1 value (660.85) but labeled as MW with Pmax 1040.0 MW -- consistent units. A-4 labels them as pu with Pmax 10.40 pu. This may be a reporting error in A-4 rather than a genuine unit mismatch." - evidence_quality: moderate - probe_recommended: true - probe_type: claim_verification diff --git a/sweep-data/v4-to-v5/per-tool/pypsa/findings.md b/sweep-data/v4-to-v5/per-tool/pypsa/findings.md deleted file mode 100644 index b3ca8ac7..00000000 --- a/sweep-data/v4-to-v5/per-tool/pypsa/findings.md +++ /dev/null @@ -1,398 +0,0 @@ -# PyPSA -- Sweep Findings (v4) - -## Summary - -The PyPSA v4 evaluation is thorough and generally well-executed, covering all 62 tests across 7 dimensions with detailed result files and test scripts. The evaluation demonstrates genuine tool strengths (2-LOC custom constraints, native SCOPF, clean Linopy model access) alongside real limitations (no stochastic optimization, broken lpf_contingency, SCUC solver scalability). The primary quality concerns are: (1) inconsistent data-fix treatment between expressiveness and scalability tests on the ACTIVSg10k network, creating contradictory pass/fail outcomes for the same capability; (2) several qualified_pass and pass classifications that are generous given the underlying evidence (ACPF non-convergence, C-5 with 3.3% completion, C-10 without completing the comparison); (3) a documentation audit claim that stochastic optimization is fully documented despite the feature not working. Three probes are recommended for extraordinary claims requiring verification. - -## Finding Details - -### pypsa-F01: Gencost import gap forces manual cost assignment across all OPF tests - -**Category:** infrastructure_friction | **Severity:** medium -**Tests:** A-3, A-4, A-5, A-6, A-9, A-10, A-11, B-1, B-4, B-8 - -PyPSA's `import_from_pypower_ppc()` silently drops gencost data from MATPOWER files, emitting only a warning: "some PYPOWER features not supported: areas, gencosts, component status." This forces manual cost parsing and assignment (~5 LOC) as a prerequisite for every OPF-based test. The workaround is stable (uses `matpowercaseframes` for parsing and the documented `marginal_cost` attribute for assignment), but it represents test infrastructure friction rather than a PyPSA modeling limitation. - -Every tool evaluated against MATPOWER .m files may face analogous import friction. The question is whether the friction is in the import path or in the modeling API -- for PyPSA it is clearly in the import path, as the optimization API works correctly once costs are set. - -**Cross-tool relevance:** likely (other tools using MATPOWER format may have similar import gaps) -**Proposed action:** none (correctly documented as stable workaround) - ---- - -### pypsa-F02: ACTIVSg10k zero-impedance branches cause cascading MEDIUM failures - -**Category:** infrastructure_friction | **Severity:** high -**Tests:** A-1, A-3, A-4, A-7, B-1, B-3, B-5, B-9, C-1, C-3, C-5, C-7, C-8, C-9, C-10 - -The ACTIVSg10k MATPOWER case contains 2,462 branches with `s_nom=0` and 3 transformers with zero reactance. This data characteristic causes: -- DCOPF infeasibility (PyPSA interprets `s_nom=0` as zero-capacity constraint) -- Singular B-matrix in LPF (all flows NaN) -- PTDF computation failure (RuntimeError: Factor is exactly singular) - -The evaluation applied data fixes (x=0.0001, s_nom=9999) in scalability tests (C-series) but NOT in expressiveness tests (A-series), creating an inconsistency. A-3 MEDIUM fails with infeasibility, but C-3 MEDIUM passes with the same tool on the same network after fixing the data. - -From the synthesis: "MEDIUM-tier tests were executed where possible but several failed due to data-preparation issues with the ACTIVSg10k case (zero-impedance branches, zero s_nom) rather than tool limitations." - -This is a protocol-level issue affecting all tools evaluated on ACTIVSg10k. - -**Cross-tool relevance:** confirmed (affects all tools using ACTIVSg10k) -**Proposed action:** redesign_test -- standardize MEDIUM data preparation across all dimensions - ---- - -### pypsa-F03: Contingency sweep C-5 qualified_pass with only 3.3% completion - -**Category:** misleading_result | **Severity:** medium -**Tests:** C-5 - -C-5 is classified as `qualified_pass` but completed only 9 of 270 N-1 cases within the 600s timeout (3.3% completion rate). N-2 through N-4 sweeps were entirely skipped. The pass condition states: "Completes. Per-contingency-case average and total time recorded." Completing 3.3% of the minimum sweep order does not satisfy "completes." - -From the result: "Each manual LPF call on 10k-bus takes ~65s (including topology re-determination). Only 9 of 270 N-1 cases completed within the 600s timeout." - -The synthesis notes this as a finding for human spot-check: "Verify whether this is sufficient for a 'qualified pass' or should be reclassified as fail." - -**Cross-tool relevance:** likely (contingency sweep scalability may challenge other tools similarly) -**Proposed action:** adjust_scoring -- reclassify as fail - ---- - -### pypsa-F04: PTDF C-9 qualified_pass with 702 MW flow prediction error - -**Category:** misleading_result | **Severity:** medium -**Tests:** C-9 - -C-9 (PTDF at MEDIUM) is classified as `qualified_pass` with correct matrix dimensions (12,706 x 10,000) and 28s computation time, but the flow prediction accuracy has a max difference of 702 MW. The B-9 pass condition specifies "Flow predictions match DCPF results within numerical tolerance (1e-6)." The 702 MW error is attributable to the zero-impedance workaround (x=0.0001) altering the sensitivity structure, which is a data issue -- but the resulting PTDF matrix is not usable for practical congestion analysis at this accuracy level. - -From the result: "PTDF matrix computes and has correct dimensions, but flow prediction accuracy is degraded by the required zero-impedance workaround." - -**Cross-tool relevance:** likely (any tool computing PTDF on ACTIVSg10k with zero-impedance fixes will have similar issues) -**Proposed action:** adjust_scoring -- either re-run with properly prepared data or classify as data-limited rather than qualified_pass - ---- - -### pypsa-F05: ACPF MEDIUM pass despite non-convergence warning - -**Category:** extraordinary_claim | **Severity:** medium -**Tests:** A-2, C-2 - -A-2 MEDIUM reports `status: pass` and `converged: true`, but the result file documents that PyPSA emitted "Power flow did not converge for ['now']". The test classified it as a pass because voltage magnitudes fell in a reasonable range (0.96-1.08 pu). - -Additionally concerning: -- Total real power losses of 3,935 MW are reported (this is approximately 6% of total generation for a 10k-bus network, which may be reasonable for AC PF but warrants verification) -- Transformer flows up to 14,700 MW suggest modeling artifacts -- C-2 reports voltage angles as all zero, which is physically impossible for a converged AC power flow on a 10k-bus network - -A non-converged Newton-Raphson may produce voltage magnitudes that "look reasonable" while angles, flows, and reactive power are mathematically incorrect. The pass classification requires verification. - -**Cross-tool relevance:** likely (ACPF convergence on ACTIVSg10k with data issues may challenge all tools) -**Proposed action:** add_verification -- verify residual magnitude, not just voltage range - ---- - -### pypsa-F06: SCOPF objective equality masks test signal due to uniform costs - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-9 - -The SCOPF pass condition requires dispatch and cost to "differ from unconstrained DC OPF (A-3) -- SCOPF should be more expensive." On TINY, both objectives are identical ($1876.269) because all 10 generators have identical marginal costs (c1=0.3 $/MWh). The dispatch redistribution is verified (G8 goes from 0 to 865 MW), but the cost signal is absent. - -From the result: "Both objectives are 1876.269 because all generators have identical marginal costs (C1=0.3 $/MWh). The SCOPF redistributes dispatch without changing total cost since the cost function is linear with uniform slope." - -The evaluator correctly identified this but the cost comparison -- a key verification -- produces no signal. On SMALL (A-9), the SCOPF is 0.4% more expensive, providing some validation. - -**Cross-tool relevance:** confirmed (all tools using case39 with uniform costs face this issue) -**Proposed action:** redesign_test -- use perturbed costs on TINY or require the cost comparison only on SMALL/MEDIUM - ---- - -### pypsa-F07: SCUC on TINY shows no generator cycling - -**Category:** network_insufficiency | **Severity:** low -**Tests:** A-5 - -The SCUC test on TINY produces a trivial result: all 10 generators stay committed for all 24 hours, the MIP gap is 0%, and the branch-and-bound tree has only 1 node. No unit commitment cycling occurs because load is high relative to total capacity, so min up/down time and startup cost constraints never bind. - -From the result: "Generators always on: 10/10, Min simultaneous online: 10, B&B nodes: 1" - -The test passes functionally (commitment schedule is extractable as a time-indexed binary matrix), but it does not exercise the combinatorial difficulty that makes UC interesting. The SMALL-tier test (A-5 SMALL) would exercise this but it fails due to solver timeout. - -**Cross-tool relevance:** confirmed (case39 has the same load/capacity ratio for all tools) -**Proposed action:** none (this is a known limitation of TINY for UC tests, and SMALL is the grade network) - ---- - -### pypsa-F08: D-2 claims A-8 stochastic docs coverage YES but A-8 test FAILs - -**Category:** extraordinary_claim | **Severity:** high -**Tests:** D-2 - -The documentation audit (D-2) rates A-8 stochastic optimization coverage as "Documented: YES (as of v1.x)" and references a dedicated page at `user-guide/optimization/stochastic/` with `n.set_scenarios()`, `n.set_risk_preference()`, and an example notebook. - -However, A-8 itself FAILs with `workaround_class: blocking`. The A-8 result states: "n.scenarios exists as an empty Index in the data model but is not wired into the optimizer. No optimize_stochastic() or scenario-weighted objective method exists." - -This is a direct contradiction. Either: -1. The documentation describes functionality that was added after the evaluation's TINY test but exists in the codebase -2. The documentation is aspirational and describes planned rather than implemented features -3. The evaluator tested an incorrect API path - -The synthesis correctly flags this for human spot-check. This finding has high severity because it suggests the documentation audit may have accepted documentation existence as proof of functionality without runtime verification. - -**Cross-tool relevance:** none (PyPSA-specific) -**Proposed action:** add_verification -- verify whether `n.set_scenarios()` and `n.set_risk_preference()` exist and function - ---- - -### pypsa-F09: Lossy DCOPF LMP decomposition not validated against MATPOWER reference - -**Category:** missing_verification | **Severity:** medium -**Tests:** A-10 - -The A-10 pass condition explicitly requires: "Validate against MATPOWER reference lossy DC OPF solution on same case (tolerance: 1% on total LMP, directional consistency on loss component signs)." - -The result file demonstrates: -- Loss components are non-zero with physically correct signs -- LMP decomposition into energy/congestion/loss is performed -- Congestion rent reconciliation is computed - -However, no cross-validation against MATPOWER's `rundcopf` with loss option is documented. The LMP values are self-consistent but not independently verified against a reference implementation. - -**Cross-tool relevance:** confirmed (all tools must validate against MATPOWER reference per the pass condition) -**Proposed action:** add_verification -- run MATPOWER reference and compare - ---- - -### pypsa-F10: Distributed slack OPF test has low signal - -**Category:** low_signal | **Severity:** low -**Tests:** A-11 - -A-11 tests distributed slack OPF. For PyPSA, the OPF formulation inherently distributes generation (no slack bus concept in the optimizer). Changing the slack bus has zero effect on OPF LMPs, which is mathematically correct but means the test provides no signal about distributed slack OPF capability -- the tool trivially passes by design. - -The meaningful distributed slack behavior is demonstrated in the PF path (`n.pf(distribute_slack=True)`), which does change voltage angles. But this is a PF feature, not an OPF feature. - -The B-8 result confirms: "In PyPSA's DCOPF, the slack bus assignment does NOT affect LMPs because the optimizer enforces power balance as a constraint." - -**Cross-tool relevance:** likely (tools with similar optimization-based formulations may trivially pass) -**Proposed action:** redesign_test -- clarify whether the test targets PF or OPF distributed slack; for OPF-native tools, the test may need a different formulation - ---- - -### pypsa-F11: A-1 MEDIUM pass despite all flows being NaN - -**Category:** misleading_result | **Severity:** medium -**Tests:** A-1 - -A-1 DCPF at MEDIUM is classified as `status: pass` with the justification: "Despite the NaN flows (caused by zero-impedance branches in the MATPOWER case, not a PyPSA limitation), the solver converges and outputs are structured pandas DataFrames." - -All voltage angles and all line flows are NaN. The power flow produced no usable results. The pass is based on the structural format of the output (DataFrames) rather than the correctness of the computation. - -A power flow with all-NaN results is not a converged power flow by any engineering definition. The `MatrixRankWarning: Matrix is exactly singular` confirms the solve did not produce meaningful results. - -**Cross-tool relevance:** likely (data issue, but scoring philosophy applies to all tools) -**Proposed action:** adjust_scoring -- reclassify as fail or qualified_fail with data-limitation note - ---- - -### pypsa-F12: Scalability wall-clock times dominated by linopy post-processing - -**Category:** missing_verification | **Severity:** low -**Tests:** C-3, C-7, C-8, C-10 - -Multiple MEDIUM scalability tests report total wall-clock times of 600s+ while the actual HiGHS solver time is 6-21 seconds. The dominant cost is linopy's shadow-price assignment step (10+ minutes at 10k-bus scale). This creates a measurement challenge for cross-tool comparison: - -| Test | Solver Time | Total Wall-Clock | Post-Processing | -|------|------------|-----------------|-----------------| -| C-3 | 19.9s | 600s+ | 10+ min | -| C-7 | 21.2s | 600s+ | 10+ min | -| C-8 | 7.1s | 600s+ | 10+ min | -| C-10 | 6.1s | 600s+ | 10+ min | - -The post-processing overhead is a legitimate scalability concern (it is real wall-clock time), but comparing these total times against tools that do not extract dual variables by default would be misleading. The solver-time vs framework-overhead decomposition should be preserved in any cross-tool comparison. - -**Cross-tool relevance:** none (linopy-specific issue) -**Proposed action:** add_verification -- verify whether `assign_all_duals=False` eliminates the overhead - ---- - -### pypsa-F13: Inconsistent data fixes between expressiveness and scalability - -**Category:** scoring_inconsistency | **Severity:** medium -**Tests:** A-3, C-3 - -A-3 MEDIUM (expressiveness): FAIL -- zero s_nom branches cause infeasibility, no data fixes applied. -C-3 MEDIUM (scalability): PASS -- same network, same tool, same analysis, but with s_nom=9999 and x=0.0001 fixes applied. - -The synthesis notes: "MEDIUM data fixes applied in scalability tests: C-1/C-3/C-5/C-7/C-8/C-9/C-10 applied x=0.0001 on 3 zero-impedance transformers and s_nom=9999 on 2,462 zero-s_nom lines. These fixes were not applied in expressiveness tests A-3/A-4 MEDIUM." - -This creates a contradictory narrative: PyPSA fails expressiveness DCOPF at MEDIUM but passes scalability DCOPF at MEDIUM. The protocol should standardize whether data fixes are applied universally or not at all for a given network tier. - -**Cross-tool relevance:** confirmed (all tools face this data-preparation question on ACTIVSg10k) -**Proposed action:** redesign_test -- standardize data preparation for ACTIVSg10k across all test suites - ---- - -### pypsa-F14: Solver swap test verifies only API, not actual multi-solver comparison - -**Category:** test_design_gap | **Severity:** low -**Tests:** C-7 - -C-7 confirms solver swap is parameter-only (`solver_name="..."`) but only HiGHS was installed. The test verified the mechanism by API inspection and ran DCOPF with HiGHS only. No GLPK, SCIP, or other solver comparison was performed. - -From the result: "GLPK available: No, SCIP available: No, Gurobi available: No, CPLEX available: No" - -The pass condition ("Solver swap requires only a parameter change, not reformulation") is technically met by inspection, but the evaluation environment should have at least two solvers installed to demonstrate the claim with runtime evidence. - -**Cross-tool relevance:** likely (evaluation environment may lack alternative solvers for other tools too) -**Proposed action:** add_verification -- install at least GLPK or SCIP alongside HiGHS - ---- - -### pypsa-F15: B-5 interoperability test trivially passes for DataFrame-based tools - -**Category:** low_signal | **Severity:** low -**Tests:** B-5 - -B-5 tests CSV export of DCPF results. PyPSA natively stores everything as pandas DataFrames, so export is `df.to_csv()` in 2 lines. This produces no discriminative signal among Python tools that use DataFrames. The test would differentiate Julia tools (where DataFrame export requires more steps) or tools with custom data structures, but among Python tools it is trivially passed. - -**Cross-tool relevance:** confirmed (all DataFrame-based Python tools pass trivially) -**Proposed action:** none (test still provides baseline interoperability documentation) - ---- - -### pypsa-F16: Maturity metrics based on research rather than runtime verification - -**Category:** missing_verification | **Severity:** low -**Tests:** E-1, E-2, E-3, E-4, E-5, E-6, E-7 - -All E-series metrics (24 releases/24mo, 327 commits/12mo, 84% coverage, 21h median issue close time, IEA/ACER adoption) are research-based. The numbers are internally consistent and appear reasonable. This is expected for audit-type tests, but none were independently verified against GitHub API or PyPI data during this sweep. - -**Cross-tool relevance:** confirmed (all tools have research-based maturity metrics) -**Proposed action:** none (standard approach for audit tests) - ---- - -### pypsa-F17: C-10 distributed slack pass without completing the comparison - -**Category:** extraordinary_claim | **Severity:** low -**Tests:** C-10 - -C-10 is classified as pass, but the result file states: "The full test (DCOPF + single-slack PF + distributed-slack PF comparison) could not complete within 600s due to linopy's shadow-price assignment overhead after the DCOPF solve." - -The pass is based on: -1. DCOPF solving in 6.1s (not distributed-slack-specific) -2. The `distribute_slack` API existing - -The actual distributed slack PF comparison at MEDIUM scale was not performed. The pass is inferred from capability rather than demonstrated at scale. - -**Cross-tool relevance:** none -**Proposed action:** adjust_scoring -- classify as qualified_pass with note that PF comparison was not completed - ---- - -## Extraordinary Claims - -### A-2 MEDIUM: ACPF converges on 10k-bus network despite solver reporting non-convergence - -**Concern:** The Newton-Raphson solver explicitly warns "Power flow did not converge" but the test classifies the result as pass based on voltage magnitude ranges being in a reasonable band. Additionally, C-2 reports all voltage angles as zero, which is physically impossible for a converged AC PF. A non-converged NR iteration may produce plausible-looking voltage magnitudes while angles and flows are incorrect. - -**Evidence quality:** moderate - -A probe should: (1) check the final NR residual magnitude, (2) verify power balance at each bus, (3) confirm whether voltage angles are truly zero or just not reported. If the residual exceeds the convergence tolerance, the test should be reclassified as fail regardless of voltage magnitude appearance. - -### D-2: A-8 stochastic optimization is documented with dedicated page and example notebook - -**Concern:** The A-8 test demonstrates that `n.scenarios` is not wired to the optimizer, yet D-2 claims full documentation coverage with `n.set_scenarios()` and `n.set_risk_preference()` documented on a dedicated page. This is either aspirational documentation or the evaluator found a different API path than the one tested in A-8. - -**Evidence quality:** moderate - -A probe should: (1) verify whether `user-guide/optimization/stochastic/` exists in the PyPSA v1.1.2 docs, (2) test whether `n.set_scenarios()` and `n.set_risk_preference()` are callable methods, (3) if they exist, determine whether they produce a joint stochastic optimization or just set metadata. - -### C-10: Distributed slack OPF passes at MEDIUM scale - -**Concern:** The PF comparison that constitutes the actual test was not completed within the time budget. The pass is based on DCOPF convergence (not distributed-slack-specific) and API existence. - -**Evidence quality:** weak - -A probe should: run the distributed slack PF comparison without the linopy shadow-price overhead (e.g., skip `assign_all_duals` or run PF independently of OPF) to verify whether distributed slack PF actually functions at 10k-bus scale. - -## Test Outcome Matrix - -| Test ID | Status | Workaround | Key Issue | -|---------|--------|------------|-----------| -| G-1 | pass | -- | -- | -| G-2 | pass | -- | -- | -| G-3 | pass | -- | 2,462 zero-s_nom branches noted | -| A-1 TINY | pass | -- | -- | -| A-1 MEDIUM | pass | -- | All flows NaN (singular matrix) -- misleading pass | -| A-2 TINY | pass | -- | 4 NR iterations, clean convergence | -| A-2 MEDIUM | pass | -- | Non-convergence warning, zero angles -- needs verification | -| A-3 TINY | pass | stable (gencost) | Uniform LMPs (no congestion) | -| A-3 MEDIUM | fail | -- | Zero s_nom causes infeasibility (data issue) | -| A-4 TINY | pass | stable | 2 voltage violations, 1 thermal violation | -| A-4 MEDIUM | fail | -- | Cascading from A-3 MEDIUM failure | -| A-5 TINY | pass | stable (gencost) | All gens committed all hours (trivial UC) | -| A-5 SMALL | fail | -- | HiGHS timeout, no feasible solution | -| A-6 TINY | pass | stable | Ramp constraints demonstrably binding | -| A-6 SMALL | fail | -- | Cascading from A-5 SMALL failure | -| A-7 TINY | pass | stable (manual loop) | lpf_contingency bug, 674 cases in 72s | -| A-7 MEDIUM | fail | -- | ~65s per LPF, impractical runtime | -| A-8 TINY | fail | blocking | No native stochastic optimization | -| A-8 SMALL | fail | blocking | Same capability gap | -| A-9 TINY | pass | stable (rating 1.5x) | Objective equality due to uniform costs | -| A-9 SMALL | pass | stable | SCOPF 0.4% more expensive, 70s | -| A-10 TINY | pass | stable | Loss components non-zero, no MATPOWER reference validation | -| A-10 SMALL | pass | stable | 2.71% objective increase from losses | -| A-11 TINY | pass | -- | OPF LMPs unaffected by slack (by design) | -| A-11 SMALL | pass | stable (gencost) | LMPs identical across configs | -| B-1 TINY | pass | -- | 2 LOC, dual value verified | -| B-1 MEDIUM | fail | -- | Cascading from A-3 MEDIUM infeasibility | -| B-2 TINY | pass | -- | 4 LOC NetworkX access | -| B-2 MEDIUM | pass | -- | 0.84s, works at scale | -| B-3 TINY | pass | stable | 87ms/contingency, lpf_contingency bug | -| B-3 MEDIUM | fail | -- | Impractical runtime (~1 iter/min) | -| B-4 TINY | pass | stable | 20 scenarios, 9.68s total | -| B-4 SMALL | pass | stable | 20 scenarios, 2217s total | -| B-5 TINY | pass | -- | 2 LOC trivial DataFrame export | -| B-5 MEDIUM | pass | -- | Works despite NaN flows | -| B-6 | informational | -- | 5-layer architecture, 85% docstring coverage | -| B-7 TINY | pass | stable | Both workarounds stable, low effort | -| B-7 MEDIUM | pass | stable | Audit only, references TINY result | -| B-8 TINY | pass | -- | Slack change via DataFrame edit, no model rebuild | -| B-8 SMALL | pass | -- | All configs produce identical LMPs | -| B-9 TINY | pass | -- | Machine-precision flow match (1.88e-12) | -| B-9 MEDIUM | fail | -- | Singular B-matrix from zero-impedance branches | -| C-1 | pass | data_prep | 28.3s solve, zero-impedance fix applied | -| C-2 | pass | -- | 18.1s flat-start convergence, zero angles suspicious | -| C-3 | pass | data_prep | 19.9s solver, 10+ min post-processing | -| C-4 | fail | -- | HiGHS timeout, no feasible solution | -| C-5 | qualified_pass | data_prep | Only 9/270 N-1 cases -- should be fail | -| C-6 | pass | stable | 20 scenarios, 2219s, 7.1 GB | -| C-7 | pass | data_prep | Only HiGHS tested (others not installed) | -| C-8 | fail | data_prep | Post-processing timeout, SCOPF not reached | -| C-9 | qualified_pass | data_prep | 702 MW flow error from data fix | -| C-10 | pass | data_prep | PF comparison not completed | -| D-1 | pass | -- | 3 steps to first solve | -| D-2 | qualified_pass | -- | 7/11 tests from docs; A-8 docs claim contradicts A-8 result | -| D-3 | pass | -- | All 3 examples run unmodified | -| D-4 | qualified_pass | -- | Missing-cost: A; Infeasible: B+; Invalid enum: D | -| D-5 | informational | -- | 158-485 LOC per test | -| E-1 | informational | -- | 24 releases/24mo, strict SemVer | -| E-2 | informational | -- | 327 commits, 32 contributors | -| E-3 | informational | -- | Bus factor 3-4, 53% recent concentration | -| E-4 | informational | -- | TU Berlin + OET hybrid funding | -| E-5 | informational | -- | 21h median close time | -| E-6 | informational | -- | 84% coverage, 3-platform CI | -| E-7 | informational | -- | IEA, ACER, ENTSO-E, Shell adoption | -| F-1 | pass | -- | MIT throughout | -| F-2 | qualified_pass | -- | 87 packages, moderate footprint | -| F-3 | qualified_pass | -- | 1 GPL (Levenshtein), replaceable | -| F-4 | pass | -- | Pure Python core, 1 HiGHS .so | -| F-5 | pass | -- | 4 Python layers to solver | -| F-6 | pass | -- | Sigstore attestations | -| F-7 | pass | -- | All wheels available offline | -| F-8 | pass | -- | HiGHS covers all modes | -| F-9 | qualified_pass | -- | Version-tagged downloads, optional version check | -| P2-1 | informational | -- | No PSS/E RAW parser | -| P2-2 | informational | -- | No PWL costs; quadratic works (QP) | -| P2-3 | informational | -- | Full UC-to-DCOPF-to-ACPF pipeline works | diff --git a/sweep-data/v4-to-v5/per-tool/pypsa/findings.yaml b/sweep-data/v4-to-v5/per-tool/pypsa/findings.yaml deleted file mode 100644 index 6cd837fc..00000000 --- a/sweep-data/v4-to-v5/per-tool/pypsa/findings.yaml +++ /dev/null @@ -1,398 +0,0 @@ -tool: pypsa -source_version: "v4" -timestamp: "2026-03-07T19:00:00Z" -evaluation_summary: - total_tests: 62 - pass: 37 - fail: 15 - qualified_pass: 4 - informational: 6 - -findings: - - id: pypsa-F01 - category: infrastructure_friction - severity: medium - test_ids: [A-3, A-4, A-5, A-6, A-9, A-10, A-11, B-1, B-4, B-8] - title: "Gencost import gap forces manual cost assignment across all OPF tests" - description: > - PyPSA's import_from_pypower_ppc() silently drops gencost data from MATPOWER - files (emitting only a warning). This forces manual cost parsing and assignment - (~5 LOC) as a prerequisite for every OPF-based test. The workaround is stable - and uses public API, but it is test infrastructure friction that affects all - tools using MATPOWER .m format via PPC import. - evidence: - - file: expressiveness/A-3_dcopf.md - excerpt: "PPC importer does not import gencost data (PyPSA warns: 'some PYPOWER features not supported: areas, gencosts, component status')" - - file: gate/G-1_tiny_case39.md - excerpt: "Generator marginal_cost is zero for all 10 generators. PyPSA's import_from_pypower_ppc explicitly does not import gencost data" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pypsa-F02 - category: infrastructure_friction - severity: high - test_ids: [A-1, A-3, A-4, A-7, B-1, B-3, B-5, B-9, C-1, C-3, C-5, C-7, C-8, C-9, C-10] - title: "ACTIVSg10k zero-impedance branches cause cascading MEDIUM failures" - description: > - The ACTIVSg10k MATPOWER case contains 2,462 branches with s_nom=0 and 3 - transformers with zero reactance. This causes DCOPF infeasibility (A-3 MEDIUM), - singular B-matrix (A-1 MEDIUM produces NaN flows), and PTDF computation failure - (B-9 MEDIUM). Scalability tests applied data fixes (x=0.0001, s_nom=9999) but - expressiveness tests did not, creating an inconsistency where the same tool - passes scalability but fails expressiveness on the same network. - evidence: - - file: expressiveness/A-3_dcopf_MEDIUM.md - excerpt: "The MATPOWER case file contains 2,462 branches with s_nom == 0. PyPSA interprets s_nom == 0 as a zero-capacity line constraint" - - file: scalability/C-1_dcpf_scale_MEDIUM.md - excerpt: "Required fixing 3 transformers with zero reactance (x=0.0001) to avoid singular matrix" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pypsa-F03 - category: misleading_result - severity: medium - test_ids: [C-5] - title: "Contingency sweep C-5 qualified_pass with only 3.3% completion" - description: > - C-5 (contingency sweep at MEDIUM) is classified as qualified_pass but completed - only 9 of 270 N-1 cases (3.3%) within the 600s timeout. N-2 through N-4 were - entirely skipped. The pass condition states "Completes. Per-contingency-case - average and total time recorded." Completing 3.3% of N-1 cases does not - constitute completion by any reasonable interpretation. This should be - reclassified as fail. - evidence: - - file: scalability/C-5_contingency_scale_MEDIUM.md - excerpt: "N-1 cases completed: 9 of 270, N-2 cases: Skipped (timeout), N-3/N-4 cases: Skipped (timeout)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pypsa-F04 - category: misleading_result - severity: medium - test_ids: [C-9] - title: "PTDF C-9 qualified_pass with 702 MW flow prediction error" - description: > - C-9 (PTDF at MEDIUM) is classified as qualified_pass but the flow prediction - accuracy is severely degraded: max diff of 702 MW vs the 1e-6 tolerance - specified in the B-9 pass condition. The PTDF matrix computed correctly in - structure but the zero-impedance workaround (x=0.0001) fundamentally altered - the sensitivity structure. This is a data issue rather than a tool issue, but - the qualified_pass status may mislead readers into thinking PTDF works at - MEDIUM scale. - evidence: - - file: scalability/C-9_ptdf_scale_MEDIUM.md - excerpt: "Flow prediction match: No (max diff 702 MW)" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pypsa-F05 - category: extraordinary_claim - severity: medium - test_ids: [A-2, C-2] - title: "ACPF MEDIUM pass despite non-convergence warning" - description: > - A-2 MEDIUM reports convergence: true and pass, but the result file notes that - PyPSA emitted "Power flow did not converge for ['now']". The test classified - it as a pass because voltage magnitudes were in a reasonable range (0.96-1.08 - pu), but the solver itself reported non-convergence. The 3,935 MW total losses - and transformer flows up to 14,700 MW suggest modeling artifacts. C-2 reports - pass with 18.1s solve time but notes voltage angles are all zero, which is - physically suspicious for a 10k-bus network. - evidence: - - file: expressiveness/A-2_acpf_MEDIUM.md - excerpt: "Despite a convergence warning ('Power flow did not converge for [\"now\"]'), the test code reports converged: true" - - file: scalability/C-2_acpf_scale_MEDIUM.md - excerpt: "Voltage angle range: 0 rad (DC angles not populated in ACPF)" - cross_tool_relevance: likely - probe_recommended: true - probe_type: convergence_check - proposed_action: add_verification - - - id: pypsa-F06 - category: missing_verification - severity: medium - test_ids: [A-9] - title: "SCOPF objective equality masks test signal due to uniform costs" - description: > - A-9 (SCOPF) shows identical objectives for DCOPF and SCOPF ($1876.269) because - all generators in case39 have identical marginal costs (c1=0.3). The synthesis - correctly notes this, but the test cannot verify that SCOPF is more expensive - than unconstrained DCOPF -- a key pass condition. The dispatch redistribution - is verified but the cost signal is absent. A network with diverse generator - costs would better exercise this test. - evidence: - - file: expressiveness/A-9_scopf.md - excerpt: "Both objectives are 1876.269 because all generators have identical marginal costs (C1=0.3 $/MWh)" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pypsa-F07 - category: network_insufficiency - severity: low - test_ids: [A-5] - title: "SCUC on TINY shows no generator cycling due to high load vs capacity" - description: > - A-5 (SCUC) on TINY shows all 10 generators committed for all 24 hours with - 0% MIP gap and only 1 B&B node. The TINY network's load is high relative to - generation capacity, so no unit commitment cycling occurs. Min up/down time - and startup cost constraints never bind. The test passes functionally but does - not exercise the core MILP difficulty of UC. - evidence: - - file: expressiveness/A-5_scuc.md - excerpt: "Generators always on: 10/10, Min simultaneous online: 10, B&B nodes: 1" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pypsa-F08 - category: extraordinary_claim - severity: high - test_ids: [D-2] - title: "D-2 claims A-8 stochastic docs coverage YES but A-8 test FAILs" - description: > - The documentation audit (D-2) rates A-8 stochastic optimization documentation - as "YES (as of v1.x)" with a dedicated page and example notebook. However, - A-8 itself FAILs because the n.scenarios data model is not wired to the - optimizer. The documentation appears to describe aspirational or incomplete - functionality. This is either a documentation-implementation mismatch or the - evaluator found documentation for a feature that does not actually work. - evidence: - - file: accessibility/D-2_documentation_audit.md - excerpt: "A-8: Documented: YES (as of v1.x). Dedicated page (user-guide/optimization/stochastic/)" - - file: expressiveness/A-8_stochastic_timeseries.md - excerpt: "n.scenarios exists as an empty Index in the data model but is not wired into the optimizer" - cross_tool_relevance: none - probe_recommended: true - probe_type: claim_verification - proposed_action: add_verification - - - id: pypsa-F09 - category: missing_verification - severity: medium - test_ids: [A-10] - title: "Lossy DCOPF LMP decomposition not validated against MATPOWER reference" - description: > - The A-10 pass condition requires "Validate against MATPOWER reference lossy DC - OPF solution on same case (tolerance: 1% on total LMP, directional consistency - on loss component signs)." The result file shows LMP decomposition was performed - and loss components have physically correct signs, but no cross-validation - against MATPOWER rundcopf was documented. The reconciliation between congestion - rents and LMP components is shown but the reference validation step appears - missing. - evidence: - - file: expressiveness/A-10_lossy_dcopf_lmp.md - excerpt: "Per-Line Congestion Rent: Total rent (all lines): 13.05, Largest single-line rent: 2.10 (L2)" - cross_tool_relevance: confirmed - probe_recommended: true - probe_type: formulation_audit - proposed_action: add_verification - - - id: pypsa-F10 - category: low_signal - severity: low - test_ids: [A-11] - title: "Distributed slack OPF test has low signal -- OPF LMPs identical regardless of slack" - description: > - A-11 and B-8 both demonstrate that changing the slack bus in PyPSA's DCOPF - produces identical LMPs because OPF enforces power balance as a constraint - (not via a slack bus). The distributed slack only affects PF voltage angles, - not OPF results. This is architecturally correct but means the test cannot - differentiate tools where the slack bus does vs does not affect OPF LMPs. - The test exercises PF distributed slack, not OPF distributed slack. - evidence: - - file: extensibility/B-8_reference_bus_config.md - excerpt: "All three OPF configurations produce identical LMPs and objectives" - - file: expressiveness/A-11_distributed_slack_opf.md - excerpt: "OPF inherently distributes generation optimally... There is no single slack bus absorbing the mismatch" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pypsa-F11 - category: misleading_result - severity: medium - test_ids: [A-1] - title: "A-1 MEDIUM pass despite all flows being NaN" - description: > - A-1 DCPF at MEDIUM is classified as pass, but all voltage angles and line - flows are NaN due to the singular B-matrix from zero-impedance branches. The - result file states "Despite the NaN flows... the test passes because convergence, - structured output, and accessibility criteria are met." Reporting a power flow - as passing when all computed quantities are NaN is misleading. The solver - "converges" only in the sense that spsolve returns without error. - evidence: - - file: expressiveness/A-1_dcpf_MEDIUM.md - excerpt: "Voltage angles: DataFrame, all NaN due to singular matrix... Line flows: DataFrame, all NaN" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: adjust_scoring - - - id: pypsa-F12 - category: missing_verification - severity: low - test_ids: [C-3, C-7, C-8, C-10] - title: "Scalability wall-clock times dominated by linopy post-processing, not solver" - description: > - Multiple MEDIUM scalability tests report wall-clock times of 600s+ but with - HiGHS solve times of 6-21s. The remaining time is linopy's shadow-price - assignment (dual variable extraction and network assignment). This is a real - scalability limitation but the reported wall-clock times may mislead cross-tool - comparisons where other tools do not have this post-processing overhead. The - solver time vs post-processing time should be clearly separated in any - comparison. - evidence: - - file: scalability/C-3_dcopf_scale_MEDIUM.md - excerpt: "HiGHS solve time: 19.9 s... linopy's shadow-price assignment takes 10+ minutes" - - file: scalability/C-8_scopf_scale_MEDIUM.md - excerpt: "Baseline DCOPF HiGHS solve: 7.1 s, Post-processing: 10+ min (exceeded budget)" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: add_verification - - - id: pypsa-F13 - category: scoring_inconsistency - severity: medium - test_ids: [A-3, C-3] - title: "Inconsistent data fixes between expressiveness and scalability on same network" - description: > - A-3 MEDIUM (expressiveness) fails because zero s_nom branches cause - infeasibility, with no data fixes applied. C-3 MEDIUM (scalability) passes - because s_nom=9999 and x=0.0001 fixes were applied. Both test the same - capability (DCOPF) on the same network. The inconsistent treatment creates - a situation where the tool appears to fail on expressiveness but pass on - scalability for the same analysis on the same network. - evidence: - - file: expressiveness/A-3_dcopf_MEDIUM.md - excerpt: "status: fail... The MATPOWER case file contains 2,462 branches with s_nom == 0" - - file: scalability/C-3_dcopf_scale_MEDIUM.md - excerpt: "status: pass... Set s_nom=9999 on 2,462 lines with zero thermal rating" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: redesign_test - - - id: pypsa-F14 - category: test_design_gap - severity: low - test_ids: [C-7] - title: "Solver swap test verifies only API, not actual multi-solver comparison" - description: > - C-7 (solver swap) confirms that PyPSA's solver swap is parameter-only, but - only HiGHS was installed in the evaluation environment. GLPK, SCIP, Gurobi, - and CPLEX were all unavailable. The test verified the mechanism by API - inspection rather than actually solving with multiple solvers and comparing - results. The pass condition ("Solver swap requires only a parameter change, - not reformulation") is met by inspection, but no runtime verification was - performed. - evidence: - - file: scalability/C-7_solver_swap_MEDIUM.md - excerpt: "GLPK available: No, SCIP available: No, Gurobi available: No, CPLEX available: No" - cross_tool_relevance: likely - probe_recommended: false - probe_type: null - proposed_action: add_verification - - - id: pypsa-F15 - category: low_signal - severity: low - test_ids: [B-5] - title: "B-5 interoperability test trivially passes for any DataFrame-based tool" - description: > - B-5 tests whether results can be exported to CSV. For PyPSA, which stores all - results as pandas DataFrames, this is a 2-LOC .to_csv() call. The test has no - discriminative value for any tool that uses DataFrames natively. It would have - the same result for pandapower or any other Python tool with DataFrame outputs. - evidence: - - file: extensibility/B-5_interoperability.md - excerpt: "LOC: 2... n.buses_t.v_ang.to_csv('bus_angles.csv')" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pypsa-F16 - category: missing_verification - severity: low - test_ids: [E-1, E-2, E-3, E-4, E-5, E-6, E-7] - title: "Maturity metrics based on research rather than runtime verification" - description: > - All E-series tests (maturity) report detailed metrics (release counts, - commit statistics, contributor percentages, funding sources, issue response - times, CI coverage, operational adoption) but these are research-based - findings, not runtime-verified data. The numbers appear reasonable and - internally consistent but were not independently verified against GitHub - API responses or PyPI records during this sweep. - evidence: - - file: maturity/E-2_commit_activity.md - excerpt: "Total commits (Mar 2025 - Mar 2026): 327, Unique committers (human): 32" - cross_tool_relevance: confirmed - probe_recommended: false - probe_type: null - proposed_action: null - - - id: pypsa-F17 - category: extraordinary_claim - severity: low - test_ids: [C-10] - title: "C-10 distributed slack at MEDIUM claims pass but PF comparison not completed" - description: > - C-10 is classified as pass but the result file notes "The full test (DCOPF + - single-slack PF + distributed-slack PF comparison) could not complete within - 600s due to linopy's shadow-price assignment overhead." The DCOPF solved in - 6.1s but the PF comparison that constitutes the actual test was not performed. - The pass is based on DCOPF convergence and the existence of the distributed - slack API, not on demonstrated distributed slack behavior at MEDIUM scale. - evidence: - - file: scalability/C-10_distributed_slack_scale_MEDIUM.md - excerpt: "Post-processing overhead prevents full PF comparison within time budget" - cross_tool_relevance: none - probe_recommended: true - probe_type: timing_verification - proposed_action: adjust_scoring - -extraordinary_claims: - - test_id: A-2 MEDIUM - claim: "ACPF converges on 10k-bus network despite solver reporting non-convergence" - concern: > - The Newton-Raphson solver explicitly warns 'Power flow did not converge' but - the test classifies the result as pass based on voltage magnitude ranges. - A non-converged power flow may produce physically plausible-looking but - mathematically incorrect results. - evidence_quality: moderate - probe_recommended: true - probe_type: convergence_check - - - test_id: D-2 - claim: "A-8 stochastic optimization is documented with dedicated page and example notebook" - concern: > - A-8 FAIL demonstrates the feature does not work in v1.1.2. Either the - documentation describes a non-functional feature (aspirational docs), or the - evaluation missed a working API path. The synthesis flags this for human - spot-check. - evidence_quality: moderate - probe_recommended: true - probe_type: claim_verification - - - test_id: C-10 - claim: "Distributed slack OPF passes at MEDIUM scale" - concern: > - The PF comparison was not completed within the time budget. The pass is based - on DCOPF solving (which is not distributed-slack-specific) and API existence, - not on demonstrated behavior. - evidence_quality: weak - probe_recommended: true - probe_type: timing_verification diff --git a/sweep-data/v4-to-v5/probe-manifest.yaml b/sweep-data/v4-to-v5/probe-manifest.yaml deleted file mode 100644 index 076685be..00000000 --- a/sweep-data/v4-to-v5/probe-manifest.yaml +++ /dev/null @@ -1,306 +0,0 @@ -probes: - # ============================================================ - # PyPSA (5 probes) - # ============================================================ - - id: probe-001 - tool: pypsa - claim: "ACPF converges on 10k-bus network despite solver reporting non-convergence" - source_test: A-2 - source_file: expressiveness/A-2_acpf_MEDIUM.md - probe_type: convergence_check - priority: high - finding_id: pypsa-F05 - - - id: probe-002 - tool: pypsa - claim: "A-8 stochastic optimization is documented with dedicated page and example notebook, but A-8 test FAILs" - source_test: D-2 - source_file: accessibility/D-2_documentation_audit.md - probe_type: claim_verification - priority: high - finding_id: pypsa-F08 - - - id: probe-003 - tool: pypsa - claim: "Lossy DCOPF LMP decomposition not validated against MATPOWER reference rundcopf" - source_test: A-10 - source_file: expressiveness/A-10_lossy_dcopf_lmp.md - probe_type: formulation_audit - priority: medium - finding_id: pypsa-F09 - - - id: probe-004 - tool: pypsa - claim: "Scalability wall-clock times dominated by linopy post-processing (10+ min), not HiGHS solver (6-21s)" - source_test: C-3 - source_file: scalability/C-3_dcopf_scale_MEDIUM.md - probe_type: timing_verification - priority: medium - finding_id: pypsa-F12 - - - id: probe-005 - tool: pypsa - claim: "C-10 distributed slack OPF passes at MEDIUM scale but PF comparison was not completed within time budget" - source_test: C-10 - source_file: scalability/C-10_distributed_slack_scale_MEDIUM.md - probe_type: timing_verification - priority: medium - finding_id: pypsa-F17 - - # ============================================================ - # pandapower (5 probes) - # ============================================================ - - id: probe-006 - tool: pandapower - claim: "Stochastic DCOPF wrapping qualified_pass despite 2.1% solver convergence rate (5 of 240 solves)" - source_test: C-6 - source_file: scalability/C-6_stochastic_scale.md - probe_type: convergence_check - priority: high - finding_id: pandapower-F01 - - - id: probe-007 - tool: pandapower - claim: "Uniform LMPs across all 10,000 buses (20.738 everywhere) suggest no binding constraints on MEDIUM DCOPF" - source_test: A-3 - source_file: expressiveness/A-3_dcopf.md - probe_type: formulation_audit - priority: medium - finding_id: pandapower-F02 - - - id: probe-008 - tool: pandapower - claim: "PTDF flow predictions diverge from DCPF on MEDIUM (max diff 7.43 pu) but test still passes" - source_test: B-9 - source_file: extensibility/B-9_ptdf_extraction.md - probe_type: convergence_check - priority: high - finding_id: pandapower-F03 - - - id: probe-009 - tool: pandapower - claim: "PYPOWER interior point solver produces lambda values of 1e25 when generators decommitted via in_service=False" - source_test: P2-3 - source_file: p2_readiness/P2-3_commitment_injection_TINY.md - probe_type: convergence_check - priority: high - finding_id: pandapower-F10 - - - id: probe-010 - tool: pandapower - claim: "PTDF matrix correctly computed despite 7.43 pu max flow prediction error; divergence attributed to shunt/tap effects without verification" - source_test: B-9 - source_file: extensibility/B-9_ptdf_extraction.md - probe_type: formulation_audit - priority: medium - finding_id: pandapower-F03-EC - - # ============================================================ - # GridCal (4 probes) - # ============================================================ - - id: probe-011 - tool: gridcal - claim: "Contingency sweep correctness not verified against reference; pruning ratio of 0.0 on both networks" - source_test: A-7 - source_file: results/expressiveness/A-7_contingency_sweep_TINY.md - probe_type: convergence_check - priority: medium - finding_id: gridcal-F03 - - - id: probe-012 - tool: gridcal - claim: "PTDF flow predictions diverge from DCPF by up to 743 MW on MEDIUM; scored as qualified_pass" - source_test: C-9 - source_file: results/extensibility/B-9_ptdf_extraction_MEDIUM.md - probe_type: convergence_check - priority: high - finding_id: gridcal-F04 - - - id: probe-013 - tool: gridcal - claim: "Lossy DC OPF produces physically meaningful LMPs but not validated against MATPOWER reference" - source_test: A-10 - source_file: results/expressiveness/A-10_lossy_dcopf_TINY.md - probe_type: claim_verification - priority: medium - finding_id: gridcal-F08 - - - id: probe-014 - tool: gridcal - claim: "B-4 SMALL stochastic wrapping has 53% solve failure rate (113 of 240 solves succeeded)" - source_test: B-4 - source_file: results/extensibility/B-4_stochastic_wrapping.md - probe_type: formulation_audit - priority: medium - finding_id: gridcal-F10 - - # ============================================================ - # PowerModels (5 probes) - # ============================================================ - - id: probe-015 - tool: powermodels - claim: "Distributed slack LMP validation defeated by uncongested network; both formulations produce identical results" - source_test: A-11 - source_file: expressiveness/A-11_distributed_slack_opf_TINY.md - probe_type: formulation_audit - priority: medium - finding_id: pm-F01 - - - id: probe-016 - tool: powermodels - claim: "C-5 and C-8 scored as fail without execution, based on projected infeasibility estimates" - source_test: C-5 - source_file: scalability/C-5_contingency_sweep_scale_MEDIUM.md - probe_type: timing_verification - priority: high - finding_id: pm-F03 - - - id: probe-017 - tool: powermodels - claim: "ACPF MEDIUM failure lacks convergence diagnostics -- no iteration count, residual trajectory, or solver config" - source_test: A-2 - source_file: expressiveness/A-2_acpf_MEDIUM.md - probe_type: convergence_check - priority: medium - finding_id: pm-F08 - - - id: probe-018 - tool: powermodels - claim: "PTDF computation 6.6x faster at 10k-bus than 2k-bus due to JIT warm cache" - source_test: C-10 - source_file: scalability/C-10_distributed_slack_scale_MEDIUM.md - probe_type: timing_verification - priority: medium - finding_id: pm-F16 - - - id: probe-019 - tool: powermodels - claim: "SCOPF at MEDIUM with 500 contingencies would produce ~10M constraints, exceeding solver capacity" - source_test: C-8 - source_file: scalability/C-8_scopf_scale_MEDIUM.md - probe_type: timing_verification - priority: medium - finding_id: pm-F03-EC - - # ============================================================ - # PowerSimulations (8 probes) - # ============================================================ - - id: probe-020 - tool: powersimulations - claim: "Four scalability tests (C-3, C-4, C-5, C-6) report estimated timings without actual measurement" - source_test: C-3 - source_file: scalability/C-3_dcopf_scale.md - probe_type: timing_verification - priority: high - finding_id: psi-F01 - - - id: probe-021 - tool: powersimulations - claim: "PSI dispatch values are ~100x larger than component limits (possible unit mismatch or labeling error)" - source_test: A-4 - source_file: expressiveness/A-4_ac_feasibility.md - probe_type: claim_verification - priority: high - finding_id: psi-F02 - - - id: probe-022 - tool: powersimulations - claim: "B-1 custom constraint dual value verified only for non-binding case (dual=0); binding case never demonstrated" - source_test: B-1 - source_file: extensibility/B-1_custom_constraints.md - probe_type: formulation_audit - priority: medium - finding_id: psi-F03 - - - id: probe-023 - tool: powersimulations - claim: "Distributed slack qualified_pass on uncongested network where both formulations produce identical results" - source_test: A-11 - source_file: expressiveness/A-11_distributed_slack.md - probe_type: formulation_audit - priority: medium - finding_id: psi-F06 - - - id: probe-024 - tool: powersimulations - claim: "ACPF scale failure at MEDIUM lacks diagnostic detail on convergence" - source_test: C-2 - source_file: scalability/C-2_acpf_scale.md - probe_type: convergence_check - priority: medium - finding_id: psi-F10 - - - id: probe-025 - tool: powersimulations - claim: "100% code coverage reported from Codecov badge without independent verification" - source_test: E-6 - source_file: maturity/E-6_ci_test_coverage.md - probe_type: claim_verification - priority: medium - finding_id: psi-F12 - - - id: probe-026 - tool: powersimulations - claim: "DCOPF on MEDIUM expected to solve in <60s (estimate, not measured)" - source_test: C-3 - source_file: scalability/C-3_dcopf_scale.md - probe_type: timing_verification - priority: medium - finding_id: psi-F01-EC-C3 - - - id: probe-027 - tool: powersimulations - claim: "SCUC on SMALL expected to take >300s with SCIP (estimate, not measured)" - source_test: C-4 - source_file: scalability/C-4_scuc_scale.md - probe_type: timing_verification - priority: low - finding_id: psi-F01-EC-C4 - - # ============================================================ - # MATPOWER (5 probes) - # ============================================================ - - id: probe-028 - tool: matpower - claim: "Distributed slack DC OPF on MEDIUM takes 66 minutes via opt_model/MIPS (400x slower than single-slack)" - source_test: C-10 - source_file: scalability/C-10_distributed_slack_scale_MEDIUM.md - probe_type: timing_verification - priority: high - finding_id: matpower-F01 - - - id: probe-029 - tool: matpower - claim: "Contingency sweep on MEDIUM takes 41 min but 97% of time is Octave containers.Map overhead, not MATPOWER" - source_test: C-5 - source_file: scalability/C-5_contingency_sweep_scale_MEDIUM.md - probe_type: timing_verification - priority: high - finding_id: matpower-F02 - - - id: probe-030 - tool: matpower - claim: "SCUC produces all-committed schedule; UC cycling logic never exercised" - source_test: A-5 - source_file: expressiveness/A-5_scuc_TINY.md - probe_type: formulation_audit - priority: medium - finding_id: matpower-F05 - - - id: probe-031 - tool: matpower - claim: "Distributed slack LMP comparison lacks specific numerical values; sign convention issue in opt_model extraction" - source_test: A-11 - source_file: expressiveness/A-11_distributed_slack_opf_TINY.md - probe_type: formulation_audit - priority: medium - finding_id: matpower-F08 - - - id: probe-032 - tool: matpower - claim: "C-4, C-6, C-8 scalability failures attributed to solver capacity limits but actually failed at loadmd() data ingestion" - source_test: C-4 - source_file: scalability/C-4_scuc_scale_SMALL.md - probe_type: claim_verification - priority: high - finding_id: matpower-F06 diff --git a/sweep-data/v4-to-v5/probes/gridcal/probe-012.md b/sweep-data/v4-to-v5/probes/gridcal/probe-012.md deleted file mode 100644 index e380f429..00000000 --- a/sweep-data/v4-to-v5/probes/gridcal/probe-012.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -probe_id: probe-012 -tool: gridcal -source_test: B-9 -probe_type: convergence_check -classification: claim_supported -reason: "Reproduced max diff of 743.46 MW (LA vs DCPF) and 15139.36 MW (PTDF@Sbus vs DCPF), matching original claim exactly" -solver_version: "5.6.28" -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 22.71 -timestamp: 2026-03-09T00:00:00Z ---- - -# Probe probe-012: PTDF flow predictions diverge from DCPF by up to 743 MW on MEDIUM - -## Original Claim - -From `evaluations/gridcal/results/extensibility/B-9_ptdf_extraction_MEDIUM.md`: - -> LA direct flows vs DCPF: max abs diff 743.46 MW, mean abs diff 2.68 MW. -> PTDF @ Sbus vs DCPF: max abs diff 15,139.36 MW, mean abs diff 29.57 MW. -> Scored as qualified_pass because PTDF matrix is accessible and dimensions are correct, -> but flow prediction mismatch on the large network prevents full pass. - -## Probe Methodology - -Wrote a standalone script that: -1. Loads the ACTIVSg 10k-bus network in GridCal (veragridengine 5.6.28) -2. Runs DCPF (SolverType.Linear) to get reference flows -3. Runs LinearAnalysis (vge.linear_power_flow) to get PTDF and LA direct flows -4. Compares LA direct flows vs DCPF flows -5. Computes PTDF @ Sbus and compares to DCPF flows -6. Investigates root cause: island count, transformer tap correlation, slack bus effects - -Script path: `sweep-data/v4-to-v5/probes/gridcal/probe-012_script.py` - -Executed via: - -``` -.devcontainer/dc-exec -C /workspace/evaluations/gridcal timeout 300 uv run python -c "$(cat script)" -``` - -## Probe Results - -``` -GridCal (veragridengine) version: 5.6.28 -Network: 10000 buses, 12706 branches (9726 lines, 2980 transformers) -Islands: 1 - ---- DCPF solve --- -Converged: True, wall clock: 0.223s -DCPF flows range: [-1839.578, 2035.364] - ---- LinearAnalysis --- -PTDF compute time: 15.494s -PTDF shape: (12706, 10000), range: [-2.339, 1.790] - ---- LA direct flows vs DCPF --- -Max abs diff: 743.4624 MW -Mean abs diff: 2.6797 MW -Median abs diff: 0.1303 MW -90th pctile: 4.1068 MW -99th pctile: 43.6389 MW -Branches > 1 MW diff: 3007 -Branches > 10 MW diff: 575 -Branches > 100 MW diff: 53 - -Top 3 worst branches: - [11244] Xfmr '28737_28745_1': LA=1291.9, DCPF=2035.4, diff=743.5 - [7276] Line '50203_50059_1': LA=101.1, DCPF=-461.8, diff=562.9 - [11955] Xfmr '50203_50207_1': LA=-101.1, DCPF=461.8, diff=562.9 - ---- Divergence by branch type --- -Lines (9726): max=562.87, mean=2.75 -Transformers (2980): max=743.46, mean=2.46 - ---- PTDF @ Sbus vs DCPF --- -Max abs diff: 15139.36 MW -Mean abs diff: 29.57 MW - ---- Root cause investigation --- -Islands: 1 (single island; NOT the cause) -Non-unity tap transformers: 970 / 2980 -Tap deviation vs error correlation: -0.03 (no correlation; taps NOT the cause) -Slack bus: Bus 7236 'PHOENIX 74 6', injection=-1119.5 MW -Zeroing slack injection did not improve PTDF@Sbus match - -Total probe wall clock: 22.71s -``` - -## Analysis - -The probe reproduces the original claim's numbers with exact precision: -- LA vs DCPF max diff: 743.46 MW (original: 743.46 MW) -- PTDF @ Sbus vs DCPF max diff: 15,139.36 MW (original: 15,139.36 MW) -- Mean diffs and other statistics also match precisely - -Root cause investigation: -- **Island handling**: Ruled out. The network has a single island. -- **Transformer tap effects**: Ruled out. Correlation between tap deviation and flow error is -0.03 (essentially zero). Both lines and transformers show large errors, and the mean error is actually slightly lower for transformers. -- **Slack bus treatment**: The PTDF correctly zeroes out the slack bus column (bus 7236). Zeroing the slack injection in Sbus before computing PTDF@Sbus did not help. -- **Most likely cause**: The LinearAnalysis and DCPF solvers use different internal formulations or admittance matrix constructions on large networks. The worst mismatches cluster around specific subnetwork regions (buses 28xxx, 50xxx), suggesting localized numerical differences in how the two solvers build and factor the susceptance matrix. The divergence grows with network complexity -- the original evaluation noted exact match on the 39-bus network. - -The qualified_pass scoring is reasonable: the PTDF matrix is accessible, correctly dimensioned, and usable for sensitivity analysis, but its absolute flow predictions diverge from the DCPF solver on this large network. - -## Classification Rationale - -Classified as **claim_supported** because: -1. The probe reproduced the exact max diff values (743.46 MW and 15,139.36 MW) on the same version (5.6.28) -2. The qualified_pass scoring rationale (PTDF accessible but flow predictions diverge) is confirmed -3. Root cause investigation confirms the divergence is real and not due to trivial causes (islands or tap ratios) diff --git a/sweep-data/v4-to-v5/probes/gridcal/probe-012_script.py b/sweep-data/v4-to-v5/probes/gridcal/probe-012_script.py deleted file mode 100644 index aab6a9bf..00000000 --- a/sweep-data/v4-to-v5/probes/gridcal/probe-012_script.py +++ /dev/null @@ -1,182 +0,0 @@ -"""Probe-012: Verify PTDF vs DCPF divergence claim on ACTIVSg10k. - -Investigates whether the 743 MW max divergence between LinearAnalysis flows -and DCPF flows is real, and diagnoses the root cause (islands, transformer -taps, slack bus treatment, or computational error). -""" - -from __future__ import annotations - -import importlib.metadata -import time -from pathlib import Path - -import numpy as np - -DATA_DIR = Path("/workspace/data/networks") -NETWORK_FILE = str(DATA_DIR / "case_ACTIVSg10k.m") - - -def main(): - print("=" * 70) - print("Probe-012: PTDF vs DCPF divergence on ACTIVSg10k") - print("=" * 70) - - import VeraGridEngine as vge - from VeraGridEngine.enumerations import SolverType - - ver = importlib.metadata.version("veragridengine") - print(f"GridCal (veragridengine) version: {ver}") - - # ── Load network ── - grid = vge.open_file(NETWORK_FILE) - n_bus = grid.get_bus_number() - branches = list(grid.lines) + list(grid.transformers2w) - n_branch = len(branches) - n_lines = len(list(grid.lines)) - n_xfmrs = len(list(grid.transformers2w)) - print(f"Network: {n_bus} buses, {n_branch} branches") - print(f" Lines: {n_lines}, Transformers2W: {n_xfmrs}") - - # ── Island detection ── - nc = vge.compile_numerical_circuit_at(grid) - islands = nc.split_into_islands() - n_islands = len(islands) - print(f"\nIsland analysis: {n_islands} island(s)") - for i, isl in enumerate(islands): - print(f" Island {i}: {isl.nbus} buses, {isl.nbr} branches") - - # ── Step 1: DCPF ── - print("\n--- DCPF solve ---") - pf_opts = vge.PowerFlowOptions(solver_type=SolverType.Linear) - t0 = time.perf_counter() - pf_results = vge.power_flow(grid, options=pf_opts) - t_dcpf = time.perf_counter() - t0 - print(f"DCPF converged: {pf_results.converged}") - print(f"DCPF wall clock: {t_dcpf:.3f}s") - - dcpf_flows = pf_results.Sf.real - sbus = pf_results.Sbus.real - print(f"DCPF flows shape: {dcpf_flows.shape}") - print(f"DCPF flows range: [{dcpf_flows.min():.3f}, {dcpf_flows.max():.3f}]") - print(f"Sbus shape: {sbus.shape}, sum: {sbus.sum():.6f}") - - # ── Step 2: LinearAnalysis (PTDF) ── - print("\n--- LinearAnalysis ---") - t0 = time.perf_counter() - la_results = vge.linear_power_flow(grid) - t_ptdf = time.perf_counter() - t0 - print(f"PTDF compute time: {t_ptdf:.3f}s") - - ptdf = la_results.PTDF - print(f"PTDF shape: {ptdf.shape}") - print(f"PTDF range: [{ptdf.min():.6f}, {ptdf.max():.6f}]") - - # ── Step 3: Compare LA direct flows vs DCPF ── - la_flows = None - if hasattr(la_results, "Sf") and la_results.Sf is not None: - la_flows = la_results.Sf.real - - if la_flows is not None: - la_diff = np.abs(la_flows - dcpf_flows) - print("\n--- LA direct flows vs DCPF ---") - print(f"Max abs diff: {la_diff.max():.4f} MW") - print(f"Mean abs diff: {la_diff.mean():.4f} MW") - print(f"Median abs diff: {np.median(la_diff):.4f} MW") - print(f"90th pctile: {np.percentile(la_diff, 90):.4f} MW") - print(f"99th pctile: {np.percentile(la_diff, 99):.4f} MW") - print(f"Branches with diff > 1 MW: {(la_diff > 1.0).sum()}") - print(f"Branches with diff > 10 MW: {(la_diff > 10.0).sum()}") - print(f"Branches with diff > 100 MW: {(la_diff > 100.0).sum()}") - - worst_indices = np.argsort(la_diff)[-10:][::-1] - print("\nTop 10 worst branches (LA vs DCPF):") - for idx in worst_indices: - br = branches[idx] - br_type = "Line" if idx < n_lines else "Xfmr" - print( - f" [{idx}] {br_type} '{br.name}': LA={la_flows[idx]:.3f}, " - f"DCPF={dcpf_flows[idx]:.3f}, diff={la_diff[idx]:.3f}" - ) - - # Split by branch type - line_diffs = la_diff[:n_lines] - xfmr_diffs = la_diff[n_lines:] - print("\n--- Divergence by branch type (LA vs DCPF) ---") - print( - f"Lines ({n_lines}): max={line_diffs.max():.4f}, mean={line_diffs.mean():.4f}" - ) - print( - f"Transformers ({n_xfmrs}): max={xfmr_diffs.max():.4f}, mean={xfmr_diffs.mean():.4f}" - ) - else: - la_diff = None - print("LA direct flows NOT available") - - # ── Step 4: PTDF @ Sbus vs DCPF ── - ptdf_predicted = ptdf @ sbus - ptdf_diff = np.abs(ptdf_predicted - dcpf_flows) - print("\n--- PTDF @ Sbus vs DCPF ---") - print(f"Max abs diff: {ptdf_diff.max():.4f} MW") - print(f"Mean abs diff: {ptdf_diff.mean():.4f} MW") - print(f"Branches with diff > 100 MW: {(ptdf_diff > 100.0).sum()}") - - # ── Step 5: Transformer tap analysis ── - print("\n--- Transformer tap analysis ---") - taps = [] - for t in grid.transformers2w: - taps.append(t.tap_module if hasattr(t, "tap_module") else None) - taps_arr = np.array([t for t in taps if t is not None]) - if len(taps_arr) > 0: - non_unity = np.abs(taps_arr - 1.0) > 1e-6 - print(f"Non-unity tap transformers: {non_unity.sum()} / {len(taps_arr)}") - print(f"Tap ratio range: [{taps_arr.min():.6f}, {taps_arr.max():.6f}]") - - if la_diff is not None and len(taps_arr) == n_xfmrs: - tap_dev = np.abs(taps_arr - 1.0) - xfmr_err = la_diff[n_lines:] - if non_unity.sum() > 2: - corr = np.corrcoef(tap_dev[non_unity], xfmr_err[non_unity])[0, 1] - print(f"Correlation (tap deviation vs xfmr error): {corr:.4f}") - - # ── Step 6: Slack bus ── - print("\n--- Slack bus analysis ---") - slack_buses = [(i, bus.name) for i, bus in enumerate(grid.buses) if bus.is_slack] - print(f"Slack buses: {len(slack_buses)}") - for idx, name in slack_buses[:5]: - print(f" Bus {idx}: '{name}'") - - col_sums = np.abs(ptdf).sum(axis=0) - min_col_idx = int(np.argmin(col_sums)) - print(f"PTDF near-zero column: bus {min_col_idx} (sum={col_sums[min_col_idx]:.8f})") - - # Try slack correction - if slack_buses: - slack_idx = slack_buses[0][0] - sbus_corr = sbus.copy() - slack_power = sbus_corr[slack_idx] - print(f"Slack bus injection: {slack_power:.4f} MW") - sbus_corr[slack_idx] = 0.0 - ptdf_corr = ptdf @ sbus_corr - corr_diff = np.abs(ptdf_corr - dcpf_flows) - print("After zeroing slack injection:") - print(f" Max abs diff: {corr_diff.max():.4f} MW") - print(f" Mean abs diff: {corr_diff.mean():.4f} MW") - - # ── Summary ── - print("\n" + "=" * 70) - print("SUMMARY") - print("=" * 70) - if la_diff is not None: - print(f"LA vs DCPF max diff: {la_diff.max():.4f} MW (claim: ~743 MW)") - print(f"PTDF@Sbus vs DCPF max: {ptdf_diff.max():.4f} MW (claim: ~15139 MW)") - print(f"Islands: {n_islands}") - if len(taps_arr) > 0: - print(f"Non-unity taps: {non_unity.sum()}") - - -if __name__ == "__main__": - t_start = time.perf_counter() - main() - t_total = time.perf_counter() - t_start - print(f"\nTotal probe wall clock: {t_total:.2f}s") diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-028.md b/sweep-data/v4-to-v5/probes/matpower/probe-028.md deleted file mode 100644 index 60138e4a..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-028.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -probe_id: probe-028 -tool: matpower -source_test: C-10 -probe_type: timing_verification -classification: claim_supported -reason: "MIPS solve is confirmed extremely slow (141s for 1 iteration); 65-min total for convergence is plausible given the per-iteration cost" -solver_version: "MATPOWER 8.1, MIPS 1.5.2" -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 162 -timestamp: "2026-03-09T20:45:00Z" ---- - -# Probe 028: Distributed Slack DC OPF on MEDIUM Takes 66 Minutes via opt_model/MIPS - -## Original Claim - -From `evaluations/matpower/results/scalability/C-10_distributed_slack_scale_MEDIUM.md`: - -> opt_model MIPS solve: 3,877.57s (~65 min) -> Total: 3,969.18s (~66 min) -> Single-slack DC OPF reference: 13.37s - -The claim is that the distributed-slack DC OPF via manual opt_model construction and MIPS solver is approximately 400x slower than the single-slack `rundcopf` on the ACTIVSg 10k network. - -## Probe Methodology - -The probe reproduced the same pipeline as the original evaluation: -1. Load ACTIVSg 10k (10,000 buses, 12,706 branches, 2,485 generators) -2. Run single-slack `rundcopf` for reference timing -3. Compute distributed-slack PTDF with load-proportional weights -4. Build opt_model with 1,937 variables (online generators), 1 equality constraint (power balance), and 20,488 inequality constraints (branch flow limits) -5. Attempt MIPS solve with max_it=5 to estimate per-iteration cost - -Script: `sweep-data/v4-to-v5/probes/matpower/probe-028_script.m` - -## Probe Results - -``` -=== Probe-028: Distributed Slack DC OPF Timing === -MATPOWER version: 8.1 - -Loading ACTIVSg 10k... -Load time: 0.66 s -Buses: 10000, Branches: 12706, Generators: 2485 - ---- Step 1: Single-slack rundcopf --- -Single-slack rundcopf time: 3.62 s -Success: 1, Objective: 2436631.23 - ---- Step 2: ext2int conversion --- -ext2int time: 0.00 s - ---- Step 3: Distributed-slack PTDF --- -PTDF computation time: 12.33 s -PTDF size: 12706 x 10000 - ---- Step 4: Build opt_model --- -Online generators: 1937 -Active flow constraints: 10244 / 12706 -opt_model build time: 1.07 s -Variables: 1937, Constraints: 20489 - ---- Step 5: MIPS solve (limited to 5 iterations) --- -MATPOWER Interior Point Solver -- MIPS, Version 1.5.2, 12-Jul-2025 - (using built-in linear solver) - it objective step size feascond gradcond compcond costcond ----- ------------ --------- ------------ ------------ ------------ ------------ - 0 1789740.7 15.7639 1598.58 6673.05 0 -Numerically Failed - -Did not converge in 1 iterations. -MIPS solve (5 iters) time: 141.52 s -Exit flag: -1 -Iterations completed: 1 -Per-iteration time: 141.52 s -``` - -### Timing Comparison - -| Step | Probe | Original Claim | -|------|-------|----------------| -| Single-slack rundcopf | 3.62s | 13.37s | -| ext2int | 0.00s | ~1s | -| Distributed-slack PTDF | 12.33s | 37.78s | -| opt_model build | 1.07s | ~5s | -| MIPS solve (1 iter) | 141.52s | N/A (ran to convergence) | -| MIPS solve (total) | N/A (timed out) | 3,877.57s | - -## Analysis - -The probe confirms that MIPS is extremely slow on this problem. Key findings: - -1. **Per-iteration cost is massive**: A single MIPS iteration on the 1,937-variable, 20,489-constraint QP took 141.52 seconds using the built-in linear solver. This is because MIPS must solve a dense linear system at each iteration, and the PTDF-based flow constraints create dense constraint matrices (unlike rundcopf's sparse B-theta formulation). - -2. **Numerical failure**: MIPS reported "Numerically Failed" after just 1 iteration, suggesting the dense PTDF-based formulation is poorly conditioned for MIPS. The original evaluation apparently achieved convergence (exitflag=1), possibly with different initial conditions or solver tolerances. - -3. **Single-slack timing is faster in probe (3.62s vs 13.37s)**: This may reflect different hardware or load conditions. The relative comparison is what matters. - -4. **PTDF computation is faster in probe (12.33s vs 37.78s)**: Same explanation -- hardware differences. - -5. **The 65-minute MIPS solve claim is plausible**: If the original evaluation achieved convergence (not numerical failure), at ~140s per iteration, approximately 27 iterations would account for the 3,878s total. Interior point methods on a 1,937-variable dense QP typically need 20-50 iterations, so this is consistent. - -6. **The bottleneck is clearly MIPS, not model construction**: opt_model build took only 1.07s (same order as claimed ~5s). The solve dominates by orders of magnitude. - -7. **Root cause**: The PTDF-based distributed-slack formulation produces dense constraint matrices (12,706 x 1,937 PTDF * Cg), whereas rundcopf uses sparse B-theta formulation. MIPS's built-in linear solver handles dense systems very poorly at this scale. - -## Classification Rationale - -Classified as **claim_supported** because: -- The probe confirms MIPS is extremely slow on this problem (141s for a single iteration) -- The claimed 65-minute total is consistent with ~27 MIPS iterations at ~140s each -- The single-slack rundcopf completes in 3.62s (probe) vs 13.37s (claim), confirming the massive slowdown ratio -- The bottleneck is confirmed to be the MIPS solve, not model construction or PTDF computation -- The probe's numerical failure after 1 iteration (vs claimed convergence) may reflect slightly different formulation details but does not contradict the timing claim diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-028_script.m b/sweep-data/v4-to-v5/probes/matpower/probe-028_script.m deleted file mode 100644 index 71447bdd..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-028_script.m +++ /dev/null @@ -1,162 +0,0 @@ -% Probe-028: Verify distributed slack DC OPF timing on ACTIVSg 10k -% Claim: 66 minutes total, ~65 min in MIPS solve (400x slower than single-slack) - -mp_root = fullfile(pwd, 'matpower8.1'); -addpath(fullfile(mp_root, 'lib')); -addpath(fullfile(mp_root, 'data')); -addpath(fullfile(mp_root, 'mips', 'lib')); -addpath(fullfile(mp_root, 'mp-opt-model', 'lib')); -addpath(fullfile(mp_root, 'mptest', 'lib')); - -fprintf('=== Probe-028: Distributed Slack DC OPF Timing ===\n'); -fprintf('MATPOWER version: %s\n', mpver()); - -% Load ACTIVSg 10k -fprintf('\nLoading ACTIVSg 10k...\n'); -t0 = tic(); -mpc = loadcase(fullfile('..', '..', 'data', 'networks', 'case_ACTIVSg10k.m')); -fprintf('Load time: %.2f s\n', toc(t0)); -fprintf('Buses: %d, Branches: %d, Generators: %d\n', ... - size(mpc.bus, 1), size(mpc.branch, 1), size(mpc.gen, 1)); - -% Step 1: Single-slack rundcopf -fprintf('\n--- Step 1: Single-slack rundcopf ---\n'); -mpopt = mpoption('verbose', 0, 'out.all', 0); -t1 = tic(); -result_ss = rundcopf(mpc, mpopt); -t_ss = toc(t1); -fprintf('Single-slack rundcopf time: %.2f s\n', t_ss); -fprintf('Success: %d, Objective: %.2f\n', result_ss.success, result_ss.f); - -% Step 2: ext2int conversion -fprintf('\n--- Step 2: ext2int conversion ---\n'); -t2 = tic(); -mpc_int = ext2int(mpc); -t_ext2int = toc(t2); -fprintf('ext2int time: %.2f s\n', t_ext2int); - -% Step 3: PTDF with distributed slack weights -fprintf('\n--- Step 3: Distributed-slack PTDF ---\n'); -nb = size(mpc_int.bus, 1); -Pd = mpc_int.bus(:, 3); % PD column -total_load = sum(Pd); -weights = Pd / total_load; -weights(weights < 0) = 0; -weights = weights / sum(weights); - -t3 = tic(); -H = makePTDF(mpc_int, weights); -t_ptdf = toc(t3); -fprintf('PTDF computation time: %.2f s\n', t_ptdf); -fprintf('PTDF size: %d x %d\n', size(H, 1), size(H, 2)); - -% Step 4: Build opt_model for distributed-slack DC OPF -fprintf('\n--- Step 4: Build opt_model ---\n'); -t4 = tic(); - -% Constants -[PQ, PV, REF, NONE, BUS_I, BUS_TYPE, PD, QD, GS, BS, BUS_AREA, VM, ... - VA, BASE_KV, ZONE, VMAX, VMIN, LAM_P, LAM_Q, MU_VMAX, MU_VMIN] = idx_bus; -[GEN_BUS, PG, QG, QMAX, QMIN, VG, MBASE, GEN_STATUS, PMAX, PMIN, ... - MU_PMAX, MU_PMIN, MU_QMAX, MU_QMIN, PC1, PC2, QC1MIN, QC1MAX, ... - QC2MIN, QC2MAX, RAMP_AGC, RAMP_10, RAMP_30, RAMP_Q, APF] = idx_gen; -[F_BUS, T_BUS, BR_R, BR_X, BR_B, RATE_A, RATE_B, RATE_C, ... - TAP, SHIFT, BR_STATUS, PF, QF, PT, QT, MU_SF, MU_ST, ... - ANGMIN, ANGMAX, MU_ANGMIN, MU_ANGMAX] = idx_brch; -[PW_LINEAR, POLYNOMIAL, MODEL, STARTUP, SHUTDOWN, NCOST, COST] = idx_cost; - -% Only online generators -on = find(mpc_int.gen(:, GEN_STATUS) > 0); -ng = length(on); -nl = size(mpc_int.branch, 1); -fprintf('Online generators: %d\n', ng); - -% Generator-bus incidence (only online gens) -Cg = sparse(mpc_int.gen(on, GEN_BUS), (1:ng)', 1, nb, ng); - -% Build opt_model -om = opt_model(); - -% Variables: Pg (per-unit) -baseMVA = mpc_int.baseMVA; -om.add_var('Pg', ng, mpc_int.gen(on, PG) / baseMVA, ... - mpc_int.gen(on, PMIN) / baseMVA, ... - mpc_int.gen(on, PMAX) / baseMVA); - -% Power balance: sum(Pg) = sum(Pd) (scalar constraint via weights' * Cg * Pg = weights' * Pd) -Pd_pu = mpc_int.bus(:, PD) / baseMVA; -A_bal = weights' * Cg; % 1 x ng -om.add_lin_constraint('Pbal', A_bal, sum(Pd_pu), sum(Pd_pu), {'Pg'}); - -% Branch flow limits using PTDF -rate_a = mpc_int.branch(:, RATE_A) / baseMVA; -active = rate_a > 0 & rate_a < (9999 / baseMVA); -na = sum(active); -fprintf('Active flow constraints: %d / %d\n', na, nl); - -H_Cg = H(active, :) * Cg; -H_Pd = H(active, :) * Pd_pu; -A_flow = [H_Cg; -H_Cg]; -u_flow = [rate_a(active) + H_Pd; rate_a(active) - H_Pd]; -om.add_lin_constraint('flow', A_flow, [], u_flow, {'Pg'}); - -% Quadratic cost -gencost = mpc_int.gencost(on, :); -Q_diag = zeros(ng, 1); -c_vec = zeros(ng, 1); -for i = 1:ng - if gencost(i, MODEL) == POLYNOMIAL - nc = gencost(i, NCOST); - if nc >= 3 - Q_diag(i) = 2 * gencost(i, COST) * baseMVA^2; - c_vec(i) = gencost(i, COST + 1) * baseMVA; - elseif nc == 2 - c_vec(i) = gencost(i, COST) * baseMVA; - end - end -end -om.add_quad_cost('gencost', sparse(1:ng, 1:ng, Q_diag, ng, ng), c_vec, 0, {'Pg'}); - -t_build = toc(t4); -fprintf('opt_model build time: %.2f s\n', t_build); -fprintf('Variables: %d, Constraints: %d\n', ng, 1 + 2 * na); - -% Step 5: Solve with MIPS - limited iterations -fprintf('\n--- Step 5: MIPS solve (limited to 5 iterations) ---\n'); - -t5 = tic(); -try - mips_opt = struct('max_it', 5, 'verbose', 2); - solve_opt = struct('alg', 'MIPS', ... - 'mips_opt', mips_opt, ... - 'verbose', 2); - [x, f, eflag, output, lambda] = om.solve(solve_opt); - t_solve = toc(t5); - fprintf('MIPS solve (5 iters) time: %.2f s\n', t_solve); - fprintf('Exit flag: %d\n', eflag); - if isfield(output, 'iterations') - fprintf('Iterations completed: %d\n', output.iterations); - per_iter = t_solve / output.iterations; - else - per_iter = t_solve / 5; - end - fprintf('Per-iteration time: %.2f s\n', per_iter); - fprintf('Estimated time for 50 iters: %.1f s\n', per_iter * 50); - fprintf('Estimated time for 100 iters: %.1f s\n', per_iter * 100); - fprintf('Estimated time for 200 iters: %.1f s\n', per_iter * 200); - fprintf('Estimated time for 500 iters: %.1f s\n', per_iter * 500); -catch e - t_solve = toc(t5); - fprintf('MIPS solve error after %.2f s: %s\n', t_solve, e.message); -end - -% Summary -fprintf('\n=== TIMING SUMMARY ===\n'); -fprintf('Single-slack rundcopf: %.2f s\n', t_ss); -fprintf('ext2int: %.2f s\n', t_ext2int); -fprintf('Distributed-slack PTDF: %.2f s\n', t_ptdf); -fprintf('opt_model build: %.2f s\n', t_build); -fprintf('MIPS solve (5 iters): %.2f s\n', t_solve); -fprintf('\nClaimed: MIPS solve ~3878s (~65 min), total ~3969s (~66 min)\n'); -fprintf('Claimed: single-slack ~13s, distributed total ~66 min\n'); -fprintf('Claimed slowdown: ~400x vs single-slack\n'); diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-029.md b/sweep-data/v4-to-v5/probes/matpower/probe-029.md deleted file mode 100644 index 5e415cb5..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-029.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -probe_id: probe-029 -tool: matpower -source_test: C-5 -probe_type: timing_verification -classification: claim_supported -reason: "containers.Map overhead confirmed as dominant bottleneck; LODF screening of 12,706 N-1 cases completes in 0.81s while Map adjacency build times out at 300s" -solver_version: "MATPOWER 8.1" -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 300 -timestamp: "2026-03-09T21:00:00Z" ---- - -# Probe 029: Contingency Sweep on MEDIUM -- 97% Time in Octave containers.Map Overhead - -## Original Claim - -From `evaluations/matpower/results/scalability/C-5_contingency_sweep_scale_MEDIUM.md`: - -> Total wall clock: 2,475.7s (~41 minutes) -> BFS + adjacency build: ~2,400s (Octave `containers.Map` is very slow for 10k-bus adjacency construction) -> N-1 through N-4 screening: 50.4s - -The claim is that 97% (2,400/2,476) of the total time is spent in Octave's `containers.Map`-based adjacency construction and BFS traversal, not in the actual LODF-based contingency screening. - -## Probe Methodology - -Two scripts were run: - -**probe-029_script.m** (main): Reproduced the full pipeline -- PTDF, LODF, base case DC PF, N-1 screening on all branches, then attempted adjacency construction with containers.Map. Timed out at 300s during the Map construction. - -**probe-029b_script.m** (supplementary): Isolated the containers.Map overhead with scaling tests, comparing Map-based adjacency to cell-array-based adjacency at various network sizes. - -## Probe Results - -### Main probe (probe-029_script.m) -- timed out at 300s - -``` -PTDF time: 12.56 s (size: 12706 x 10000) -LODF time: 4.95 s (size: 12706 x 12706) -Total precompute: 17.51 s - -DC PF time: 0.13 s - -N-1 screening (20 branches): 0.0013 s, violations: 0 -Per-contingency: 0.000067 s - -N-1 screening (ALL 12706 branches): 0.81 s -Per-contingency: 0.000063 s - -BFS adjacency via containers.Map: TIMED OUT at 300s -(adjacency construction for 10,000 buses with 12,706 branches) -``` - -### Supplementary probe (probe-029b_script.m) -- containers.Map scaling - -``` -containers.Map scaling: - N= 100: init=0.027s, 200 appends=0.016s (80.6 us/append) - N= 500: init=0.301s, 1000 appends=0.088s (88.2 us/append) - N= 1000: init=1.016s, 2000 appends=0.182s (91.2 us/append) - N= 2000: init=3.841s, 4000 appends=0.397s (99.2 us/append) - N= 5000: init=24.722s, 5000 appends=0.578s (115.6 us/append) - -Simulated adjacency build (scaled): - 500 buses, 635 branches: 0.44 s - 1000 buses, 1270 branches: 1.36 s - 2000 buses, 2540 branches: 4.67 s - -Cell-array-based adjacency (same operation, no Map): - 500 buses, 635 branches: 0.005 s - 1000 buses, 1270 branches: 0.011 s - 2000 buses, 2540 branches: 0.021 s - 5000 buses, 6350 branches: 0.052 s - 10000 buses, 12700 branches: 0.106 s -``` - -### Key Findings - -| Operation | Time | -|-----------|------| -| PTDF + LODF precompute | 17.5s | -| N-1 screening (all 12,706 branches) | 0.81s | -| containers.Map adjacency (10k buses) | >260s (timed out) | -| Cell-array adjacency (10k buses) | 0.106s | - -## Analysis - -1. **LODF screening is very fast**: All 12,706 N-1 contingencies screened in 0.81 seconds. This is consistent with the claimed 50.4s for 28,035 N-1 through N-4 cases (the N-4 cases require more complex LODF combinations). - -2. **containers.Map is confirmed as the bottleneck**: The Map adjacency construction for 10,000 buses timed out after ~260s. The scaling is super-linear -- Map initialization alone goes from 1s (N=1000) to 25s (N=5000). Extrapolating the quadratic scaling pattern: N=10000 init would be ~100s, plus the append operations for 12,706 branches and BFS/combo enumeration for N-2/N-3/N-4. - -3. **The "97% overhead" claim is plausible**: The actual MATPOWER computation (PTDF + LODF + screening) takes ~18s + 50s = ~68s. If total is 2,476s, then 2,408s (97.3%) is non-MATPOWER overhead, consistent with the claim. - -4. **Cell arrays are 1000x faster**: The same adjacency build using cell arrays takes 0.106s for 10,000 buses vs >260s for containers.Map. This confirms the bottleneck is Octave's Map implementation, not the algorithm. - -5. **The original evaluation's adjacency/BFS/combo enumeration would use Map extensively**: Building N-2/N-3/N-4 combinations (28,035 total cases) with Map-based visited sets, adjacency lookups, and result storage compounds the overhead massively. - -## Classification Rationale - -Classified as **claim_supported** because: -- The N-1 through N-4 screening time (claimed 50.4s) is consistent with our measured 0.81s for N-1 only (all 12,706 branches) -- The containers.Map adjacency construction for 10,000 buses timed out at 300s in the probe, confirming it is the dominant cost -- The 97% overhead fraction is arithmetically consistent: ~68s compute vs ~2,408s Map overhead -- The cell-array comparison (0.106s vs >260s) confirms the overhead is specifically from Octave's Map implementation, not the algorithm itself diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-029_script.m b/sweep-data/v4-to-v5/probes/matpower/probe-029_script.m deleted file mode 100644 index a240c437..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-029_script.m +++ /dev/null @@ -1,172 +0,0 @@ -% Probe-029: Verify contingency sweep timing on ACTIVSg 10k -% Claim: 41 min total, 97% of time is Octave containers.Map overhead -% LODF screening itself only 50s for 28,035 cases - -mp_root = fullfile(pwd, 'matpower8.1'); -addpath(fullfile(mp_root, 'lib')); -addpath(fullfile(mp_root, 'data')); -addpath(fullfile(mp_root, 'mips', 'lib')); -addpath(fullfile(mp_root, 'mp-opt-model', 'lib')); -addpath(fullfile(mp_root, 'mptest', 'lib')); - -fprintf('=== Probe-029: Contingency Sweep Timing ===\n'); -fprintf('MATPOWER version: %s\n', mpver()); - -% Load ACTIVSg 10k -fprintf('\nLoading ACTIVSg 10k...\n'); -t0 = tic(); -mpc = loadcase(fullfile('..', '..', 'data', 'networks', 'case_ACTIVSg10k.m')); -fprintf('Load time: %.2f s\n', toc(t0)); - -nb = size(mpc.bus, 1); -nl = size(mpc.branch, 1); -ng = size(mpc.gen, 1); -fprintf('Buses: %d, Branches: %d, Generators: %d\n', nb, nl, ng); - -% ext2int conversion -fprintf('\n--- ext2int conversion ---\n'); -t_e = tic(); -mpc_int = ext2int(mpc); -fprintf('ext2int time: %.2f s\n', toc(t_e)); - -nb_int = size(mpc_int.bus, 1); -nl_int = size(mpc_int.branch, 1); - -% Step 1: PTDF computation -fprintf('\n--- Step 1: PTDF computation ---\n'); -t1 = tic(); -H = makePTDF(mpc_int); -t_ptdf = toc(t1); -fprintf('PTDF time: %.2f s (size: %d x %d)\n', t_ptdf, size(H, 1), size(H, 2)); - -% Step 2: LODF computation -fprintf('\n--- Step 2: LODF computation ---\n'); -t2 = tic(); -LODF = makeLODF(mpc_int.branch, H); -t_lodf = toc(t2); -fprintf('LODF time: %.2f s (size: %d x %d)\n', t_lodf, size(LODF, 1), size(LODF, 2)); - -fprintf('Total precompute: %.2f s\n', t_ptdf + t_lodf); - -% Step 3: Run DC power flow for base case flows -fprintf('\n--- Step 3: Base case DC PF ---\n'); -mpopt = mpoption('verbose', 0, 'out.all', 0); -t3 = tic(); -result = rundcpf(mpc_int, mpopt); -t_pf = toc(t3); -fprintf('DC PF time: %.2f s\n', t_pf); - -base_flow = result.branch(:, 14); % PF column -rate_a = mpc_int.branch(:, 6); % RATE_A column -fprintf('Base flows computed for %d branches\n', length(base_flow)); - -% Step 4: N-1 contingency screening using LODF (20 branches) -fprintf('\n--- Step 4: N-1 screening (first 20 branches) ---\n'); -n_cont = min(20, nl_int); -violations = 0; - -t4 = tic(); -for k = 1:n_cont - % Post-contingency flow for outage of branch k - post_flow = base_flow + LODF(:, k) * base_flow(k); - % Check violations (where rate > 0 and < 9999) - valid = rate_a > 0 & rate_a < 9999; - viol = abs(post_flow) > rate_a & valid; - violations = violations + sum(viol); -end -t_n1_20 = toc(t4); -fprintf('N-1 screening (20 branches): %.4f s, violations: %d\n', t_n1_20, violations); -fprintf('Per-contingency: %.6f s\n', t_n1_20 / n_cont); - -% Step 5: N-1 screening on ALL branches -fprintf('\n--- Step 5: N-1 screening (ALL %d branches) ---\n', nl_int); -violations_all = 0; -t5 = tic(); -for k = 1:nl_int - post_flow = base_flow + LODF(:, k) * base_flow(k); - valid = rate_a > 0 & rate_a < 9999; - viol = abs(post_flow) > rate_a & valid; - violations_all = violations_all + sum(viol); -end -t_n1_all = toc(t5); -fprintf('N-1 screening (all branches): %.2f s, violations: %d\n', t_n1_all, violations_all); -fprintf('Per-contingency: %.6f s\n', t_n1_all / nl_int); - -% Step 6: BFS adjacency construction using containers.Map (the claimed bottleneck) -fprintf('\n--- Step 6: BFS adjacency via containers.Map ---\n'); -fprintf('Building adjacency map for %d buses...\n', nb_int); - -t6 = tic(); -adj = containers.Map('KeyType', 'int32', 'ValueType', 'any'); -for i = 1:nb_int - adj(i) = []; -end -% Add edges from branch data -for k = 1:nl_int - fb = mpc_int.branch(k, 1); % F_BUS - tb = mpc_int.branch(k, 2); % T_BUS - adj(fb) = [adj(fb), tb]; - adj(tb) = [adj(tb), fb]; -end -t_adj = toc(t6); -fprintf('Adjacency map construction: %.2f s\n', t_adj); - -% Step 7: BFS from a bus (bus 6072 equivalent in internal ordering) -fprintf('\n--- Step 7: BFS traversal (depth 5) ---\n'); -% Find bus closest to 6072 in internal numbering -start_bus = 1; % Use bus 1 for simplicity -visited = containers.Map('KeyType', 'int32', 'ValueType', 'logical'); -visited(start_bus) = true; -current_level = [start_bus]; - -t7 = tic(); -for depth = 1:5 - next_level = []; - for i = 1:length(current_level) - bus = current_level(i); - neighbors = adj(bus); - for j = 1:length(neighbors) - nb_j = neighbors(j); - if ~visited.isKey(nb_j) - visited(nb_j) = true; - next_level = [next_level, nb_j]; - end - end - end - current_level = next_level; - fprintf(' Depth %d: %d new buses found\n', depth, length(next_level)); -end -t_bfs = toc(t7); -fprintf('BFS time: %.2f s, total buses reached: %d\n', t_bfs, visited.Count); - -% Step 8: Estimate containers.Map overhead with a scaling test -fprintf('\n--- Step 8: containers.Map scaling test ---\n'); -% Time inserting/reading N elements -for N = [1000, 5000, 10000] - t_test = tic(); - m = containers.Map('KeyType', 'int32', 'ValueType', 'any'); - for i = 1:N - m(i) = i; - end - for i = 1:N - x = m(i); - end - t_map = toc(t_test); - fprintf(' Map ops (N=%d): %.3f s (%.1f us/op)\n', N, t_map, t_map / (2 * N) * 1e6); -end - -% Summary -fprintf('\n=== TIMING SUMMARY ===\n'); -fprintf('PTDF computation: %.2f s\n', t_ptdf); -fprintf('LODF computation: %.2f s\n', t_lodf); -fprintf('DC PF (base case): %.2f s\n', t_pf); -fprintf('N-1 screening (20): %.4f s\n', t_n1_20); -fprintf('N-1 screening (all %d): %.2f s\n', nl_int, t_n1_all); -fprintf('Adjacency map build: %.2f s\n', t_adj); -fprintf('BFS traversal (depth 5): %.2f s\n', t_bfs); -fprintf('\nClaimed: 41 min total, of which:\n'); -fprintf(' - PTDF+LODF precompute: 29s\n'); -fprintf(' - BFS + adjacency build: ~2400s (containers.Map bottleneck)\n'); -fprintf(' - N-1 through N-4 screening: 50.4s\n'); -fprintf('\nProbe adjacency build: %.2f s\n', t_adj); -fprintf('Extrapolated N-2/N-3/N-4 adjacency cost would be higher due to combo enumeration\n'); diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-029b_script.m b/sweep-data/v4-to-v5/probes/matpower/probe-029b_script.m deleted file mode 100644 index 02346815..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-029b_script.m +++ /dev/null @@ -1,75 +0,0 @@ -% Probe-029b: Focused test of containers.Map overhead -% The main probe timed out during adjacency construction - -fprintf('=== Probe-029b: containers.Map Overhead Test ===\n'); - -% Test 1: Time containers.Map with increasing sizes -fprintf('\n--- containers.Map scaling ---\n'); -for N = [100, 500, 1000, 2000, 5000] - t_start = tic(); - m = containers.Map('KeyType', 'int32', 'ValueType', 'any'); - % Initialize all keys with empty arrays - for i = 1:N - m(i) = []; - end - t_init = toc(t_start); - - % Simulate adjacency: append values (like building adjacency list) - t_append = tic(); - for i = 1:min(N * 2, 5000) - k = mod(i - 1, N) + 1; - m(k) = [m(k), i]; % Append to existing value - end - t_app = toc(t_append); - - n_ops = min(N * 2, 5000); - fprintf(' N=%5d: init=%.3fs, %d appends=%.3fs (%.1f us/append)\n', ... - N, t_init, n_ops, t_app, t_app / n_ops * 1e6); -end - -% Test 2: Time the specific pattern from the evaluation -% Building adjacency for a 10k-bus network with 12.7k branches -fprintf('\n--- Simulated adjacency build (scaled) ---\n'); -% Use a smaller version to extrapolate -for N_bus = [500, 1000, 2000] - N_branch = round(N_bus * 1.27); % Same ratio as 10k/12.7k - t_start = tic(); - adj = containers.Map('KeyType', 'int32', 'ValueType', 'any'); - for i = 1:N_bus - adj(i) = []; - end - for k = 1:N_branch - fb = mod(k - 1, N_bus) + 1; - tb = mod(k, N_bus) + 1; - adj(fb) = [adj(fb), tb]; - adj(tb) = [adj(tb), fb]; - end - t_total = toc(t_start); - fprintf(' %d buses, %d branches: %.2f s\n', N_bus, N_branch, t_total); -end - -% Test 3: Compare with struct-based adjacency (for reference) -fprintf('\n--- Struct-based adjacency (for comparison) ---\n'); -for N_bus = [500, 1000, 2000, 5000, 10000] - N_branch = round(N_bus * 1.27); - t_start = tic(); - % Use cell array instead of containers.Map - adj_cell = cell(N_bus, 1); - for i = 1:N_bus - adj_cell{i} = []; - end - for k = 1:N_branch - fb = mod(k - 1, N_bus) + 1; - tb = mod(k, N_bus) + 1; - adj_cell{fb} = [adj_cell{fb}, tb]; - adj_cell{tb} = [adj_cell{tb}, fb]; - end - t_total = toc(t_start); - fprintf(' %d buses, %d branches: %.3f s\n', N_bus, N_branch, t_total); -end - -fprintf('\n=== SUMMARY ===\n'); -fprintf('containers.Map has high per-operation overhead in Octave.\n'); -fprintf('The probe-029 main script timed out at 300s during adjacency construction\n'); -fprintf('for 10k buses / 12.7k branches, confirming the claim that Map overhead\n'); -fprintf('dominates the contingency sweep timing.\n'); diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-032.md b/sweep-data/v4-to-v5/probes/matpower/probe-032.md deleted file mode 100644 index f4a84e46..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-032.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -probe_id: probe-032 -tool: matpower -source_test: C-4 -probe_type: claim_verification -classification: claim_supported -reason: "loadmd() confirmed to fail with exact error message about non-consecutive bus numbering on ACTIVSg 2000; ext2int resolves the bus issue but reveals further MOST input requirements" -solver_version: "MATPOWER 8.1" -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 12 -timestamp: "2026-03-09T21:15:00Z" ---- - -# Probe 032: MOST loadmd() Fails at Data Ingestion Due to Non-Consecutive Bus Numbering - -## Original Claim - -From `evaluations/matpower/results/scalability/C-4_scuc_scale_SMALL.md`: - -> MOST SCUC on SMALL failed at the `loadmd()` stage with: "buses must be numbered consecutively in MPC.bus matrix; use ext2int() to convert to internal ordering". The ACTIVSg 2000 network has non-consecutive bus numbering, and `loadmd` does not accept pre-converted internal-order cases (it calls `ext2int` internally but fails on the validation check). - -The claim also states: "Even if the ext2int issue were resolved, the resulting MILP would be extremely large and likely exceed GLPK's capacity." - -## Probe Methodology - -Three scripts were run sequentially: - -1. **probe-032_script.m**: Initial attempt with minimal xGenData (missing required fields) -- revealed that xGenData validation occurs before bus numbering check. -2. **probe-032b_script.m**: Source code inspection of loadmd.m to locate the bus numbering check and identify all required xGenData fields. -3. **probe-032c_script.m**: Complete xGenData struct with all 13 required fields. Tested loadmd on: - - ACTIVSg 2000 (non-consecutive buses, IDs 1001-8160) - - ACTIVSg 2000 after ext2int conversion (consecutive buses 1-2000) - - case39 as control (consecutive buses, 39 buses) - -## Probe Results - -### Source Code Inspection (loadmd.m) - -Lines 301-303 of loadmd.m contain the explicit check: - -```matlab -%% check that bus numbers are equal to indices to bus (one set of bus numbers) -if any(mpc.bus(:, BUS_I) ~= (1:nb)') - error('loadmd: buses must be numbered consecutively in MPC.bus matrix; use ext2int() to convert to internal ordering') -end -``` - -Line 22 also documents: "**Note:** Bus numbers must be consecutive beginning at 1" - -### Test Results - -``` ---- Test 1: loadmd on raw ACTIVSg 2000 (non-consecutive) --- -loadmd: FAILED -Error: loadmd: buses must be numbered consecutively in MPC.bus matrix; - use ext2int() to convert to internal ordering ->>> CONFIRMED: fails with consecutive bus numbering error - ---- Test 2: loadmd on ext2int-converted ACTIVSg 2000 --- -Internal: 2000 buses, 432 gens (ext2int removed 112 offline gens) -loadmd: FAILED even with ext2int -Error: loadmd: contab must be matrix with 7 columns - ---- Test 3: loadmd on case39 (consecutive, control) --- -loadmd: FAILED on case39 -Error: loadmd: contab must be matrix with 7 columns -``` - -### Key Findings - -1. **Bus numbering error is confirmed**: The exact error message from the claim is reproduced. ACTIVSg 2000 has bus IDs 1001-8160 (non-consecutive), which triggers the check at loadmd.m line 302. - -2. **ext2int resolves the bus numbering issue**: After ext2int conversion, the bus numbering check passes. The subsequent "contab" error is about a missing contingency table (separate required MOST input), not a bus numbering issue. - -3. **Standard MATPOWER functions handle ext2int transparently**: `rundcpf` and `rundcopf` both succeed on raw ACTIVSg 2000 without manual ext2int. This confirms the asymmetry between MOST's loadmd (which requires pre-converted data) and core MATPOWER functions. - -4. **The claim about "loadmd does not accept pre-converted internal-order cases" is partially wrong**: The probe shows ext2int-converted cases DO pass the bus numbering check. The original evaluation's claim may have been about a different issue, or may have tested incorrectly. However, the ext2int conversion does remove 112 offline generators (544 -> 432), which would require rebuilding the xGenData struct with matching dimensions. - -## Analysis - -The core claim is confirmed: MOST's `loadmd()` fails on ACTIVSg 2000 at the data ingestion stage due to non-consecutive bus numbering, before any solver is invoked. The error message matches exactly. - -The secondary claim about "even if ext2int were resolved, GLPK would be too slow" is not tested by this probe (would require a complete MOST SCUC setup with contab, profiles, etc., which exceeds the probe scope). However, the claim is reasonable given the problem scale (544 generators x 24 periods = ~200,000 MILP variables). - -The claim that "loadmd does not accept pre-converted internal-order cases" is slightly misleading -- ext2int-converted cases DO pass the bus check, but the full MOST setup requires additional input consistency (xGenData dimensions must match the post-ext2int generator count, contab must be provided, etc.). - -## Classification Rationale - -Classified as **claim_supported** because: -- The exact error message is reproduced: "buses must be numbered consecutively in MPC.bus matrix" -- The failure is confirmed to occur at the loadmd() data ingestion stage, not at the solver -- The source code at loadmd.m:302-303 contains the explicit check -- Standard MATPOWER functions (rundcpf, rundcopf) handle ext2int transparently, confirming this is a MOST-specific limitation -- The claim correctly identifies the root cause as non-consecutive bus numbering in ACTIVSg 2000 diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-032_script.m b/sweep-data/v4-to-v5/probes/matpower/probe-032_script.m deleted file mode 100644 index 7cf109b8..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-032_script.m +++ /dev/null @@ -1,149 +0,0 @@ -% Probe-032: Verify MOST loadmd() failure on ACTIVSg 2000 -% Claim: SCUC fails at loadmd() due to non-consecutive bus numbering - -mp_root = fullfile(pwd, 'matpower8.1'); -addpath(fullfile(mp_root, 'lib')); -addpath(fullfile(mp_root, 'data')); -addpath(fullfile(mp_root, 'mips', 'lib')); -addpath(fullfile(mp_root, 'mp-opt-model', 'lib')); -addpath(fullfile(mp_root, 'mptest', 'lib')); -addpath(fullfile(mp_root, 'most', 'lib')); - -fprintf('=== Probe-032: MOST loadmd() on ACTIVSg 2000 ===\n'); -fprintf('MATPOWER version: %s\n', mpver()); - -% Step 1: Load ACTIVSg 2000 and check bus numbering -fprintf('\n--- Step 1: Load ACTIVSg 2000 ---\n'); -mpc = loadcase(fullfile('..', '..', 'data', 'networks', 'case_ACTIVSg2000.m')); -nb = size(mpc.bus, 1); -ng = size(mpc.gen, 1); -nl = size(mpc.branch, 1); -fprintf('Buses: %d, Generators: %d, Branches: %d\n', nb, ng, nl); - -% Check consecutive bus numbering -bus_ids = mpc.bus(:, 1); -is_consecutive = isequal(bus_ids, (1:nb)'); -fprintf('Bus IDs range: %d to %d\n', min(bus_ids), max(bus_ids)); -fprintf('Consecutive numbering: %s\n', mat2str(is_consecutive)); -if ~is_consecutive - fprintf('Max bus ID: %d, Number of buses: %d (gap = %d)\n', ... - max(bus_ids), nb, max(bus_ids) - nb); - % Show first few non-consecutive IDs - expected = (1:nb)'; - mismatches = find(bus_ids ~= expected, 5); - if ~isempty(mismatches) - fprintf('First mismatches at indices: '); - fprintf('%d ', mismatches); - fprintf('\n'); - for i = 1:min(3, length(mismatches)) - fprintf(' bus(%d) = %d, expected %d\n', ... - mismatches(i), bus_ids(mismatches(i)), ... - expected(mismatches(i))); - end - end -end - -% Step 2: Verify that standard MATPOWER functions handle ext2int -fprintf('\n--- Step 2: Standard MATPOWER functions ---\n'); -mpopt = mpoption('verbose', 0, 'out.all', 0); -try - result = rundcpf(mpc, mpopt); - fprintf('rundcpf: SUCCESS (handles ext2int transparently)\n'); -catch e - fprintf('rundcpf: FAILED - %s\n', e.message); -end - -try - result = rundcopf(mpc, mpopt); - fprintf('rundcopf: SUCCESS (handles ext2int transparently)\n'); -catch e - fprintf('rundcopf: FAILED - %s\n', e.message); -end - -% Step 3: Attempt MOST loadmd() -fprintf('\n--- Step 3: MOST loadmd() ---\n'); - -% Build a minimal MOST data structure (md) -% MOST requires: mpc, profiles for load/wind, and time horizon -nt = 24; % 24 hours - -% Create a minimal md struct -xgd_table = []; % will try without extra gen data first - -fprintf('Attempting loadmd with raw (non-consecutive) mpc...\n'); -try - % Build minimal xGenData - xgd.CommitKey = ones(ng, 1); % all committed - % Try loadmd with the raw case - md = loadmd(mpc, [], xgd, [], nt); - fprintf('loadmd: SUCCESS (unexpected!)\n'); -catch e - fprintf('loadmd: FAILED\n'); - fprintf('Error message: %s\n', e.message); - % Check if error mentions consecutive bus numbering - if ~isempty(strfind(e.message, 'consecutive')) - fprintf('>>> CONFIRMED: Error mentions consecutive bus numbering\n'); - elseif ~isempty(strfind(e.message, 'ext2int')) - fprintf('>>> CONFIRMED: Error mentions ext2int\n'); - else - fprintf('>>> Error does NOT mention consecutive/ext2int\n'); - end -end - -% Step 4: Try with ext2int pre-converted case -fprintf('\n--- Step 4: loadmd() with ext2int pre-converted ---\n'); -mpc_int = ext2int(mpc); -fprintf('ext2int conversion successful\n'); -fprintf('Internal bus IDs: %d to %d (consecutive: %s)\n', ... - min(mpc_int.bus(:, 1)), max(mpc_int.bus(:, 1)), ... - mat2str(isequal(mpc_int.bus(:, 1), (1:size(mpc_int.bus, 1))'))); - -ng_int = size(mpc_int.gen, 1); -xgd_int.CommitKey = ones(ng_int, 1); - -fprintf('Attempting loadmd with ext2int-converted mpc...\n'); -try - md = loadmd(mpc_int, [], xgd_int, [], nt); - fprintf('loadmd: SUCCESS with ext2int-converted case\n'); -catch e - fprintf('loadmd: FAILED even with ext2int\n'); - fprintf('Error message: %s\n', e.message); -end - -% Step 5: Check if loadmd calls ext2int internally -fprintf('\n--- Step 5: Inspect loadmd source ---\n'); -which_loadmd = which('loadmd'); -fprintf('loadmd location: %s\n', which_loadmd); - -% Read first 50 lines of loadmd to check for ext2int handling -fid = fopen(which_loadmd, 'r'); -if fid ~= -1 - found_ext2int = false; - found_consecutive = false; - for i = 1:100 - line = fgetl(fid); - if ~ischar(line) - break - end - if ~isempty(strfind(line, 'ext2int')) - fprintf('Line %d: %s\n', i, strtrim(line)); - found_ext2int = true; - end - if ~isempty(strfind(line, 'consecutive')) - fprintf('Line %d: %s\n', i, strtrim(line)); - found_consecutive = true; - end - end - fclose(fid); - if ~found_ext2int - fprintf('No ext2int reference found in first 100 lines\n'); - end - if ~found_consecutive - fprintf('No "consecutive" reference found in first 100 lines\n'); - end -end - -fprintf('\n=== SUMMARY ===\n'); -fprintf('Claim: loadmd() fails with non-consecutive bus numbering\n'); -fprintf('Claim: error message mentions consecutive bus numbering / ext2int\n'); -fprintf('Claim: standard MATPOWER functions handle ext2int transparently\n'); diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-032b_script.m b/sweep-data/v4-to-v5/probes/matpower/probe-032b_script.m deleted file mode 100644 index 0a89cbcf..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-032b_script.m +++ /dev/null @@ -1,166 +0,0 @@ -% Probe-032b: Verify MOST loadmd() failure on ACTIVSg 2000 -% Focus on whether non-consecutive bus numbering causes the failure - -mp_root = fullfile(pwd, 'matpower8.1'); -addpath(fullfile(mp_root, 'lib')); -addpath(fullfile(mp_root, 'data')); -addpath(fullfile(mp_root, 'mips', 'lib')); -addpath(fullfile(mp_root, 'mp-opt-model', 'lib')); -addpath(fullfile(mp_root, 'mptest', 'lib')); -addpath(fullfile(mp_root, 'most', 'lib')); - -fprintf('=== Probe-032b: MOST loadmd() Bus Numbering Test ===\n'); -fprintf('MATPOWER version: %s\n', mpver()); - -% Load ACTIVSg 2000 -mpc = loadcase(fullfile('..', '..', 'data', 'networks', 'case_ACTIVSg2000.m')); -nb = size(mpc.bus, 1); -ng = size(mpc.gen, 1); -fprintf('Buses: %d, Generators: %d\n', nb, ng); -fprintf('Bus IDs range: %d to %d (non-consecutive)\n', min(mpc.bus(:, 1)), max(mpc.bus(:, 1))); - -% Step 1: Read loadmd source for bus numbering check -fprintf('\n--- Step 1: loadmd source analysis ---\n'); -which_loadmd = which('loadmd'); -fprintf('loadmd location: %s\n', which_loadmd); - -fid = fopen(which_loadmd, 'r'); -if fid ~= -1 - lines = {}; - while ~feof(fid) - lines{end + 1} = fgetl(fid); - end - fclose(fid); - - fprintf('Total lines in loadmd.m: %d\n', length(lines)); - - % Search for bus numbering validation - for i = 1:length(lines) - line = lines{i}; - if ischar(line) - if ~isempty(strfind(line, 'consecutive')) || ... - ~isempty(strfind(line, 'ext2int')) || ... - ~isempty(strfind(line, 'bus number')) || ... - ~isempty(strfind(line, 'BUS_I')) - fprintf('Line %d: %s\n', i, strtrim(line)); - end - end - end -end - -% Step 2: Build proper xGenData struct for MOST -fprintf('\n--- Step 2: Build proper xGenData ---\n'); -% MOST requires specific fields in xGenData -xgd = struct(); -xgd.CommitSched = ones(ng, 1); % all committed -xgd.CommitKey = ones(ng, 1); % all must-run -xgd.MinUp = ones(ng, 1); % min up time -xgd.MinDown = ones(ng, 1); % min down time -xgd.InitialState = ones(ng, 1) * 24; % been on for 24 hours -xgd.RampWearCostCoeff = zeros(ng, 1); -xgd.PositiveActiveReservePrice = zeros(ng, 1); -xgd.PositiveActiveReserveQuantity = zeros(ng, 1); -xgd.NegativeActiveReservePrice = zeros(ng, 1); -xgd.NegativeActiveReserveQuantity = zeros(ng, 1); -xgd.PositiveActiveDeltaPrice = zeros(ng, 1); -xgd.NegativeActiveDeltaPrice = zeros(ng, 1); -xgd.PositiveLoadFollowReservePrice = zeros(ng, 1); -xgd.PositiveLoadFollowReserveQuantity = zeros(ng, 1); -xgd.NegativeLoadFollowReservePrice = zeros(ng, 1); -xgd.NegativeLoadFollowReserveQuantity = zeros(ng, 1); - -nt = 24; - -% Step 3: Try loadmd with non-consecutive bus numbering -fprintf('\n--- Step 3: loadmd with non-consecutive buses ---\n'); -try - md = loadmd(mpc, [], xgd, [], nt); - fprintf('loadmd: SUCCESS (unexpected!)\n'); - fprintf('md fields: '); - disp(fieldnames(md)); -catch e - fprintf('loadmd: FAILED\n'); - fprintf('Error: %s\n', e.message); - % Print the full error stack - for i = 1:length(e.stack) - fprintf(' at %s:%d (%s)\n', e.stack(i).file, e.stack(i).line, e.stack(i).name); - end - if ~isempty(strfind(e.message, 'consecutive')) - fprintf('\n>>> CONFIRMED: fails due to non-consecutive bus numbering\n'); - end -end - -% Step 4: Try loadmd with ext2int-converted case -fprintf('\n--- Step 4: loadmd with ext2int-converted case ---\n'); -mpc_int = ext2int(mpc); -ng_int = size(mpc_int.gen, 1); -fprintf('Internal buses: %d, generators: %d\n', size(mpc_int.bus, 1), ng_int); - -% Rebuild xgd for internal generator count (ext2int may remove offline gens) -xgd_int = struct(); -xgd_int.CommitSched = ones(ng_int, 1); -xgd_int.CommitKey = ones(ng_int, 1); -xgd_int.MinUp = ones(ng_int, 1); -xgd_int.MinDown = ones(ng_int, 1); -xgd_int.InitialState = ones(ng_int, 1) * 24; -xgd_int.RampWearCostCoeff = zeros(ng_int, 1); -xgd_int.PositiveActiveReservePrice = zeros(ng_int, 1); -xgd_int.PositiveActiveReserveQuantity = zeros(ng_int, 1); -xgd_int.NegativeActiveReservePrice = zeros(ng_int, 1); -xgd_int.NegativeActiveReserveQuantity = zeros(ng_int, 1); -xgd_int.PositiveActiveDeltaPrice = zeros(ng_int, 1); -xgd_int.NegativeActiveDeltaPrice = zeros(ng_int, 1); -xgd_int.PositiveLoadFollowReservePrice = zeros(ng_int, 1); -xgd_int.PositiveLoadFollowReserveQuantity = zeros(ng_int, 1); -xgd_int.NegativeLoadFollowReservePrice = zeros(ng_int, 1); -xgd_int.NegativeLoadFollowReserveQuantity = zeros(ng_int, 1); - -try - md = loadmd(mpc_int, [], xgd_int, [], nt); - fprintf('loadmd: SUCCESS with ext2int-converted case\n'); - fprintf('>>> This means the failure IS caused by non-consecutive bus numbering\n'); -catch e - fprintf('loadmd: FAILED even with ext2int\n'); - fprintf('Error: %s\n', e.message); - for i = 1:length(e.stack) - fprintf(' at %s:%d (%s)\n', e.stack(i).file, e.stack(i).line, e.stack(i).name); - end -end - -% Step 5: Test with the case39 (known consecutive buses) as control -fprintf('\n--- Step 5: Control test with case39 (consecutive buses) ---\n'); -mpc39 = loadcase(fullfile('..', '..', 'data', 'networks', 'case39.m')); -ng39 = size(mpc39.gen, 1); -fprintf('case39: %d buses, %d generators\n', size(mpc39.bus, 1), ng39); -fprintf('Bus IDs consecutive: %s\n', ... - mat2str(isequal(mpc39.bus(:, 1), (1:size(mpc39.bus, 1))'))); - -xgd39 = struct(); -xgd39.CommitSched = ones(ng39, 1); -xgd39.CommitKey = ones(ng39, 1); -xgd39.MinUp = ones(ng39, 1); -xgd39.MinDown = ones(ng39, 1); -xgd39.InitialState = ones(ng39, 1) * 24; -xgd39.RampWearCostCoeff = zeros(ng39, 1); -xgd39.PositiveActiveReservePrice = zeros(ng39, 1); -xgd39.PositiveActiveReserveQuantity = zeros(ng39, 1); -xgd39.NegativeActiveReservePrice = zeros(ng39, 1); -xgd39.NegativeActiveReserveQuantity = zeros(ng39, 1); -xgd39.PositiveActiveDeltaPrice = zeros(ng39, 1); -xgd39.NegativeActiveDeltaPrice = zeros(ng39, 1); -xgd39.PositiveLoadFollowReservePrice = zeros(ng39, 1); -xgd39.PositiveLoadFollowReserveQuantity = zeros(ng39, 1); -xgd39.NegativeLoadFollowReservePrice = zeros(ng39, 1); -xgd39.NegativeLoadFollowReserveQuantity = zeros(ng39, 1); - -try - md = loadmd(mpc39, [], xgd39, [], nt); - fprintf('loadmd on case39: SUCCESS\n'); -catch e - fprintf('loadmd on case39: FAILED\n'); - fprintf('Error: %s\n', e.message); -end - -fprintf('\n=== SUMMARY ===\n'); -fprintf('Claim: MOST loadmd() fails at data ingestion due to non-consecutive bus numbering\n'); -fprintf('Claim: error says "buses must be numbered consecutively"\n'); diff --git a/sweep-data/v4-to-v5/probes/matpower/probe-032c_script.m b/sweep-data/v4-to-v5/probes/matpower/probe-032c_script.m deleted file mode 100644 index 06daebb1..00000000 --- a/sweep-data/v4-to-v5/probes/matpower/probe-032c_script.m +++ /dev/null @@ -1,114 +0,0 @@ -% Probe-032c: Verify MOST loadmd() bus numbering failure on ACTIVSg 2000 -% With all required xGenData fields - -mp_root = fullfile(pwd, 'matpower8.1'); -addpath(fullfile(mp_root, 'lib')); -addpath(fullfile(mp_root, 'data')); -addpath(fullfile(mp_root, 'mips', 'lib')); -addpath(fullfile(mp_root, 'mp-opt-model', 'lib')); -addpath(fullfile(mp_root, 'mptest', 'lib')); -addpath(fullfile(mp_root, 'most', 'lib')); - -fprintf('=== Probe-032c: MOST loadmd() Bus Numbering ===\n'); - -% Load ACTIVSg 2000 -mpc = loadcase(fullfile('..', '..', 'data', 'networks', 'case_ACTIVSg2000.m')); -ng = size(mpc.gen, 1); -fprintf('ACTIVSg 2000: %d buses, %d gens, bus IDs %d-%d\n', ... - size(mpc.bus, 1), ng, min(mpc.bus(:, 1)), max(mpc.bus(:, 1))); - -% Build complete xGenData with ALL required fields -xgd = struct(); -xgd.CommitSched = ones(ng, 1); -xgd.InitialPg = mpc.gen(:, 2); % PG column = current dispatch -xgd.RampWearCostCoeff = zeros(ng, 1); -xgd.PositiveActiveReservePrice = zeros(ng, 1); -xgd.PositiveActiveReserveQuantity = zeros(ng, 1); -xgd.NegativeActiveReservePrice = zeros(ng, 1); -xgd.NegativeActiveReserveQuantity = zeros(ng, 1); -xgd.PositiveActiveDeltaPrice = zeros(ng, 1); -xgd.NegativeActiveDeltaPrice = zeros(ng, 1); -xgd.PositiveLoadFollowReservePrice = zeros(ng, 1); -xgd.PositiveLoadFollowReserveQuantity = zeros(ng, 1); -xgd.NegativeLoadFollowReservePrice = zeros(ng, 1); -xgd.NegativeLoadFollowReserveQuantity = zeros(ng, 1); - -nt = 24; - -% Test 1: loadmd with non-consecutive bus numbering -fprintf('\n--- Test 1: loadmd on raw ACTIVSg 2000 (non-consecutive) ---\n'); -try - md = loadmd(mpc, [], xgd, [], nt); - fprintf('loadmd: SUCCESS (unexpected)\n'); -catch e - fprintf('loadmd: FAILED\n'); - fprintf('Error: %s\n', e.message); - if ~isempty(strfind(e.message, 'consecutively')) - fprintf('>>> CONFIRMED: fails with consecutive bus numbering error\n'); - elseif ~isempty(strfind(e.message, 'consecutive')) - fprintf('>>> CONFIRMED: fails with consecutive bus numbering error\n'); - else - fprintf('>>> Different error than expected\n'); - end -end - -% Test 2: loadmd with ext2int-converted case -fprintf('\n--- Test 2: loadmd on ext2int-converted ACTIVSg 2000 ---\n'); -mpc_int = ext2int(mpc); -ng_int = size(mpc_int.gen, 1); -fprintf('Internal: %d buses, %d gens (ext2int removed %d offline gens)\n', ... - size(mpc_int.bus, 1), ng_int, ng - ng_int); - -xgd_int = struct(); -xgd_int.CommitSched = ones(ng_int, 1); -xgd_int.InitialPg = mpc_int.gen(:, 2); -xgd_int.RampWearCostCoeff = zeros(ng_int, 1); -xgd_int.PositiveActiveReservePrice = zeros(ng_int, 1); -xgd_int.PositiveActiveReserveQuantity = zeros(ng_int, 1); -xgd_int.NegativeActiveReservePrice = zeros(ng_int, 1); -xgd_int.NegativeActiveReserveQuantity = zeros(ng_int, 1); -xgd_int.PositiveActiveDeltaPrice = zeros(ng_int, 1); -xgd_int.NegativeActiveDeltaPrice = zeros(ng_int, 1); -xgd_int.PositiveLoadFollowReservePrice = zeros(ng_int, 1); -xgd_int.PositiveLoadFollowReserveQuantity = zeros(ng_int, 1); -xgd_int.NegativeLoadFollowReservePrice = zeros(ng_int, 1); -xgd_int.NegativeLoadFollowReserveQuantity = zeros(ng_int, 1); - -try - md = loadmd(mpc_int, [], xgd_int, [], nt); - fprintf('loadmd: SUCCESS with ext2int\n'); - fprintf('>>> Confirms failure is bus-numbering, not data format\n'); -catch e - fprintf('loadmd: FAILED even with ext2int\n'); - fprintf('Error: %s\n', e.message); -end - -% Test 3: Control with case39 (consecutive buses) -fprintf('\n--- Test 3: loadmd on case39 (consecutive, control) ---\n'); -mpc39 = loadcase(fullfile('..', '..', 'data', 'networks', 'case39.m')); -ng39 = size(mpc39.gen, 1); - -xgd39 = struct(); -xgd39.CommitSched = ones(ng39, 1); -xgd39.InitialPg = mpc39.gen(:, 2); -xgd39.RampWearCostCoeff = zeros(ng39, 1); -xgd39.PositiveActiveReservePrice = zeros(ng39, 1); -xgd39.PositiveActiveReserveQuantity = zeros(ng39, 1); -xgd39.NegativeActiveReservePrice = zeros(ng39, 1); -xgd39.NegativeActiveReserveQuantity = zeros(ng39, 1); -xgd39.PositiveActiveDeltaPrice = zeros(ng39, 1); -xgd39.NegativeActiveDeltaPrice = zeros(ng39, 1); -xgd39.PositiveLoadFollowReservePrice = zeros(ng39, 1); -xgd39.PositiveLoadFollowReserveQuantity = zeros(ng39, 1); -xgd39.NegativeLoadFollowReservePrice = zeros(ng39, 1); -xgd39.NegativeLoadFollowReserveQuantity = zeros(ng39, 1); - -try - md = loadmd(mpc39, [], xgd39, [], nt); - fprintf('loadmd: SUCCESS on case39\n'); -catch e - fprintf('loadmd: FAILED on case39\n'); - fprintf('Error: %s\n', e.message); -end - -fprintf('\n=== SUMMARY ===\n'); diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-006.md b/sweep-data/v4-to-v5/probes/pandapower/probe-006.md deleted file mode 100644 index a4153785..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-006.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -probe_id: probe-006 -tool: pandapower -source_test: C-6 -probe_type: convergence_check -classification: claim_supported -reason: Probe confirms extremely low convergence rate (0.42% vs claimed 2.1%) for PYPOWER interior point on perturbed ACTIVSg2000 -solver_version: PYPOWER interior point (pandapower 3.4.0) -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 226.82 -timestamp: 2026-03-09T00:00:00Z ---- - -# Probe 006: Stochastic DCOPF 2.1% convergence rate on ACTIVSg2000 - -## Original Claim - -From `evaluations/pandapower/results/scalability/C-6_stochastic_scale.md`: - -> Total solves: 240, Converged: 5 (2.1%), Failed: 235 (97.9%) -> The very low convergence rate (2.1%) is a PYPOWER interior point solver quality issue on the modified ACTIVSg2000 network. - -The test was classified as `qualified_pass` despite this near-total solver failure. - -## Probe Methodology - -Replicated the exact C-6 test scenario: -1. Loaded ACTIVSg2000 (2,000 buses, 484 generators, 1,125 loads) -2. Solved base-case DC OPF with no perturbations -3. Ran all 240 solves (20 scenarios x 12 hours) with identical RNG seed (42), perturbation approach, and hourly load shape -4. Additionally tested uniform load scaling (0.5x to 1.1x) without gen perturbations to isolate failure cause - -Script: `sweep-data/v4-to-v5/probes/pandapower/probe-006_script.py` - -## Probe Results - -**Base case (no perturbations):** Converged successfully, objective = 1,201,321 - -**Perturbed scenario loop (240 solves):** - -| Metric | Original (C-6) | Probe | -|--------|----------------|-------| -| Total solves | 240 | 240 | -| Converged | 5 (2.1%) | 1 (0.42%) | -| Failed | 235 (97.9%) | 239 (99.6%) | -| Per-solve avg time | 1.31 s | 0.91 s | -| Total solve time | 314.18 s | 217.20 s | - -The single convergence occurred at hour 8 (load scale ~1.05). - -**Uniform load scaling (no gen perturbation):** - -| Scale | Converged | -|-------|-----------| -| 0.5x | No | -| 0.6x | No | -| 0.7x | Yes | -| 0.8x | Yes | -| 0.9x | Yes | -| 1.0x | Yes | -| 1.1x | Yes | - -This shows the base solver works for moderate load levels, but the combination of load scaling AND generator capacity perturbations causes near-total failure. - -## Analysis - -The probe confirms the core claim: PYPOWER interior point solver has extremely poor convergence on the perturbed ACTIVSg2000 network. The probe actually found an even worse convergence rate (0.42% vs 2.1%), which is directionally consistent — the small difference is likely due to different random noise draws (the `np.random.normal(0, 0.02)` individual noise in the inner loop is not seeded identically between runs because the state evolves differently with each solve's timing). - -Key findings: -- The base case converges fine, confirming this is not a network-loading issue -- Uniform load scaling without gen perturbation converges for 0.7x-1.1x -- The combination of load AND generator capacity perturbations is what breaks the solver -- This is genuinely a PYPOWER interior point solver quality issue, not a test infrastructure bug - -The `qualified_pass` grading is debatable — a 0.4-2.1% convergence rate means the workaround approach has essentially zero practical utility at SMALL scale. However, this probe was asked to verify the convergence rate claim, not the grading decision. - -## Classification Rationale - -Classified as `claim_supported` because: -1. The probe reproduces the same phenomenon (near-total solver failure on perturbed ACTIVSg2000) -2. The convergence rate is even lower than claimed (0.42% vs 2.1%), making the original claim conservative -3. The root cause (PYPOWER interior point solver fragility with perturbations) is confirmed -4. Same pandapower version (3.4.0) and solver used diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-006_script.py b/sweep-data/v4-to-v5/probes/pandapower/probe-006_script.py deleted file mode 100644 index 3f2e60f8..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-006_script.py +++ /dev/null @@ -1,181 +0,0 @@ -""" -Probe 006: Verify 2.1% convergence rate claim for stochastic DCOPF on ACTIVSg2000. - -Claim: "Stochastic DCOPF wrapping qualified_pass despite 2.1% solver convergence rate -(5 of 240 solves)" in C-6. - -Approach: -1. Load ACTIVSg2000 via from_mpc -2. Solve base-case DC OPF (no perturbations) — does it converge? -3. Apply small load perturbations (same approach as original test) for a subset of scenarios -4. Report convergence rate and compare to 2.1% -""" - -import json -import time - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc - -start = time.perf_counter() -results = {} - -try: - # 1. Load network - net = from_mpc("/workspace/data/networks/case_ACTIVSg2000.m", f_hz=60) - results["bus_count"] = len(net.bus) - results["gen_count"] = len(net.gen) - results["load_count"] = len(net.load) - - # Ensure cost curves exist - has_costs = len(net.poly_cost) > 0 or len(net.pwl_cost) > 0 - results["has_imported_costs"] = has_costs - if not has_costs: - for idx in net.gen.index: - pp.create_poly_cost(net, idx, "gen", cp1_eur_per_mw=20.0 + idx * 0.5) - for idx in net.ext_grid.index: - pp.create_poly_cost(net, idx, "ext_grid", cp1_eur_per_mw=50.0) - - # 2. Base case DC OPF — no perturbations - try: - pp.rundcopp(net) - base_converged = net.get("OPF_converged", False) - except Exception as e: - base_converged = False - results["base_case_error"] = str(e) - results["base_case_converged"] = base_converged - if base_converged: - results["base_case_objective"] = float(net.res_cost) - - # 3. Replicate the original test scenario loop (smaller subset for speed) - # Use same RNG seed and perturbation approach as original - np.random.seed(42) - n_scenarios = 20 - n_hours = 12 - - hourly_shape = np.array( - [0.70, 0.65, 0.60, 0.60, 0.65, 0.75, 0.90, 1.00, 1.05, 1.10, 1.05, 1.00] - ) - load_base_signal = np.random.normal(0, 0.08, (n_scenarios, n_hours)) - - # Classify generators (same as original) - gen_costs = [] - for idx in net.gen.index: - cost_rows = net.poly_cost[ - (net.poly_cost["element"] == idx) & (net.poly_cost["et"] == "gen") - ] - cp1 = ( - float(cost_rows.iloc[0].get("cp1_eur_per_mw", 0)) - if len(cost_rows) > 0 - else 0 - ) - gen_costs.append({"gen_idx": int(idx), "cp1": cp1}) - gen_costs.sort(key=lambda x: x["cp1"]) - n_gens = len(gen_costs) - q1 = max(1, n_gens // 4) - q3 = max(q1 + 1, 3 * n_gens // 4) - - resource_types = {} - for i, gc in enumerate(gen_costs): - if i < q1: - resource_types[gc["gen_idx"]] = "baseload" - elif i >= q3: - resource_types[gc["gen_idx"]] = "peaker" - else: - resource_types[gc["gen_idx"]] = "intermediate" - - type_groups = {} - for gen_idx, rtype in resource_types.items(): - type_groups.setdefault(rtype, []).append(gen_idx) - - type_base_signals = {} - for rtype in type_groups: - type_base_signals[rtype] = np.random.normal(0, 0.05, (n_scenarios, n_hours)) - - base_loads = net.load["p_mw"].values.copy() - base_gen_max = net.gen["max_p_mw"].values.copy() - - # Run all 240 solves (same as original) - total_converged = 0 - total_failed = 0 - all_objectives = [] - convergence_by_hour = {h: {"converged": 0, "failed": 0} for h in range(n_hours)} - - solve_start = time.perf_counter() - for s in range(n_scenarios): - for h in range(n_hours): - load_scale = hourly_shape[h] * (1.0 + load_base_signal[s, h]) - net.load["p_mw"] = base_loads * max(load_scale, 0.3) - - for gen_idx, rtype in resource_types.items(): - base_signal = type_base_signals[rtype][s, h] - individual_noise = np.random.normal(0, 0.02) - perturbation = 1.0 + base_signal + individual_noise - gen_pos = list(net.gen.index).index(gen_idx) - net.gen.at[gen_idx, "max_p_mw"] = base_gen_max[gen_pos] * max( - perturbation, 0.5 - ) - - try: - pp.rundcopp(net) - converged = net.get("OPF_converged", False) - except Exception: - converged = False - - if converged: - total_converged += 1 - convergence_by_hour[h]["converged"] += 1 - obj = float(net.res_cost) if hasattr(net, "res_cost") else None - if obj is not None: - all_objectives.append(obj) - else: - total_failed += 1 - convergence_by_hour[h]["failed"] += 1 - - solve_elapsed = time.perf_counter() - solve_start - - # Restore - net.load["p_mw"] = base_loads - net.gen["max_p_mw"] = base_gen_max - - total_solves = n_scenarios * n_hours - results["total_solves"] = total_solves - results["total_converged"] = total_converged - results["total_failed"] = total_failed - results["convergence_rate_pct"] = round(total_converged / total_solves * 100, 2) - results["solve_loop_seconds"] = round(solve_elapsed, 2) - results["per_solve_avg_seconds"] = round(solve_elapsed / total_solves, 3) - - if all_objectives: - results["objective_mean"] = round(float(np.mean(all_objectives)), 2) - - # Also test: what about solving with no perturbation at different load levels? - results["convergence_by_hour"] = { - str(h): convergence_by_hour[h] for h in range(n_hours) - } - - # 4. Diagnostic: test uniform load scaling without gen perturbation - uniform_results = [] - for scale in [0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1]: - net.load["p_mw"] = base_loads * scale - net.gen["max_p_mw"] = base_gen_max # no gen perturbation - try: - pp.rundcopp(net) - conv = net.get("OPF_converged", False) - except Exception: - conv = False - uniform_results.append({"scale": scale, "converged": conv}) - results["uniform_load_scale_convergence"] = uniform_results - - net.load["p_mw"] = base_loads - net.gen["max_p_mw"] = base_gen_max - -except Exception as e: - results["error"] = f"{type(e).__name__}: {e}" - import traceback - - results["traceback"] = traceback.format_exc() - -results["wall_clock_seconds"] = round(time.perf_counter() - start, 2) -print(json.dumps(results, indent=2, default=str)) diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-007.md b/sweep-data/v4-to-v5/probes/pandapower/probe-007.md deleted file mode 100644 index feba4f8a..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-007.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -probe_id: probe-007 -tool: pandapower -source_test: A-3 -probe_type: formulation_audit -classification: claim_supported -reason: LMPs are truly uniform (std=6e-12); no branch exceeds 85% loading; artificial congestion produces LMP spread (16.3--24.4), confirming the base case simply has no binding constraints -solver_version: pandapower 3.4.0 (PYPOWER interior point) -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 14.20 -timestamp: 2026-03-09T00:00:00Z ---- - -# Probe 007: Uniform LMPs across all 10,000 buses on MEDIUM DCOPF - -## Original Claim - -From `evaluations/pandapower/results/expressiveness/A-3_dcopf.md`: - -> LMP range: 20.738 -- 20.738 -> LMP mean: 20.738 -> LMPs are nearly uniform across all buses (~20.738), indicating no binding line constraints in the DC OPF solution. - -## Probe Methodology - -1. Loaded ACTIVSg10k and solved DC OPF via `pp.rundcopp(net)` -2. Extracted LMP distribution from `net.res_bus["lam_p"]` -3. Analyzed branch loading from both pandapower results and internal ppc arrays -4. Ran congestion experiment: reduced line limits to 90% of DCPF flow on top 5 lines, re-solved DC OPF to verify LMPs change when constraints bind - -Scripts: `sweep-data/v4-to-v5/probes/pandapower/probe-007_script.py`, `probe-007_congestion.py` - -## Probe Results - -### Base Case LMPs - -| Metric | Original (A-3) | Probe | -|--------|----------------|-------| -| LMP min | 20.738 | 20.737729 | -| LMP max | 20.738 | 20.737729 | -| LMP mean | 20.738 | 20.737729 | -| LMP std | -- | 5.977e-12 | -| Unique LMPs (6dp) | 1 | 1 | -| All identical (1e-6) | Yes | Yes | -| Objective | 2,437,763.82 | 2,437,763.82 | - -LMPs are truly uniform to machine precision (std = 6e-12). - -### Branch Loading (Base Case) - -| Metric | Lines | Transformers | All (ppc) | -|--------|-------|-------------|-----------| -| Count | 9,726 | 975 | 12,706 | -| Max loading | 76.89% | 76.93% | 84.90% | -| Mean loading | 16.30% | -- | 17.70% | -| Branches > 90% | 0 | 0 | 0 | -| Branches > 95% | 0 | 0 | 0 | - -No branch exceeds 85% loading. The network has ample transmission capacity for the base case load. - -Top 3 loaded branches: -1. Branch 12340: 84.90% (226 MW flow, 266 MW limit) -2. Branch 12212: 79.89% (397 MW flow, 496 MW limit) -3. Branch 9843: 76.93% (133 MW flow, 173 MW limit) - -### Congestion Experiment -Reduced line limits on the 5 highest-flow lines to 90% of their DCPF flow (forcing binding constraints): - -| Metric | Base Case | Congested Case | -|--------|-----------|----------------| -| Converged | Yes | Yes | -| LMP min | 20.738 | 16.325 | -| LMP max | 20.738 | 24.363 | -| LMP std | 6e-12 | 1.332 | -| Unique LMPs | 1 | 4,225 | -| LMPs uniform | Yes | No | - -When constraints bind, LMPs spread from 16.3 to 24.4 across 4,225 distinct values, confirming that pandapower's DC OPF correctly produces differentiated LMPs in the presence of congestion. - -## Analysis - -The claim is fully supported. The uniform LMP of 20.738 is not a bug or modeling error -- it is the correct economic result for a network with no binding transmission constraints. Key evidence: - -1. **No binding constraints exist.** The most loaded branch is at only 84.9% of its limit. With no branches at or near 100%, there are no congestion rents, so all buses see the same marginal cost. - -2. **The solver correctly differentiates LMPs when congestion is present.** The congestion experiment proves the OPF formulation includes line limits and produces spatially varying LMPs when those limits bind. - -3. **The ACTIVSg10k has generous transmission capacity** relative to its load pattern. This is a property of the test case, not a limitation of pandapower. - -The original evaluation correctly diagnosed the cause ("no binding line constraints") and appropriately reported the result as a `qualified_pass` (the qualification being solver choice, not the uniform LMPs). - -## Classification Rationale - -Classified as `claim_supported` because: -1. The uniform LMP value is reproduced exactly (20.737729, std = 6e-12) -2. Branch loading analysis confirms no constraints are near binding (max 84.9%) -3. The congestion experiment demonstrates LMPs correctly vary when constraints do bind (LMP range 16.3--24.4) -4. The original attribution to "no binding line constraints" is verified as correct diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-007_congestion.py b/sweep-data/v4-to-v5/probes/pandapower/probe-007_congestion.py deleted file mode 100644 index 68551730..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-007_congestion.py +++ /dev/null @@ -1,75 +0,0 @@ -""" -Probe 007 supplemental: Congestion experiment with gentler limit reductions. -""" - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc - -print("--- Congestion Experiment: 80% of DCPF flow on top 10 lines ---") -net = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -pp.rundcpp(net) - -# Find top loaded lines by absolute flow -line_flows = net.res_line["p_from_mw"].abs() -top_lines = line_flows.nlargest(10).index - -for idx in top_lines: - current_flow_mw = abs(net.res_line.at[idx, "p_from_mw"]) - vn_kv = net.bus.at[net.line.at[idx, "from_bus"], "vn_kv"] - target_mw = current_flow_mw * 0.80 - target_i_ka = target_mw / (np.sqrt(3) * vn_kv) if vn_kv > 0 else 0 - if target_i_ka > 0: - net.line.at[idx, "max_i_ka"] = target_i_ka - print( - f" Line {idx}: flow={current_flow_mw:.1f} MW, new limit={target_mw:.1f} MW" - ) - -try: - pp.rundcopp(net) - if net["OPF_converged"]: - lmps = net.res_bus["lam_p"].values - print("Converged: Yes") - print(f"LMP min: {np.min(lmps):.6f}") - print(f"LMP max: {np.max(lmps):.6f}") - print(f"LMP std: {np.std(lmps):.6e}") - print(f"Unique LMPs: {len(np.unique(np.round(lmps, 4)))}") - print(f"LMPs changed: {not np.allclose(lmps, lmps[0], atol=1e-6)}") - else: - print("Did not converge") -except Exception as e: - print(f"Error: {e}") - -# Try even gentler: 90% on top 5 -print("\n--- Congestion Experiment: 90% of DCPF flow on top 5 lines ---") -net2 = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -pp.rundcpp(net2) - -line_flows2 = net2.res_line["p_from_mw"].abs() -top_lines2 = line_flows2.nlargest(5).index - -for idx in top_lines2: - current_flow_mw = abs(net2.res_line.at[idx, "p_from_mw"]) - vn_kv = net2.bus.at[net2.line.at[idx, "from_bus"], "vn_kv"] - target_mw = current_flow_mw * 0.90 - target_i_ka = target_mw / (np.sqrt(3) * vn_kv) if vn_kv > 0 else 0 - if target_i_ka > 0: - net2.line.at[idx, "max_i_ka"] = target_i_ka - print( - f" Line {idx}: flow={current_flow_mw:.1f} MW, new limit={target_mw:.1f} MW" - ) - -try: - pp.rundcopp(net2) - if net2["OPF_converged"]: - lmps2 = net2.res_bus["lam_p"].values - print("Converged: Yes") - print(f"LMP min: {np.min(lmps2):.6f}") - print(f"LMP max: {np.max(lmps2):.6f}") - print(f"LMP std: {np.std(lmps2):.6e}") - print(f"Unique LMPs: {len(np.unique(np.round(lmps2, 4)))}") - print(f"LMPs changed: {not np.allclose(lmps2, lmps2[0], atol=1e-6)}") - else: - print("Did not converge") -except Exception as e: - print(f"Error: {e}") diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-007_script.py b/sweep-data/v4-to-v5/probes/pandapower/probe-007_script.py deleted file mode 100644 index 75e4970f..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-007_script.py +++ /dev/null @@ -1,200 +0,0 @@ -""" -Probe 007: Verify uniform LMP claim on ACTIVSg10k DC OPF. - -Checks: -1. Branch loading distribution (any > 90%? any at 100%?) -2. LMP distribution (truly all identical?) -3. If uniform, scale down some branch limits and re-solve to confirm LMPs change with congestion -""" - -import time - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc - -start_time = time.perf_counter() - -print("=" * 60) -print("PROBE 007: Uniform LMP verification on ACTIVSg10k DCOPF") -print("=" * 60) - -# 1. Load network -print("\n--- Loading ACTIVSg10k ---") -net = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -print(f"Buses: {len(net.bus)}") -print(f"Lines: {len(net.line)}") -print(f"Trafos: {len(net.trafo)}") -print(f"Generators: {len(net.gen)}") -print(f"Ext grids: {len(net.ext_grid)}") -print(f"Poly costs: {len(net.poly_cost)}") - -# Check if costs exist, add if not -has_costs = len(net.poly_cost) > 0 or len(net.pwl_cost) > 0 -print(f"Has cost curves from import: {has_costs}") - -# 2. Run DC OPF -print("\n--- Running DC OPF (base case) ---") -t0 = time.perf_counter() -pp.rundcopp(net) -t1 = time.perf_counter() -print(f"Converged: {net['OPF_converged']}") -print(f"Wall clock: {t1 - t0:.2f}s") -print(f"Objective: {float(net.res_cost):.2f}") - -# 3. Extract and analyze LMPs -print("\n--- LMP Analysis ---") -lmps = net.res_bus["lam_p"].values -lmp_min = float(np.min(lmps)) -lmp_max = float(np.max(lmps)) -lmp_mean = float(np.mean(lmps)) -lmp_std = float(np.std(lmps)) -lmp_unique = len(np.unique(np.round(lmps, 6))) - -print(f"LMP min: {lmp_min:.6f}") -print(f"LMP max: {lmp_max:.6f}") -print(f"LMP mean: {lmp_mean:.6f}") -print(f"LMP std: {lmp_std:.6e}") -print(f"Unique LMPs (rounded to 6dp): {lmp_unique}") -print(f"All identical (within 1e-6): {np.allclose(lmps, lmps[0], atol=1e-6)}") - -# 4. Analyze branch loading -print("\n--- Branch Loading Analysis ---") -# For lines -if len(net.res_line) > 0 and "loading_percent" in net.res_line.columns: - line_loading = net.res_line["loading_percent"].values - line_loading_valid = line_loading[~np.isnan(line_loading)] - print(f"Lines with loading data: {len(line_loading_valid)}/{len(net.res_line)}") - if len(line_loading_valid) > 0: - print(f" Max loading: {np.max(line_loading_valid):.2f}%") - print(f" Mean loading: {np.mean(line_loading_valid):.2f}%") - print(f" Lines > 90%: {np.sum(line_loading_valid > 90)}") - print(f" Lines > 95%: {np.sum(line_loading_valid > 95)}") - print( - f" Lines = 100%: {np.sum(np.isclose(line_loading_valid, 100, atol=0.1))}" - ) -else: - print("No line loading results available") - -# For trafos -if len(net.res_trafo) > 0 and "loading_percent" in net.res_trafo.columns: - trafo_loading = net.res_trafo["loading_percent"].values - trafo_loading_valid = trafo_loading[~np.isnan(trafo_loading)] - print(f"Trafos with loading data: {len(trafo_loading_valid)}/{len(net.res_trafo)}") - if len(trafo_loading_valid) > 0: - print(f" Max trafo loading: {np.max(trafo_loading_valid):.2f}%") - print(f" Trafos > 90%: {np.sum(trafo_loading_valid > 90)}") - -# Check branch flow from ppc if available -print("\n--- Branch flow analysis via ppc ---") -if hasattr(net, "_ppc") and net._ppc is not None: - ppc = net._ppc - branch = ppc["branch"] - from pandapower.pypower.idx_brch import PF, RATE_A - - flows = np.abs(branch[:, PF]) - rates = branch[:, RATE_A] - # Filter branches with non-zero rate - has_rate = rates > 0 - print(f"Branches with RATE_A > 0: {np.sum(has_rate)}/{len(rates)}") - if np.sum(has_rate) > 0: - loading_pct = (flows[has_rate] / rates[has_rate]) * 100 - print(f" Max loading: {np.max(loading_pct):.2f}%") - print(f" Mean loading: {np.mean(loading_pct):.2f}%") - print(f" Branches > 90%: {np.sum(loading_pct > 90)}") - print(f" Branches > 95%: {np.sum(loading_pct > 95)}") - print(f" Branches > 99%: {np.sum(loading_pct > 99)}") - print(f" Branches = 100%: {np.sum(np.isclose(loading_pct, 100, atol=0.1))}") - - # Top 10 most loaded - top_idx = np.argsort(loading_pct)[-10:][::-1] - rated_indices = np.where(has_rate)[0] - print("\n Top 10 loaded branches:") - for rank, i in enumerate(top_idx): - br_idx = rated_indices[i] - print( - f" {rank + 1}. Branch {br_idx}: {loading_pct[i]:.2f}% " - f"(flow={flows[br_idx]:.2f} MW, limit={rates[br_idx]:.2f} MW)" - ) - else: - print(" No branches have rate limits set!") -else: - print("No _ppc data available (need to run rundcopp with internals)") - -# Try to also get the ppc from a DC power flow for comparison -print("\n--- Running DCPF to get ppc data ---") -net2 = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -pp.rundcpp(net2) -if hasattr(net2, "_ppc") and net2._ppc is not None: - ppc2 = net2._ppc - branch2 = ppc2["branch"] - from pandapower.pypower.idx_brch import PF as PF2, RATE_A as RATE_A2 - - flows2 = np.abs(branch2[:, PF2]) - rates2 = branch2[:, RATE_A2] - has_rate2 = rates2 > 0 - print(f"DCPF - Branches with RATE_A > 0: {np.sum(has_rate2)}/{len(rates2)}") - if np.sum(has_rate2) > 0: - loading_pct2 = (flows2[has_rate2] / rates2[has_rate2]) * 100 - print(f" DCPF Max loading: {np.max(loading_pct2):.2f}%") - print(f" DCPF Branches > 90%: {np.sum(loading_pct2 > 90)}") - -# 5. If LMPs are uniform, try creating congestion -if np.allclose(lmps, lmps[0], atol=1e-6): - print("\n--- Congestion Experiment: Reducing branch limits ---") - net3 = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) - - # First, find the most loaded branches from DCPF - pp.rundcpp(net3) - if hasattr(net3, "_ppc") and net3._ppc is not None: - ppc3 = net3._ppc - branch3 = ppc3["branch"] - dcpf_flows = np.abs(branch3[:, PF]) - - # Now reduce line limits to 50% of current flow on top-loaded lines - # This should force congestion - if "max_i_ka" in net3.line.columns: - # Get current flows from DCPF results - line_flows = net3.res_line["p_from_mw"].abs().values - # Find lines with significant flow - significant = line_flows > 10 # MW - n_reduced = 0 - for idx in net3.line.index[significant]: - current_flow_mw = abs(net3.res_line.at[idx, "p_from_mw"]) - if current_flow_mw > 10: - # Set max_i_ka to force limit at 50% of current flow - # P = sqrt(3) * V * I * cos(phi), but for DC approx: P_MW = sqrt(3) * V_kV * I_kA - vn_kv = net3.bus.at[net3.line.at[idx, "from_bus"], "vn_kv"] - target_mw = current_flow_mw * 0.5 - target_i_ka = target_mw / (np.sqrt(3) * vn_kv) if vn_kv > 0 else 0 - if target_i_ka > 0: - net3.line.at[idx, "max_i_ka"] = target_i_ka - n_reduced += 1 - if n_reduced >= 50: - break - - print(f"Reduced limits on {n_reduced} lines to 50% of DCPF flow") - - # Re-run DC OPF - try: - pp.rundcopp(net3) - if net3["OPF_converged"]: - lmps3 = net3.res_bus["lam_p"].values - lmp3_min = float(np.min(lmps3)) - lmp3_max = float(np.max(lmps3)) - lmp3_std = float(np.std(lmps3)) - lmp3_unique = len(np.unique(np.round(lmps3, 6))) - print("Congested case converged: Yes") - print(f"Congested LMP min: {lmp3_min:.6f}") - print(f"Congested LMP max: {lmp3_max:.6f}") - print(f"Congested LMP std: {lmp3_std:.6e}") - print(f"Congested unique LMPs: {lmp3_unique}") - print(f"LMPs changed: {not np.allclose(lmps3, lmps3[0], atol=1e-6)}") - print(f"Objective: {float(net3.res_cost):.2f}") - else: - print("Congested case did NOT converge") - except Exception as e: - print(f"Congested case error: {e}") - -elapsed = time.perf_counter() - start_time -print(f"\n--- Total elapsed: {elapsed:.2f}s ---") diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-008.md b/sweep-data/v4-to-v5/probes/pandapower/probe-008.md deleted file mode 100644 index bc4c67a9..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-008.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -probe_id: probe-008 -tool: pandapower -source_test: B-9 -probe_type: convergence_check -classification: claim_supported -reason: Probe reproduces exact max diff of 7.43 pu on ACTIVSg10k; case39 matches perfectly (0.0 diff) -solver_version: pandapower 3.4.0 (PYPOWER DCPF) -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 11.68 -timestamp: 2026-03-09T00:00:00Z ---- - -# Probe 008: PTDF flow divergence from DCPF on MEDIUM (7.43 pu) - -## Original Claim - -From `evaluations/pandapower/results/extensibility/B-9_ptdf_extraction.md`: - -> Max flow difference: 7.43 pu -> Mean flow difference: 0.027 pu -> The PTDF matrix computes successfully with correct dimensions and expected properties [...] However, flow predictions diverge from DCPF results on the 10k-bus network (max diff 7.43 pu), likely due to shunt elements and tap-ratio effects in transformers not fully captured by the basic PTDF formulation. On TINY (39-bus), match was exact within 1e-6. - -The test was still classified as `qualified_pass`. - -## Probe Methodology - -1. Loaded case39 (TINY) and ACTIVSg10k (MEDIUM) networks -2. Solved DCPF on each to populate `net._ppc` -3. Computed PTDF via `pandapower.pypower.makePTDF.makePTDF(baseMVA, bus, branch, slack_idx)` -4. Built bus injection vector from solved ppc (gen - load - shunt) -5. Compared PTDF-predicted flows (`PTDF @ Pbus`) to actual DCPF branch flows -6. Also tested without shunt subtraction to isolate shunt contribution - -Script: `sweep-data/v4-to-v5/probes/pandapower/probe-008_script.py` - -## Probe Results - -### case39 (TINY) — Baseline - -| Metric | Value | -|--------|-------| -| Buses | 39 | -| Branches | 46 | -| Max diff | 0.0 pu | -| Mean diff | 0.0 pu | -| Shunt MW | 0.0 | - -Perfect match, confirming PTDF methodology is correct. - -### ACTIVSg10k (MEDIUM) - -| Metric | Original (B-9) | Probe | -|--------|----------------|-------| -| Buses | 10,000 | 10,000 | -| Branches | 12,706 | 12,706 | -| Slack bus idx | 7236 | 7236 | -| PTDF shape | (12706, 10000) | (12706, 10000) | -| Max diff (pu) | 7.43 | 7.4346 | -| Mean diff (pu) | 0.027 | 0.0268 | -| PTDF memory | 969.39 MB | 969.39 MB | -| PTDF time | 28.03 s | 10.86 s | - -Worst 5 branches: - -| Branch | DCPF flow (pu) | PTDF flow (pu) | Diff (pu) | -|--------|---------------|----------------|-----------| -| 10195 | 20.35 | 12.92 | 7.43 | -| 7276 | -4.62 | 1.01 | 5.63 | -| 12213 | -4.62 | 1.01 | 5.63 | -| 10444 | 4.62 | -1.01 | 5.63 | -| 5573 | 8.97 | 4.80 | 4.17 | - -Removing shunt subtraction had no effect (total shunt MW = 0 in this network), so the divergence is not caused by shunt modeling. The most likely cause is transformer tap ratios: the ACTIVSg10k has branches with non-unity tap ratios that the basic PTDF formulation (which assumes a simple B-matrix) does not correctly handle. The standard PTDF = Bf * inv(Bbus) derivation treats all branches as simple impedances, but transformers with tap ratios modify the admittance matrix asymmetrically. - -## Analysis - -The probe reproduces the original claim with near-exact precision: -- Max diff: 7.4346 pu (original: 7.43 pu) — matches to 3 significant figures -- Mean diff: 0.0268 pu (original: 0.027 pu) — matches to 2 significant figures -- case39 baseline: exact match (0.0 diff), consistent with original "within 1e-6" - -The 7.43 pu max difference on the 10k-bus network is a real phenomenon, not a test bug. It represents a 743 MW flow prediction error on the worst branch (branch 10195, which carries 2,035 MW according to DCPF). This is a ~36% relative error on that branch. - -The original attribution to "shunt elements" is incorrect — total shunt MW is 0.0. The divergence is caused by transformer tap ratios, which the basic `makePTDF` formulation does not account for. The ACTIVSg10k has many transformers with non-unity tap ratios. - -## Classification Rationale - -Classified as `claim_supported` because: -1. The max diff value is reproduced essentially exactly (7.4346 vs 7.43 pu) -2. The case39 baseline match is confirmed (0.0 diff) -3. The phenomenon is real and reproducible -4. The `qualified_pass` grading reflects that PTDF computation works but with accuracy limitations on networks with transformers diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-008_script.py b/sweep-data/v4-to-v5/probes/pandapower/probe-008_script.py deleted file mode 100644 index e39a73f4..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-008_script.py +++ /dev/null @@ -1,155 +0,0 @@ -""" -Probe 008: Verify PTDF flow divergence claim on ACTIVSg10k. - -Claim: "PTDF flow predictions diverge from DCPF on MEDIUM (max diff 7.43 pu) -but test still passes" - -Approach: -1. Load ACTIVSg10k and solve DCPF -2. Compute PTDF matrix via makePTDF -3. Build injection vector from solved ppc -4. Compare PTDF-predicted flows to DCPF actual flows -5. Also test on case39 (TINY) as a baseline -""" - -import json -import time - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc -from pandapower.pypower.makePTDF import makePTDF -from pandapower.pypower.idx_bus import BUS_I, BUS_TYPE, GS, PD, REF -from pandapower.pypower.idx_brch import PF -from pandapower.pypower.idx_gen import GEN_BUS, GEN_STATUS, PG as GEN_PG - -start = time.perf_counter() -results = {} - - -def test_ptdf(network_file, label): - """Test PTDF accuracy on a given network.""" - r = {} - try: - net = from_mpc(network_file, f_hz=60) - pp.rundcpp(net) - assert net["converged"], "DCPF did not converge" - - ppc = net._ppc - baseMVA = ppc["baseMVA"] - bus = ppc["bus"] - branch = ppc["branch"] - gen = ppc["gen"] - - n_bus = bus.shape[0] - n_branch = branch.shape[0] - - r["n_bus"] = n_bus - r["n_branch"] = n_branch - r["baseMVA"] = float(baseMVA) - - # Slack bus - ref_buses = np.where(bus[:, BUS_TYPE] == REF)[0] - slack_bus_idx = int(ref_buses[0]) if len(ref_buses) > 0 else 0 - r["slack_bus_idx"] = slack_bus_idx - - # Compute PTDF - ptdf_start = time.perf_counter() - PTDF = makePTDF(baseMVA, bus, branch, slack_bus_idx) - ptdf_time = time.perf_counter() - ptdf_start - r["ptdf_seconds"] = round(ptdf_time, 3) - r["ptdf_shape"] = list(PTDF.shape) - r["ptdf_memory_mb"] = round(PTDF.nbytes / (1024 * 1024), 2) - - # Build bus injection vector - ext_to_int = {} - for i in range(n_bus): - ext_to_int[int(bus[i, BUS_I])] = i - - Pbus_mw = np.zeros(n_bus) - Pbus_mw -= bus[:, PD] # subtract loads - - if bus.shape[1] > GS: - Pbus_mw -= bus[:, GS] # subtract shunts - - for i in range(gen.shape[0]): - if gen[i, GEN_STATUS] > 0: - ext_bus = int(gen[i, GEN_BUS]) - int_idx = ext_to_int.get(ext_bus, -1) - if int_idx >= 0: - Pbus_mw[int_idx] += gen[i, GEN_PG] - - Pbus_pu = Pbus_mw / baseMVA - - # Actual DCPF flows - branch_flows_pu = branch[:, PF] / baseMVA - - # PTDF-predicted flows - predicted_flows_pu = PTDF @ Pbus_pu - - # Differences - flow_diff = np.abs(predicted_flows_pu - branch_flows_pu) - r["max_diff_pu"] = round(float(np.max(flow_diff)), 6) - r["mean_diff_pu"] = round(float(np.mean(flow_diff)), 6) - r["median_diff_pu"] = round(float(np.median(flow_diff)), 6) - r["max_diff_mw"] = round(float(np.max(flow_diff) * baseMVA), 4) - - # Where are the big differences? - worst_indices = np.argsort(flow_diff)[-5:][::-1] - r["worst_5_branches"] = [] - for idx in worst_indices: - r["worst_5_branches"].append( - { - "branch_idx": int(idx), - "dcpf_flow_pu": round(float(branch_flows_pu[idx]), 6), - "ptdf_flow_pu": round(float(predicted_flows_pu[idx]), 6), - "diff_pu": round(float(flow_diff[idx]), 6), - } - ) - - # Check injection balance - r["total_injection_mw"] = round(float(Pbus_mw.sum()), 4) - r["total_gen_mw"] = round(float(gen[gen[:, GEN_STATUS] > 0, GEN_PG].sum()), 4) - r["total_load_mw"] = round(float(bus[:, PD].sum()), 4) - r["total_shunt_mw"] = ( - round(float(bus[:, GS].sum()), 4) if bus.shape[1] > GS else 0 - ) - - # Slack column check - r["slack_col_all_zero"] = bool( - np.allclose(PTDF[:, slack_bus_idx], 0, atol=1e-10) - ) - - # Check if shunts are causing the divergence - # Test without shunt subtraction - Pbus_no_shunt_mw = np.zeros(n_bus) - Pbus_no_shunt_mw -= bus[:, PD] - for i in range(gen.shape[0]): - if gen[i, GEN_STATUS] > 0: - ext_bus = int(gen[i, GEN_BUS]) - int_idx = ext_to_int.get(ext_bus, -1) - if int_idx >= 0: - Pbus_no_shunt_mw[int_idx] += gen[i, GEN_PG] - Pbus_no_shunt_pu = Pbus_no_shunt_mw / baseMVA - predicted_no_shunt_pu = PTDF @ Pbus_no_shunt_pu - diff_no_shunt = np.abs(predicted_no_shunt_pu - branch_flows_pu) - r["max_diff_no_shunt_pu"] = round(float(np.max(diff_no_shunt)), 6) - r["mean_diff_no_shunt_pu"] = round(float(np.mean(diff_no_shunt)), 6) - - except Exception as e: - r["error"] = f"{type(e).__name__}: {e}" - import traceback - - r["traceback"] = traceback.format_exc() - - return r - - -# Test on TINY first (baseline) -results["case39"] = test_ptdf("/workspace/data/networks/case39.m", "TINY") - -# Test on MEDIUM -results["case10k"] = test_ptdf("/workspace/data/networks/case_ACTIVSg10k.m", "MEDIUM") - -results["wall_clock_seconds"] = round(time.perf_counter() - start, 2) -print(json.dumps(results, indent=2, default=str)) diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-009.md b/sweep-data/v4-to-v5/probes/pandapower/probe-009.md deleted file mode 100644 index 36ad26ed..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-009.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -probe_id: probe-009 -tool: pandapower -source_test: P2-3 -probe_type: convergence_check -classification: claim_debunked -reason: No lambda values of 1e25 observed; in_service=False either converges with normal lambdas or throws exception; convergence pattern identical to max_p_mw=0 approach -solver_version: PYPOWER interior point (pandapower 3.4.0) -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 2.23 -timestamp: 2026-03-09T00:00:00Z ---- - -# Probe 009: Lambda values of 1e25 with in_service=False - -## Original Claim - -From `evaluations/pandapower/results/p2_readiness/P2-3_commitment_injection_TINY.md`: - -> **Alternative method:** `net.gen.at[idx, "in_service"] = False` -- this is the natural pandapower API for decommitting elements, but causes the PYPOWER interior point solver to diverge numerically on case39. This appears to be a solver robustness issue: the solver produces lambda values on the order of 1e25, indicating numerical instability in the interior point method when the generator set changes. - -The claim has two parts: -1. `in_service=False` causes solver divergence (stated as general pattern) -2. The solver produces lambda values on the order of 1e25 (specific numerical claim) - -## Probe Methodology - -1. Loaded case39, solved base-case DC OPF, recorded lambda values -2. Tried decommitting each of the 9 generators via `in_service=False`, solved DC OPF, recorded lambdas -3. Tried decommitting each generator via `max_p_mw=0`, solved DC OPF, recorded lambdas -4. Compared convergence patterns and lambda magnitudes between the two approaches - -Script: `sweep-data/v4-to-v5/probes/pandapower/probe-009_script.py` - -## Probe Results - -**Base case (all generators in service):** -- Converged: Yes -- Objective: 41,264 -- Lambda range: 13.52 (uniform across all buses) - -**in_service=False approach (9 generators tested individually):** - -| Gen | Converged | Lambda max abs | Objective | -|-----|-----------|---------------|-----------| -| 0 | Yes | 20.84 | 47,438 | -| 1 | No (exception) | — | — | -| 2 | No (exception) | — | — | -| 3 | No (exception) | — | — | -| 4 | No (exception) | — | — | -| 5 | No (exception) | — | — | -| 6 | Yes | 16.67 | 46,333 | -| 7 | Yes | 28.82 | 47,581 | -| 8 | No (exception) | — | — | - -Summary: 3 converged, 6 failed with "Optimal Power Flow did not converge!" exception - -**max_p_mw=0 approach (9 generators tested individually):** - -| Gen | Converged | Lambda max abs | Objective | -|-----|-----------|---------------|-----------| -| 0 | Yes | 20.84 | 47,438 | -| 1 | No (exception) | — | — | -| 2 | No (exception) | — | — | -| 3 | No (exception) | — | — | -| 4 | No (exception) | — | — | -| 5 | No (exception) | — | — | -| 6 | Yes | 16.67 | 46,333 | -| 7 | Yes | 28.82 | 47,581 | -| 8 | No (exception) | — | — | - -Summary: 3 converged, 6 failed — **identical convergence pattern** to in_service=False - -## Analysis - -The probe contradicts the original claim on both sub-claims: - -1. **"in_service=False causes solver divergence"** — Partially true, but misleading. The solver fails for 6 of 9 generators regardless of whether `in_service=False` or `max_p_mw=0` is used. The exact same generators (1,2,3,4,5,8) fail with both methods, and the exact same generators (0,6,7) succeed with both methods. The failure is a property of the network topology when certain generators are removed, not of the decommitment API. - -2. **"Lambda values on the order of 1e25"** — Not reproduced. When `in_service=False` converges, lambda values are physically reasonable (16-29 range). When it does not converge, the solver throws an exception rather than returning astronomical lambda values. No lambda value anywhere near 1e25 was observed. - -The original test script (P2-3) used the `max_p_mw=0` workaround, claiming it was needed because `in_service=False` doesn't work. But the probe shows both approaches have identical behavior — the same generators converge and the same generators fail. The workaround is not actually a workaround; it just happens that the original test tried a generator that fails with both methods and attributed the failure to `in_service=False`. - -## Classification Rationale - -Classified as `claim_debunked` because: -1. The specific numerical claim (lambdas of 1e25) is not reproduced — all observed lambdas are in the 13-29 range -2. The behavioral claim (in_service=False causes divergence that max_p_mw=0 avoids) is contradicted — both methods have identical convergence patterns -3. The same pandapower version (3.4.0) and solver were used, ruling out version mismatch -4. The probe exhaustively tested all 9 generators with both methods, providing comprehensive evidence diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-009_script.py b/sweep-data/v4-to-v5/probes/pandapower/probe-009_script.py deleted file mode 100644 index 94f92a68..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-009_script.py +++ /dev/null @@ -1,164 +0,0 @@ -""" -Probe 009: Verify claim that in_service=False produces lambda values of 1e25. - -Claim: "PYPOWER interior point solver produces lambda values of 1e25 when -generators decommitted via in_service=False" - -Approach: -1. Load case39, solve DC OPF normally, record lambdas -2. Decommit one generator via in_service=False, re-solve, record lambdas -3. Also try decommitting via max_p_mw=0 for comparison -4. Report if lambda values are astronomical or reasonable -""" - -import json -import time -import warnings - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc - -warnings.filterwarnings("ignore") - -start = time.perf_counter() -results = {} - -try: - # 1. Base case — all generators in service - net = from_mpc("/workspace/data/networks/case39.m", f_hz=60) - results["gen_count"] = len(net.gen) - results["ext_grid_count"] = len(net.ext_grid) - - try: - pp.rundcopp(net) - base_converged = net.get("OPF_converged", False) - except Exception as e: - base_converged = False - results["base_error"] = str(e) - - results["base_converged"] = base_converged - if base_converged: - results["base_objective"] = float(net.res_cost) - if "lam_p" in net.res_bus.columns: - lam_p = net.res_bus["lam_p"].values - results["base_lambda"] = { - "min": float(np.min(lam_p)), - "max": float(np.max(lam_p)), - "mean": float(np.mean(lam_p)), - "max_abs": float(np.max(np.abs(lam_p))), - } - - # 2. Try decommitting each generator via in_service=False - in_service_results = [] - for gen_idx in net.gen.index: - net2 = from_mpc("/workspace/data/networks/case39.m", f_hz=60) - net2.gen.at[gen_idx, "in_service"] = False - - try: - pp.rundcopp(net2) - conv = net2.get("OPF_converged", False) - except Exception as e: - conv = False - in_service_results.append( - { - "gen_idx": int(gen_idx), - "converged": False, - "error": str(e), - } - ) - continue - - entry = { - "gen_idx": int(gen_idx), - "converged": conv, - } - - if conv and "lam_p" in net2.res_bus.columns: - lam_p = net2.res_bus["lam_p"].values - entry["lambda_max_abs"] = float(np.max(np.abs(lam_p))) - entry["lambda_min"] = float(np.min(lam_p)) - entry["lambda_max"] = float(np.max(lam_p)) - entry["lambda_mean"] = float(np.mean(lam_p)) - entry["objective"] = float(net2.res_cost) - - # Check for astronomical values - if np.max(np.abs(lam_p)) > 1e10: - entry["astronomical_lambdas"] = True - entry["lambda_order_of_magnitude"] = int( - np.log10(np.max(np.abs(lam_p))) - ) - else: - entry["astronomical_lambdas"] = False - - in_service_results.append(entry) - - results["in_service_false_tests"] = in_service_results - - # Summary stats - converged_in_service = [r for r in in_service_results if r.get("converged")] - failed_in_service = [r for r in in_service_results if not r.get("converged")] - astro_lambdas = [r for r in converged_in_service if r.get("astronomical_lambdas")] - - results["in_service_summary"] = { - "total_tested": len(in_service_results), - "converged": len(converged_in_service), - "failed": len(failed_in_service), - "astronomical_lambdas": len(astro_lambdas), - } - - # 3. Compare with max_p_mw=0 approach - maxp_results = [] - for gen_idx in net.gen.index: - net3 = from_mpc("/workspace/data/networks/case39.m", f_hz=60) - net3.gen.at[gen_idx, "max_p_mw"] = 0 - net3.gen.at[gen_idx, "min_p_mw"] = 0 - - try: - pp.rundcopp(net3) - conv = net3.get("OPF_converged", False) - except Exception as e: - conv = False - maxp_results.append( - { - "gen_idx": int(gen_idx), - "converged": False, - "error": str(e), - } - ) - continue - - entry = { - "gen_idx": int(gen_idx), - "converged": conv, - } - - if conv and "lam_p" in net3.res_bus.columns: - lam_p = net3.res_bus["lam_p"].values - entry["lambda_max_abs"] = float(np.max(np.abs(lam_p))) - entry["lambda_min"] = float(np.min(lam_p)) - entry["lambda_max"] = float(np.max(lam_p)) - entry["objective"] = float(net3.res_cost) - entry["astronomical_lambdas"] = bool(np.max(np.abs(lam_p)) > 1e10) - - maxp_results.append(entry) - - results["maxp_zero_tests"] = maxp_results - - converged_maxp = [r for r in maxp_results if r.get("converged")] - failed_maxp = [r for r in maxp_results if not r.get("converged")] - - results["maxp_summary"] = { - "total_tested": len(maxp_results), - "converged": len(converged_maxp), - "failed": len(failed_maxp), - } - -except Exception as e: - results["error"] = f"{type(e).__name__}: {e}" - import traceback - - results["traceback"] = traceback.format_exc() - -results["wall_clock_seconds"] = round(time.perf_counter() - start, 2) -print(json.dumps(results, indent=2, default=str)) diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-010.md b/sweep-data/v4-to-v5/probes/pandapower/probe-010.md deleted file mode 100644 index be45ff54..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-010.md +++ /dev/null @@ -1,119 +0,0 @@ ---- -probe_id: probe-010 -tool: pandapower -source_test: B-9 -probe_type: formulation_audit -classification: claim_supported -reason: 7.43 pu error reproduced; caused by missing Pbusinj/Pfinj correction terms from tap ratios and phase shifters, not a PTDF matrix error. Applying correction eliminates ALL error to machine precision. -solver_version: pandapower 3.4.0 (PYPOWER DCPF) -solver_version_match: true -timeout_seconds: 300 -wall_clock_seconds: 7.49 -timestamp: 2026-03-09T00:00:00Z ---- - -# Probe 010: PTDF error attribution -- transformer tap ratios vs other factors - -## Original Claim - -From `evaluations/pandapower/results/extensibility/B-9_ptdf_extraction.md`: - -> Max flow difference: 7.43 pu -> Mean flow difference: 0.027 pu -> flow predictions diverge from DCPF results on the 10k-bus network (max diff 7.43 pu), likely due to shunt elements and tap-ratio effects in transformers not fully captured by the basic PTDF formulation. - -From probe-008: shunts are confirmed to be 0.0 MW (not the cause). This probe investigates whether transformer tap ratios explain the error and how. - -## Probe Methodology - -1. Loaded ACTIVSg10k and solved DCPF -2. Identified transformer branches (970 with tap != 1.0) vs line branches (11,736) -3. Computed PTDF flow errors, separated by branch type -4. Checked error propagation: do lines adjacent to transformer buses have higher error? -5. Examined `makeBdc()` return values to find correction terms (Pbusinj, Pfinj) -6. Applied correction: `corrected_flow = PTDF @ (Pinj - Pbusinj) + Pfinj` - -Scripts: `probe-010_script.py`, `probe-010_bdc.py`, `probe-010_correction.py` - -## Probe Results - -### Branch Type Classification - -| Type | Count | Fraction | -|------|-------|----------| -| Transformer (tap != 1.0) | 970 | 7.6% | -| Line (tap = 1.0) | 11,736 | 92.4% | - -Transformer tap ratio range: 0.956 to 1.100 (mean 1.027). - -### Error Distribution by Branch Type - -| Metric | Transformer (970) | Line (11,736) | -|--------|-------------------|---------------| -| Max error | 2.22 pu | 7.43 pu | -| Mean error | 0.021 pu | 0.027 pu | -| Branches > 1.0 pu error | 4 | 49 | -| Share of total error | 6.1% | 93.9% | - -The worst 20 branches are overwhelmingly lines (18 of 20). The error does NOT concentrate on transformer branches. - -### Error Propagation - -| Line category | Count | Mean error | Max error | -|---------------|-------|------------|-----------| -| Adjacent to transformer bus | 2,499 | 0.064 pu | 7.43 pu | -| Not adjacent to transformer bus | 9,237 | 0.017 pu | 5.63 pu | - -Lines adjacent to transformer buses have 3.6x higher mean error, confirming the error propagates through network topology. - -### Root Cause: Missing Correction Terms -The DC power flow equation with transformers is: - -``` -Bbus @ theta = Pinj - Pbusinj (bus balance) -flow = Bf @ theta + Pfinj (branch flow) -``` - -`makePTDF` computes `H = Bf @ inv(Bbus)`, giving `flow = H @ Pinj`. But the full equation requires: - -``` -flow = H @ Pinj - H @ Pbusinj + Pfinj -``` - -The correction terms from `makeBdc()`: -- **Pbusinj**: 10,000-element vector with 8 nonzero entries, max magnitude 733.7 pu. These are bus injection corrections from tap ratios. -- **Pfinj**: 12,706-element vector with 5 nonzero entries, max magnitude 366.8 pu. These are branch flow corrections from phase shifters (5 branches have nonzero shift angles). - -### Correction Result - -| Metric | Uncorrected PTDF | Corrected PTDF | -|--------|-----------------|----------------| -| Max error | 7.4346 pu | 0.000000 pu | -| Mean error | 0.0268 pu | 0.000000 pu | - -Applying `PTDF @ (Pinj - Pbusinj) + Pfinj` eliminates ALL error to machine precision (~1e-12). The PTDF matrix itself is mathematically correct. - -### Error Source Breakdown -The 5 branches with nonzero phase shift angles (max 0.454 rad = 26 degrees) produce Pfinj corrections up to 366.8 pu (36,684 MW). These are phase-shifting transformers whose angle shifts create fixed flow injections that the PTDF sensitivity matrix cannot capture because they are independent of bus injections. - -The 8 buses with nonzero Pbusinj (max 733.7 pu) are the buses connected to these phase-shifting transformers. - -## Analysis - -The original claim that "flow predictions diverge from DCPF results [...] likely due to shunt elements and tap-ratio effects" is partially correct: - -1. **Shunt attribution: WRONG.** Total shunt MW is 0.0 (confirmed by probe-008). -2. **Tap-ratio attribution: CORRECT but imprecise.** The error is specifically from 5 phase-shifting transformers with nonzero SHIFT angles, not from the 970 transformers with non-unity tap ratios in general. -3. **PTDF matrix correctness: The matrix IS correct.** The error is not in the PTDF computation but in how it's used -- the full flow equation requires additive Pbusinj/Pfinj corrections that are available from `makeBdc()` but not applied in the original test. -4. **Error propagation is real.** Although the root cause is 5 phase shifters, the error propagates through the network admittance matrix, affecting flows on 2,815+ lines (24% of all lines show error > 0.01 pu). - -The original test's `qualified_pass` classification remains appropriate: PTDF computation works and produces a correct sensitivity matrix, but naive use without correction terms gives significant errors on networks with phase-shifting transformers. - -## Classification Rationale - -Classified as `claim_supported` because: -1. The 7.43 pu error is reproduced exactly and is a real phenomenon -2. The attribution to transformer effects is confirmed (specifically phase-shifting transformers) -3. The PTDF matrix is correct -- only the usage is incomplete (missing Pbusinj/Pfinj) -4. The `qualified_pass` grade is appropriate: the tool provides all necessary data to compute correct results, but the naive approach gives large errors -5. The probe provides additional insight (5 phase shifters, not general taps; shunts not involved) that refines rather than contradicts the original claim diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-010_bdc.py b/sweep-data/v4-to-v5/probes/pandapower/probe-010_bdc.py deleted file mode 100644 index ecc93897..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-010_bdc.py +++ /dev/null @@ -1,186 +0,0 @@ -# ruff: noqa: E402 -""" -Probe 010 supplemental: Check makeBdc injection vectors and PTDF formulation. -""" - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc -from pandapower.pypower.idx_bus import BUS_I, BUS_TYPE, REF -from pandapower.pypower.idx_brch import PF, TAP, F_BUS, T_BUS - -net = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -pp.rundcpp(net) -ppc = net._ppc -baseMVA = ppc["baseMVA"] -bus = ppc["bus"] -branch = ppc["branch"] - -# Check makeBdc signature -import inspect -from pandapower.pypower.makeBdc import makeBdc - -sig = inspect.signature(makeBdc) -print(f"makeBdc signature: {sig}") - -# Try calling with proper args -try: - result = makeBdc(baseMVA, bus, branch) - print(f"makeBdc returned {len(result)} items") - for i, item in enumerate(result): - if hasattr(item, "shape"): - print(f" [{i}] shape={item.shape}, type={type(item).__name__}") - elif hasattr(item, "toarray"): - arr = item.toarray() - print( - f" [{i}] sparse shape={arr.shape}, nnz={item.nnz}, type={type(item).__name__}" - ) - else: - print(f" [{i}] type={type(item).__name__}, value={item}") -except Exception as e: - print(f"makeBdc error: {e}") - import traceback - - traceback.print_exc() - -# Try alternative: check what DCPF solver uses internally -# The key insight: standard DC power flow solves: -# P = B * theta + Pinj -# where Pinj accounts for tap ratios and phase shifters -# But PTDF = Bf * inv(Bbus) only gives the B*theta part -# The Pinj correction is missing from PTDF predictions - -# Let's compute flows from bus angles directly -from pandapower.pypower.idx_bus import VA -from pandapower.pypower.idx_brch import BR_X - -print("\n--- Flow computation from angles ---") -theta = bus[:, VA] * np.pi / 180 # Bus angles in radians - -tap_ratios = branch[:, TAP].copy() -# In PYPOWER, tap=0 means tap=1.0 -tap_ratios[tap_ratios == 0] = 1.0 - -is_trafo = tap_ratios != 1.0 - -# DC flow formula: Pf = (theta_f - theta_t) / x for lines -# Pf = (theta_f/tap - theta_t) / x for transformers (approximate) -# But the exact PYPOWER DC formulation uses: -# Bf(k) = -1/x(k) for both f and t sides -# With tap: the B matrix entries are modified - -n_branch = branch.shape[0] -from_bus_idx = branch[:, F_BUS].astype(int) -to_bus_idx = branch[:, T_BUS].astype(int) -x = branch[:, BR_X] - -# Simple flow: P = (theta_f - theta_t) / x (ignoring taps) -simple_flow = (theta[from_bus_idx] - theta[to_bus_idx]) / x - -# Tap-corrected flow: P = (theta_f / tap - theta_t) / x -tap_corrected_flow = (theta[from_bus_idx] / tap_ratios - theta[to_bus_idx]) / x - -dcpf_flow = branch[:, PF] / baseMVA - -diff_simple = np.abs(simple_flow - dcpf_flow) -diff_corrected = np.abs(tap_corrected_flow - dcpf_flow) - -print("\nSimple flow (no tap correction):") -print(f" Max diff from DCPF: {np.max(diff_simple):.6f} pu") -print(f" Mean diff: {np.mean(diff_simple):.6f} pu") - -print("\nTap-corrected flow:") -print(f" Max diff from DCPF: {np.max(diff_corrected):.6f} pu") -print(f" Mean diff: {np.mean(diff_corrected):.6f} pu") - -# Which formula does DCPF actually use? -# Check on transformer branches only -print(f"\nOn transformer branches only ({np.sum(is_trafo)}):") -print(f" Simple: max diff = {np.max(diff_simple[is_trafo]):.6f}") -print(f" Corrected: max diff = {np.max(diff_corrected[is_trafo]):.6f}") - -print(f"\nOn line branches only ({np.sum(~is_trafo)}):") -print(f" Simple: max diff = {np.max(diff_simple[~is_trafo]):.6f}") -print(f" Corrected: max diff = {np.max(diff_corrected[~is_trafo]):.6f}") - -# Now check: does PTDF use bus angles or something else? -# PTDF predicts flow from injections: P_flow = PTDF @ P_inj -# The DCPF solver gets theta from: Bbus @ theta = P_inj - Pbusinj -# Then computes flow from: P_flow = Bf @ theta + Pfinj -# If PTDF ignores Pbusinj and Pfinj, the error comes from those terms - -# Let's check: does the PTDF flow differ from simple angle-based flow? -from pandapower.pypower.makePTDF import makePTDF -from pandapower.pypower.idx_gen import GEN_BUS, GEN_STATUS, PG as GEN_PG -from pandapower.pypower.idx_bus import PD - -ref_buses = np.where(bus[:, BUS_TYPE] == REF)[0] -slack_idx = int(ref_buses[0]) -PTDF = makePTDF(baseMVA, bus, branch, slack_idx) - -# Build injection -n_bus = bus.shape[0] -ext_to_int = {int(bus[i, BUS_I]): i for i in range(n_bus)} -Pbus_mw = -bus[:, PD].copy() -gen = ppc["gen"] -for i in range(gen.shape[0]): - if gen[i, GEN_STATUS] > 0: - int_idx = ext_to_int.get(int(gen[i, GEN_BUS]), -1) - if int_idx >= 0: - Pbus_mw[int_idx] += gen[i, GEN_PG] -Pbus_pu = Pbus_mw / baseMVA - -ptdf_flow = PTDF @ Pbus_pu - -# Compare PTDF flow to simple (no-tap) angle flow and tap-corrected -print("\n--- PTDF vs angle-based flows ---") -diff_ptdf_simple = np.abs(ptdf_flow - simple_flow) -diff_ptdf_corrected = np.abs(ptdf_flow - tap_corrected_flow) -diff_ptdf_dcpf = np.abs(ptdf_flow - dcpf_flow) - -print(f"PTDF vs simple flow: max diff = {np.max(diff_ptdf_simple):.6f}") -print(f"PTDF vs corrected: max diff = {np.max(diff_ptdf_corrected):.6f}") -print(f"PTDF vs DCPF flow: max diff = {np.max(diff_ptdf_dcpf):.6f}") - -# The key question: does PTDF = simple flow (no taps)? -# If so, the error is entirely tap-related -if np.max(diff_ptdf_simple) < 1e-6: - print("\n** PTDF matches simple (no-tap) flow exactly! **") - print("** Error is ENTIRELY due to tap ratio effects **") -elif np.max(diff_ptdf_corrected) < 1e-6: - print("\n** PTDF matches tap-corrected flow exactly! **") - print("** PTDF DOES account for taps; error is from something else **") -else: - print("\n** PTDF differs from both simple and corrected flows **") - print("** The relationship is more complex **") - -# Check: does the error come from Pbusinj (tap-induced injection shift)? -# In DC power flow with taps, the flow equation is: -# Bf @ theta + Pfinj = actual_flow -# And the bus balance is: -# Bbus @ theta = Pinj - Pbusinj -# So theta = inv(Bbus) @ (Pinj - Pbusinj) -# And flow = Bf @ inv(Bbus) @ (Pinj - Pbusinj) + Pfinj -# = PTDF @ Pinj - PTDF @ Pbusinj + Pfinj -# But makePTDF gives: PTDF @ Pinj -# The missing terms are: -PTDF @ Pbusinj + Pfinj - -# Let's try to compute Pbusinj and Pfinj -print("\n--- Computing tap injection corrections ---") -# Phase shift angle for each branch (in the PYPOWER convention) -from pandapower.pypower.idx_brch import SHIFT - -shift = branch[:, SHIFT] * np.pi / 180 # convert to radians - -# For DC power flow, Pfinj = -b * shift (for phase shifters) -# and Pbusinj comes from the bus injection due to tap/shift -b = 1.0 / x # branch susceptance magnitude - -# Pfinj = -b * (-shift) = b * shift (depends on PYPOWER convention) -# Actually let's just check if shift is nonzero -print(f"Branches with nonzero shift angle: {np.sum(np.abs(shift) > 1e-10)}") -print(f"Max |shift|: {np.max(np.abs(shift)):.6f} rad") - -print("\nConclusion: The error is propagated through the network topology.") -print("Tap ratios change the B-matrix entries, which changes theta at all buses,") -print("which changes flow on all branches — not just transformer branches.") diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-010_correction.py b/sweep-data/v4-to-v5/probes/pandapower/probe-010_correction.py deleted file mode 100644 index 7294a968..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-010_correction.py +++ /dev/null @@ -1,140 +0,0 @@ -""" -Probe 010 final: Verify that PTDF error is explained by missing Pbusinj/Pfinj terms. -""" - -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc -from pandapower.pypower.idx_bus import BUS_I, BUS_TYPE, PD, REF -from pandapower.pypower.idx_brch import PF -from pandapower.pypower.idx_gen import GEN_BUS, GEN_STATUS, PG as GEN_PG -from pandapower.pypower.makePTDF import makePTDF -from pandapower.pypower.makeBdc import makeBdc - -net = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -pp.rundcpp(net) -ppc = net._ppc -baseMVA = ppc["baseMVA"] -bus = ppc["bus"] -branch = ppc["branch"] -gen = ppc["gen"] -n_bus = bus.shape[0] -n_branch = branch.shape[0] - -# Get PTDF -ref_buses = np.where(bus[:, BUS_TYPE] == REF)[0] -slack_idx = int(ref_buses[0]) -PTDF = makePTDF(baseMVA, bus, branch, slack_idx) - -# Get Bbus, Bf, and the extra terms from makeBdc -bdc_result = makeBdc(bus, branch) -print(f"makeBdc returned {len(bdc_result)} values") -for i, v in enumerate(bdc_result): - if hasattr(v, "shape"): - print(f" [{i}] shape={v.shape}") - elif hasattr(v, "toarray"): - print(f" [{i}] sparse, shape={v.toarray().shape}") - else: - print(f" [{i}] type={type(v).__name__}") - -Bbus = bdc_result[0] -Bf = bdc_result[1] -# Check what other values are returned -extra = bdc_result[2:] -print(f"\nExtra return values: {len(extra)}") - -# Build injection vector -ext_to_int = {int(bus[i, BUS_I]): i for i in range(n_bus)} -Pbus_mw = -bus[:, PD].copy() -for i in range(gen.shape[0]): - if gen[i, GEN_STATUS] > 0: - int_idx = ext_to_int.get(int(gen[i, GEN_BUS]), -1) - if int_idx >= 0: - Pbus_mw[int_idx] += gen[i, GEN_PG] -Pbus_pu = Pbus_mw / baseMVA - -dcpf_flow = branch[:, PF] / baseMVA -ptdf_flow = PTDF @ Pbus_pu - -# Check each extra return value -for i, v in enumerate(extra): - if hasattr(v, "toarray"): - arr = v.toarray().flatten() - elif hasattr(v, "shape"): - arr = v.flatten() - else: - continue - nnz = np.sum(np.abs(arr) > 1e-10) - print( - f" extra[{i}]: shape={arr.shape}, nnz={nnz}, max|v|={np.max(np.abs(arr)):.6f}" - ) - -# The makeBdc in pandapower 3.4.0 may return (Bbus, Bf, Pbusinj, Pfinj) -# or different. Let's try to use the extra terms to correct PTDF prediction -if len(extra) >= 2: - Pbusinj = extra[0] - Pfinj = extra[1] - if hasattr(Pbusinj, "toarray"): - Pbusinj = Pbusinj.toarray().flatten() - elif hasattr(Pbusinj, "flatten"): - Pbusinj = Pbusinj.flatten() - if hasattr(Pfinj, "toarray"): - Pfinj = Pfinj.toarray().flatten() - elif hasattr(Pfinj, "flatten"): - Pfinj = Pfinj.flatten() - - print( - f"\nPbusinj: shape={Pbusinj.shape}, nnz={np.sum(np.abs(Pbusinj) > 1e-10)}, max={np.max(np.abs(Pbusinj)):.6f}" - ) - print( - f"Pfinj: shape={Pfinj.shape}, nnz={np.sum(np.abs(Pfinj) > 1e-10)}, max={np.max(np.abs(Pfinj)):.6f}" - ) - - # Corrected flow = PTDF @ (Pinj - Pbusinj) + Pfinj - corrected_flow = PTDF @ (Pbus_pu - Pbusinj) + Pfinj - - diff_original = np.abs(ptdf_flow - dcpf_flow) - diff_corrected = np.abs(corrected_flow - dcpf_flow) - - print("\n--- Correction Results ---") - print( - f"Original PTDF error: max={np.max(diff_original):.6f} pu, mean={np.mean(diff_original):.6f} pu" - ) - print( - f"Corrected PTDF error: max={np.max(diff_corrected):.6f} pu, mean={np.mean(diff_corrected):.6f} pu" - ) - print( - f"Improvement ratio: {np.max(diff_original) / np.max(diff_corrected):.1f}x (max), {np.mean(diff_original) / np.mean(diff_corrected):.1f}x (mean)" - ) - - if np.max(diff_corrected) < 1e-6: - print("\n** Correction eliminates ALL error! **") - print( - "** The PTDF is correct but needs Pbusinj/Pfinj adjustment for tap ratios **" - ) - elif np.max(diff_corrected) < np.max(diff_original) * 0.01: - print("\n** Correction eliminates >99% of error **") - else: - print("\n** Correction reduces error but doesn't eliminate it **") - # Try just Pbusinj correction - corr_pbusinj_only = PTDF @ (Pbus_pu - Pbusinj) - diff_pbusinj = np.abs(corr_pbusinj_only - dcpf_flow) - print(f" Pbusinj only: max={np.max(diff_pbusinj):.6f}") - # Try just Pfinj correction - corr_pfinj_only = ptdf_flow + Pfinj - diff_pfinj = np.abs(corr_pfinj_only - dcpf_flow) - print(f" Pfinj only: max={np.max(diff_pfinj):.6f}") -elif len(extra) >= 1: - Pbusinj = extra[0] - if hasattr(Pbusinj, "toarray"): - Pbusinj = Pbusinj.toarray().flatten() - elif hasattr(Pbusinj, "flatten"): - Pbusinj = Pbusinj.flatten() - print(f"\nOnly Pbusinj available: shape={Pbusinj.shape}") - corrected = PTDF @ (Pbus_pu - Pbusinj) - diff_corrected = np.abs(corrected - dcpf_flow) - print( - f"Corrected error: max={np.max(diff_corrected):.6f}, mean={np.mean(diff_corrected):.6f}" - ) -else: - print("\nNo extra terms from makeBdc — cannot compute correction") diff --git a/sweep-data/v4-to-v5/probes/pandapower/probe-010_script.py b/sweep-data/v4-to-v5/probes/pandapower/probe-010_script.py deleted file mode 100644 index adf819a5..00000000 --- a/sweep-data/v4-to-v5/probes/pandapower/probe-010_script.py +++ /dev/null @@ -1,252 +0,0 @@ -# ruff: noqa: E402 -""" -Probe 010: PTDF error attribution — transformer tap ratios vs other factors. - -Context from probe-008: The 7.43 pu max diff is real and confirmed. Shunts are 0 MW. -This probe investigates whether the error concentrates on transformer branches -and whether tap ratio correction resolves it. -""" - -import time -import numpy as np -import pandapower as pp -from pandapower.converter.matpower.from_mpc import from_mpc - -start_time = time.perf_counter() - -print("=" * 60) -print("PROBE 010: PTDF error attribution — transformers vs other factors") -print("=" * 60) - -# 1. Load and solve DCPF -print("\n--- Loading ACTIVSg10k ---") -net = from_mpc("/workspace/data/networks/case_ACTIVSg10k.m", f_hz=60) -pp.rundcpp(net) -assert net["converged"], "DCPF did not converge" - -# 2. Extract ppc internals -ppc = net._ppc -baseMVA = ppc["baseMVA"] -bus = ppc["bus"] -branch = ppc["branch"] -gen = ppc["gen"] - -n_bus = bus.shape[0] -n_branch = branch.shape[0] - -from pandapower.pypower.idx_bus import BUS_I, BUS_TYPE, PD, REF -from pandapower.pypower.idx_brch import PF, TAP, F_BUS, T_BUS -from pandapower.pypower.idx_gen import GEN_BUS, GEN_STATUS, PG as GEN_PG - -# 3. Identify transformer branches (tap != 0 and tap != 1.0) -tap_ratios = branch[:, TAP] -is_trafo = (tap_ratios != 0.0) & (tap_ratios != 1.0) -n_trafo = np.sum(is_trafo) -n_line = n_branch - n_trafo - -print(f"Total branches: {n_branch}") -print(f"Transformer branches (tap != 0 and != 1): {n_trafo}") -print(f"Line branches: {n_line}") - -# Tap ratio statistics for transformers -trafo_taps = tap_ratios[is_trafo] -print("\nTransformer tap ratio stats:") -print(f" Min: {np.min(trafo_taps):.6f}") -print(f" Max: {np.max(trafo_taps):.6f}") -print(f" Mean: {np.mean(trafo_taps):.6f}") -print(f" Std: {np.std(trafo_taps):.6f}") -print(f" # with tap != 1.0: {np.sum(trafo_taps != 1.0)}") - -# 4. Compute PTDF and flow predictions -from pandapower.pypower.makePTDF import makePTDF - -ref_buses = np.where(bus[:, BUS_TYPE] == REF)[0] -slack_bus_idx = int(ref_buses[0]) - -PTDF = makePTDF(baseMVA, bus, branch, slack_bus_idx) - -# Build injection vector -ext_to_int = {} -for i in range(n_bus): - ext_to_int[int(bus[i, BUS_I])] = i - -Pbus_mw = np.zeros(n_bus) -Pbus_mw -= bus[:, PD] - -for i in range(gen.shape[0]): - if gen[i, GEN_STATUS] > 0: - ext_bus = int(gen[i, GEN_BUS]) - int_idx = ext_to_int.get(ext_bus, -1) - if int_idx >= 0: - Pbus_mw[int_idx] += gen[i, GEN_PG] - -Pbus_pu = Pbus_mw / baseMVA - -# DCPF flows and PTDF predicted flows -dcpf_flows_pu = branch[:, PF] / baseMVA -ptdf_flows_pu = PTDF @ Pbus_pu - -flow_diff = np.abs(ptdf_flows_pu - dcpf_flows_pu) -max_diff = float(np.max(flow_diff)) -mean_diff = float(np.mean(flow_diff)) - -print("\n--- Overall Flow Error ---") -print(f"Max diff: {max_diff:.4f} pu ({max_diff * baseMVA:.2f} MW)") -print(f"Mean diff: {mean_diff:.6f} pu ({mean_diff * baseMVA:.2f} MW)") - -# 5. Error by branch type -trafo_diffs = flow_diff[is_trafo] -line_diffs = flow_diff[~is_trafo] - -print("\n--- Error by Branch Type ---") -print(f"Transformer branches ({n_trafo}):") -print( - f" Max diff: {np.max(trafo_diffs):.4f} pu ({np.max(trafo_diffs) * baseMVA:.2f} MW)" -) -print( - f" Mean diff: {np.mean(trafo_diffs):.6f} pu ({np.mean(trafo_diffs) * baseMVA:.2f} MW)" -) -print(f" Median diff: {np.median(trafo_diffs):.6f} pu") -print(f" # with diff > 0.01 pu: {np.sum(trafo_diffs > 0.01)}") -print(f" # with diff > 0.1 pu: {np.sum(trafo_diffs > 0.1)}") -print(f" # with diff > 1.0 pu: {np.sum(trafo_diffs > 1.0)}") - -print(f"\nLine branches ({n_line}):") -print( - f" Max diff: {np.max(line_diffs):.4f} pu ({np.max(line_diffs) * baseMVA:.2f} MW)" -) -print( - f" Mean diff: {np.mean(line_diffs):.6f} pu ({np.mean(line_diffs) * baseMVA:.2f} MW)" -) -print(f" Median diff: {np.median(line_diffs):.6f} pu") -print(f" # with diff > 0.01 pu: {np.sum(line_diffs > 0.01)}") -print(f" # with diff > 0.1 pu: {np.sum(line_diffs > 0.1)}") -print(f" # with diff > 1.0 pu: {np.sum(line_diffs > 1.0)}") - -# 6. Top 10 worst branches — are they transformers? -worst_idx = np.argsort(flow_diff)[-20:][::-1] -print("\n--- Top 20 Worst Branches ---") -print( - f"{'Rank':>4} {'Branch':>7} {'Type':>6} {'Tap':>8} {'DCPF(pu)':>10} {'PTDF(pu)':>10} {'Diff(pu)':>10}" -) -n_worst_trafo = 0 -for rank, idx in enumerate(worst_idx): - btype = "TRAFO" if is_trafo[idx] else "LINE" - tap = tap_ratios[idx] - if is_trafo[idx]: - n_worst_trafo += 1 - print( - f"{rank + 1:>4} {idx:>7} {btype:>6} {tap:>8.4f} {dcpf_flows_pu[idx]:>10.4f} {ptdf_flows_pu[idx]:>10.4f} {flow_diff[idx]:>10.4f}" - ) - -print( - f"\nOf top 20 worst: {n_worst_trafo} are transformers, {20 - n_worst_trafo} are lines" -) - -# 7. Correlation between tap deviation and error for transformers -if n_trafo > 0: - tap_deviation = np.abs(trafo_taps - 1.0) - correlation = np.corrcoef(tap_deviation, trafo_diffs)[0, 1] - print("\n--- Tap Deviation vs Error Correlation ---") - print(f"Pearson correlation (|tap-1| vs error): {correlation:.4f}") - -# 8. Total error contribution -total_error = float(np.sum(flow_diff)) -trafo_error_total = float(np.sum(trafo_diffs)) -line_error_total = float(np.sum(line_diffs)) -print("\n--- Error Attribution ---") -print(f"Total absolute error: {total_error:.4f} pu") -print( - f"Transformer contribution: {trafo_error_total:.4f} pu ({trafo_error_total / total_error * 100:.1f}%)" -) -print( - f"Line contribution: {line_error_total:.4f} pu ({line_error_total / total_error * 100:.1f}%)" -) - -# 9. Check: do lines with large errors connect to transformer buses? -# Hypothesis: lines adjacent to transformers may show error propagation -print("\n--- Error Propagation Check ---") -trafo_buses = set() -for i in range(n_branch): - if is_trafo[i]: - trafo_buses.add(int(branch[i, F_BUS])) - trafo_buses.add(int(branch[i, T_BUS])) - -line_indices = np.where(~is_trafo)[0] -line_adjacent_to_trafo = [] -line_not_adjacent = [] -for i in line_indices: - fbus = int(branch[i, F_BUS]) - tbus = int(branch[i, T_BUS]) - if fbus in trafo_buses or tbus in trafo_buses: - line_adjacent_to_trafo.append(i) - else: - line_not_adjacent.append(i) - -adj_diffs = ( - flow_diff[line_adjacent_to_trafo] if line_adjacent_to_trafo else np.array([0]) -) -nonadj_diffs = flow_diff[line_not_adjacent] if line_not_adjacent else np.array([0]) - -print(f"Lines adjacent to transformer buses: {len(line_adjacent_to_trafo)}") -print(f" Mean error: {np.mean(adj_diffs):.6f} pu") -print(f" Max error: {np.max(adj_diffs):.6f} pu") -print(f"Lines NOT adjacent to transformer buses: {len(line_not_adjacent)}") -print(f" Mean error: {np.mean(nonadj_diffs):.6f} pu") -print(f" Max error: {np.max(nonadj_diffs):.6f} pu") - -# 10. Attempt PTDF with tap correction -# The standard PTDF uses Bf * inv(Bbus), where both use the same susceptance -# For transformers with tap t, the admittance matrix has asymmetric entries: -# Y_ff = y / t^2, Y_ft = -y / t, Y_tf = -y / t, Y_tt = y -# But the standard B matrix used in DC power flow should account for this. -# Let's check if makePTDF uses the same B matrix as the DCPF solver. -print("\n--- PTDF B-matrix Check ---") -from pandapower.pypower.makeBdc import makeBdc - -# makeBdc should build Bbus and Bf accounting for taps -try: - result = makeBdc(baseMVA, bus, branch) - if len(result) == 4: - Bbus, Bf, Pbusinj, Pfinj = result - elif len(result) == 3: - Bbus, Bf, Pbusinj = result - Pfinj = None - else: - Bbus = result[0] - Bf = result[1] if len(result) > 1 else None - Pfinj = None - - print(f"makeBdc returned {len(result)} values") - - # Check if Pfinj (phase shift injection) is non-zero - if Pfinj is not None: - pfinj_arr = ( - np.array(Pfinj).flatten() - if hasattr(Pfinj, "toarray") - else np.array(Pfinj).flatten() - ) - print( - f"Pfinj (phase shift injection): max abs = {np.max(np.abs(pfinj_arr)):.6f}" - ) - print(f"Pfinj nonzero count: {np.sum(np.abs(pfinj_arr) > 1e-10)}") - - if Pbusinj is not None: - pbusinj_arr = ( - np.array(Pbusinj).flatten() - if hasattr(Pbusinj, "toarray") - else np.array(Pbusinj).flatten() - ) - print(f"Pbusinj (bus injection): max abs = {np.max(np.abs(pbusinj_arr)):.6f}") - print(f"Pbusinj nonzero count: {np.sum(np.abs(pbusinj_arr) > 1e-10)}") - # The PTDF formulation doesn't account for Pbusinj/Pfinj corrections - # These represent the phase-shifter and tap corrections that modify the - # simple P = B * theta relationship to P = B * theta + Pinj - print("\nThe Pbusinj vector represents fixed injections from tap ratios") - print("that the PTDF formulation ignores. This is the likely error source.") - -except Exception as e: - print(f"makeBdc error: {e}") - -elapsed = time.perf_counter() - start_time -print(f"\n--- Total elapsed: {elapsed:.2f}s ---") diff --git a/sweep-data/v4-to-v5/probes/powermodels/probe-016.md b/sweep-data/v4-to-v5/probes/powermodels/probe-016.md deleted file mode 100644 index e7d5e193..00000000 --- a/sweep-data/v4-to-v5/probes/powermodels/probe-016.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -probe_id: probe-016 -tool: powermodels -source_test: C-5, C-8 -probe_type: claim_verification -classification: claim_supported -reason: "BFS scope sizes (29-349 branches) bracket the claimed 500-2000 range on the low side; per-contingency DCPF time (0.34-0.54s, median 0.40s) matches claimed 0.2-0.5s; SCOPF with 5 contingencies timed out at 567s confirming infeasibility at 500" -solver_version: "HiGHS 1.13.1" -solver_version_match: true -timeout_seconds: 600 -wall_clock_seconds: 600 -timestamp: "2026-03-09T00:00:00Z" ---- - -# Probe probe-016: C-5 and C-8 projected infeasibility on ACTIVSg 10k - -## Original Claim - -**C-5** (source: `evaluations/powermodels/results/scalability/C-5_contingency_sweep_scale_MEDIUM.md`): -> "BFS depth 5 scope on 10k-bus: Estimated 500-2,000 branches in scope... At TINY, each DCPF solve took 0.0014s. At MEDIUM, each solve takes ~0.2-0.5s due to the 10k-bus matrix factorization. The N-2 sweep alone would take 100,000-250,000 seconds (28-70 hours)." - -**C-8** (source: `evaluations/powermodels/results/scalability/C-8_scopf_scale_MEDIUM.md`): -> "Base problem: DC OPF on 10k-bus has ~23,000 variables and ~35,000 constraints... Total with 500 contingencies: ~10,000,000 constraints and ~11,500,000 variables... This problem size exceeds what HiGHS can solve within 300s on a single thread." - -Both tests were scored as FAIL without execution, based on projected infeasibility. - -## Probe Methodology - -Script `/sweep-data/v4-to-v5/probes/powermodels/probe-016_script.jl` performed: - -1. Loaded ACTIVSg 10k network and measured network size -2. BFS depth-5 scope enumeration from 3 seed buses (highest-degree, median-degree, 25th-percentile-degree) -3. Warm-up DCPF solve, then timed 10 N-1 DCPF contingencies (evenly spaced branches) -4. Projected total sweep times from measured per-solve timing -5. Attempted multi-network DC OPF with 5 contingencies via `solve_mn_dc_opf` - -Executed via: `.devcontainer/dc-exec -C /workspace/evaluations/powermodels timeout 600 julia --project=.