Skip to content

feat(evaluation): add single-arm baseline comparisons - #1328

Open
frf12 wants to merge 2 commits into
masterfrom
codex/swe-single-arm-baselines
Open

feat(evaluation): add single-arm baseline comparisons#1328
frf12 wants to merge 2 commits into
masterfrom
codex/swe-single-arm-baselines

Conversation

@frf12

@frf12 frf12 commented Aug 23, 2026

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

Follow-up to #1247. No standalone issue.

Rationale for this change

The evaluation console previously required every SWE-bench Pro task to execute both OFF and ON, even when an existing historical result was the intended comparison point. That doubled execution cost and latency and provided no durable way to reuse trusted results across batches.

What changes are included in this PR?

  • Add off_on, on_only, and off_only execution modes across runner, worker, artifacts, reports, API contracts, and persisted batch requests.
  • Preserve the existing paired OFF/ON comparison when both Arms run, while omitting unavailable Arm data from single-Arm reports.
  • Add immutable single-Arm baseline snapshots with task outcomes, token metrics, exact attempt provenance, dataset/harness revisions, model settings, and PowerContext revision.
  • Add APIs for saving baselines, listing newest-first baseline records, checking compatibility, selecting multiple baselines, and computing historical comparisons without rerunning evaluation.
  • Add report-page controls for saving ON or OFF baselines, selecting historical comparisons, and viewing a richer baseline library.
  • Migrate the evaluation runtime schema from 2 to 3 and retain backward compatibility for existing paired report artifacts.
  • Accept standard JSON arrays in strict baseline-selection requests, covering the deployed 422 regression.
  • Update the evaluation README and focused backend/frontend tests.

Are there any user-facing changes?

Yes. A batch may now run OFF + ON, ON only, or OFF only. Completed aggregate reports can save any executed Arm as one of multiple immutable baselines. Later reports can compare an executed Arm with multiple compatible historical baselines without rerunning tasks. The baseline library is ordered by save time descending.

Existing paired evaluation and report behavior remains available. The SQLite runtime schema migrates automatically; this does not change the public PowerContext HTTP API outside the evaluation console.

How was this change tested?

  • uv run pytest -q tests --disable-warnings — 1033 passed
  • uv run ruff check src tests
  • uv run ty check src
  • npm test -- --run — 15 files, 96 tests passed
  • npm run build
  • git diff --check
  • Deployed to the m0 evaluation environment and smoke-tested schema consistency, all three treatment modes, baseline candidates, selection PUT, historical comparison, existing paired reports, and frontend assets.

AI usage statement

Implemented and tested with OpenAI Codex (GPT-5) under human-directed product review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license-eye has checked 805 files.

Valid Invalid Ignored Fixed
608 3 194 0
Click to see the invalid file list
  • evaluation/src/powercontext_eval/web/baselines.py
  • evaluation/web/src/components/BaselineLibrary.tsx
  • evaluation/web/src/components/BatchLauncher.test.tsx
Use this command to fix any missing license headers
```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

@@ -0,0 +1,221 @@
# Copyright (c) 2026 OceanBase.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Copyright (c) 2026 OceanBase.
# Copyright (c) 2026 OceanBase.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright (c) 2026 OceanBase.

@@ -0,0 +1,84 @@
/*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/*
/*
* Copyright (c) 2026 OceanBase.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/*

@@ -1,264 +1,175 @@
/*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/*
/*
* Copyright (c) 2026 OceanBase.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/*

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant