feat(evaluation): add single-arm baseline comparisons - #1328
Open
frf12 wants to merge 2 commits into
Open
Conversation
There was a problem hiding this comment.
license-eye has checked 805 files.
| Valid | Invalid | Ignored | Fixed |
|---|---|---|---|
| 608 | 3 | 194 | 0 |
Click to see the invalid file list
- evaluation/src/powercontext_eval/web/baselines.py
- evaluation/web/src/components/BaselineLibrary.tsx
- evaluation/web/src/components/BatchLauncher.test.tsx
Use this command to fix any missing license headers
```bash
docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix
</details>
| @@ -0,0 +1,221 @@ | |||
| # Copyright (c) 2026 OceanBase. | |||
There was a problem hiding this comment.
Suggested change
| # Copyright (c) 2026 OceanBase. | |
| # Copyright (c) 2026 OceanBase. | |
| # | |
| # Licensed under the Apache License, Version 2.0 (the "License"); | |
| # you may not use this file except in compliance with the License. | |
| # You may obtain a copy of the License at | |
| # | |
| # http://www.apache.org/licenses/LICENSE-2.0 | |
| # | |
| # Unless required by applicable law or agreed to in writing, software | |
| # distributed under the License is distributed on an "AS IS" BASIS, | |
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| # See the License for the specific language governing permissions and | |
| # limitations under the License. | |
| # Copyright (c) 2026 OceanBase. |
| @@ -0,0 +1,84 @@ | |||
| /* | |||
There was a problem hiding this comment.
Suggested change
| /* | |
| /* | |
| * Copyright (c) 2026 OceanBase. | |
| * | |
| * Licensed under the Apache License, Version 2.0 (the "License"); | |
| * you may not use this file except in compliance with the License. | |
| * You may obtain a copy of the License at | |
| * | |
| * http://www.apache.org/licenses/LICENSE-2.0 | |
| * | |
| * Unless required by applicable law or agreed to in writing, software | |
| * distributed under the License is distributed on an "AS IS" BASIS, | |
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| * See the License for the specific language governing permissions and | |
| * limitations under the License. | |
| */ | |
| /* |
| @@ -1,264 +1,175 @@ | |||
| /* | |||
There was a problem hiding this comment.
Suggested change
| /* | |
| /* | |
| * Copyright (c) 2026 OceanBase. | |
| * | |
| * Licensed under the Apache License, Version 2.0 (the "License"); | |
| * you may not use this file except in compliance with the License. | |
| * You may obtain a copy of the License at | |
| * | |
| * http://www.apache.org/licenses/LICENSE-2.0 | |
| * | |
| * Unless required by applicable law or agreed to in writing, software | |
| * distributed under the License is distributed on an "AS IS" BASIS, | |
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |
| * See the License for the specific language governing permissions and | |
| * limitations under the License. | |
| */ | |
| /* |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue or RFC does this PR close?
Follow-up to #1247. No standalone issue.
Rationale for this change
The evaluation console previously required every SWE-bench Pro task to execute both OFF and ON, even when an existing historical result was the intended comparison point. That doubled execution cost and latency and provided no durable way to reuse trusted results across batches.
What changes are included in this PR?
off_on,on_only, andoff_onlyexecution modes across runner, worker, artifacts, reports, API contracts, and persisted batch requests.Are there any user-facing changes?
Yes. A batch may now run OFF + ON, ON only, or OFF only. Completed aggregate reports can save any executed Arm as one of multiple immutable baselines. Later reports can compare an executed Arm with multiple compatible historical baselines without rerunning tasks. The baseline library is ordered by save time descending.
Existing paired evaluation and report behavior remains available. The SQLite runtime schema migrates automatically; this does not change the public PowerContext HTTP API outside the evaluation console.
How was this change tested?
uv run pytest -q tests --disable-warnings— 1033 passeduv run ruff check src testsuv run ty check srcnpm test -- --run— 15 files, 96 tests passednpm run buildgit diff --checkAI usage statement
Implemented and tested with OpenAI Codex (GPT-5) under human-directed product review.