Skip to content

fix(benchmark): handle non-object baseline as CLI validation error #144

Description

@97Three

问题摘要

reposteward benchmark run --baseline raises an uncaught TypeError when the baseline file contains valid JSON whose top-level value is not an object. The invalid baseline is not handled as a normal CLI input error.

最小复现

  1. Create a baseline file containing a JSON array:
tmp_dir="$(mktemp -d)"
printf '[]\n' > "$tmp_dir/baseline.json"

  1. Run:
uv run reposteward benchmark run \
  --scenario context.utf8_estimate \
  --baseline "$tmp_dir/baseline.json"
  1. Observe an unhandled traceback ending with:

TypeError: baseline benchmark report must be an object

预期行为

The CLI should report that the baseline must be a JSON object, return exit code 2, and not display a Python traceback.

实际行为

The exception is not handled, and the command displays a traceback ending with:
TypeError: baseline benchmark report must be an object

环境

RepoSteward commit: a6dc8c6
OS: Ubuntu 24.04.4 LTS on WSL2
Kernel: Linux 6.18.33.2-microsoft-standard-WSL2 x86_64
Python: 3.12.3
uv: 0.12.10
Harness: Codex

补充证据

load_benchmark_report() raises TypeError when the decoded JSON value is not a dictionary. The CLI's top-level error handler handles ValueError and several other expected errors, but not TypeError, so this exception escapes the normal CLI error path.

提交前确认

  • 我已搜索已有 Issue 和 PR。
  • 这不是需要私下报告的安全问题。
  • 我已经删除凭据、账号缓存和个人信息。

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions