Skip to content

Guard static demo report semantics in Pages artifact check #207

@luoyuctl

Description

@luoyuctl

Background

site/demo-report.html is a public evaluation artifact, while generated demo reports now expose newer report evidence such as incident timelines and conservative tool authority categories. The current Pages artifact check can pass even when the static demo report is semantically stale.

Evidence

  • Current master f354b39 generated demo HTML from /tmp/agenttrace-product-check --demo --overview -f html includes an Incident timeline section.
  • Current committed site/demo-report.html does not contain Incident, timeline, baseline, or authority wording when searched locally.
  • scripts/ci/check-report-semantics.sh validates generated demo JSON/Markdown/HTML semantics, including authority categories and baseline comparison fields.
  • scripts/ci/check-pages-artifact.sh site validates static Pages artifact presence, current version metadata or static-sample disclosure, local asset references, and required assets, but does not validate whether site/demo-report.html demonstrates current report semantics.
  • Duplicate search for demo-report stale site artifact semantics found no exact open issue.

User value

Users evaluating the public demo report should see a representative current report shape, not an older static artifact that misses the evidence model shipped in the binary.

Adoption rationale

The public sample report is part of onboarding and developer experience. A small CI guard reduces the chance that docs/site artifacts understate current report capabilities after report-format changes.

Suggested scope

  • Extend the Pages/static artifact validation so site/demo-report.html either contains current report evidence sections or is clearly marked as a static sample that intentionally does not cover them.
  • Keep the check narrow and deterministic. Candidate assertions: current-version static sample should include an incident timeline section and tool-authority wording, or a clear static-sample disclaimer.
  • Coordinate wording expectations with Surface report regression evidence in public docs #206, which owns public docs packaging for baseline comparison, incident timelines, and tool authority categories.

Non-goals

  • Do not regenerate or redesign the site from this issue unless needed for the guard to pass.
  • Do not add release, npm publication, Homebrew, or external posting behavior.
  • Do not claim security enforcement from tool authority categories.
  • Do not require volatile live-machine aggregate totals.

Acceptance criteria

  • CI fails if site/demo-report.html silently lacks the current report evidence expected from generated demo reports.
  • The guard does not depend on live local session counts or costs.
  • scripts/ci/check-pages-artifact.sh site and scripts/ci/check-report-semantics.sh remain complementary rather than duplicating the full generated-report semantic suite.
  • Any static demo report copy remains local-first and does not imply hosted tracing or uploaded logs.

Suggested lane

lane/quality

Risk

Low to medium. The main risk is making the static page check too brittle; prefer a small set of stable semantic markers or an explicit static-sample disclaimer.

Source

Product surface audit on 2026-05-17 comparing generated demo HTML, committed site/demo-report.html, and CI script coverage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions