Onboarding attachment for Rome maintainers.
Status: all 38 cases catalogued in section 5 are defined as AI-native YAML workflows under
tests/midscene/cases/and form the mock suite, which runs as one CI job. Each case usesaiActfor interaction andaiAssertfor its visible outcome. A second suite, the backend suite, runs against a real Rome container and shares the same report flow.
Rome is a conversation-centric personal AI OS that integrates routines, approvals, an activity feed and installed apps. Much of its product experience uses rich interactive cards. A routine proposal changes state inside Chat, an approval completes inside its card, and an app link opens beside the current conversation.
This kind of experience has two testing problems:
- Selector-based tests are brittle: card markup, animations, streaming renders and virtualized layouts change often.
- The real backend is not testable: features depend on personal accounts, IM channels (Telegram/WhatsApp) and third-party services, none of which can be reproduced reliably in CI.
This proposal uses Midscene visual-semantic driving
for problem 1, and Rome's strict MSW E2E mode (pnpm dev:mock:e2e) for
problem 2: every case runs only against in-repo synthetic fixtures, with no
real accounts and no external side effects.
Goals:
- Establish a long-lived AI-native E2E baseline for Rome's core product stories.
- Serve as a reference implementation for offering Midscene CI to open-source projects (deterministic fixtures + visual-semantic assertions + one report flow).
- Keep browser traffic local, repeatable and mutually non-polluting. The Node-side Midscene agent still calls the configured model endpoint.
GitHub Actions (job: mock)
└─ tests/midscene (private pnpm workspace package)
├─ midscene.config.ts # Playwright + Midscene config
│ └─ project setup
│ ├─ pnpm --filter rome-web dev:mock:e2e # strict MSW mode, localhost:3200
│ └─ browser network boundary check
├─ cases/**.yaml # cases (topic tags)
└─ Playwright Chromium (1440×900, en-US)
└─ fresh BrowserContext per case
├─ localStorage: rome.lang=en
├─ localStorage: rome-sidebar-pins=<full set>
└─ in-memory MSW state reset with the context
tests/midscene is a private pnpm workspace package. It declares its own
Midscene and browser dependencies, resolved by the root pnpm-lock.yaml.
- One root install prepares the product and test packages.
- The backend CI job installs only the Midscene package with
pnpm --filter. - Dependency scripts follow the root
allowBuildspolicy. CI installs with scripts disabled, then rebuildsesbuildandsharpfor Midscene. - The runtime workspace excludes the test package.
- Fresh BrowserContext per case: MSW handlers run in a Service Worker
(
setupWorker) and keep writes in memory. A new context is a cold start with fixtures reset. Zero shared state between cases. - Pinned English UI: an init script writes
rome.lang=en, so a zh-CN CI machine locale cannot drift the copy. A localized Chinese-shell case is future coverage. None is included in the current 38-case suite. - Full sidebar-pin injection: the mock guardian pins only Apps/Chat/
Projects by default. Other entries live behind the "all apps" popover. Cases
write
rome-sidebar-pins(the shell's own localStorage contract) so every built-in entry is expanded and cross-page sidebar clicks are deterministic. - AI-native interaction:
aiActperforms navigation, typing, scrolling, menu selection, and other user actions from a goal.aiAssertchecks visible outcomes.app.openhandles isolated setup. - Env-driven selection:
MIDSCENE_INCLUDE_TAGS/MIDSCENE_EXCLUDE_TAGS(comma-separated, OR semantics),MIDSCENE_RETRY,HEADLESS— the same entry point serves local single-case iteration and the CI job.
The harness registers two custom nodes used by committed cases. Each case
uses one app.open. Run pnpm run nodes in tests/midscene to generate a
local reference for the full node list.
| Node | Purpose |
|---|---|
app.open |
Open a route in a fresh context, seed language/pins, wait for the sidebar or login page |
app.expectRenameConflict |
Check that FILE-03's rename request received HTTP 409 after aiAct submitted it |
Every case relies on the behavioral facts below. They document the current implementation. If the implementation changes, the cases must change with it.
- Fixed cold-start state:
/api/health,/api/bootstrapand/api/auth/medrop the app straight into an authenticated shell with the synthetic guardian user. The browser is authenticated on open. - Writes are in-memory only: POST/PATCH operations (enabling a routine, approving an approval, installing an app) land in MSW memory and persist across client routes within the same BrowserContext. A hard reload, closing the context, or opening a new one restores fixture defaults.
- Unmatched API requests fail locally: strict E2E mode has no backend
proxy, and MSW returns 503 for unmocked
/apicalls. Cases must not depend on such capabilities (listed as gaps in section 6). - Synthetic fixture data only: no real names, accounts, tokens or chat content may be introduced.
- Waiting strategy:
app.openwaits for the shell (a[href="/chat"]in the sidebar), which doubles as mock-readiness. Async settling after a card mounts (e.g. a routine card re-runningGET /api/routines) uses explicitwaitsteps. - Chat stick-to-bottom: the transcript snaps to the latest message and
programmatic scrolling is pushed back by
useStickToBottom. Only a scroll within 300 ms of a trusted gesture (wheel/touch/keyboard) releases it. Describe the target message and the scrolling goal inaiAct. - In-chat app links:
/apps/<id>links in markdown are intercepted byChatLinkand open in a workspace tile beside the chat while the URL stays at/chat/.... This is real product behavior. Assert on the tile, not on a route change.
The workflow file is the source of truth for step-level details (action pins, timeouts, cache flags). This section records the design decisions and the reasons behind them.
- Triggers: the independent Midscene workflow runs the full model-backed
suite nightly against the latest upstream
maincommit. Its daily schedule is0 6 * * *(06:00 UTC / 14:00 Beijing), matchingnightly.yml. It does not depend on the ordinary CI or Nightly workflow. Pushes and pull requests do not trigger it. Each suite job type-checks the package before it runs cases, and themockproject setup checks MSW, browser HTTP, and WebSocket blocking before the first case. A broken harness or mock boundary fails the next nightly run, so a PR check would only report it a day earlier. Manualworkflow_dispatchalso runs the full suite on upstreammain. A fork owner may manually dispatch a fork branch using that fork's secrets. Report publishing requires the repository to matchMIDSCENE_PUBLISH_REPO. The suite jobs do not run onpull_request: model calls are billable and a PR job checks out contributor-controlled code. Note that merely naming a protected environment in the workflow does not create its reviewer gate — an absent environment is provisioned open — so until a maintainer actually configures a protected environment with required reviewers (ideally with environment-scoped secrets), the secret-bearing suite jobs stay fail-closed and runs only against code on the protected default branch. To enable model-backed PR runs later, create the protected environment in Settings first, then add environment-gatedpull_requestsuite jobs back to the workflow under apull_requesttrigger. - Runtime: Ubuntu + Node.js 24. Pnpm is enabled (reading the root
packageManagerversion) beforesetup-node, whosecache: pnpmrequires pnpm to already exist on a clean runner. Workspace dependencies install withpnpm install --frozen-lockfile --ignore-scripts. The backend job filters this install torome-midscene-e2e. Both jobs rebuildesbuildandsharp, then runpnpm exec playwright install --with-deps chromium. - Secret handling and trust boundary: the four
MIDSCENE_MODEL_*values come from repository secrets and are scoped only to the steps that actually call the model (configuration check,midscene-test model verify, suite run). Dependency installation and the mock server never see them. The job starts with a non-empty check andmodel verify, which checks connectivity and Midscene compatibility, so a missing secret or bad endpoint fails in seconds instead of burning the job's full time budget. - Network reachability (important):
model verifydialsMIDSCENE_MODEL_BASE_URLdirectly from the runner. Hosted GitHub runners can only reach public endpoints. If the model gateway lives on an intranet (for example a corporate-internal hostname),model verifyfails with a connect timeout. The workflow then requires a self-hosted runner that can reach that intranet (change theruns-onlabel) or a publicly reachable model endpoint. Correct credentials with no network route fail atmodel verifyjust as quickly. - Mock server: the
mockproject setup startspnpm --filter rome-web dev:mock:e2ewithout the model variables and stops it in teardown. It runsbuild:kitfirst, so readiness is polled for up to six minutes. If a server already answers atROME_E2E_BASE_URL, setup uses it. The log ismidscene_run/rome-devmock.log, part of the suite artifact. - Suites: each suite is one job. The
mockjob runs every mock case in one Midscene project (web-mock) with a 90-minute job timeout and 2 case-level retries. Thebackendjob runs after it with the same step order, pulling the published image instead of starting the mock server. Each suite's config selects its cases by path:cases/formock,backend/cases/forbackend. - Evidence: each suite job uploads its
midscene_run/and.midscene/directories asmidscene-suite-<suite>, including failed runs, and writes Midscene's nativeresult.md(counts, failed cases, execution errors) to its own job Summary. Thepublishjob copies each suite's native HTML report to Pages underruns/<run-id>/<suite>/and links the reports that exist. If Pages fails or is skipped, the suite Summaries and artifacts remain available. - Timeouts:
timeout --signal=TERMstops a suite before its job limit. Midscene handlesSIGTERM: it stops the running case, marks the rest not run, and still writessummary.json,result.md, and the HTML report. - Network stability:
NODE_OPTIONS=--dns-result-order=ipv4first --no-network-family-autoselectionworks around runner-side IPv6 racing when the model endpoint is only stable over IPv4.
Case ID convention:
<SUITE>-<number>. ✅ = mock-drivable (all drafted in this iteration),⚠️ = blocked by a mock gap from section 6 (registered only, no YAML).
The suite contains 38 ✅ mock-drivable cases in 15 YAML files. The case catalog lists their names and suite assignments.
| Topic | Candidate cases | Blocking handler |
|---|---|---|
| chat | Real send round-trip, streaming replies, question-card submit success state | turn SSE |
| sessions | /sessions Overview metrics page (success-rate/duration charts) |
sessions metrics |
| projects | Right-hand projects dashboard summary rendering | projects dashboard |
| chat/sessions | Session fork, share links, archive/delete success paths | fork/share/archive |
| settings/activity | New-channel connection wizard, connection-request Approve success state | connection setup |
The product capabilities below are not covered by the mock today and block the
packages/web/mock/handlers/,
matching the existing house style:
All of them should follow the existing handler conventions: synthetic fixtures + in-memory writes + cold-start reset, with zero interaction with real external services.
- Today: sending a message and submitting a question/approval card share
POST /api/.../turns, which the mock does not serve. The UI can only reach "Failed to send message". CHAT-04/05/20 therefore assert the designed failure state. - Suggested: a turn handler that accepts a synthetic prompt and returns a fixed synthetic reply over SSE (message-start → content deltas → tool/card block → done), with preset branches for an answered question card and a completed approval execution.
- Unblocks: successful send round-trips, streaming-typing assertions, the locked success state after question submission, sending/stop button states.
- Today:
/sessions(the Overview segment) depends onPOST /api/sessions/metrics. Unmocked, the whole page is unusable, so all sessions cases land on/sessions/all. - Suggested: aggregate the existing sessions fixture into fixed-window metrics (run counts, success rate, token/cost totals, per-day/per-type distributions).
- Unblocks: Overview charts/summary rendering, time-window linkage,
metrics-vs-list consistency assertions (roughly 3–4
⚠️ cases).
- Today: the left-hand file tree on
/projectsis fully usable, but the right-hand dashboard's summary request is unmocked and permanently shows "Network error / Retry" (FILE-01 locks this current state in as a baseline). - Suggested: return a synthetic summary for fixture projects (recent files, activity, usage and other static aggregates).
- Unblocks: normal dashboard rendering, file-selection/summary linkage, Retry disappearing after recovery.
- Today: session writes — fork creation, share links, archive/delete — have no handlers, so only failure copy can be verified.
- Suggested: in-memory handlers for session writes: fork produces a copy with a new id. Share issues a synthetic read-only link (fixed token). Archive/delete removes the row from the list and can be restored in memory.
- Unblocks: content inheritance after fork, read-only share rendering, list disappearance/view filters after archiving, the delete confirmation flow.
- Today: the three channel connection requests in Activity (Telegram/Discord/Feishu) render and can be rejected, but Approve and the in-settings new-channel wizard have no success-path handlers.
- Suggested: Approve writes to memory and returns a pairing code/success state. Add a multi-step GET/POST wizard (choose channel → generate code → verify → complete), fully synthetic.
- Unblocks: request disappearing and banner decrement after approval, a full wizard walkthrough, the connection card appearing in Channels/Settings.
- File-browser watch SSE (
packages/web/mock/handlers/file-browser.ts): an/eventsSSE stream (subscribers/emit/ready, events mirroring the frontend's chokidar watch events). In-mock POST/PATCH/DELETE writes now pushadd/addDir/changeimmediately, so the file tree reflects writes without a full page reload (required by the files/memory cases). - Desktop workspace placeholder
(
packages/web/mock/public/desktop-vnc.html): the noVNC document embedded by the Browser workspace iframe is served by @rome/core in production. In mock mode the missing static asset fell through the SPA fallback into a nested /chat. A blank dark placeholder was added at the dev-server static layer (an MSW browser Service Worker cannot intercept an iframe's initial document navigation, because the new frame has not registered the worker yet). The placeholder is mock infrastructure only. A desktop iframe assertion is future coverage and is not included in the current suite.
See tests/midscene/README.md. In short:
pnpm install --frozen-lockfile # from the repo root
cd tests/midscene # setup starts the mock server on localhost:3200
pnpm exec playwright install chromium
cp .env.example .env # fill in model credentials; .env is gitignored
pnpm test
MIDSCENE_INCLUDE_TAGS=poc pnpm test # PoC stories only- Every case starts with exactly one
app.open. UseaiActfor navigation within the case. - Every case contains at least one
aiActand oneaiAssert. Do not use atomic AI nodes or operationalapp.*nodes beyond the two listed above. - Write each
aiActas a user goal. Combine related clicks, typing, scrolling, and navigation when they serve one intent. - Write
aiAssertprompts against visible outcomes and stable product text. - Add functional tags such as
chatorroutinesas needed. The directory, not a tag, decides the suite. - New cases must pass repeatedly in a fresh context. Inter-case dependencies are forbidden.
- When fixtures change, update case assertions in lockstep. When a new product
capability is added, add the mock handler before the
⚠️ case.