Skip to content

fix(web-ai): honor --family and bound assistant DOM reads by the polling deadline - #89

Closed
dbc-hbin wants to merge 2 commits into
lidge-jun:mainfrom
dbc-hbin:fix/web-ai-family-cli-wiring
Closed

fix(web-ai): honor --family and bound assistant DOM reads by the polling deadline#89
dbc-hbin wants to merge 2 commits into
lidge-jun:mainfrom
dbc-hbin:fix/web-ai-family-cli-wiring

Conversation

@dbc-hbin

@dbc-hbin dbc-hbin commented Jul 27, 2026

Copy link
Copy Markdown

Fixes #87
Fixes #88

Two related web-ai defects, one commit each, so they can still be reviewed independently:

They are combined because both touch web-ai/chatgpt.mjs and both bump line counts in structure/str_func.md, so merging them separately would need a npm run fix:counts conflict resolution in between. Verified together below.


#87--family was accepted and silently ignored

--family is advertised by the top-level help, README, the bundled web-ai skill, and the web_ai_submit_prompt MCP schema, but the value never reached model selection. The send/query parser runs with strict: false, so the flag was swallowed:

agbrowse web-ai render --vendor chatgpt --family gpt-5.6-sol \
  --model thinking --effort high --inline-only --prompt "check" --json
# → status: rendered

--family bogus-family returned the same rendered/exit 0, so neither output proved anything about the family. Agent workflows could report that GPT-5.6 Sol was explicitly enforced while using whatever family happened to be selected in the browser.

web-ai/chatgpt-model.mjs already implemented family selection and post-click verification (selectChatGptFamily, normalizeChatGptFamilyChoice); only the wiring was missing, so the fix is small.

Changes

  • declare family in the send/query parser and carry it into the normalized input
  • pass { family, effort } into selectChatGptModel and the model capability probe
  • forward family in web_ai_submit_prompt, failing closed when sent to a provider with no Chat family axis
  • validate the alias in CLI preflight, before any browser mutation
  • document --family in web-ai help, which omitted it while the top-level help advertised it

Omitting --family is unchanged: the family currently selected in the UI is preserved and no submenu is touched.

--family gpt-5.6-sol|gpt-5.5|gpt-5.4|gpt-5.3|o3   accepted, selected and verified
--family gpt-5.6-luna                              unsupported ChatGPT family: gpt-5.6-luna
--vendor gemini --family gpt-5.6-sol               --family is supported only for ChatGPT

An unsupported family also makes the model capability probe report fail, so a probe result is no longer mistaken for proof that the family was enforced.

#88 — polling could outlive --timeout

pollWebAi checks its deadline only at the while boundary, then awaits readAssistantMessages(), which called page.evaluate() with no per-call bound. Playwright's page.evaluate() accepts no timeout option (only locator.evaluate does), so when assistant-message extraction stalled on a large conversation, control never reached the next deadline check. The process and session lock stayed alive while the stderr heartbeat went silent — matching the report where sessions doctor showed a valid target, a healthy command-lock heartbeat, and no CDP or login failure.

The same function also serialized every historical assistant turn on each 500 ms tick, then sliced off everything before baseline.assistantCount.

Changes

  • add withAssistantReadTimeout() and resolveAssistantReadBudgetMs(); each read races the smaller of the remaining deadline and a per-read ceiling
  • bound the page.evaluate() path, the locator fallback, and the post-timeout recovery read, which otherwise re-hung the command it exists to rescue
  • add readAssistantTextsAfterIndex(): count turns in-page, serialize only turns after the baseline
  • keep emitting the poll heartbeat while reads stall, since silence with a live process was the original symptom
  • surface assistant-dom-read-timeout:<n> on the timeout envelope so a stalled read is distinguishable from ongoing generation

A read that exceeds its budget retries on the next tick; at the deadline the existing recoverable provider.poll-timeout envelope is returned, so poll and sessions resume semantics are unchanged.

Two judgment calls worth review:

  • Exhausting the budget exactly at the loop boundary is treated as the deadline itself, not a DOM-read failure, so ordinary timeouts are not mislabeled as stalls.
  • When reads were already stalling, the recovery read uses a shorter (2s) budget so recovery cannot extend the command by another full ceiling.

The trimmed read falls back to the full read whenever it observes no turns, keeping pages that cannot serialize the object argument on the previous behavior.


Verification

Full suite, integrated, with Playwright's pinned Chromium installed so nothing is skipped:

npx playwright-core install chromium     # matches playwright-core 1.58.2 (build 1208)
npx vitest run test/unit test/integration test/e2e

  main (a03de27):  161 files, 1437 passed
  this branch:     163 files, 1454 passed, 0 skipped

All 1437 baseline tests still pass; the delta is the 17 new tests. Also green:

npm run gate:all                  # all 16 release gates
npm run typecheck                 # clean
npm run check:strict-baseline     # OK
npm run test:contract-drift       # 1 passed
npm run test:eval                 # 32 passed
npm run test:eval-fixtures        # regressions: []
npm run test:mcp                  # 45 passed
npm run test:trace-policy         # 42 passed
npm run test:source-audit         # 14 passed
npm run smoke:bins                # ok
npm run check:module-graph        # 356 files, max tier 11
bash structure/check-doc-drift.sh # 164 passed
bash structure/verify-counts.sh   # 76 passed (str_func.md refreshed via fix:counts)

typecheck:checkjs / typecheck:checkjs-dom have pre-existing errors on main; I diffed per-file error counts against the a03de27 baseline and this branch adds none.

New tests

  • test/unit/web-ai-chatgpt-family-wiring.test.mjs — asserts at the module boundary that the alias reaches selectChatGptModel and the probe, that omitting it keeps the zero-mutation contract, and that an unsupported family fails closed without touching the page
  • test/unit/web-ai-assistant-read-deadline.test.mjs — drives pollWebAi with a page.evaluate() that never resolves and asserts the command still honors its deadline and reports the stall; also covers the budget helpers and proves historical turns are no longer re-serialized

Both were checked against the unfixed code: dropping the family argument fails the wiring test, and removing the read bound makes the poll test hang until the runner's own timeout.

Live Chrome check

Because the deterministic tests use stub pages, I also ran both read paths against real Chrome over CDP on a ~2 MB, 40-turn conversation:

full read (old) post-baseline read (new)
evaluate time 64 ms 2 ms
serialized 1,980,271 chars 49,507 chars

That is ~1.93 MB of text no longer serialized per 500 ms tick, and the trimmed read still reports the correct total turn count (40).

End-to-end on a real Playwright page whose assistant read never resolves:

requested timeout: 5s
actual elapsed:    7.1s
status:            timeout (ok: false, recoverable: true, retryHint: poll-or-resume)
warnings:          ["assistant-dom-read-timeout:2"]

Before this change that call did not return at all. The 2.1s overshoot is the intentional post-timeout recovery read under its 2s budget.

Not verified

I did not reproduce the original failure against a live ChatGPT conversation with 40–60 KB assistant answers, so the real-world stall threshold remains unmeasured; the deadline behavior itself is covered deterministically and against real Chrome.

One existing fake page in web-ai-provider-session.test.mjs keyed its answer sequence off a raw evaluate call count, so it now keys off the assistant-read argument shape instead. Its assertions are unchanged.

@dbc-hbin dbc-hbin changed the title fix(web-ai): carry --family through the CLI and MCP send paths fix(web-ai): honor --family and bound assistant DOM reads by the polling deadline Jul 28, 2026
hanbinnoh added 2 commits July 28, 2026 09:17
…-jun#87)

`--family` was advertised by the top-level help, README, the bundled web-ai
skill, and the `web_ai_submit_prompt` MCP schema, but the value never reached
ChatGPT model selection. Because `parseArgs` runs with `strict: false`, the flag
was accepted and silently dropped, so a successful `render` or capability probe
looked like proof that the requested family had been enforced.

`chatgpt-model.mjs` already implemented family selection and verification; only
the wiring was missing.

- declare `family` in the send/query parser and carry it into the normalized
  input
- pass `{ family, effort }` into `selectChatGptModel` and the model capability
  probe so selection and verification actually run
- forward `family` in `web_ai_submit_prompt`, and fail closed when it is sent to
  a provider without a Chat family axis
- validate the alias in CLI preflight, before any browser mutation, so an
  unsupported value such as `gpt-5.6-luna` errors instead of being ignored
- document `--family` in `web-ai help`, which previously omitted it

Omitting `--family` keeps the existing zero-mutation behavior: the family
currently selected in the UI is preserved and no submenu is touched.

Tests: new `test/unit/web-ai-chatgpt-family-wiring.test.mjs` asserts at the
module boundary that the alias reaches the selector and that an unsupported
family fails closed without touching the page; CLI contract tests cover every
canonical alias, the invalid alias, non-ChatGPT rejection, and help/parser
agreement.
…-jun#88)

ChatGPT polling could stay alive with no progress well past `--timeout`. The
poll loop only re-checked its deadline at the `while` boundary, then awaited
`readAssistantMessages()`, which called `page.evaluate()` with no per-call bound.
Playwright's `page.evaluate()` accepts no timeout option, so when assistant
message extraction stalled on a very large conversation, control never reached
the next deadline check: the process and the session lock stayed alive while the
stderr heartbeat went silent.

- add `withAssistantReadTimeout()` and `resolveAssistantReadBudgetMs()`; every
  assistant read is raced against the smaller of the remaining command deadline
  and a per-read ceiling
- bound the `page.evaluate()` path, the locator fallback, and the post-timeout
  recovery read, which otherwise re-hung the command it exists to rescue
- add `readAssistantTextsAfterIndex()`: count turns in-page but serialize only
  the turns after the baseline, instead of re-serializing the whole conversation
  on every 500ms tick
- keep emitting the poll heartbeat while reads stall, since silence with a live
  process was the original symptom
- report a stalled read distinctly from ongoing generation via an
  `assistant-dom-read-timeout:<n>` warning on the timeout envelope

A read that exceeds its budget now retries on the next tick and, at the
deadline, returns the existing recoverable `provider.poll-timeout` envelope, so
`poll` and `sessions resume` behavior is unchanged. Exhausting the budget at the
loop boundary is treated as the deadline itself, not as a DOM-read failure, so
ordinary timeouts are not mislabeled.

Tests: new `test/unit/web-ai-assistant-read-deadline.test.mjs` drives
`pollWebAi` with a `page.evaluate()` that never resolves and asserts the command
still honors its deadline and reports the stall; it also covers the budget
helpers and proves historical turns are no longer re-serialized. Verified that
removing the bound makes the poll test hang until the runner's own timeout.
@dbc-hbin
dbc-hbin force-pushed the fix/web-ai-family-cli-wiring branch from 79db5d1 to 4ada9cb Compare July 28, 2026 00:22
@dbc-hbin

Copy link
Copy Markdown
Author

Rebased onto main at v0.1.19 (1463a53) to clear the merge conflict, which was only the structure/str_func.md line-count rows; resolved with npm run fix:counts. No code changes in the rebase, and both commits are still separate.

Re-verified on top of v0.1.19:

npx vitest run test/unit test/integration test/e2e

  origin/main (1463a53):  162 files, 1448 passed
  this branch  (4ada9cb): 164 files, 1465 passed, 0 skipped

All 1448 baseline tests still pass; the delta is the 17 new tests. Also green: npm run gate:all (16/16), typecheck, check:strict-baseline, check-doc-drift.sh (164), verify-counts.sh (76).

The Contract Drift Check workflow shows action_required because this is a fork PR and needs maintainer approval to run. I ran its four steps locally against this commit:

npm run test:contract-drift                              # 1 passed
npm run test:eval                                        # 32 passed
npm run test:eval-fixtures                               # regressions: []
AGBROWSE_DRIFT_MODE=fixture npm run eval:web-ai:fixtures # ok: true, regressions: []

Note for whichever order you merge in: nothing here touches the new postinstall star-prompt work from v0.1.19.

lidge-jun added a commit that referenced this pull request Aug 1, 2026
WP1 docs-only 사이클. dev 기준 실태 조사와 계획 문서 8종.

- 002: PR #89 두 커밋(d5d9475, 4ada9cb)의 dev 대조 판정. #87은
  f8e8b9b로 부분 충족(probe/MCP 갭 잔존), #88은 미충족.
- 001: devlog _plan 11개 유닛 판정과 00_index.md 드리프트 인벤토리.
- 003: A 페이즈 3라운드 FAIL의 근본 원인 분석. #88 방어 범위를
  열거로 확정하려는 시도가 매 라운드 새 누락을 낳아, WP3를
  경계 인벤토리 확정으로 축소하고 구현은 후속 WP로 분리.

코드 변경 없음.
lidge-jun added a commit that referenced this pull request Aug 1, 2026
WP5 A 게이트 6건. 리뷰어가 실제 오류를 셋 잡았다.

- 커밋 목록에서 b524453(WP2 계획 보강)이 빠져 16개로 셌다. 실제
  17개다(git rev-list --count c7e87c1..HEAD).
- 감사 통계가 틀렸다. '리뷰어 4명 16라운드'가 아니라 5명 14라운드
  (WP1 9 + WP2 1 + WP3 4)이고 분포는 FAIL 11 / GO-WITH-FIXES 3이다.
  출처 없는 '그 외 2라운드'를 지웠다.
- main-only 커밋을 6개로 적었으나 실측 5개다.

c2 정합: '전수 열거'를 요구하는 criterion을 met으로 두면서 증거는
'표본 36개'라 서로 반박했다. scenario를 표본+예산 계약 기준으로
개정했다 — 전수는 달성 조건이 아니라고 명시했다. 040의 '경계 확정
DONE'도 '부분'으로 고쳤다.

diagnostics 배정이 040(DOM 유닛)과 021(artifact 유닛)에서 달랐다.
021 기준으로 통일하고 각 행에 출처를 달았다.

마감 절차: _plan 표에는 이 유닛 행이 애초에 없어서 제거 diff가
적용 불가였다. _fin 행 추가만 남겼다. git mv 후 goalplan
capturedEvidence 경로도 갱신하라는 단계를 추가했다 — 증거가
가리키는 곳이 없으면 증거가 아니다.

fresh 게이트를 마감 시점에 다시 돌려 기록했다(verify-counts 76,
doc-drift 164, typecheck 0). 문서 수정이 카운트를 밀므로 이관
직전 fix:counts 재실행을 절차에 박았다.

LOOP-PESSIMIST에 WP2의 죽은 가설 둘 추가(fixture 거짓 양성,
guard hole). PR #89 mergeStateStatus UNSTABLE도 인계에 적었다.
lidge-jun added a commit that referenced this pull request Aug 1, 2026
유닛을 _fin으로 이관하고 00_index에 closeout 행을 추가했다.

WP5 A 게이트 2라운드 잔여 4건도 접었다:
- 커밋 목록에 기준점을 달았다(WP5 착수 전 c7e87c1..9c9ea88, 17개).
  자기 커밋 해시를 자기 안에 적을 수 없으니 범위를 한정하는 게 맞다.
- 감사 통계를 WP1~WP3(5명 14라운드)로 한정하고 WP5 2라운드를 별도
  줄로 뒀다. 유닛 전체는 6명 16라운드, FAIL 12 / GO-WITH-FIXES 4.
- 040과 goalplan wp3 태스크에 남아 있던 '전수' 표현을 '확인된 표본
  전체와 접근 방식'으로 통일했다. c2를 표본 기준으로 고쳐놓고
  다른 기록에 전수성이 남아 있으면 같은 모순이다.
- diagnostics를 c7(유닛 A)에서 c8(유닛 B)로 옮겼다. 021이 B10/B28을
  유닛 B에 배정했는데 criterion이 A를 가리키고 있었다.
  readActivityState도 '그때 결정'에서 '유닛 A 3번 phase 확정'으로.

git mv 후 c1/c2의 capturedEvidence 경로를 _fin으로 갱신했다.
증거가 가리키는 곳이 없으면 증거가 아니다.

최종 게이트 (이관·수정 후 재실행):
  typecheck exit 0
  unit 5파일 129건, integration 3파일 70건
  gate:all 16/16, doc-drift 164, verify-counts 76
@lidge-jun

Copy link
Copy Markdown
Owner

Thanks for this PR — both fixes were real, and both shaped what landed. We could not take the patches directly: dev's assistant reader was already refactored to the split snapshot reader (web-ai/chatgpt-response-dom.mjs:297), so the #88 hunks no longer apply, and the trimmed read would break the wrapped/wrapperless correlation. Instead dev carries independent implementations covering both commits: --family now reaches the model probe fail-closed and MCP rejects non-ChatGPT family combinations (76e4793), and every assistant DOM read, recovery read, and session-store write is bounded by the polling deadline (8a971ff through 107233e — see devlog/_plan/260731_webai_poll_deadline/). Closing in favor of those; the campaign devlog credits this PR as the trigger.

@lidge-jun lidge-jun closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pollWebAi can hang past --timeout when assistant DOM evaluation stalls web-ai CLI silently ignores documented --family

2 participants