diff --git a/README.md b/README.md index 008f22aa..9e240567 100644 --- a/README.md +++ b/README.md @@ -433,7 +433,7 @@ Tab limits: | Setting | Default | Env var | | --- | --- | --- | -| Max tabs | 10 | `AGBROWSE_MAX_TABS` | +| Max tabs | 20 | `AGBROWSE_MAX_TABS` | | Idle timeout | 30 min | `AGBROWSE_TAB_IDLE` | `send` and `query` run tab cleanup before opening another tab. Cleanup never @@ -454,14 +454,20 @@ navigation, reload, tab switch, or any major page mutation. Use `vision-click` only when a target is visible in a screenshot but has no usable DOM/ref target, such as canvas/WebGL-heavy UIs. +The normal order is ref click first, coordinate click last. Vision results are +treated as bbox candidates with confidence; low-confidence or legacy point-only +results require verification instead of clicking directly. ```bash agbrowse screenshot --json agbrowse-vision-click "the visible Submit button" + +agbrowse observe-bundle --screenshot --boxes --json > /tmp/bundle.json +agbrowse-vision-click "Submit button" --bundle /tmp/bundle.json --verify-before-click ``` -The vision path handles device-pixel-ratio correction before sending -`page.mouse.click()` coordinates. +The vision path handles device-pixel-ratio correction and clip-origin evidence +before sending `page.mouse.click()` coordinates. ## Web AI diff --git a/devlog/00_index.md b/devlog/00_index.md index 87fa2b3d..351c36cb 100644 --- a/devlog/00_index.md +++ b/devlog/00_index.md @@ -13,21 +13,53 @@ must be treated as historical do not edit them after release.evidence | Folder | Purpose | | --- | --- | +| `_fin/` | Completed non-MVP closeouts, research outcomes, and shipped implementation plans. | | `_fin/mvp/` | Shipped MVP phases, grouped by topic (read-only history). | | `_plan/` | Active or deferred plans not yet shipped. | | `_legacy/` | Pre-rewrite changelogs / plans / research dumps. | | `context/` | Verbatim Pro / Grok peer reviews and gap audits. | -## `_plan/` active research +## `_plan/` active or deferred work | Topic | Folder | Status | | --- | --- | --- | +| Post-MVP competitive gap closeout | `_plan/260506_post_mvp_gap_closeout/` | Deferred backlog; not fully shipped. | +| Oracle ZIP browser bundle proposal | `_plan/260513_oracle_zip_bundle_proposal/` | External upstream proposal draft; no local agbrowse implementation authority. | +| Adaptive Fetch v2 hardening | `_plan/260515_adaptive_fetch_v2_hardening/` | Follow-up planning after v2; not closed. | +| Provider expansion | `_plan/260519_provider_expansion/` | Deferred provider roadmap for Claude, Perplexity, NotebookLM. | +| Codebase audit backlog | `_plan/260603_codebase_audit/` | Historical audit + issue tracker; keep grouped until separately closed. | | K-BrowseComp search gap analysis | `_plan/260608_kbrowsecomp_search_gap/` | Research/spec complete; implementation deferred. | -| web-ai GPT Code Mode (artifact zip 회수) | `_plan/260611_webai_gpt_code_mode/` | Implemented as ChatGPT-only beta: single zip, multi-zip, and JS-only artifact retrieval verified. | -| web-ai code mode GPT dev-agent context | `_plan/260611_code_mode_gpt_agent_context/` | Active: auto-upload saved GPT dev-agent context zip, require `PLAN.md`/`00_plan.md`, and mirror independent runtime into cli-jaw. | -| web-ai skill + cli-jaw mirror closeout | `_plan/260611_webai_skill_cli_jaw_mirror/` | Active: update agent-facing skill docs and mirror the simplified ChatGPT Intelligence picker into cli-jaw. | -| Background runtime hook research | `_plan/260611_background_runtime_hook/` | Research complete; findings reflected in `skills/web-ai/SKILL.md` (Long-Running section). Hook design relocated to cli-jaw `devlog/_plan/260611_bgtask_background_runtime/` for implementation planning. | -| ChatGPT composer tool selection live probe | `_plan/260615_chatgpt_composer_tools_live_probe.md` | Active: PR #78 live DOM/UX evidence for explicit-only tool/plugin/model gating. | +| Oracle stability gap analysis | `_plan/260608_oracle_stability_gap/` | Partial follow-up backlog; includes 2026-06-20 Oracle 0.15 delta and 31/32 PABCD follow-up plans. | +| Legacy MVP phase plans | `_plan/legacy_mvp_phase_plans/` | Pre-closeout phase 8.1/9 planning references; grouped for audit hygiene. | +| Strict migration | `_plan/strict-migration/` | Deferred migration planning and arbitration notes. | + +Other grouped planning folders under `_plan/` remain until they receive a +separate closeout audit. + +## Recent `_fin/` closeouts + +| Topic | Folder | Closeout signal | +| --- | --- | --- | +| UX blocker fixes | `_fin/260507_ux-blockers-p0p1/` | README maps fixes to implemented commits `ccb7051`, `1a4743b`, and `f7b0e97`. | +| Oracle parity feature batch | `_fin/260508_oracle_parity/` | Implemented by `fe359a9` and follow-up commits. | +| web-ai session rebinding hardening | `_fin/260510_webai_session_rebind_diff_plan.md` | Implemented by `276aeac`. | +| Oracle follow-up guardrails | `_fin/260513_oracle_followup_guardrails_diff_plan.md` | Implemented by `085cc83`. | +| Adaptive Fetch v1 / Insane Search mirror | `_fin/260514_insane_search_adaptive_fetch/` | README status `implemented-v1`; shipped by `39708a3` and follow-ups. | +| Adaptive Fetch v2 | `_fin/260515_adaptive_fetch_v2/` | Index status `implemented`; hardening follow-up remains in `_plan/`. | +| Shared web-ai target lock | `_fin/260525_shared_web_ai_target_lock/` | Implemented by `602a700` and `e28f66e`. | +| Runway MCP parity expansion | `_fin/260528_runway_mcp_parity_expansion/` | Implemented by `7458f64` and Runway continuity follow-ups. | +| Defuddle reader candidate | `_fin/260610_defuddle_reader/` | Implementation result recorded; shipped by `631615d`. | +| Background runtime hook research | `_fin/260611_background_runtime_hook/` | Research complete; cli-jaw implementation planning relocated. | +| web-ai GPT Code Mode | `_fin/260611_webai_gpt_code_mode/` | ChatGPT-only beta implemented: single zip, multi-zip, JS-only retrieval. | +| Code Mode GPT dev-agent context | `_fin/260611_code_mode_gpt_agent_context/` | Implemented by `7ef4955`, `81af74f`, and `864ae41`. | +| web-ai multi/mixed attachments | `_fin/260611_webai_multi_attach/` | Implemented and live verified by `ef01881`. | +| web-ai skill + cli-jaw mirror | `_fin/260611_webai_skill_cli_jaw_mirror/` | Agent-facing docs and simplified picker mirror closed. | +| Docs Pages and code-mode overhaul | `_fin/260611_docs_pages_overhaul/` | Final goal audit proves local gates, push, and live Pages deployment. | +| ChatGPT composer tool selection probe | `_fin/260615_chatgpt_composer_tools_live_probe.md` | PR #78 evidence and follow-up patches applied. | +| Computer-use contract hardening / vision upgrade | `_fin/260617_computer_use_contract_hardening/` | `dev-vision-upgrade` verification report records implementation and live smoke evidence. | +| Web-AI stability and concurrency closeout | `_fin/260619_webai_stability/`, `_fin/260619_tab_parallel_stability/` | Timeout/watch/skill-envelope closed earlier; tab MVV closed by active lease cap + PID reaper + record-before-bind. | +| MCP wait response recovery | `_fin/260621_mcp_wait_response_recovery/` | GitHub #79 PABCD: session-bound MCP wait/resume recovery and monotonic timeout handling. | +| Tab stability MVV closeout | `_fin/260621_tab_stability_mvv_closeout/` | Final branch closeout plan for tab MVV, verification, push, and PR body `Closes #79`. | ## `_fin/mvp/` topics diff --git a/devlog/_plan/260507_ux-blockers-p0p1/README.md b/devlog/_fin/260507_ux-blockers-p0p1/README.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/README.md rename to devlog/_fin/260507_ux-blockers-p0p1/README.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p0-fix4-stale-target-ownership.md b/devlog/_fin/260507_ux-blockers-p0p1/p0-fix4-stale-target-ownership.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p0-fix4-stale-target-ownership.md rename to devlog/_fin/260507_ux-blockers-p0p1/p0-fix4-stale-target-ownership.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p0-fix5-stale-answer-pickup.md b/devlog/_fin/260507_ux-blockers-p0p1/p0-fix5-stale-answer-pickup.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p0-fix5-stale-answer-pickup.md rename to devlog/_fin/260507_ux-blockers-p0p1/p0-fix5-stale-answer-pickup.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p1-fix1-file-upload-policy.md b/devlog/_fin/260507_ux-blockers-p0p1/p1-fix1-file-upload-policy.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p1-fix1-file-upload-policy.md rename to devlog/_fin/260507_ux-blockers-p0p1/p1-fix1-file-upload-policy.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p1-fix3-crash-recovery.md b/devlog/_fin/260507_ux-blockers-p0p1/p1-fix3-crash-recovery.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p1-fix3-crash-recovery.md rename to devlog/_fin/260507_ux-blockers-p0p1/p1-fix3-crash-recovery.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p1-fix7-thinking-placeholder.md b/devlog/_fin/260507_ux-blockers-p0p1/p1-fix7-thinking-placeholder.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p1-fix7-thinking-placeholder.md rename to devlog/_fin/260507_ux-blockers-p0p1/p1-fix7-thinking-placeholder.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p2-fix2-same-tab-reuse.md b/devlog/_fin/260507_ux-blockers-p0p1/p2-fix2-same-tab-reuse.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p2-fix2-same-tab-reuse.md rename to devlog/_fin/260507_ux-blockers-p0p1/p2-fix2-same-tab-reuse.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p2-fix6-session-url-reuse.md b/devlog/_fin/260507_ux-blockers-p0p1/p2-fix6-session-url-reuse.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p2-fix6-session-url-reuse.md rename to devlog/_fin/260507_ux-blockers-p0p1/p2-fix6-session-url-reuse.md diff --git a/devlog/_plan/260507_ux-blockers-p0p1/p2-fix8-zip-default.md b/devlog/_fin/260507_ux-blockers-p0p1/p2-fix8-zip-default.md similarity index 100% rename from devlog/_plan/260507_ux-blockers-p0p1/p2-fix8-zip-default.md rename to devlog/_fin/260507_ux-blockers-p0p1/p2-fix8-zip-default.md diff --git a/devlog/_plan/260508_oracle_parity/00_index.md b/devlog/_fin/260508_oracle_parity/00_index.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/00_index.md rename to devlog/_fin/260508_oracle_parity/00_index.md diff --git a/devlog/_plan/260508_oracle_parity/01_artifacts.md b/devlog/_fin/260508_oracle_parity/01_artifacts.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/01_artifacts.md rename to devlog/_fin/260508_oracle_parity/01_artifacts.md diff --git a/devlog/_plan/260508_oracle_parity/02_images.md b/devlog/_fin/260508_oracle_parity/02_images.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/02_images.md rename to devlog/_fin/260508_oracle_parity/02_images.md diff --git a/devlog/_plan/260508_oracle_parity/03_tab_harvest.md b/devlog/_fin/260508_oracle_parity/03_tab_harvest.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/03_tab_harvest.md rename to devlog/_fin/260508_oracle_parity/03_tab_harvest.md diff --git a/devlog/_plan/260508_oracle_parity/04_multi_turn.md b/devlog/_fin/260508_oracle_parity/04_multi_turn.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/04_multi_turn.md rename to devlog/_fin/260508_oracle_parity/04_multi_turn.md diff --git a/devlog/_plan/260508_oracle_parity/05_deep_research.md b/devlog/_fin/260508_oracle_parity/05_deep_research.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/05_deep_research.md rename to devlog/_fin/260508_oracle_parity/05_deep_research.md diff --git a/devlog/_plan/260508_oracle_parity/06_archive.md b/devlog/_fin/260508_oracle_parity/06_archive.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/06_archive.md rename to devlog/_fin/260508_oracle_parity/06_archive.md diff --git a/devlog/_plan/260508_oracle_parity/07_project_sources.md b/devlog/_fin/260508_oracle_parity/07_project_sources.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/07_project_sources.md rename to devlog/_fin/260508_oracle_parity/07_project_sources.md diff --git a/devlog/_plan/260508_oracle_parity/08_control_summary.md b/devlog/_fin/260508_oracle_parity/08_control_summary.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/08_control_summary.md rename to devlog/_fin/260508_oracle_parity/08_control_summary.md diff --git a/devlog/_plan/260508_oracle_parity/10_gap_closeout_index.md b/devlog/_fin/260508_oracle_parity/10_gap_closeout_index.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/10_gap_closeout_index.md rename to devlog/_fin/260508_oracle_parity/10_gap_closeout_index.md diff --git a/devlog/_plan/260508_oracle_parity/11_generated_images_public_contract.md b/devlog/_fin/260508_oracle_parity/11_generated_images_public_contract.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/11_generated_images_public_contract.md rename to devlog/_fin/260508_oracle_parity/11_generated_images_public_contract.md diff --git a/devlog/_plan/260508_oracle_parity/12_project_sources_hardening.md b/devlog/_fin/260508_oracle_parity/12_project_sources_hardening.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/12_project_sources_hardening.md rename to devlog/_fin/260508_oracle_parity/12_project_sources_hardening.md diff --git a/devlog/_plan/260508_oracle_parity/13_multi_turn_lifecycle.md b/devlog/_fin/260508_oracle_parity/13_multi_turn_lifecycle.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/13_multi_turn_lifecycle.md rename to devlog/_fin/260508_oracle_parity/13_multi_turn_lifecycle.md diff --git a/devlog/_plan/260508_oracle_parity/14_deep_research_contract.md b/devlog/_fin/260508_oracle_parity/14_deep_research_contract.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/14_deep_research_contract.md rename to devlog/_fin/260508_oracle_parity/14_deep_research_contract.md diff --git a/devlog/_plan/260508_oracle_parity/15_artifacts_archive_contract.md b/devlog/_fin/260508_oracle_parity/15_artifacts_archive_contract.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/15_artifacts_archive_contract.md rename to devlog/_fin/260508_oracle_parity/15_artifacts_archive_contract.md diff --git a/devlog/_plan/260508_oracle_parity/16_attachment_chip_hardening.md b/devlog/_fin/260508_oracle_parity/16_attachment_chip_hardening.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/16_attachment_chip_hardening.md rename to devlog/_fin/260508_oracle_parity/16_attachment_chip_hardening.md diff --git a/devlog/_plan/260508_oracle_parity/17_upload_size_cap.md b/devlog/_fin/260508_oracle_parity/17_upload_size_cap.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/17_upload_size_cap.md rename to devlog/_fin/260508_oracle_parity/17_upload_size_cap.md diff --git a/devlog/_plan/260508_oracle_parity/18_mcp_advanced_surface.md b/devlog/_fin/260508_oracle_parity/18_mcp_advanced_surface.md similarity index 100% rename from devlog/_plan/260508_oracle_parity/18_mcp_advanced_surface.md rename to devlog/_fin/260508_oracle_parity/18_mcp_advanced_surface.md diff --git a/devlog/_plan/260510_webai_session_rebind_diff_plan.md b/devlog/_fin/260510_webai_session_rebind_diff_plan.md similarity index 100% rename from devlog/_plan/260510_webai_session_rebind_diff_plan.md rename to devlog/_fin/260510_webai_session_rebind_diff_plan.md diff --git a/devlog/_plan/260513_oracle_followup_guardrails_diff_plan.md b/devlog/_fin/260513_oracle_followup_guardrails_diff_plan.md similarity index 99% rename from devlog/_plan/260513_oracle_followup_guardrails_diff_plan.md rename to devlog/_fin/260513_oracle_followup_guardrails_diff_plan.md index 5da83aab..9a4cedf5 100644 --- a/devlog/_plan/260513_oracle_followup_guardrails_diff_plan.md +++ b/devlog/_fin/260513_oracle_followup_guardrails_diff_plan.md @@ -27,7 +27,7 @@ Out of scope for this first slice: - Direct `--file` browser max-size cap. - Nested current-UI attachment-chip hardening from Oracle PR #192. - Oracle ZIP proposal. That is tracked separately in - `devlog/_plan/260513_oracle_zip_bundle_proposal.md`. + `devlog/_plan/260513_oracle_zip_bundle_proposal/00_proposal.md`. ## Part 1 — Easy Explanation diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/01_agbrowse_first_mirror_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/01_agbrowse_first_mirror_plan.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/01_agbrowse_first_mirror_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/01_agbrowse_first_mirror_plan.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/02_surface_design_chrome_boundaries.md b/devlog/_fin/260514_insane_search_adaptive_fetch/02_surface_design_chrome_boundaries.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/02_surface_design_chrome_boundaries.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/02_surface_design_chrome_boundaries.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/03_agbrowse_skill_reinforcement_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/03_agbrowse_skill_reinforcement_plan.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/03_agbrowse_skill_reinforcement_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/03_agbrowse_skill_reinforcement_plan.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/04_search_keyword_consolidation_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/04_search_keyword_consolidation_plan.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/04_search_keyword_consolidation_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/04_search_keyword_consolidation_plan.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/05_diff_level_browser_fetch_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/05_diff_level_browser_fetch_plan.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/05_diff_level_browser_fetch_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/05_diff_level_browser_fetch_plan.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/06_engine_flow.md b/devlog/_fin/260514_insane_search_adaptive_fetch/06_engine_flow.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/06_engine_flow.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/06_engine_flow.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/07_safety_and_risk.md b/devlog/_fin/260514_insane_search_adaptive_fetch/07_safety_and_risk.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/07_safety_and_risk.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/07_safety_and_risk.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/08_test_strategy.md b/devlog/_fin/260514_insane_search_adaptive_fetch/08_test_strategy.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/08_test_strategy.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/08_test_strategy.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/09_eli5_visual_explanation.md b/devlog/_fin/260514_insane_search_adaptive_fetch/09_eli5_visual_explanation.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/09_eli5_visual_explanation.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/09_eli5_visual_explanation.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/10_upstream_inventory.md b/devlog/_fin/260514_insane_search_adaptive_fetch/10_upstream_inventory.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/10_upstream_inventory.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/10_upstream_inventory.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/11_principles.md b/devlog/_fin/260514_insane_search_adaptive_fetch/11_principles.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/11_principles.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/11_principles.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/12_cli_jaw_fit_baseline.md b/devlog/_fin/260514_insane_search_adaptive_fetch/12_cli_jaw_fit_baseline.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/12_cli_jaw_fit_baseline.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/12_cli_jaw_fit_baseline.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md b/devlog/_fin/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/14_open_questions.md b/devlog/_fin/260514_insane_search_adaptive_fetch/14_open_questions.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/14_open_questions.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/14_open_questions.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md b/devlog/_fin/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/16_gpt_pro_validation_report.md b/devlog/_fin/260514_insane_search_adaptive_fetch/16_gpt_pro_validation_report.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/16_gpt_pro_validation_report.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/16_gpt_pro_validation_report.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/17_grok_similar_repo_research.md b/devlog/_fin/260514_insane_search_adaptive_fetch/17_grok_similar_repo_research.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/17_grok_similar_repo_research.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/17_grok_similar_repo_research.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/18_grok_borrowed_patterns_integration_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/18_grok_borrowed_patterns_integration_plan.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/18_grok_borrowed_patterns_integration_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/18_grok_borrowed_patterns_integration_plan.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/19_phased_diff_implementation_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/19_phased_diff_implementation_plan.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/19_phased_diff_implementation_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/19_phased_diff_implementation_plan.md diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/20_phased_diff_browser_docs_closeout_plan.md b/devlog/_fin/260514_insane_search_adaptive_fetch/20_phased_diff_browser_docs_closeout_plan.md similarity index 95% rename from devlog/_plan/260514_insane_search_adaptive_fetch/20_phased_diff_browser_docs_closeout_plan.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/20_phased_diff_browser_docs_closeout_plan.md index 2af4d0a6..b22b10d1 100644 --- a/devlog/_plan/260514_insane_search_adaptive_fetch/20_phased_diff_browser_docs_closeout_plan.md +++ b/devlog/_fin/260514_insane_search_adaptive_fetch/20_phased_diff_browser_docs_closeout_plan.md @@ -164,7 +164,7 @@ cookies unless explicitly requested. npm test -- browser-fetch-command bash structure/verify-counts.sh bash structure/check-doc-drift.sh -git diff --check HEAD -- skills/browser README.md structure devlog/_plan/260514_insane_search_adaptive_fetch +git diff --check HEAD -- skills/browser README.md structure devlog/_fin/260514_insane_search_adaptive_fetch ``` ## Phase 07 — Integrated Gates And Mirror Readiness @@ -174,9 +174,9 @@ git diff --check HEAD -- skills/browser README.md structure devlog/_plan/260514_ MODIFY: ```text -devlog/_plan/260514_insane_search_adaptive_fetch/README.md -devlog/_plan/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md -devlog/_plan/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md +devlog/_fin/260514_insane_search_adaptive_fetch/README.md +devlog/_fin/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md +devlog/_fin/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md ``` Optional new closeout note: diff --git a/devlog/_plan/260514_insane_search_adaptive_fetch/README.md b/devlog/_fin/260514_insane_search_adaptive_fetch/README.md similarity index 100% rename from devlog/_plan/260514_insane_search_adaptive_fetch/README.md rename to devlog/_fin/260514_insane_search_adaptive_fetch/README.md diff --git a/devlog/_fin/260515_adaptive_fetch_v1.md b/devlog/_fin/260515_adaptive_fetch_v1.md index 11bc943f..97c08717 100644 --- a/devlog/_fin/260515_adaptive_fetch_v1.md +++ b/devlog/_fin/260515_adaptive_fetch_v1.md @@ -45,9 +45,9 @@ Updated source-of-truth surfaces: - `skills/browser/SKILL.md` - `structure/commands.md` - `structure/CAPABILITY_TRUTH_TABLE.md` -- `devlog/_plan/260514_insane_search_adaptive_fetch/README.md` -- `devlog/_plan/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md` -- `devlog/_plan/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md` +- `devlog/_fin/260514_insane_search_adaptive_fetch/README.md` +- `devlog/_fin/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md` +- `devlog/_fin/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md` The documented product boundary is: diff --git a/devlog/_plan/260515_adaptive_fetch_v2/00_index.md b/devlog/_fin/260515_adaptive_fetch_v2/00_index.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/00_index.md rename to devlog/_fin/260515_adaptive_fetch_v2/00_index.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/01_principles.md b/devlog/_fin/260515_adaptive_fetch_v2/01_principles.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/01_principles.md rename to devlog/_fin/260515_adaptive_fetch_v2/01_principles.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/02_escalation_ladder.md b/devlog/_fin/260515_adaptive_fetch_v2/02_escalation_ladder.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/02_escalation_ladder.md rename to devlog/_fin/260515_adaptive_fetch_v2/02_escalation_ladder.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/03_session_identity.md b/devlog/_fin/260515_adaptive_fetch_v2/03_session_identity.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/03_session_identity.md rename to devlog/_fin/260515_adaptive_fetch_v2/03_session_identity.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/04_challenge_system.md b/devlog/_fin/260515_adaptive_fetch_v2/04_challenge_system.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/04_challenge_system.md rename to devlog/_fin/260515_adaptive_fetch_v2/04_challenge_system.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/05_content_scoring.md b/devlog/_fin/260515_adaptive_fetch_v2/05_content_scoring.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/05_content_scoring.md rename to devlog/_fin/260515_adaptive_fetch_v2/05_content_scoring.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/06_cli_result_schema.md b/devlog/_fin/260515_adaptive_fetch_v2/06_cli_result_schema.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/06_cli_result_schema.md rename to devlog/_fin/260515_adaptive_fetch_v2/06_cli_result_schema.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/07_vs_insane_search.md b/devlog/_fin/260515_adaptive_fetch_v2/07_vs_insane_search.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/07_vs_insane_search.md rename to devlog/_fin/260515_adaptive_fetch_v2/07_vs_insane_search.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/08_safety_model.md b/devlog/_fin/260515_adaptive_fetch_v2/08_safety_model.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/08_safety_model.md rename to devlog/_fin/260515_adaptive_fetch_v2/08_safety_model.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/09_file_map.md b/devlog/_fin/260515_adaptive_fetch_v2/09_file_map.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/09_file_map.md rename to devlog/_fin/260515_adaptive_fetch_v2/09_file_map.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/10_phase_01_core_safety.md b/devlog/_fin/260515_adaptive_fetch_v2/10_phase_01_core_safety.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/10_phase_01_core_safety.md rename to devlog/_fin/260515_adaptive_fetch_v2/10_phase_01_core_safety.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/11_phase_02_http_endpoints.md b/devlog/_fin/260515_adaptive_fetch_v2/11_phase_02_http_endpoints.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/11_phase_02_http_endpoints.md rename to devlog/_fin/260515_adaptive_fetch_v2/11_phase_02_http_endpoints.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/12_phase_03_reader_scorer.md b/devlog/_fin/260515_adaptive_fetch_v2/12_phase_03_reader_scorer.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/12_phase_03_reader_scorer.md rename to devlog/_fin/260515_adaptive_fetch_v2/12_phase_03_reader_scorer.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/13_phase_04_challenge_waf.md b/devlog/_fin/260515_adaptive_fetch_v2/13_phase_04_challenge_waf.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/13_phase_04_challenge_waf.md rename to devlog/_fin/260515_adaptive_fetch_v2/13_phase_04_challenge_waf.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/14_phase_05_browser_isolated.md b/devlog/_fin/260515_adaptive_fetch_v2/14_phase_05_browser_isolated.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/14_phase_05_browser_isolated.md rename to devlog/_fin/260515_adaptive_fetch_v2/14_phase_05_browser_isolated.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/15_phase_06_user_session_human.md b/devlog/_fin/260515_adaptive_fetch_v2/15_phase_06_user_session_human.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/15_phase_06_user_session_human.md rename to devlog/_fin/260515_adaptive_fetch_v2/15_phase_06_user_session_human.md diff --git a/devlog/_plan/260515_adaptive_fetch_v2/16_phase_07_cli_docs_gates.md b/devlog/_fin/260515_adaptive_fetch_v2/16_phase_07_cli_docs_gates.md similarity index 100% rename from devlog/_plan/260515_adaptive_fetch_v2/16_phase_07_cli_docs_gates.md rename to devlog/_fin/260515_adaptive_fetch_v2/16_phase_07_cli_docs_gates.md diff --git a/devlog/_plan/260525_shared_web_ai_target_lock/00_overview.md b/devlog/_fin/260525_shared_web_ai_target_lock/00_overview.md similarity index 100% rename from devlog/_plan/260525_shared_web_ai_target_lock/00_overview.md rename to devlog/_fin/260525_shared_web_ai_target_lock/00_overview.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/00_overview.md b/devlog/_fin/260528_runway_mcp_parity_expansion/00_overview.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/00_overview.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/00_overview.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/01_mcp_full_dissection.md b/devlog/_fin/260528_runway_mcp_parity_expansion/01_mcp_full_dissection.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/01_mcp_full_dissection.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/01_mcp_full_dissection.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/02_agbrowse_current_surface.md b/devlog/_fin/260528_runway_mcp_parity_expansion/02_agbrowse_current_surface.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/02_agbrowse_current_surface.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/02_agbrowse_current_surface.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/03_gap_analysis.md b/devlog/_fin/260528_runway_mcp_parity_expansion/03_gap_analysis.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/03_gap_analysis.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/03_gap_analysis.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/04_args_expansion_plan.md b/devlog/_fin/260528_runway_mcp_parity_expansion/04_args_expansion_plan.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/04_args_expansion_plan.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/04_args_expansion_plan.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/05_diff_level_implementation.md b/devlog/_fin/260528_runway_mcp_parity_expansion/05_diff_level_implementation.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/05_diff_level_implementation.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/05_diff_level_implementation.md diff --git a/devlog/_plan/260528_runway_mcp_parity_expansion/06_verification_plan.md b/devlog/_fin/260528_runway_mcp_parity_expansion/06_verification_plan.md similarity index 100% rename from devlog/_plan/260528_runway_mcp_parity_expansion/06_verification_plan.md rename to devlog/_fin/260528_runway_mcp_parity_expansion/06_verification_plan.md diff --git a/devlog/_plan/260610_defuddle_reader/00_plan.md b/devlog/_fin/260610_defuddle_reader/00_plan.md similarity index 100% rename from devlog/_plan/260610_defuddle_reader/00_plan.md rename to devlog/_fin/260610_defuddle_reader/00_plan.md diff --git a/devlog/_plan/260611_background_runtime_hook/00_plan.md b/devlog/_fin/260611_background_runtime_hook/00_plan.md similarity index 90% rename from devlog/_plan/260611_background_runtime_hook/00_plan.md rename to devlog/_fin/260611_background_runtime_hook/00_plan.md index 135b28a3..cde20b23 100644 --- a/devlog/_plan/260611_background_runtime_hook/00_plan.md +++ b/devlog/_fin/260611_background_runtime_hook/00_plan.md @@ -72,10 +72,10 @@ All research-only — no source code modifications. | Path | Purpose | |------|---------| -| `devlog/_plan/260611_background_runtime_hook/00_plan.md` | This plan (you're reading it) | -| `devlog/_plan/260611_background_runtime_hook/01_runtime_survey.md` | Task 1 결과: 런타임별 비교표 | -| `devlog/_plan/260611_background_runtime_hook/02_agbrowse_sufficiency.md` | Task 2 결과: agbrowse 현재 구현 평가 | -| `devlog/_plan/260611_background_runtime_hook/03_hook_design.md` | Task 3 결과: cli-jaw background hook 설계 초안 | +| `devlog/_fin/260611_background_runtime_hook/00_plan.md` | This plan (you're reading it) | +| `devlog/_fin/260611_background_runtime_hook/01_runtime_survey.md` | Task 1 결과: 런타임별 비교표 | +| `devlog/_fin/260611_background_runtime_hook/02_agbrowse_sufficiency.md` | Task 2 결과: agbrowse 현재 구현 평가 | +| `devlog/_fin/260611_background_runtime_hook/03_hook_design.md` | Task 3 결과: cli-jaw background hook 설계 초안 | ### MODIFY files diff --git a/devlog/_plan/260611_background_runtime_hook/01_runtime_survey.md b/devlog/_fin/260611_background_runtime_hook/01_runtime_survey.md similarity index 100% rename from devlog/_plan/260611_background_runtime_hook/01_runtime_survey.md rename to devlog/_fin/260611_background_runtime_hook/01_runtime_survey.md diff --git a/devlog/_plan/260611_background_runtime_hook/02_agbrowse_sufficiency.md b/devlog/_fin/260611_background_runtime_hook/02_agbrowse_sufficiency.md similarity index 100% rename from devlog/_plan/260611_background_runtime_hook/02_agbrowse_sufficiency.md rename to devlog/_fin/260611_background_runtime_hook/02_agbrowse_sufficiency.md diff --git a/devlog/_plan/260611_background_runtime_hook/03_hook_design.md b/devlog/_fin/260611_background_runtime_hook/03_hook_design.md similarity index 100% rename from devlog/_plan/260611_background_runtime_hook/03_hook_design.md rename to devlog/_fin/260611_background_runtime_hook/03_hook_design.md diff --git a/devlog/_plan/260611_code_mode_gpt_agent_context/00_plan.md b/devlog/_fin/260611_code_mode_gpt_agent_context/00_plan.md similarity index 99% rename from devlog/_plan/260611_code_mode_gpt_agent_context/00_plan.md rename to devlog/_fin/260611_code_mode_gpt_agent_context/00_plan.md index cac6259f..50645fce 100644 --- a/devlog/_plan/260611_code_mode_gpt_agent_context/00_plan.md +++ b/devlog/_fin/260611_code_mode_gpt_agent_context/00_plan.md @@ -63,7 +63,7 @@ agbrowse/ │ ├── web-ai-code-mode-prompt.test.mjs │ ├── web-ai-code-mode.test.mjs │ └── web-ai-code-dev-context.test.mjs NEW -└── devlog/_plan/260611_code_mode_gpt_agent_context/ +└── devlog/_fin/260611_code_mode_gpt_agent_context/ └── 00_plan.md cli-jaw/ diff --git a/devlog/_plan/260611_docs_pages_overhaul/00_plan.md b/devlog/_fin/260611_docs_pages_overhaul/00_plan.md similarity index 100% rename from devlog/_plan/260611_docs_pages_overhaul/00_plan.md rename to devlog/_fin/260611_docs_pages_overhaul/00_plan.md diff --git a/devlog/_plan/260611_docs_pages_overhaul/20_final_goal_audit.md b/devlog/_fin/260611_docs_pages_overhaul/20_final_goal_audit.md similarity index 100% rename from devlog/_plan/260611_docs_pages_overhaul/20_final_goal_audit.md rename to devlog/_fin/260611_docs_pages_overhaul/20_final_goal_audit.md diff --git a/devlog/_plan/260611_webai_gpt_code_mode/00_plan.md b/devlog/_fin/260611_webai_gpt_code_mode/00_plan.md similarity index 100% rename from devlog/_plan/260611_webai_gpt_code_mode/00_plan.md rename to devlog/_fin/260611_webai_gpt_code_mode/00_plan.md diff --git a/devlog/_plan/260611_webai_gpt_code_mode/01_prompt_contract.md b/devlog/_fin/260611_webai_gpt_code_mode/01_prompt_contract.md similarity index 100% rename from devlog/_plan/260611_webai_gpt_code_mode/01_prompt_contract.md rename to devlog/_fin/260611_webai_gpt_code_mode/01_prompt_contract.md diff --git a/devlog/_plan/260611_webai_gpt_code_mode/probe_js_retrieval.md b/devlog/_fin/260611_webai_gpt_code_mode/probe_js_retrieval.md similarity index 100% rename from devlog/_plan/260611_webai_gpt_code_mode/probe_js_retrieval.md rename to devlog/_fin/260611_webai_gpt_code_mode/probe_js_retrieval.md diff --git a/devlog/_plan/260611_webai_multi_attach/00_plan.md b/devlog/_fin/260611_webai_multi_attach/00_plan.md similarity index 100% rename from devlog/_plan/260611_webai_multi_attach/00_plan.md rename to devlog/_fin/260611_webai_multi_attach/00_plan.md diff --git a/devlog/_plan/260611_webai_skill_cli_jaw_mirror/00_plan.md b/devlog/_fin/260611_webai_skill_cli_jaw_mirror/00_plan.md similarity index 100% rename from devlog/_plan/260611_webai_skill_cli_jaw_mirror/00_plan.md rename to devlog/_fin/260611_webai_skill_cli_jaw_mirror/00_plan.md diff --git a/devlog/_plan/260615_chatgpt_composer_tools_live_probe.md b/devlog/_fin/260615_chatgpt_composer_tools_live_probe.md similarity index 100% rename from devlog/_plan/260615_chatgpt_composer_tools_live_probe.md rename to devlog/_fin/260615_chatgpt_composer_tools_live_probe.md diff --git a/devlog/_fin/260617_computer_use_contract_hardening/00_browser_alignment.md b/devlog/_fin/260617_computer_use_contract_hardening/00_browser_alignment.md new file mode 100644 index 00000000..dcf829d6 --- /dev/null +++ b/devlog/_fin/260617_computer_use_contract_hardening/00_browser_alignment.md @@ -0,0 +1,111 @@ +# Browser Computer Control Alignment + +Date: 2026-06-17 +Status: P-phase plan + +## Objective + +Align agbrowse browser control with the shared computer-use contract while preserving its existing ref-first design. + +Shared master spec: + +- `/Users/jun/Developer/codex/23_computer_use/devlog/_plan/260617_computer_use_contract_hardening/00_shared_contract_spec.md` + +## Scope + +Target repository: + +- `/Users/jun/Developer/new/700_projects/agbrowse` + +Primary surfaces: + +- `agbrowse snapshot --interactive` +- `agbrowse observe-bundle --screenshot --boxes --json` +- `agbrowse observe-actions --json` +- `agbrowse-vision-click` +- `agbrowse mouse-click` + +## Current State + +agbrowse already has the right top-level preference order: + +1. accessibility snapshot refs +2. locator/ref actions +3. observe-bundle boxes +4. screenshot/coordinate fallback +5. vision-click for no-ref targets + +The hardening work should keep that order and strengthen the fallback path. + +## Contract Mapping + +### Observation + +Existing equivalents: + +- `snapshot --interactive` -> refs +- `observe-bundle` -> refs, boxes, optional screenshot, viewport, DPR +- `screenshot --json` -> screenshot path, viewport, DPR, optional clip + +Required additions: + +- stable observation/capture id inside vision-click flow +- explicit target id/url freshness check for coordinate fallback +- screenshot clip metadata preserved through candidate resolution + +### Candidate + +Existing equivalents: + +- snapshot ref +- observe-action candidate +- vision-click coordinate result + +Required additions: + +- `vision_bbox` candidate with bbox, point, confidence, and reason +- candidate can reconcile to a nearby ref when possible +- low confidence becomes a failure, not a click + +### Action + +Existing actions: + +- `click ` +- `mouse-click ` + +Required policy: + +- prefer ref click +- use coordinate click only when no usable ref exists +- require vision bbox verification before coordinate click + +### Verification + +Existing verification: + +- optional verify crop in vision-click +- post-click snapshot best effort + +Required additions: + +- pre-click viewport/clip bounds validation +- bbox center must remain inside verified crop +- optional `elementFromPoint` or nearby-ref reconciliation before click +- post-click snapshot or caller-provided assertion for risky actions + +## Non-Goals + +- Do not replace Playwright locator/ref actions. +- Do not make vision-click the default browser action path. +- Do not add browser-level desktop permissions. +- Do not modify JWC or jawcode. + +## Open Design Defaults + +Adopt these defaults unless audit finds a blocker: + +- bbox confidence threshold: 0.75 for normal targets +- require verify crop for coordinate fallback in dense UI +- reject ambiguous target descriptions instead of clicking +- keep existing `--clip` and `--region` flags, but preserve their coordinate basis explicitly diff --git a/devlog/_fin/260617_computer_use_contract_hardening/10_browser_phase_backlog.md b/devlog/_fin/260617_computer_use_contract_hardening/10_browser_phase_backlog.md new file mode 100644 index 00000000..566804e4 --- /dev/null +++ b/devlog/_fin/260617_computer_use_contract_hardening/10_browser_phase_backlog.md @@ -0,0 +1,137 @@ +# Browser Computer Control Phase Backlog + +Date: 2026-06-17 +Status: P-phase plan + +## Scope + +Target repository: + +- `/Users/jun/Developer/new/700_projects/agbrowse` + +Companion desktop plan: + +- `/Users/jun/Developer/codex/23_computer_use/devlog/_plan/260617_computer_use_contract_hardening/10_desktop_phase_backlog.md` + +## Phase 1: Vision Candidate Contract + +### Planned Files + +Modify: + +- `/Users/jun/Developer/new/700_projects/agbrowse/skills/vision-click/vision-core.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/skills/vision-click/vision-click.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/skills/vision-click/SKILL.md` +- `/Users/jun/Developer/new/700_projects/agbrowse/README.md` + +Add: + +- `/Users/jun/Developer/new/700_projects/agbrowse/test/fixtures/vision-candidates.json` +- `/Users/jun/Developer/new/700_projects/agbrowse/test/fixtures/browser-dpr-clip.json` + +### Required Changes + +- Change vision output contract from point-only to bbox candidate plus confidence. +- Keep backward-compatible parsing only if needed, but mark point-only as lower confidence. +- Validate bbox and click point are finite and inside viewport or clip. +- Return structured failure when target is not found, low confidence, or ambiguous. + +### Verification + +- `vitest run test/unit/vision-core.test.mjs` +- new fixture tests for bbox/confidence parsing + +## Phase 2: Observation-Bundle Reconciliation + +### Planned Files + +Modify: + +- `/Users/jun/Developer/new/700_projects/agbrowse/skills/browser/browser.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/web-ai/observation-bundle.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/test/unit/g06-observation-bundle.test.mjs` + +Add if needed: + +- `/Users/jun/Developer/new/700_projects/agbrowse/web-ai/candidate-reconcile.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/test/unit/candidate-reconcile.test.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/test/fixtures/browser-observation-stale.json` +- `/Users/jun/Developer/new/700_projects/agbrowse/test/fixtures/browser-ref-vs-coordinate.json` + +### Required Changes + +- Reconcile vision bbox with nearby snapshot refs when boxes are available. +- Prefer ref click if reconciliation is strong. +- Preserve observation target id and url in coordinate fallback evidence. +- Reject stale ref/target mismatch before coordinate fallback. +- Map shared browser fixture cases to tests: + - `browser-ref-vs-coordinate.json` -> ref candidate beats coordinate candidate + - `browser-observation-stale.json` -> target id or URL mismatch rejects stale fallback + - `browser-dpr-clip.json` -> DPR plus clip origin produces correct CSS click point + +### Verification + +- observation bundle fixture tests +- candidate reconciliation fixture tests +- `vitest run test/unit/candidate-reconcile.test.mjs test/unit/g06-observation-bundle.test.mjs` + +## Phase 3: Safer Coordinate Fallback + +### Planned Files + +Modify: + +- `/Users/jun/Developer/new/700_projects/agbrowse/skills/vision-click/vision-click.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/skills/browser/browser.mjs` +- `/Users/jun/Developer/new/700_projects/agbrowse/structure/commands.md` + +### Required Changes + +- Require verify crop by default for low/medium confidence coordinate fallbacks. +- Preserve clip origin and DPR in final clicked evidence. +- Add explicit failure envelopes for stale observation, invalid coordinate, low confidence, and ambiguous target. +- Document ref-first and coordinate-last routing. + +### Verification + +- unit tests for coordinate conversion and clip handling +- focused browser primitive test if parser/help changes +- `npm run docs:drift` + +## Phase 4: Real Browser Smoke + +### Planned Files + +Add: + +- `/Users/jun/Developer/new/700_projects/agbrowse/devlog/_fin/260617_computer_use_contract_hardening/20_verification_report.md` + +### Smoke Cases + +- accessible button: snapshot ref click, no vision fallback +- no-ref target: vision bbox -> verify crop -> coordinate click +- ambiguous target: reject without click +- DPR/clip target: final CSS coordinate evidence is correct + +### Verification + +- record exact local commands +- record browser state preconditions +- no fake pass for unavailable Chrome/CDP state + +## Release Documentation Impact + +If implementation changes public command behavior: + +- update `/Users/jun/Developer/new/700_projects/agbrowse/README.md` +- update `/Users/jun/Developer/new/700_projects/agbrowse/skills/vision-click/SKILL.md` +- update `/Users/jun/Developer/new/700_projects/agbrowse/structure/commands.md` +- run `npm run docs:drift` + +## Implementation Readiness + +Ready for Build only after: + +- desktop plan and browser plan are audited together +- shared fixture schema is accepted as the source of contract tests +- no additional user decision is required diff --git a/devlog/_fin/260617_computer_use_contract_hardening/20_verification_report.md b/devlog/_fin/260617_computer_use_contract_hardening/20_verification_report.md new file mode 100644 index 00000000..db06b073 --- /dev/null +++ b/devlog/_fin/260617_computer_use_contract_hardening/20_verification_report.md @@ -0,0 +1,114 @@ +# Browser Vision Upgrade Verification Report + +Date: 2026-06-17 +Status: B-phase implementation verification +Branch: `dev-vision-upgrade` + +## Implementation Commit + +- `/Users/jun/Developer/new/700_projects/agbrowse`: `3392adc feat: harden vision coordinate fallback` +- `/Users/jun/Developer/new/700_projects/agbrowse`: `2b03e74 fix: wire vision candidate reconciliation` +- `/Users/jun/Developer/new/700_projects/agbrowse`: `f7e8cd0 refactor: split vision candidate helpers` + +## Implemented Scope + +- Added `vision-candidate-v1` parsing with bbox, point, confidence, and risk flags. +- Kept legacy `{found,x,y}` parsing but marks it as point-only and lower confidence. +- Added candidate validation for finite bbox/point and viewport/clip bounds. +- Added `candidate-reconcile.mjs` for bbox-to-ref reconciliation. +- Extended ObservationBundleV1 with `observationId`, `targetId`, and `basis`. +- Wired `--bundle` into `agbrowse-vision-click` so a vision bbox can prefer a matching ref, fail on ambiguous refs, or fall back to coordinates. +- Added stale ObservationBundle rejection before coordinate fallback. +- Added `url` and `targetId` to `screenshot --json` so freshness checks use the same basis as `observe-bundle`. +- Updated docs for ref-first, coordinate-last browser control. +- Added focused fixtures and unit tests. +- Split vision candidate parsing/validation helpers into `skills/vision-click/vision-candidate.mjs` so touched vision-click modules remain under 500 lines. + +## Commands Run + +From `/Users/jun/Developer/new/700_projects/agbrowse`: + +```bash +git branch --show-current +``` + +Result: + +```text +dev-vision-upgrade +``` + +```bash +npx vitest run test/unit/vision-core.test.mjs test/unit/candidate-reconcile.test.mjs test/unit/g06-observation-bundle.test.mjs +``` + +Result: + +```text +PASS: 3 files, 36 tests, 0 failures +``` + +```bash +npm run typecheck:checkjs +``` + +Result: + +```text +PASS +``` + +```bash +npm run docs:drift +``` + +Result: + +```text +PASS: 144 checks +``` + +```bash +git diff --check +``` + +Result: + +```text +PASS +``` + +```bash +wc -l skills/vision-click/vision-core.mjs skills/vision-click/vision-candidate.mjs skills/vision-click/vision-click.mjs web-ai/candidate-reconcile.mjs +``` + +Result: + +```text +PASS: touched vision-click modules are <= 500 lines (`vision-core.mjs` 341, `vision-candidate.mjs` 249, `vision-click.mjs` 434, `candidate-reconcile.mjs` 74) +``` + +## Real Browser Smoke + +Run on 2026-06-17 with linked global `agbrowse` binaries. + +Setup: + +- `npm ls -g --depth=0 agbrowse` shows `agbrowse@0.1.14 -> /Users/jun/Developer/new/700_projects/agbrowse`. +- `agbrowse start` launched Chrome on CDP `9222`. +- Test page was a data URL with one accessible button and one canvas-only green `CANVAS` target. + +Smoke results: + +- `agbrowse screenshot --json` returned `url`, `targetId: cdp:9222`, `dpr: 2`, viewport, and path; `sips` confirmed screenshot pixels were `2880x1604` for viewport `1440x802`. +- `agbrowse observe-bundle --screenshot --boxes --json` initially exposed a bug: snapshot refs were `eN`, while `ObservationBundle` kept only `@eN`, producing `refs: []`. +- After patch, `observe-bundle` preserved `e2` and captured its box: `{ x: 71, y: 119, width: 222, height: 67 }`. +- Stale bundle check failed closed before click with `COMPUTER_OBSERVATION_STALE: observation URL does not match current page`. +- Canvas no-ref fallback succeeded: `agbrowse-vision-click "green CANVAS rectangle" --clip 300 90 500 330 --verify-before-click` clicked `(515,224)`, and `agbrowse text` changed to `canvas-clicked`. +- Accessible button with bundle initially showed a second bug: reconciliation compared raw image pixels to CSS boxes and ran before verification, so a full-page candidate missed the ref box and coordinate fallback did not click the button. +- After patch, bundle click succeeded: `agbrowse-vision-click "Blue Test Button" --bundle /tmp/agbrowse-vision-bundle-fixed2.json` verified the crop, re-reconciled in CSS space, used the ref path, and `agbrowse text` changed to `blue-clicked`. + +Smoke limitations: + +- `agbrowse evaluate` was denied by policy, so DOM state was checked via `agbrowse text`. +- Ambiguous target live smoke remains future work; ambiguity behavior is covered by `test/unit/candidate-reconcile.test.mjs`. diff --git a/devlog/_fin/260617_computer_use_contract_hardening/30_dev_vision_upgrade_branch_plan.md b/devlog/_fin/260617_computer_use_contract_hardening/30_dev_vision_upgrade_branch_plan.md new file mode 100644 index 00000000..8eb7cb20 --- /dev/null +++ b/devlog/_fin/260617_computer_use_contract_hardening/30_dev_vision_upgrade_branch_plan.md @@ -0,0 +1,225 @@ +# dev-vision-upgrade Implementation Plan + +Date: 2026-06-17 +Status: P-phase implementation plan +Target repo: `/Users/jun/Developer/new/700_projects/agbrowse` +Target branch: `dev-vision-upgrade` + +## Objective + +Implement the browser-side vision upgrade on `dev-vision-upgrade` while preserving agbrowse's ref-first control model. + +The browser implementation should make coordinate fallback safer by turning vision output into a `vision_bbox` candidate with confidence, explicit clip/DPR basis, and fail-closed behavior for low-confidence, stale, or ambiguous targets. + +## Confirmed Inputs + +- Shared desktop/browser contract: + - `/Users/jun/Developer/codex/23_computer_use/devlog/_plan/260617_computer_use_contract_hardening/00_shared_contract_spec.md` +- Browser alignment: + - `00_browser_alignment.md` +- Browser backlog: + - `10_browser_phase_backlog.md` + +## Scope + +Modify only this repository on branch `dev-vision-upgrade`. + +Primary files: + +- `skills/browser/browser.mjs` +- `skills/vision-click/vision-core.mjs` +- `skills/vision-click/vision-click.mjs` +- `skills/vision-click/SKILL.md` +- `web-ai/observation-bundle.mjs` +- new `web-ai/candidate-reconcile.mjs` +- `README.md` +- `structure/commands.md` +- targeted tests and fixtures + +Do not modify: + +- JWC +- `/Users/jun/Developer/new/700_projects/jawcode/devlog` +- default ref/locator click path except where docs need to say coordinate fallback is last + +## Diff Plan + +### 1. Vision Candidate Contract + +Modify: + +- `skills/vision-click/vision-core.mjs` + +Add exported helpers: + +- `normalizeVisionCandidate(raw, options?)` +- `extractVisionCandidateJson(text)` +- `validateVisionCandidate(candidate, observation)` +- `candidateCenter(candidate)` +- `isLowConfidence(candidate, threshold = 0.75)` + +Candidate shape: + +```js +{ + schemaVersion: 'vision-candidate-v1', + found: true, + kind: 'vision_bbox', + bbox: { x, y, width, height }, + point: { x, y }, + confidence: 0.0_to_1.0, + description, + reason, + riskFlags: [] +} +``` + +Backward compatibility: + +- existing `{found,x,y,description}` responses still parse +- point-only responses become `kind: "coordinate"` with lower confidence and `riskFlags: ["point_only"]` +- point-only responses require verification before click + +Prompt change: + +- `buildCoordPrompt` asks for bbox + confidence JSON first +- old point JSON is tolerated only as fallback parser input + +### 2. Fail-Closed Vision Click + +Modify: + +- `skills/vision-click/vision-click.mjs` + +Changes: + +- use `extractVisionCandidateJson` +- fail when `found:false` +- fail when confidence `< 0.75` unless a future explicit override is added; do not add override in this phase +- validate bbox/point are finite and inside clip/capture bounds before DPR conversion +- preserve evidence in result: + - raw candidate + - css point + - dpr + - clip + - verification crop + - `verified` +- require verification for point-only candidates and medium-confidence bbox candidates + +### 3. Observation Reconciliation + +Add: + +- `web-ai/candidate-reconcile.mjs` +- `test/unit/candidate-reconcile.test.mjs` + +Modify: + +- `skills/browser/browser.mjs` +- `skills/vision-click/vision-click.mjs` +- `web-ai/observation-bundle.mjs` + +Function: + +- `reconcileVisionCandidate({ candidate, bundle, maxDistance = 32 })` + +Behavior: + +- if candidate center falls inside a ref box, return `{ action: "ref", ref, reason }` +- if candidate center is near exactly one ref box, return that ref +- if multiple nearby refs compete, return ambiguous failure +- otherwise return coordinate fallback with reason + +Observation-bundle changes: + +- add optional `observationId` +- add optional `targetId` +- keep existing output compatible +- include `basis` summary with `url`, `viewport`, `dpr`, `capturedAt` + +Wiring requirement: + +- `browser.mjs` must expose or reuse the reconciliation path wherever an observation bundle with boxes is available. +- `vision-click.mjs` must preserve candidate evidence and use `reconcileVisionCandidate` when a bundle is supplied or produced by the command path. +- If no bundle is available, `vision-click.mjs` must explicitly record `reconciliation: "unavailable"` and continue only through the stricter bbox/confidence/verification gate. + +### 4. Fixtures and Tests + +Add: + +- `test/fixtures/vision-candidates.json` +- `test/fixtures/browser-dpr-clip.json` +- `test/fixtures/browser-observation-stale.json` +- `test/fixtures/browser-ref-vs-coordinate.json` + +Modify: + +- `test/unit/vision-core.test.mjs` +- `test/unit/g06-observation-bundle.test.mjs` +- `tsconfig.checkjs.json` + +Coverage: + +- bbox candidate parses from JSON +- point-only candidate is backward-compatible but marked risky +- low confidence rejects +- invalid bbox rejects +- DPR + clip converts to final CSS point +- ref candidate beats coordinate candidate +- ambiguous nearby refs fail +- observation basis carries URL/target/viewport/DPR evidence + +Typecheck coverage: + +- add `web-ai/observation-bundle.mjs` and `web-ai/candidate-reconcile.mjs` to `tsconfig.checkjs.json`, or replace the checkjs gate with a documented equivalent that actually covers those files. +- Preferred path: update `tsconfig.checkjs.json` so `npm run typecheck:checkjs` validates the new browser-side contract module. + +### 5. Docs + +Modify: + +- `skills/vision-click/SKILL.md` +- `README.md` +- `structure/commands.md` + +Docs changes: + +- document ref-first, coordinate-last policy +- document bbox/confidence candidate contract +- document `--verify-before-click` as required for point-only/medium-confidence fallback +- keep public command names unchanged + +### 6. Verification Report + +Add after implementation: + +- `devlog/_fin/260617_computer_use_contract_hardening/20_verification_report.md` + +Must include: + +- exact commands run +- branch name +- skipped real browser smoke reason if CDP/Chrome is unavailable +- audit result + +## Verification Commands + +Run from `/Users/jun/Developer/new/700_projects/agbrowse`: + +```bash +git branch --show-current +npx vitest run test/unit/vision-core.test.mjs test/unit/candidate-reconcile.test.mjs test/unit/g06-observation-bundle.test.mjs +npm run docs:drift +npm run typecheck:checkjs +git diff --check +``` + +## Commit Plan + +Commit on `dev-vision-upgrade` only: + +```text +feat: harden vision coordinate fallback +``` + +Do not push. diff --git a/devlog/_fin/260619_tab_parallel_stability/00_overview.md b/devlog/_fin/260619_tab_parallel_stability/00_overview.md new file mode 100644 index 00000000..79b06404 --- /dev/null +++ b/devlog/_fin/260619_tab_parallel_stability/00_overview.md @@ -0,0 +1,45 @@ +# Tab Parallel Stability + +## Problem +Multiple agents sharing one CDP instance with tab-per-session model has race conditions across tab recovery, lease binding, and concurrent send/poll operations. + +## Key Gaps + +### TOCTOU in tab recovery +- `tab-recovery.mjs:34-68` — `isTabAlive()` check and `getPageByTargetId()` are not atomic +- Tab can close between check and use + +### Lease-session binding race +- `chatgpt.mjs:200-201` — `bindSessionToTab()` (sync) and `recordActiveLease()` (async) are separate +- Watcher can start polling before lease is recorded -> reattach-mismatch + +### No send+poll mutex +- `sendWebAi()` and `pollWebAi()` can race on same session's `conversationUrl` +- `withSessionCommandLock` only partially applied + +### Crashed session lease leak +- `tab-lease-store.mjs:227` — active leases have no TTL +- Crashed sessions lock tabs forever, manual cleanup required + +## Goal +Stable multi-agent CDP sharing: multiple jaw employees or CLI agents connect to one Chrome instance, each with its own tab, no reattach-mismatch, no lease leaks. + +## Test Coverage Needed +- Concurrent send+poll on same session (mutex correctness) +- Lease acquisition under high contention (3+ agents) +- Tab recovery during active polling +- Watcher reattach during mid-flight send +- Lease TTL expiration for crashed sessions + +## Decision (locked 2026-06-19) +Concurrency target **vendor ≤ 5, global 12–16** (today 3 / 8). Investigated by a CLI sub-agent AND the Backend employee independently — both converged on the same 4 root causes. **Prerequisite:** unify the 3 colliding TTLs (session lock / active-command / active-lease) on the pro=3600s model-aware deadline from `260619_timeout_adaptive_scaling` + 60s heartbeat **before** raising caps. See `01_root_cause.md`, `10_solution_plan.md` (§C sequencing). + +## Pressure-test (2026-06-20) — DOWNSCOPE & split +Separate-process model + existing `active-command.target-owned` cross-process mutex kill claims 1 & 3; claim 5 out-of-scope. Must-have = record-before-bind reorder + active-count cap (revert `maxTabs:Infinity`) + PID reaper + doc fix. See `20_pressure_test_verdict.md`. **This is the one area with real, load-bearing fixes for the multi-agent goal.** + +## Status +- [x] Interview/requirements gathering +- [x] Plan (`10_solution_plan.md`) — superseded for scope +- [x] Pressure-test (`20_pressure_test_verdict.md`) +- [ ] Implementation (MVV must-have set, if approved) +- [ ] Verification diff --git a/devlog/_fin/260619_tab_parallel_stability/01_root_cause.md b/devlog/_fin/260619_tab_parallel_stability/01_root_cause.md new file mode 100644 index 00000000..a7674070 --- /dev/null +++ b/devlog/_fin/260619_tab_parallel_stability/01_root_cause.md @@ -0,0 +1,85 @@ +# Tab Parallel Stability — Root Cause (verified) + +> All paths under `web-ai/` unless noted. Two independent investigations (CLI sub-agent + Backend employee) reached the same four root causes; both are folded in. Line numbers observed 2026-06-19. + +## Pool defaults today + +`maxPerKey=3` (vendor cap), `globalMax=8`, TTL `30m` (`tab-lease-store.mjs:64-66`). Note: help text in `cli.mjs:213` and `skills/browser/browser.mjs:3370` says "TTL=15m" — **stale doc**, actual default is 30m. + +## Gap 1 — TOCTOU in tab recovery + +`tab-recovery.mjs:34-68` — `recoverSessionTab` checks liveness, then *separately* fetches the page; the tab can close in the gap: + +```js +const alive = await isTabAlive(port, session.targetId); // L34 +if (alive) { + const page = await getPageByTargetId(port, session.targetId); // L38 + if (page) { + const currentUrl = page.url(); // L41 — can throw "Target closed" + ... + await page.goto(targetUrl, { waitUntil: 'load', timeout: 30_000 }); // L51 — can throw +``` + +The same check→use pattern recurs in `verifySessionTab` (`:121-131`) and `resolveSessionPage` (`:294,:311`). `withSessionPage` (`:391-402`) only retries on `isPageDeathError` — a **string-match** classifier (`:179-188`, matches `'target closed'|'page closed'|'crash'`) — so any death that doesn't stringify to those tokens escapes as a hard error. The Backend employee confirmed the identical pattern in `skills/browser/tab-manager.mjs:395-401,427-444`. + +## Gap 2 — Lease↔session binding is non-atomic (sync bind + async lease) + +`chatgpt.mjs:200-209`: + +```js +if (targetId) bindSessionToTab(session.sessionId, targetId); // L200 — synchronous session-store write +if (targetId) await recordActiveLease({ ... targetId, url: page.url(), port }); // L201 — separate awaited fs-locked write +``` + +`bindSessionToTab` (`session.mjs:277`) updates the session immediately; `recordActiveLease` (`tab-lease-store.mjs:221`) takes a **different lock file** (`web-ai-tab-leases.json.lock`) than the session store. Between L200 and the lease being durably written, the session is **bound to a target with no active lease row**. A watcher running in a separate process (`watcher.mjs:114` `watchSessionOnce` → `withSessionPage` → `ensureWatcherAttached` `:398-412`) can read the session in that window; a concurrent `checkoutPooledLease`/`cleanupLeasedTabs` on the same `browserProfileKey` can then treat that target as free → **reattach-mismatch**. Same bug in `deepResearchWebAi` (`chatgpt.mjs:625-633`). The `active-command` row (`cli.mjs:1305-1318`) partially mitigates via `activeCommandTargetIds` (`tab-lease-store.mjs:345,355`) but is keyed on the command's lifetime, not the lease. + +## Gap 3 — No send+poll mutex on the same session + +The lock primitive exists (`session-store.mjs:272` `withSessionCommandLock`) and **poll holds it** (`cli.mjs:1107`), **resume-send holds it** (`cli.mjs:1267`). But it's only taken when `input.session` is set (`cli.mjs:1266`). The **initial** send creates the session *inside* `sendWebAi` and dispatches through the unlocked branch: + +```js +case 'send': return withWebAiActiveCommand(command, deps, input, () => sendWebAi(deps, input)); // cli.mjs:1361 — NO lock +``` + +`sendWebAi` (`chatgpt.mjs:160-321`) never acquires the lock, yet it mutates `conversationUrl` at `:289-291`: + +```js +const finalUrl = page.url(); +if (session && finalUrl !== session.conversationUrl) + updateSession(session.sessionId, { conversationUrl: finalUrl }); // L290 +``` + +while `poll` reads/writes the same field. Two writers on one session's `conversationUrl`, no shared mutex = the classic send/poll race. The watcher is also unprotected: its lock (`watcher.mjs:142`) wraps only the timeout→polling status flip; the actual reattach+poll (`watcher.mjs:176-188`, including `page.goto` at `:404`) runs **outside** any session lock. `queryWebAi` = send→poll with no outer lock (`chatgpt.mjs:587-597`). + +## Gap 4 — Crashed-session lease leak (no TTL on active leases) + +`recordActiveLease` writes `state: 'active-session'` with `leasedAt`/`updatedAt` but **no `poolExpiresAt`, no heartbeat** (`tab-lease-store.mjs:225-230`). Every reclamation path keys off `state === 'pooled'` + `poolExpiresAt`: + +```js +// checkoutPooledLease L254-256 — active-session never selected +.filter(lease => lease.state === 'pooled' && lease.leaseKey === key) +// selectOverflowAndExpired L465-468 — pooled only +const pooled = leases.filter(lease => lease.state === 'pooled'); +``` + +`cleanupLeasedTabs` (`:340-370`) reclaims an `active-session` lease only if the **tab is physically dead** (`!isTabAlive`, `:352`). A **crashed CLI/jaw process** leaves the tab open and the lease intact → the tab is locked forever, permanently consuming a `maxPerKey=3`/`globalMax=8` slot. `releaseCompletedLease` (`:289`) is the only `active-session → pooled` transition and runs only on the happy path. + +## Gap 5 (scaling) — No admission control on *live* concurrency + +`ensureProviderTab` (`cli.mjs:972-1011`) cleans pooled tabs, tries a pooled/reused tab, otherwise **unconditionally `createTab`** (`:1001`, with `maxTabs: Infinity` at `:978`). `globalMax`/`maxPerKey` are consulted **only** inside `selectOverflowAndExpired` over `pooled` rows — they cap the **idle pool, never the count of concurrent active-session leases**. So nothing today queues or rejects the 13th concurrent send; it just opens another tab. This is the gap that must close before scaling to vendor 5 / global 12–16. + +## ⚠️ The cross-cutting risk — three TTLs collide at pro=3600s + +Independently flagged by the Backend employee as the *single biggest risk*. A live session is governed by three TTLs, none aware of pro=3600s: + +| TTL | Value | Source | At pro=3600s | +|-----|-------|--------|--------------| +| Session command lock | 35 min | `session-store.mjs:50` | Expires mid-wait → another command seizes the session | +| Active-command heartbeat | 2 min | `active-command-store.mjs:37` | Expires mid-poll → cleanup stops protecting the tab | +| Active-session lease | none | `tab-lease-store.mjs:221-236` | Crashed session locks the tab forever | + +Fix must unify all three on the **model-aware deadline from `260619_timeout_adaptive_scaling`** + a 60s heartbeat *before* raising caps. See `10_solution_plan.md` §C/§E. + +## Evidence index + +`tab-recovery.mjs` 34-68, 121-131, 179-188, 294, 311, 391-402 · `chatgpt.mjs` 160-321, 190-209, 289-291, 587-597, 625-633 · `tab-lease-store.mjs` 64-66, 221-236, 243-282, 289-333, 340-370, 462-490 · `session-store.mjs` 46-50, 227-234, 262, 272-316 · `cli.mjs` 972-1011, 1101-1128, 1265-1319, 1326-1410 (init send 1361) · `watcher.mjs` 114-188, 398-412 · `session.mjs` 246-266, 277-288 · `active-command-store.mjs` 37 · `skills/browser/tab-manager.mjs` 395-444 · `skills/browser/tab-lifecycle.mjs`. diff --git a/devlog/_fin/260619_tab_parallel_stability/10_solution_plan.md b/devlog/_fin/260619_tab_parallel_stability/10_solution_plan.md new file mode 100644 index 00000000..ed0026b9 --- /dev/null +++ b/devlog/_fin/260619_tab_parallel_stability/10_solution_plan.md @@ -0,0 +1,120 @@ +# Tab Parallel Stability — Solution Plan + +> ⚠️ **SUPERSEDED for scope by `20_pressure_test_verdict.md` (2026-06-20).** Pressure-test: each `web-ai` cmd is a separate OS process and a cross-process tab mutex (`active-command.target-owned`) already exists → claims 1 (TOCTOU) and 3 (send/poll mutex) DROP; claim 5 (3-TTL) out-of-scope. Real must-have = record-before-bind reorder + active-count cap (revert `maxTabs:Infinity`) + PID reaper + doc fix. The full atomic-acquire/heartbeat/admission program below is downscoped. Read the verdict first; this file is the full analysis. + +> Locked decisions: concurrency target **vendor ≤ 5, global 12–16**; **pro = 3600s** is the source of truth for the model-aware deadline. Documentation only; code sketch is illustrative. Design A–E below is the merged output of the CLI sub-agent and the Backend employee (they converged). + +## Sequencing rule (read first) + +Land in this order — caps last: +1. Unify the three TTLs on the model-aware deadline + 60s heartbeat (§C). **Prerequisite for everything.** +2. Atomic lease acquire (§A) + per-session mutex (§B) + TOCTOU fix (§D). +3. Only then raise caps + add admission control (§E). + +Raising caps (§E) *before* §C multiplies the collision rate — do not reorder. + +## A. Atomic `acquire tab + record lease` + +New `acquireActiveLease(port, input)` in `tab-lease-store.mjs`: inside **one** `withLeaseLock`, (a) enforce per-key + global **active** cap, (b) write the `active-session` row with fresh `lastHeartbeat`/`ownerPid`, (c) return `{ ok, lease, reason }`. + +Ordering in `sendWebAi` (replaces `chatgpt.mjs:200-209`), all under the session lock (§B): +1. Resolve/create tab → `targetId`. +2. `acquireActiveLease(...)` — atomic cap-check + row write. If `!ok` → throw typed `provider.capacity` backpressure error (§E). +3. `bindSessionToTab(...)` — strictly **after** the durable lease exists. **Record-before-bind** closes the "bound target with no lease" window (Gap 2). +4. Compose/submit. +5. Finalize → `releaseCompletedLease` demotes `active-session → pooled`. + +Apply the same pattern to `grok-live.mjs`, `gemini-live.mjs`, and `deepResearchWebAi`. **Lock ordering: session-command lock → lease lock (never reverse)** to avoid deadlock; both fs locks have retry-limit timeouts so worst case is a thrown "failed to acquire", not a hang. + +## B. Per-session mutex (send + poll + watcher-reattach) + +The lock primitive is correct (`session-store.mjs:262,272-316`); the fix is *where it's wrapped*: + +| Path | Change | +|------|--------| +| Initial `send`/`query` | **Hoist `sessionId` minting** into `runCommand` before dispatch and wrap `sendWebAi` in `withSessionCommandLock(sessionId, …)`, making `cli.mjs:1361` symmetric with the resume path (`:1267`). (Lighter alt: lock on `tab:${targetId}` — rejected; a *reused pooled* tab could collide two sessions. Prefer the hoist.) | +| `poll` | Already correct (`cli.mjs:1107`). No change. | +| `queryWebAi` | Single outer lock around send+poll (`chatgpt.mjs:587-597`). | +| Watcher | Wrap the reattach+poll body (`watcher.mjs:176-188`) in `withSessionCommandLock(session.sessionId, …)` so a watcher `page.goto` can't fire while a live send/poll holds the session. | +| `sessions reattach` | Add the lock (`cli-sessions.mjs:123`). | + +Result: send, poll, and watcher-reattach serialize on the same session → on the same `conversationUrl`. + +## C. Unify the three TTLs (the load-bearing change) + +Add to the `active-session` lease row: `lastHeartbeat`, `ownerPid`, `expiresAt`, `phase ('send'|'poll'|'watch')`. Drive **all** TTLs from the model-aware deadline imported from `260619_timeout_adaptive_scaling` (`resolveTimeoutSeconds`) — do not re-hardcode 3600 here. + +```js +// hardcoded ceilings derived from the timeout table + 60s slack +const MODEL_LOCK_TTL_MS = { default: 1_260_000 /*21m*/, pro: 3_660_000 /*61m*/, deep: 3_660_000 }; +``` + +- **Session command lock TTL** — pass `{ ttlMs: MODEL_LOCK_TTL_MS[tier], heartbeatMs: 60_000 }`; raise the `DEFAULT_SESSION_COMMAND_LOCK_TTL_MS` ceiling (`session-store.mjs:50`) to 61m. +- **Active-command TTL** — set `ttlMs: MODEL_LOCK_TTL_MS[tier]` in `withWebAiActiveCommand` (`cli.mjs:1308-1314`) so 2-min cleanup stops fighting a live pro poll. +- **Active-lease TTL + heartbeat** — `expiresAt = session.deadlineAt`; renew `lastHeartbeat` every **60s** from the session-lock heartbeat timer (`session-store.mjs:303-308`, already fires q15s — piggyback `touchActiveLease(targetId)`), plus each watcher tick and each poll iteration. + +**Reaper** `reapStaleActiveLeases(port)` (called from `cleanupLeasedTabs`), inside `withLeaseLock`, reclaims an `active-session` lease only when **BOTH**: `now - lastHeartbeat > 180s` (3 missed beats) **AND** `!pidAlive(ownerPid)` (process dead). The pro-safe invariant: + +> A legitimate 3600s pro wait has an old `leasedAt` but a `lastHeartbeat` renewed q60s and a live PID → it is **never** reaped. The reaper keys on heartbeat freshness + process liveness, **not** on `leasedAt`. + +Numbers: heartbeat **60s**, miss threshold **180s**, grace past `expiresAt` **120s**. Even a SIGSTOP-frozen process reads alive to `process.kill(pid,0)` → not reaped (safe), at the cost of holding a slot until truly dead — acceptable. + +## D. TOCTOU fix + +1. **Typed `TabDeadError` + single-flight recovery.** Replace the brittle string-match `isPageDeathError` with a typed error; wrap `page.url()`/first-use in try/catch that throws it. Make recovery **single-flight per `targetId`** (in-process `Map`) so N concurrent ops that see the same tab die trigger exactly one `recoverSessionTab`, not N racing `createTab`s. New `resolvePageForTarget(port, targetId)` helper (3 retries, 100ms backoff) replaces every `isTabAlive → getPageByTargetId` sequence in `recoverSessionTab`/`verifySessionTab`/`resolveSessionPage`. Keep `isTabAlive` only for cheap pool-scan decisions, never for "use this page". +2. **Acquire-and-validate under the session lock** (from §B) serializes resolve→use per session, reducing residual TOCTOU to "tab dies mid-op", which is the recoverable path. Keep the `catch {}` fall-through at `tab-recovery.mjs:63-66` but emit a structured warning so leaks/reattaches are observable. + +## E. Scale to vendor 5 / global 12–16 + +Three layers, in order of necessity: +1. **Env limits (necessary, sufficient for the pool):** `AGBROWSE_PROVIDER_POOL_MAX_PER_KEY=5`, `AGBROWSE_PROVIDER_POOL_GLOBAL_MAX=14` (target 12–16 → pick 14), `AGBROWSE_MAX_TABS=20` (headroom above active+pooled). Fix the stale "TTL=15m" help text. +2. **Admission control on active sessions (the real gap):** `acquireActiveLease` (§A) counts current `active-session` leases per-key and globally and **rejects** when `activeCount >= cap` (`AGBROWSE_PROVIDER_ACTIVE_GLOBAL_MAX=16`, per-vendor 5). Revert `ensureProviderTab`'s `maxTabs: Infinity` to `AGBROWSE_MAX_TABS` after slot acquire. This moves the cap from "pool cleanup" to "admission". +3. **Backpressure / fairness:** return a typed `WebAiError{ errorCode: 'provider.capacity', retryHint: 'retry-after', evidence: { activeCount, cap, retryAfterMs } }`. Prefer **reject-with-retry-after** over unbounded queueing at the 12–16 ceiling (avoids head-of-line blocking a 3600s pro session behind a queue). Optional bounded FIFO per `leaseKey` (≤120s wait) for fairness; keep `vendor ≤ 5` as a hard per-`leaseKey` sub-cap so one vendor can't starve the global pool. jaw/CLI callers must honor `retryAfterMs`. + +## Code sketch (illustrative — NOT for commit) + +```diff +// tab-lease-store.mjs — typedef + acquire/touch/reap ++ // Lease += lastHeartbeat, ownerPid, expiresAt, activeTtlMs (reaper keys on lastHeartbeat, NOT leasedAt) ++ export async function acquireActiveLease(port, input) { return withLeaseLock(() => { ++ const active = readStore().leases.filter(l => l.state === 'active-session'); ++ if (active.filter(l => l.leaseKey === key).length >= ACTIVE_PER_KEY) return { ok:false, reason:'vendor-cap' }; ++ if (active.length >= ACTIVE_GLOBAL) return { ok:false, reason:'global-cap' }; ++ /* write active-session row w/ lastHeartbeat+ownerPid; return { ok:true, lease } */ }); } ++ export async function touchActiveLease(targetId) { /* withLeaseLock: refresh lastHeartbeat */ } ++ export async function reapStaleActiveLeases(port, now) { /* reclaim iff stale(180s) AND !pidAlive(ownerPid) */ } + +// chatgpt.mjs sendWebAi — replace L200-209 (record THEN bind) ++ const acq = await acquireActiveLease(port, { owner:'web-ai', vendor, sessionType:'send-poll', sessionId, targetId, url: page.url() }); ++ if (!acq.ok) throw new WebAiError({ errorCode:'provider.capacity', stage:'lease-acquire', retryHint:'retry-after', evidence: acq }); ++ bindSessionToTab(session.sessionId, targetId); + +// cli.mjs — initial send symmetric with resume +- case 'send': return withWebAiActiveCommand(command, deps, input, () => sendWebAi(deps, input)); ++ case 'send': return withSessionCommandLock(input.session ?? mintedSessionId, ++ () => withWebAiActiveCommand(command, deps, input, () => sendWebAi(deps, input))); + +// watcher.mjs — serialize reattach+poll (wrap L176-188) ++ return withSessionCommandLock(session.sessionId, () => withSessionPage(deps, options.sessionId, async ({ page, targetId }) => { ++ await touchActiveLease(targetId); /* ensureWatcherAttached + poll */ })); +``` + +## Test Strategy (absorbs area-5 concurrency gaps) + +Homes: `test/unit/web-ai-session-store.test.mjs` (lock FIFO at :222-225), `web-ai-watcher.test.mjs`, `tab-lifecycle.test.mjs`, `web-ai-shared-target-lock.test.mjs`. + +1. **Concurrent send+poll on one session (mutex).** Slow stub send (resolve after 200ms) + concurrent `poll --session`; assert poll's lock-acquire blocks until send releases, and `conversationUrl` ends at the send-written value, never interleaved. +2. **Lease acquisition under contention (3→5→6 agents).** N simulated PIDs call `acquireActiveLease` on one store; assert exactly `min(N, cap)` succeed, rest `{ ok:false, reason:'global-cap' }`; ≤5 per `leaseKey`. +3. **Tab recovery during active polling.** Mid-poll, `isTabAlive`→true then `getPageByTargetId`→`TabDeadError` (the TOCTOU window); assert exactly one `createTab` fires across 3 concurrent callers on the same `targetId` (single-flight), session rebinds. +4. **Watcher reattach during mid-flight send.** Hold the session lock via a fake in-flight send; invoke `watchSessionOnce`; spy `page.goto` call-count == 0 while locked. +5. **Lease TTL expiry for crashed sessions.** `active-session` lease with dead `ownerPid` + `lastHeartbeat` = now−7min → `reapStaleActiveLeases` closes the tab + removes the row. Negative: live PID ⇒ not reaped even if heartbeat old. +6. **(critical) Legit 3600s pro wait is NOT reaped.** Lease `leasedAt` = now−3000s, `lastHeartbeat` = now−45s (q60s renewal), live PID; advance `now`; assert the lease survives. Second variant: drive the real `withSessionCommandLock` heartbeat over a fake clock to prove `touchActiveLease` keeps `lastHeartbeat` fresh across the hold. +7. **Atomicity (no unleased-bound window).** Interleave a reader between record and bind; assert it never observes a bound target whose `active-session` lease row is absent (record-before-bind invariant). + +## Open Risks / Tradeoffs + +- **Lease-TTL vs 3600s pro (primary).** Hinges on `lastHeartbeat` renewal firing for the full hour. Mitigations: process-dead AND-gate (live PID never reaped regardless of heartbeat); heartbeat runs on an `.unref()`'d `setInterval` independent of awaited work; `activeTtlMs`(≈6m via 180s+grace) ≫ heartbeat (60s) → ≥3× margin. +- **Two lock files** → fixed ordering (session → lease); opposite order risks deadlock. Documented + retry-timeout-bounded. +- **Backpressure choice** — reject-with-retry-after pushes retry policy to callers (jaw + CLI must honor `retryAfterMs`); a bounded queue is fairer but risks a CLI burst delaying a jaw employee. Reject is the safer default at the 12–16 ceiling; add a `priority` field only if needed. +- **Single-flight recovery is in-process only** — two separate CLI processes seeing the same tab die can still both `createTab`; the active-lease admission cap (§E.2) is the cross-process backstop. +- **Stale docs** — `cli.mjs:213` / `browser.mjs:3370` "TTL=15m" vs code 30m; fix with the cap bump. diff --git a/devlog/_fin/260619_tab_parallel_stability/20_pressure_test_verdict.md b/devlog/_fin/260619_tab_parallel_stability/20_pressure_test_verdict.md new file mode 100644 index 00000000..d8785545 --- /dev/null +++ b/devlog/_fin/260619_tab_parallel_stability/20_pressure_test_verdict.md @@ -0,0 +1,45 @@ +# Tab Stability — Pressure-Test Verdict (2026-06-20) + +> Adversarial review of `01_root_cause.md` + `10_solution_plan.md` against the user's actual goal: "N agents, one tab each, stays stable" (vendor ≤5, global 12–16). Outcome: **1 real observed bug + 2 real-but-narrow gaps, all over-engineered; 2 theoretical claims essentially already mitigated; 1 conditional/out-of-scope.** The locked vendor5/global12-16 decision still holds — it needs env caps + a count cap, not the queue/backpressure program. + +## Decisive context the plan under-weighted + +1. **Each `agbrowse web-ai ` is a separate short-lived OS process** (`cli.mjs:447`). N agents = N processes coordinating only through fs-locked JSON stores. **In-process mechanisms (the proposed single-flight `Map`) give ZERO cross-agent protection** — so they don't even serve the multi-agent goal. +2. **The cross-process tab mutex already exists.** `registerActiveCommand` (`active-command-store.mjs:156-167`) rejects a second command on the same `targetId` with a fs-locked `active-command.target-owned` error, and it wraps *every* send/poll/query via `withWebAiActiveCommand` (`cli.mjs:1312`). This is the "send+poll can't both drive one tab" guarantee — already shipped. + +## Verdict per claim + +| Claim | Verdict | Why | +|------|---------|-----| +| 2 · Lease↔session bind race (reattach-mismatch) | **DOWNSCOPE → 2-line reorder** | Real OBSERVED window (`chatgpt.mjs:200` sync bind / `:201` awaited lease). Fix = **record-before-bind** (swap the 2 lines ×4 sites). The atomic `acquireActiveLease` cap-machinery was bundled gold-plating. | +| 4 · Crashed-session lease leak | **DOWNSCOPE → PID reaper only** | Real leak, but `!pidAlive(ownerPid)` (already implemented `session-store.mjs:214`) is a sufficient+sound cross-process reap key. The 60s-heartbeat + phase + expiresAt tuning is for a single-process-stall case the goal doesn't need. | +| 6 · Admission control vendor5/global12-16 | **DOWNSCOPE → count cap** | Real gap: active leases are NEVER capped (`ensureProviderTab` unconditionally `createTab`s, `cli.mjs:1005`; caps only touch `pooled` rows). Fix = env bumps + revert `maxTabs:Infinity` + simple active-count check. Drop FIFO/retry-after/priority. | +| 1 · TOCTOU in tab recovery | **DROP** (add a warning log only) | Theoretical; `catch{}` fall-through already degrades to fresh-tab recovery (`tab-recovery.mjs:63-71,396-401`). Proposed single-flight is in-process-only → doesn't serve the N-process goal. | +| 3 · send+poll mutex (initial send) | **DROP** | The harmful collision is already blocked cross-process by `active-command.target-owned`. The one unlocked write (`conversationUrl`, `chatgpt.mjs:289`) is monotonic convergence to the same live URL, not a corrupting interleave. | +| 5 · 3-TTL collision at pro=3600s | **DROP** (conditional/out-of-scope) | Only bites IF pro=3600s is adopted (separate decision). Not on the path to multi-agent stability. | +| — · Stale "TTL=15m" doc | **KEEP** (trivial) | Code default is 30m (`tab-lease-store.mjs:64`); fix `cli.mjs:217`, `browser.mjs:3371`. | + +## Already-mitigated / degrades gracefully today + +- Concurrent same-tab ops across agents → rejected by `active-command.target-owned` (cross-process, fs-locked). *(kills Claim 3, softens 2)* +- Tab dies mid-recovery → `catch{}` + unconditional `createTab` + one `forceRecover` retry reopen a fresh tab. *(kills Claim 1)* +- Poll on wrong tab → `buildTargetMismatchResult` typed mismatch before any write (`chatgpt.mjs:358-369`). +- Reuse stealing an active tab → `findReusableProviderTab` excludes active-session + active-command targets (`cli.mjs:1054,1060`). *(residual = only the L200/L201 ordering)* +- Dead tabs → `cleanupLeasedTabs` already drops `!isTabAlive` leases (`:350-354`). Gap = only crashed-process-with-live-tab. + +## Minimal Viable Version (ranked by service to the goal) + +1. **Record-before-bind reorder** — swap `chatgpt.mjs:200↔201` (+ `deepResearchWebAi:625`, `grok-live.mjs:210`, `gemini-live.mjs:270`). The only OBSERVED bug; the most direct cause of reattach-mismatch. ~2 lines ×4 sites, no new API. +2. **Active-lease count cap + revert `maxTabs:Infinity`** — per-key (≤5) + global (12–16) check on `active-session` rows inside the existing `withLeaseLock`; `cli.mjs:982` → finite `AGBROWSE_MAX_TABS`; bump 3 env defaults. Makes the (N+1)th bounded instead of spawning unbounded tabs. +3. **PID-based active-lease reaper** — persist `ownerPid`; `reapStaleActiveLeases` reclaims iff `!pidAlive(ownerPid)`; call from existing `cleanupLeasedTabs`. Stops a crashed agent eating a slot. +4. **Stale-doc fix** (free). + +Items 1–3 are load-bearing and each maps 1:1 to a concrete failure of "N agents, one tab each." + +## Recommendation + +DOWNSCOPE and split into must-have vs defer. +- **Must-have (ships the goal):** reorder · active-count cap + revert Infinity + env bumps · PID reaper · doc fix. +- **Defer/cut:** typed `TabDeadError`+single-flight (in-process-only), hoist-sessionId initial-send lock (cross-process `target-owned` already covers the harm), full 3-TTL unification + 60s heartbeat + phase (couple to a separate pro=3600s decision), reject-with-retry-after / FIFO / priority. + +Same pattern as skill-envelope: the load-bearing capability (cross-process tab mutex via `active-command.target-owned`; graceful fresh-tab recovery via `catch{}`) **already exists**. `10_solution_plan.md` is **superseded by this verdict** for scope; `01_root_cause.md` stays as background. diff --git a/devlog/_fin/260619_timeout_adaptive_scaling/00_overview.md b/devlog/_fin/260619_timeout_adaptive_scaling/00_overview.md new file mode 100644 index 00000000..c91d952c --- /dev/null +++ b/devlog/_fin/260619_timeout_adaptive_scaling/00_overview.md @@ -0,0 +1,33 @@ +# Timeout Adaptive Scaling + +## Problem +All vendors use fixed poll timeouts (ChatGPT/Gemini 1200s, Grok 600s) regardless of model or effort level. No adaptive scaling based on model type, reasoning effort, or response complexity. + +## Key Gaps +- `chatgpt.mjs:329` — `reasoningEffort` tracked but not used for timeout scaling +- `chatgpt.mjs:62-78` — stability heuristic based on text length, not thinking state detection +- 500ms poll interval is fixed, no exponential backoff +- Intermediate "thinking" responses can trigger false completion +- Deep Research mode uses same base timeout as regular queries + +## Goal +Model-aware timeout scaling: instant (short), thinking/standard (medium), pro/heavy/deep-research (long). Adaptive poll interval with backoff. + +## Test Coverage Needed +- Unit tests for timeout calculation per model+effort combination +- Integration test: thinking model with heavy effort completes without premature timeout +- Integration test: instant model completes without unnecessary wait +- Edge case: response pauses mid-stream then resumes + +## Decision (locked 2026-06-19) +Hardcoded timeout table (no history-learning). instant ≈ 120s, thinking/standard ≈ 600s, **pro/heavy/deep-research = 3600s**. This `resolveTimeoutSeconds` is the single source of truth for the model-aware deadline reused by `260619_tab_parallel_stability` (lease TTL). See `01_root_cause.md`, `10_solution_plan.md`. + +## Pressure-test (2026-06-20) — DOWNSCOPE ~80% +`--timeout` escape hatch + action-button done-signal already exist. Real scope = 4-entry tier-default map + export `pro=3600` + 1 doc line + 1 test. Claims 2/3/4 dropped. See `20_pressure_test_verdict.md`. + +## Status — CLOSED 2026-06-20 (implemented → _fin) +- [x] Interview/requirements gathering +- [x] Plan (`10_solution_plan.md`) — superseded for scope +- [x] Pressure-test (`20_pressure_test_verdict.md`) +- [x] Implementation (MVV): commit `1d86985` — `TIER_DEFAULT_TIMEOUT_SEC`/`resolveTimeoutDefaultSec` in `session.mjs`, applied at `cli.mjs` input builder, `--help` updated +- [x] Verification: vitest 837/837 unit + 14/14 affected + 6/6 integration, `npm run gate:all` 16/16 diff --git a/devlog/_fin/260619_timeout_adaptive_scaling/01_root_cause.md b/devlog/_fin/260619_timeout_adaptive_scaling/01_root_cause.md new file mode 100644 index 00000000..dfc90b32 --- /dev/null +++ b/devlog/_fin/260619_timeout_adaptive_scaling/01_root_cause.md @@ -0,0 +1,76 @@ +# Timeout Adaptive Scaling — Root Cause (verified) + +> All paths under `web-ai/`. Line numbers observed 2026-06-19; re-verify before editing. + +## Where model + effort + timeout are all known at once + +The single join point is the dispatch layer that builds `input` — `web-ai/cli.mjs:630-689` (CLI) and `web-ai/mcp-server.mjs:207` (MCP). `model`, `reasoningEffort`, `vendor`, and `research` are all in scope there. After this point each provider's `poll` runs blind to the tier. + +## Gap 1 — Fixed poll timeout, ignores model/effort + +`chatgpt.mjs:327-329` (poll entry) uses a flat default: + +```js +export async function pollWebAi(deps, input = {}) { + const vendor = input.vendor || 'chatgpt'; + const timeout = Math.max(1, Number(input.timeout || 1200)); // flat 1200s +``` + +Same pattern in the other providers: +- `grok-live.mjs:267` → `Number(input.timeout || input.thinkingTime || 600) * 1000` — **600s shared by `expert` AND `heavy`** (aliases resolve together in `grok-model.mjs:24-50` with no per-tier timeout). +- `gemini-live.mjs:439` → `Number(input.timeout || input.thinkingTime || 1200) * 1000` — `thinkingTime` is used only as a fallback constant, never to scale Deep Think vs Flash. + +The session-level helper is also flat per-vendor — `session.mjs:322`: + +```js +const VENDOR_DEFAULT_TIMEOUT_SEC = { chatgpt: 1200, gemini: 1200, grok: 600 }; +``` + +`resolveDeadlineAt` (`session.mjs:329-334`) only reads `input.timeout` or this table — **no model/effort dimension**. + +## Gap 2 — `reasoningEffort` is tracked but never scales the timeout + +Parsed at `cli.mjs:663` (`reasoningEffort: values.effort || values['reasoning-effort']`), threaded into send at `chatgpt.mjs:178`: + +```js +const selectedModel = await selectChatGptModel(page, input.model, { effort: input.reasoningEffort }); +``` + +Inside `chatgpt-model.mjs:6-7,105-189` it is already normalized into tier-grade enums — `ModelChoice = 'instant'|'thinking'|'pro'`, `EffortChoice = 'light'|'standard'|'extended'|'heavy'`. **The system already knows the exact tier.** But `selectChatGptModel`'s return is consumed only for warnings (`chatgpt.mjs:309-312`) and is never passed to `pollWebAi`. The signal exists and is discarded. + +## Gap 3 — Stability heuristic is text-length based → false "complete" on a mid-stream pause + +`chatgpt.mjs:386-398`: + +```js +const streaming = await isStreaming(page); +const finished = !streaming && latest ? await isResponseFinished(page) : false; +if (latest && !streaming) { + if (latest === stableText) { + const textLen = latest.length; + const minStableMs = finished + ? 1000 + : textLen < 16 ? 8000 : textLen < 40 ? 3000 : textLen < 500 ? 2000 : 3000; + if (elapsedStable >= minStableMs) { /* declare complete */ } +``` + +The grace window keys off **`textLen`, not thinking state**. Signals available: +- `isStreaming` (`chatgpt.mjs:547-553`) — checks only the **stop button** (`CHATGPT_STOP_SELECTORS`, `:106-109`). +- `isResponseFinished` (`chatgpt.mjs:558-581`) — checks post-completion action buttons (`FINISHED_ACTIONS_SELECTOR`, `:55-60`: copy / 👍 / 👎 / Share). +- `PLACEHOLDER_PATTERNS` (`chatgpt.mjs:62-78`) — only filters *pure-placeholder* turns (`/^thinking$/i`, `/^reasoning$/i`, `/^pro thinking/i`…). + +**Failure mode:** a reasoning model emits an intermediate chunk, then **hides the stop button while it thinks** before resuming. At that instant `streaming===false` and `finished===false`; if the partial text is ≥500 chars it needs only **3000ms** of stability to be falsely declared complete. A real partial answer with substantive text passes `isFinalAnswer` (`chatgpt.mjs:915-917`) and is eligible for early completion. Grok is weaker still — a flat `>= 1500ms` with no `finished` check (`grok-live.mjs:278`). + +This is the user's "이게 중간 응답도 있잖아" concern, located precisely: **absence-of-stop-button is not a done-signal by itself** for reasoning models. + +## Gap 4 — Fixed 500ms poll interval, no backoff + +`chatgpt.mjs:483` → `await page.waitForTimeout(500)` (also `grok-live.mjs:316`; Gemini flat `2_000` at `gemini-live.mjs:479`). A pro tier raised to 3600s would run **~7200 polls**, each a full DOM round-trip (`readAssistantMessages` + `isStreaming` + `isResponseFinished`) — wasteful and noisy. + +## Gap 5 — Deep Research shares the same base timeout + +`chatgpt.mjs:635` → `Number(input.timeout || 1200) * 1000`, passed to `sendDeepResearch` (own default `1_200_000` at `chatgpt-deep-research.mjs:207`, flat `waitForTimeout(2000)` poll at `:278`). Deep Research gets no dedicated tier; it inherits the same 1200s base as a thinking query. + +## Evidence index + +`chatgpt.mjs` 327-329, 178, 55-60, 62-78, 106-109, 386-398, 483, 547-553, 558-581, 635, 915-917 · `grok-live.mjs` 267, 278, 316 · `gemini-live.mjs` 439, 479 · `session.mjs` 322, 329-334 · `chatgpt-model.mjs` 6-7, 105-189 · `grok-model.mjs` 24-50 · `chatgpt-deep-research.mjs` 207, 278 · `cli.mjs` 630-689 (esp. 641, 663) · `mcp-server.mjs` 207. diff --git a/devlog/_fin/260619_timeout_adaptive_scaling/10_solution_plan.md b/devlog/_fin/260619_timeout_adaptive_scaling/10_solution_plan.md new file mode 100644 index 00000000..802c4b50 --- /dev/null +++ b/devlog/_fin/260619_timeout_adaptive_scaling/10_solution_plan.md @@ -0,0 +1,128 @@ +# Timeout Adaptive Scaling — Solution Plan + +> ⚠️ **SUPERSEDED for scope by `20_pressure_test_verdict.md` (2026-06-20).** Pressure-test found ~80% of this plan over-engineered (the `--timeout` escape hatch + action-button done-signal already exist; claims 2/3/4 dropped). Real scope = a 4-entry tier-default map + 1 export + 1 doc line + 1 test. Also note the `pollTimeoutSec` name collision flagged in the verdict. Read the verdict first; this file is retained as the full analysis. + +> Locked decision: **hardcoded mapping table** (no history-learning). instant ≈ 120s, thinking/standard ≈ 600s, **pro / heavy / deep-research = 3600s**. Documentation only; code sketch is illustrative. + +## A. Hardcoded `TIMEOUT_TABLE` (new module `web-ai/timeout-policy.mjs`) + +Static mapping keyed by `${provider}:${tier}:${effort||'-'}`, in **seconds**: + +```js +export const TIMEOUT_TABLE = { + // ChatGPT + 'chatgpt:instant:-': 120, + 'chatgpt:thinking:light': 600, + 'chatgpt:thinking:standard': 600, + 'chatgpt:thinking:extended': 1200, + 'chatgpt:thinking:heavy': 1800, + 'chatgpt:pro:standard': 3600, // pro = 3600s (user choice) + 'chatgpt:pro:extended': 3600, + 'chatgpt:deep-research:-': 3600, + // Gemini + 'gemini:flash-lite:-': 120, + 'gemini:flash:-': 600, + 'gemini:pro:-': 600, + 'gemini:deep-think:-': 3600, + // Grok + 'grok:fast:-': 120, + 'grok:auto:-': 600, + 'grok:expert:-': 600, + 'grok:heavy:-': 3600, // no longer shares 600 with expert +}; + +const TIER_DEFAULT = { instant: 120, thinking: 600, pro: 3600, 'deep-research': 3600 }; + +export function resolveTimeoutSeconds({ provider, tier, effort }, vendorDefault) { + const key = `${provider}:${tier}:${effort || '-'}`; + if (TIMEOUT_TABLE[key] != null) return { key, seconds: TIMEOUT_TABLE[key], source: 'table' }; + if (TIER_DEFAULT[tier] != null) return { key, seconds: TIER_DEFAULT[tier], source: 'tier-default' }; + return { key, seconds: vendorDefault, source: 'vendor-default' }; +} +``` + +**Precedence (must preserve current behavior):** an explicit `input.timeout` always wins; the table only supplies the *default* when `--timeout` is absent. Keeps `poll --timeout 1800` authoritative and the fake-chatgpt test's `timeout: 2` green. + +> This `resolveTimeoutSeconds` is the **single source of truth** for the model-aware deadline that `260619_tab_parallel_stability` reuses for its lease-TTL / session-lock TTL (see that folder's `10_solution_plan.md` §C). Do not duplicate the pro=3600 constant in the lease store — import it. + +## B. Tier mapping — where to derive it + +`deriveTier(input)` reuses existing normalizers, called at the dispatch layer (`cli.mjs:630-689` / `mcp-server.mjs:207`) so all three providers share one path: +- ChatGPT: `normalizeChatGptModelChoice(input.model)` + `normalizeChatGptEffortChoice(input.reasoningEffort)` (`chatgpt-model.mjs:136,146`); `input.research === 'deep'` → tier `'deep-research'`. +- Gemini: `normalizeGeminiModel` + `isGeminiDeepThinkAlias` (`gemini-model.mjs:74,83`). +- Grok: `GROK_MODEL_ALIASES` lookup (`grok-model.mjs:24`). + +Set `input.pollTimeoutSec` at dispatch; each `poll` also falls back to `resolveTimeoutSeconds(deriveTier(input), VENDOR_DEFAULT_TIMEOUT_SEC[vendor])` so a directly-invoked `pollWebAi` (session resume) still scales. + +## C. Intermediate-response handling — the hard part (replaces `chatgpt.mjs:386-398`) + +Gate completion on a **positive done-signal**, not on absence-of-stop-button. A small state machine: + +1. **Authoritative DONE** — `isResponseFinished(page)` true (action buttons on the latest turn) AND text stable → complete now (keep `finished ? 1000` fast path). Trusted regardless of tier. +2. **Authoritative STREAMING** — stop button visible → reset stability, never complete (already at `chatgpt.mjs:480-482`). +3. **Ambiguous (the trap)** — stop button gone, action buttons absent → *possibly still thinking*. Require **tier-scaled `minStableMs`** AND a **no-growth-for-N-polls** rule: + - `instant` → 1500ms is enough (no hidden thinking phases). + - `thinking`/`pro`/`deep-research` → `minStableMs` ≈ 8000ms AND text length unchanged across ≥3 polls. + - **Thinking-indicator probe** `hasThinkingIndicator(page)`: if any `PLACEHOLDER_PATTERNS` text or a reasoning/"thinking" chip is still in the latest turn's DOM, force streaming-equivalent behavior (reset stability) even with the stop button gone. This is the missing link that stops "declared done while between reasoning steps". +4. **Hard deadline** — the tier-scaled timeout (§A) bounds the loop; pro waits up to 3600s but isn't tripped early. + +## D. Adaptive poll interval with backoff (replaces fixed 500ms) + +Reset on text growth (stay responsive), grow while idle (don't hammer a 3600s wait): + +```js +let pollMs = 500; const POLL_MIN = 500, POLL_MAX = 4000; +if (latest !== lastSeen) { pollMs = POLL_MIN; lastSeen = latest; } +else { pollMs = Math.min(POLL_MAX, Math.round(pollMs * 1.5)); } +await page.waitForTimeout(pollMs); +``` + +At a 4s cap a 3600s pro wait is a few hundred polls instead of ~7200; a fast-streaming instant response still samples at 500ms. + +## Code sketch (illustrative — NOT for commit) + +```diff +# web-ai/timeout-policy.mjs (NEW) — TIMEOUT_TABLE + resolveTimeoutSeconds + deriveTier + +# web-ai/cli.mjs (~641) ++ pollTimeoutSec: values.timeout != null ? Number(values.timeout) ++ : resolveTimeoutSeconds(deriveTier({ vendor: values.vendor, model: values.model, ++ reasoningEffort: values.effort || values['reasoning-effort'], research: values.research }), ++ VENDOR_DEFAULT_TIMEOUT_SEC[values.vendor] ?? 1200).seconds, + +# web-ai/chatgpt.mjs:329 +- const timeout = Math.max(1, Number(input.timeout || 1200)); ++ const timeout = Math.max(1, Number(input.timeout ?? input.pollTimeoutSec ++ ?? resolveTimeoutSeconds(deriveTier(input), 1200).seconds)); + +# web-ai/chatgpt.mjs:386-398 (stability gate) ++ const thinking = !streaming && await hasThinkingIndicator(page); ++ if (latest && !streaming && !thinking) { ++ const tier = deriveTier(input).tier; ++ const minStableMs = finished ? 1000 : tier === 'instant' ? 1500 : 8000; ++ const enoughNoGrowth = noGrowthPolls >= (tier === 'instant' ? 1 : 3); ++ if (elapsedStable >= minStableMs && (finished || enoughNoGrowth)) { /* complete */ } + +# grok-live.mjs:267 / gemini-live.mjs:439 — same default + backoff +``` + +## Test Strategy (absorbs area-5 test gaps) + +Build on `test/integration/web-ai-fake-chatgpt.test.mjs` (its `waitForTimeout` mock already flips `'Pro thinking...'` → `'OK'`) and `test/unit/web-ai-chatgpt-model.test.mjs` (effort matrix at :54-58). + +1. **Unit — timeout table per (provider, model, effort).** `resolveTimeoutSeconds`/`deriveTier`: `chatgpt:pro:* → 3600`, `chatgpt:instant → 120`, `chatgpt:thinking:standard → 600`, `chatgpt:deep-research → 3600`, **`grok:heavy → 3600` (regression: must NOT equal `grok:expert`'s 600)**, `gemini:deep-think → 3600`. Assert explicit `input.timeout` overrides the table. +2. **Integration — heavy/pro not timing out early.** Fake page keeps stop button visible past the *old* default but within tier; assert no premature `status:'timeout'` and `deadline` derived from 3600, not 1200. +3. **Integration — instant not over-waiting.** Fake page completes fast (action buttons appear); assert completion well under 120s and `responseStableMs` ≈ short instant grace. +4. **Integration — mid-stream pause then resume (the edge case).** Sequence: partial text ≥500 chars → hide stop button + keep a thinking chip for several polls → resume → final text → action buttons. Assert the result equals the **final** text (not the partial) — proves the thinking-indicator + no-growth gate. +5. **Backoff unit.** Interval resets to 500ms on growth, climbs to 4000ms cap while idle; assert a simulated 3600s idle wait is bounded to hundreds of polls. + +## Open Risks / Tradeoffs + +- **DOM signal fragility.** The thinking-indicator probe depends on ChatGPT keeping a reasoning/placeholder chip during pauses; if removed, the gate degrades to the tier-scaled no-growth rule (still safer than today). Mitigate with selector-list resilience like the existing stop/finished selectors. +- **3600s pro hard wait** ties up a tab/lease for up to an hour on a hung response. Backoff reduces CPU but the lease is still held — this is exactly why `260619_tab_parallel_stability` must add lease TTL + 60s heartbeat keyed on the same 3600s value (cross-cut). +- **Hardcoded table drift.** Tier values are static by design; a provider tier rename needs a manual one-module edit, unit-tested. +- **Hot-path change.** Threading `pollTimeoutSec` through `cli.mjs` + `mcp-server.mjs` + three pollers is mechanical; explicit-`--timeout`-wins precedence keeps all current callers behavior-compatible. + +## References (web) + +The copy/action button being the authoritative "response complete" signal (with stop-button + text-stability as supporting layers) is corroborated by ChatGPT streaming/scraping write-ups (theodormarcu.com "How ChatGPT streams responses"; Scrapfly "How to Scrape ChatGPT 2026" / "Find Web Selectors with ChatGPT"). diff --git a/devlog/_fin/260619_timeout_adaptive_scaling/20_pressure_test_verdict.md b/devlog/_fin/260619_timeout_adaptive_scaling/20_pressure_test_verdict.md new file mode 100644 index 00000000..38d4084c --- /dev/null +++ b/devlog/_fin/260619_timeout_adaptive_scaling/20_pressure_test_verdict.md @@ -0,0 +1,39 @@ +# Timeout — Pressure-Test Verdict (2026-06-20) + +> Adversarial review of `01_root_cause.md` + `10_solution_plan.md`. Same outcome as the skill-envelope review: **the capability mostly already exists; the plan was ~80% over-engineered.** Locked decision (pro=3600s) still holds — but it needs a 4-entry default map, not a 20-row table + DOM state machine + backoff. + +## Verdict per claim + +| Claim | Verdict | Why | +|------|---------|-----| +| Fixed timeout ignores model/effort → full `TIMEOUT_TABLE` module | **DOWNSCOPE** | `--timeout ` is already a complete, plumbed, documented escape hatch (`cli.mjs:124,331,547,645` → all 3 pollers). Only real value = sane *defaults* for long tiers, deliverable as a 4-entry map. | +| Intermediate "thinking" false-completion → `hasThinkingIndicator` state machine | **DROP** | Theoretical — traced to a user *remark*, no repro/test/trace. Already guarded by `isResponseFinished` action-buttons (`chatgpt.mjs:386-398,558-581`, `finished?1000` fast path) + `PLACEHOLDER_PATTERNS`+`isFinalAnswer` filter (`:62-78,384`). Proposed DOM probe re-derives `finished` and adds a new breakage surface. | +| 500ms fixed poll, no backoff | **DROP** | Pure CPU/efficiency; zero user-visible symptom. The "7200 polls" cost only exists if you build the 3600s wait. Premature optimization. | +| Deep Research same base timeout | **DROP** | Already has its own stability gate (`chatgpt-deep-research.mjs:284-288`, 5000ms + progress indicator) and a `1_200_000`ms internal default; fully `--timeout`-overridable. No gap. | + +## ⚠️ Bug the skeptic found in MY plan + +`10_solution_plan.md` threads tier timeout as `input.pollTimeoutSec` — but **`pollTimeoutSec` is already taken**: it's the watcher's per-iteration poll timeout (`cli.mjs:685` `pollTimeoutSec: values['poll-timeout']` → `watcher.mjs:282`). Implementing the sketch as written would conflate two different timeouts. Use a different field name (e.g. `resolvedTimeoutSec`) if this is ever built. The collision is itself evidence the table design wasn't checked against existing code. + +## Already-mitigated / escape hatches + +- `--timeout ` → fully plumbed to all 3 pollers + deep-research + query; documented with a Pro example (`cli.mjs:331`). +- `--deadline ` → second override (`cli.mjs:200`, `session.mjs:330`). +- Action-button done-signal + `finished?1000` fast path = the "missing" positive done-signal, already present. +- `PLACEHOLDER_PATTERNS`+`isFinalAnswer` = already suppresses thinking/placeholder turns. + +## Minimal Viable Version + +The only user-visible defect: **a pro/heavy run left WITHOUT `--timeout` silently caps at the 1200s default.** Fix exactly that: + +1. Tier-aware **default** via a 4-entry map reusing existing normalizers (`normalizeChatGptModelChoice`/`Effort` at `chatgpt-model.mjs:136,146`): + `TIER_DEFAULT = { instant:120, thinking:600, pro:3600, 'deep-research':3600 }`. Applied in `resolveDeadlineAt` (`session.mjs:329`) + the 3 poller fallbacks. **No new `timeout-policy.mjs` module, no provider×tier×effort matrix.** +2. Export the single `pro = 3600` constant for `tab_parallel_stability` to import (that plan needs only this one value). +3. One doc line in `cli.mjs:124` (defaults auto-scale for pro/heavy/deep-think; `--timeout` overrides). +4. One regression test: `grok:heavy` default ≠ `grok:expert` 600; pro default = 3600. + +**Cut entirely:** `timeout-policy.mjs` module, the 20-row table, `hasThinkingIndicator` state machine, no-growth-N-polls rule, adaptive backoff. + +## Recommendation + +DOWNSCOPE. Keep claim 1 (as the 4-entry default map); drop claims 2/3/4. Net surface ≈ 1 map + 1 export + 1 doc line + 1 test. `01_root_cause.md` analysis stays valid as background; `10_solution_plan.md` is **superseded by this verdict** for the actual scope. diff --git a/devlog/_fin/260619_watch_notification_gaps/00_overview.md b/devlog/_fin/260619_watch_notification_gaps/00_overview.md new file mode 100644 index 00000000..0b6e7be3 --- /dev/null +++ b/devlog/_fin/260619_watch_notification_gaps/00_overview.md @@ -0,0 +1,45 @@ +# Watch/Notification Gaps + +## Problem +Watcher relies solely on 15-second interval polling. No CDP event subscriptions for real-time completion detection. Session URL tracking breaks on provider redirects. + +## Key Gaps + +### Poll-only architecture +- `watcher.mjs:26` — `DEFAULT_WATCH_INTERVAL_MS = 15_000` +- No DOM mutation observer, no CDP network idle detection +- Completion can be delayed up to 15 seconds + +### Session URL tracking fragile +- `conversationUrl` only updated at send time (`chatgpt.mjs:290`) +- `extractConversationId()` (`chatgpt.mjs:906-912`) hardcodes `/c/{id}` regex +- Provider URL format changes break session resumption + +### Reattach-mismatch common +- Watcher tab URL vs session URL mismatch when provider redirects +- Only auto-recovers with explicit `--navigate` flag +- Documented as issue #77 + +## Goal +Reduce completion detection latency. Make URL tracking format-agnostic. Evaluate CDP event-based detection feasibility vs. risk. + +## Test Coverage Needed +- Watcher behavior when provider redirects conversation URL +- Reattach recovery with and without `--navigate` +- URL extraction against different provider URL formats +- (If implemented) CDP event-based completion detection reliability + +## Decision (locked 2026-06-19) +Two tiers. **Tier 1 (safe, do-first):** adaptive poll interval (pro-safe backoff), per-provider conversation-id registry, self-healing URL via the existing `shouldPreferCurrentProviderUrl` (closes Issue #77 watch-path half). **Tier 2 (opt-in `--cdp-nudge`, default off):** CDP event only *nudges* an early poll — poll stays source of truth, so a broken event can never cause a false/missed completion. User's "리스키" stance confirmed correct (networkIdle is unreliable per devtools-protocol #154). See `01_root_cause.md`, `10_solution_plan.md`. + +## Pressure-test (2026-06-20) — DOWNSCOPE ~75%, mostly →_fin +15s latency invisible under send→watch→bgtask (adaptive polling dropped); conversation-id registry misdiagnosed (gemini/grok don't use the regex); Tier 2 dropped. Only real win: watcher already self-heals via `resolveSessionPage` but discards the healed session — ~30-line consolidation. See `20_pressure_test_verdict.md`. + +## Status — CLOSED 2026-06-20 (implemented → _fin) +- [x] Interview/requirements gathering +- [x] Plan (`10_solution_plan.md`) — superseded for scope +- [x] Pressure-test (`20_pressure_test_verdict.md`) +- [x] Implementation (MVV), two steps: + - commit `1d86985` — `watcher.mjs` feeds the resolver-healed session to `ensureWatcherAttached` (retires the watch-path half of #77 root-drift) + - follow-up — completed the MVV's "retire `urlsEquivalentForWatch`": replaced the strict hash-only compare with the canonical tolerant `urlsCompatible` (exported from `tab-recovery.mjs`, shared with the resolver) and deleted `urlsEquivalentForWatch`. Replace-not-delete preserves the genuine-mismatch guard (non-provider landing) that a blind deletion would have stripped — confirmed by an independent completion audit. +- [x] Verification: affected tests 21/21, `npm run gate:all` 16/16 (twice) diff --git a/devlog/_fin/260619_watch_notification_gaps/01_root_cause.md b/devlog/_fin/260619_watch_notification_gaps/01_root_cause.md new file mode 100644 index 00000000..0dfeee56 --- /dev/null +++ b/devlog/_fin/260619_watch_notification_gaps/01_root_cause.md @@ -0,0 +1,79 @@ +# Watch / Notification Gaps — Root Cause (verified) + +> All paths under `web-ai/` unless noted. Line numbers observed 2026-06-19. User stance (constraint): a fully event-driven "fire a hook the instant the AI finishes" is **risky** (provider DOM churn, truncated/errored responses, many edge cases). So: fix the cheap robust wins; treat CDP events as an optional, bounded enhancement *on top of* polling. + +## Gap 1 — Poll-only, fixed 15s floor → up to 15s detection latency + +`watcher.mjs:26` → `export const DEFAULT_WATCH_INTERVAL_MS = 15_000;` + +The loop sleeps a constant interval between ticks — no DOM mutation observer, no network-idle, no CDP event (`watcher.mjs:68-96`): + +```js +for (let iteration = 1; ; iteration += 1) { + lock.heartbeat({ iteration }); + const tick = await watchSessionOnce(deps, { ...options, session: options.sessionId }); + if (tick.terminal === true) { ...; break; } + if (options.once) { ...; break; } + if (options.maxIterations && iteration >= options.maxIterations) break; + await sleep(options.intervalMs); // constant; no adaptivity, no early-wake +} +``` + +`intervalMs` is parsed once (`watcher.mjs:281`) and never recomputed. Inside one tick `pollWebAi` *does* detect completion (stop-button + action-button + stability), but `watchSessionOnce` caps each poll at `DEFAULT_WATCH_POLL_TIMEOUT_SEC = 30` (`watcher.mjs:27`), and **between** ticks the loop is blind for the full 15s. Completion signals that already exist during a poll (stop button at `chatgpt.mjs:106-109,548`; copy/action buttons at `:55-60`) are not wired to wake the *watcher* early. + +## Gap 2 — `conversationUrl` captured only at send time; ID extractor is `/c/{id}`-hardcoded + +`conversationUrl` is written at session create (`chatgpt.mjs:193`) and once more at the end of `sendWebAi` (`:288-291`); nothing re-reads the live tab URL into the session during watch. If the provider redirects *after* send (root `/` → `/c/` settle, locale/share rewrite, `chatgpt.com` ↔ `chat.openai.com`), the stored URL goes stale and reattach fails. + +The only conversation-identity primitive is a single hardcoded ChatGPT regex (`chatgpt.mjs:908-912`): + +```js +function extractConversationId(url) { + if (!url) return null; + const match = url.match(/\/c\/([a-f0-9-]+)/); + return match ? match[1] : null; +} +``` + +used in the poll mismatch guard (`chatgpt.mjs:372-381`). For Gemini (`gemini.google.com/app/`) and Grok (`grok.com/chat/`) — both first-class watcher vendors (`PROVIDER_HOSTS`, `watcher.mjs:31-35`) — it returns `null`, collapsing the identity check to a raw string compare that any harmless query-param/hash change breaks. + +## Gap 3 — Reattach-mismatch only auto-recovers with explicit `--navigate` (Issue #77, P0) + +`ensureWatcherAttached` (`watcher.mjs:398-412`) does an exact (hash-stripped only) URL compare and hard-fails unless `--navigate`: + +```js +const targetUrl = session.conversationUrl || session.originalUrl; +const currentUrl = page.url?.() || ''; +if (urlsEquivalentForWatch(currentUrl, targetUrl)) return { ok: true, url: currentUrl, warnings: [] }; +if (options.navigate) { await page.goto(targetUrl, ...); return { ok: true, url: targetUrl, warnings: [`reattached:...`] }; } +return { ok: false, url: currentUrl, warnings: [`current tab ${currentUrl} does not match session conversationUrl ${targetUrl}; pass --navigate to switch tabs`] }; +``` + +`urlsEquivalentForWatch` (`watcher.mjs:559-569`) strips only the hash — not host aliases, trailing slashes, or root→conversation drift. The mismatch surfaces as a non-terminal `status: 'reattach-mismatch'` (`watcher.mjs:183-187`) that keeps polling forever, and the CLI prints "pass --navigate" (`cli-sessions.mjs:212-215`). + +**Issue #77 confirmed in-repo** — `docs/production-readiness.md:41`: *"Web-AI durable session recovery | Beta until the #77 matrix is green: closed target recovery, root-to-conversation URL drift, stale command lock, wrong active tab, and watch transient timeout recovery."* + +**Key asymmetry:** `sessions reattach`/`resume` go through `resolveSessionPage` (`tab-recovery.mjs:259`), which **already** tolerates drift via `shouldPreferCurrentProviderUrl` and self-heals by writing the live URL back (`tab-recovery.mjs:314-325`): + +```js +if (current.conversationUrl && page.url() !== current.conversationUrl) { + const liveUrl = page.url(); + if (shouldPreferCurrentProviderUrl(current.conversationUrl, liveUrl)) { + updateSession(sessionId, { conversationUrl: liveUrl }); // self-heal exists here +``` + +The watcher reimplements its own stricter compare and does NOT call this helper. **The watch path is strictly more fragile than the reattach path for the same drift** — a large part of the fix is "make the watcher reuse tolerance that already ships." + +## Gap 4 — pro=3600s-class waits hammer the page + +Deadlines from `resolveDeadlineAt` (`session.mjs:329-336`; defaults chatgpt/gemini 1200s, grok 600s; `--timeout` can push pro far higher). During a long pro/Deep-Research wait the watcher still polls every 15s, each spinning `pollWebAi`'s inner 500ms loop (`chatgpt.mjs:483`) up to the 30s poll-timeout. A naive "poll faster" change makes this worse — any adaptive scheme must **back off** to a slow floor while a long reasoning job is mid-flight (ties to `260619_timeout_adaptive_scaling`). + +## Confirmed infrastructure (good news) + +- Notifier is **pluggable/duck-typed**: `watchSession(deps, input, notifier = null)` (`watcher.mjs:42`), defaults to `createStdoutNotifier` (`:54,258-274`), invoked as bare `await notify(enriched)` (`:60`). Any `async (event) => {}` works → cli-jaw can register a bgtask notifier with no core change. +- `send`/`watch` separation is real: `send` persists a session + returns `status:'sent'`; `watch` (`cli.mjs:719-720`) is a separate command driving `watchSession`. +- CDP event infra already exists to reuse: `Network.enable` + `Network.requestWillBeSent` listener (`skills/browser/browser.mjs:1989-2003`); `Page.enable`/`DOM.enable` precedent (`chatgpt-project-sources.mjs:152,195`); `getCdpSession` via `page.context().newCDPSession(page)` (`cli-sessions.mjs:108`). + +## Evidence index + +`watcher.mjs` 26, 27, 31-35, 42, 54, 60, 68-96, 142, 163-173, 176-188, 236, 258-274, 281, 398-412, 449-453, 559-569 · `chatgpt.mjs` 55-60, 106-109, 193, 288-291, 372-381, 386-398, 483, 485-495, 547-553, 558-581, 908-912 · `tab-recovery.mjs` 259-309, 314-325 · `cli-sessions.mjs` 108, 114-145, 212-215 · `session.mjs` 322, 329-336 · `cli.mjs` 16, 230-233, 719-720 · `skills/browser/browser.mjs` 1989-2003 · `chatgpt-project-sources.mjs` 152, 195 · `docs/production-readiness.md` 41 (Issue #77). diff --git a/devlog/_fin/260619_watch_notification_gaps/10_solution_plan.md b/devlog/_fin/260619_watch_notification_gaps/10_solution_plan.md new file mode 100644 index 00000000..98629659 --- /dev/null +++ b/devlog/_fin/260619_watch_notification_gaps/10_solution_plan.md @@ -0,0 +1,113 @@ +# Watch / Notification Gaps — Solution Plan + +> ⚠️ **SUPERSEDED for scope by `20_pressure_test_verdict.md` (2026-06-20).** Pressure-test: 15s latency is invisible under the `send→watch→bgtask` workflow (adaptive polling DROP); the conversation-id registry is misdiagnosed (gemini/grok never use the `/c/{id}` regex — DROP); Tier 2 CDP nudge DROP. The one real win is even smaller than stated: the watcher already self-heals via `resolveSessionPage` but **discards the healed session** and re-checks against a stale one — fix is ~30 lines of deletion/consolidation. Read the verdict first; this file is the full analysis. + +> Documentation only; code sketch illustrative. Split into TIER 1 (safe, do-first) and TIER 2 (optional, risk-bounded). **Poll is always the source of truth**; events only ever shorten the wait, never declare completion. + +## TIER 1 — Safe wins (no event dependency) + +**1A. Adaptive poll interval (bounded, deterministic).** Replace the constant `sleep(intervalMs)` with an interval from the last tick's `status` + time-since-send, clamped to `[minMs, maxMs]`: +- Just after `watch.start` / a fresh `sent` session (completion likely soon): poll fast (~2s). +- `status === 'streaming'` (`watcher.mjs:236,449-453`): medium (~3s) — answer imminent. +- `status === 'polling'`, stable for N ticks (idle / long reasoning / pro): **back off** geometrically toward `maxMs` (5s → 10s → 15s default, cap 30s for pro-length deadlines). + +This *is* the pro=3600s mitigation: idle ⇒ slow floor, never a 500ms hammer; the fast band applies only in the short completion-likely window. + +**1B. Format-agnostic conversation identity (per-provider registry).** Replace the single `/c/{id}` regex: + +```js +const CONVERSATION_ID_EXTRACTORS = { + chatgpt: (u) => u.match(/\/c\/([a-f0-9-]+)/)?.[1] ?? null, + gemini: (u) => u.match(/\/app\/([a-z0-9-]+)/i)?.[1] ?? null, + grok: (u) => u.match(/\/chat\/([a-z0-9-]+)/i)?.[1] ?? null, +}; +function extractConversationId(url, vendor = 'chatgpt') { + if (!url) return null; + return (CONVERSATION_ID_EXTRACTORS[vendor] ?? CONVERSATION_ID_EXTRACTORS.chatgpt)(url); +} +``` + +Unknown vendor ⇒ normalized-URL compare (host alias + trailing-slash + query/hash stripped). Fixes the poll mismatch guard for all three vendors. + +**1C. Re-read `conversationUrl` from the live tab + tolerate redirects.** In `ensureWatcherAttached`, stop the strict compare-or-fail. Instead: call the **existing** `shouldPreferCurrentProviderUrl` (`tab-recovery.mjs`); if the live tab is the *same conversation id* (1B) but a drifted URL, **self-heal** — write the live URL back into the session (mirroring `tab-recovery.mjs:314-325`), no `--navigate`. Only a genuinely *different* conversation (different id, or cross-host with no id match) stays a mismatch. + +**1D. Robust reattach without manual `--navigate`.** Route the watcher's attach through `resolveSessionPage(deps, sessionId, { allowNavigate })` (`tab-recovery.mjs:259`) so watch **inherits** the drift tolerance + recovery already shipped for reattach/resume — closing the asymmetry and retiring the Issue #77 "root-to-conversation URL drift" row for the watch path. `--navigate` then governs only *cross-tab* switching (genuinely different conversation), not benign same-conversation drift. + +## TIER 2 — Optional CDP "early nudge" (opt-in `--cdp-nudge`, default off) + +A bounded enhancement layered on the Tier-1 poll. It **never declares completion**; it only resolves the sleep early so the next poll (with its robust DOM-stability logic) runs sooner. Delivered as an internal early-wake signal, orthogonal to the pluggable notifier (which still emits the authoritative `watch.complete` only after a real poll confirms). + +```js +// inside the loop, replacing await sleep(interval) +const wake = subscribeEarlyWake(deps, { floorMs: 1_500 }); // first cheap signal wins +await Promise.race([sleep(interval), wake.promise]).catch(() => {}); // a rejected wake must never crash +wake.dispose(); +``` + +Cheap signals (any one fires the nudge): stop-button present→absent (`MutationObserver` on `button[data-testid="stop-button"]`); `Page.lifecycleEvent` networkIdle; `Network.loadingFinished` on the SSE endpoint. Debounce ~250ms; **floor the early-wake at ≥1.5s** so a chatty page can't poll faster than that. + +### Honest risk table (why poll-as-safety-net contains every failure) + +| Failure mode | Trigger | If it fires wrong | Containment | +|---|---|---|---| +| `stop-button` testid renamed/removed | provider DOM change | nudge silently no-ops | falls back to Tier-1 floor (≤15s); no missed completion | +| networkIdle not fired / late | known CDP bug (devtools-protocol #154) | late/absent nudge | latency degrades to 15s floor — strictly no-worse than today | +| networkIdle fires early (mid-stream lull) | SSE keep-alive gap | an extra poll runs; sees streaming still true → `polling` | wasted poll only; poll, not event, decides completion | +| event flood | heavy DOM churn | tight-loop / CPU burn | debounce + ≥1.5s early-wake floor cap the poll rate | +| truncated/errored response leaves stop-button stuck | provider error UI | nudge never fires | Tier-1 floor + existing deadline/timeout path (`watcher.mjs:163-173`) still terminates | +| CDP session detaches / tab crash | tab death | `earlyWake()` rejects | `Promise.race(...).catch()` → sleep still resolves; poll hits existing `isPageDeathError` path | +| long pro/Deep-Research run | — | frequent benign idle nudges | each gated by floor + backoff; poll confirms "still streaming" | + +**Net contract:** a broken or hostile event can only make latency *worse-bounded-by-the-floor* or waste a poll — it can **never** cause a false or missed completion, because completion is decided exclusively by `pollWebAi`'s DOM-stability logic, unchanged. + +## Code sketch (illustrative — NOT for commit) + +```diff +// watcher.mjs — adaptive interval ++ function nextInterval(prev, tick, opts) { ++ const { minMs = 2_000, maxMs = 30_000, floorMs = opts.intervalMs } = opts; ++ if (tick.status === 'streaming') return Math.max(minMs, 3_000); ++ if (tick.justSent) return minMs; ++ if (tick.status === 'polling') return Math.min(maxMs, Math.round((prev ?? floorMs) * 1.5)); // pro-safe backoff ++ return floorMs; ++ } +- await sleep(options.intervalMs); ++ const interval = nextInterval(interval, tick, options); ++ const wake = options.cdpNudge ? subscribeEarlyWake(deps, { floorMs: 1_500 }) : null; ++ await Promise.race([sleep(interval), wake?.promise].filter(Boolean)).catch(() => {}); ++ wake?.dispose?.(); + +// watcher.mjs — ensureWatcherAttached: reuse existing tolerance, self-heal ++ if (sameConversation(currentUrl, targetUrl, session.vendor)) { // 1B registry ++ if (currentUrl !== targetUrl) updateSession(session.sessionId, { conversationUrl: currentUrl }); // self-heal ++ return { ok: true, url: currentUrl, warnings: [] }; ++ } ++ if (options.navigate) { await page.goto(targetUrl, ...); return { ok: true, ... }; } ++ return { ok: false, url: currentUrl, warnings: [`tab is a different conversation; pass --navigate`] }; +``` + +## Test Strategy (absorbs area-5 gaps) + +Homes: `test/unit/web-ai-watcher.test.mjs` (source-string contracts), `test/integration/web-ai-fake-chatgpt.test.mjs` (fake provider). + +**Tier 1** +- **URL extraction registry (unit, table-driven):** per vendor feed real + drifted URLs (`/c/`, `/app/`, `/chat/`, host alias `chat.openai.com`↔`chatgpt.com`, trailing slash, `?model=`, `#hash`) → `extractConversationId` returns the stable id; `sameConversation` true across cosmetic drift, false across different ids. +- **Watcher self-heals on provider redirect (integration):** session created at `/`, fake redirects to `/c/` after send; `watchSessionOnce` **without** `--navigate` → no `reattach-mismatch`, `session.conversationUrl` rewritten to live URL, poll proceeds. +- **Reattach with/without `--navigate` (integration):** (a) *different conversation*, no `--navigate` → `reattach-mismatch` preserved (don't hijack the user's other tab); (b) same + `--navigate` → navigates; (c) same-conversation drift, no `--navigate` → recovers silently. +- **Adaptive interval (unit, pure fn):** `nextInterval` fast after send, medium while streaming, geometric backoff while polling; a long-deadline (pro) sequence converges to the slow floor and never below `minMs`. + +**Tier 2 (if built)** +- **Nudge shortens wait (integration, fake CDP):** synthetic stop-button-removed / networkIdle mid-sleep → next poll starts before `interval` elapsed. +- **Fallback-to-poll correctness (load-bearing):** (a) never fire any event → completion still detected within the Tier-1 floor; (b) fire networkIdle *early while still streaming* → poll returns `polling`, NOT `complete`; (c) make `earlyWake()` reject (simulate CDP detach) → loop catches, `sleep` still resolves, watch continues; (d) flood events → poll rate capped by the floor. + +## Open Risks / Tradeoffs (honest CDP-event assessment) + +- **The user's "risky" call is correct, and the search confirms it.** `Page.lifecycleEvent` networkIdle is documented as "not fired in some cases" (devtools-protocol #154). A design that *trusted* it would be fragile. Using it only as an early-*nudge* with poll-as-truth converts that unreliability from a correctness bug into, at worst, latency capped at today's 15s floor. +- **Tier-1 carries the genuine behavior-change risk, not Tier-2.** Self-healing `conversationUrl` (1C/1D) means the watcher follows a drifted URL without asking. Mitigation: strict gating on *conversation-id equality* (1B) — a different id still requires `--navigate`. Be conservative here: silently attaching to the wrong conversation is worse than a 15s delay. +- **Selector coupling** (stop-button testid) is the Tier-2 liability, but unlike completion detection it incurs no correctness penalty when it rots. +- **Adaptive polling raises request volume in the fast band** — bounded by `minMs` and backoff once `polling` stabilizes; the pro/Deep-Research path explicitly backs off. +- **Scope discipline:** Tier 1 is mostly *consolidation* — `shouldPreferCurrentProviderUrl` + self-heal already exist in `tab-recovery.mjs`; the watcher just isn't using them. Ship Tier 2 behind `--cdp-nudge` (default off), with the floor and `Promise.race(...).catch()` containment as non-negotiable. + +## References (web) + +devtools-protocol #154 (networkIdle unreliable — validates poll-as-safety-net); Chrome DevTools Protocol reference (Page/Network/DOM); Chrome for Developers "Detect DOM changes with MutationObserver" (stop-button-disappearance nudge). diff --git a/devlog/_fin/260619_watch_notification_gaps/20_pressure_test_verdict.md b/devlog/_fin/260619_watch_notification_gaps/20_pressure_test_verdict.md new file mode 100644 index 00000000..ebaa78a8 --- /dev/null +++ b/devlog/_fin/260619_watch_notification_gaps/20_pressure_test_verdict.md @@ -0,0 +1,38 @@ +# Watch/Notify — Pressure-Test Verdict (2026-06-20) + +> Adversarial review of `01_root_cause.md` + `10_solution_plan.md`. Outcome: **~75% premature optimization + 1 misdiagnosis, around one genuine cheap consolidation win.** The MVV is even smaller than the plan claimed — and most of the folder should close to `_fin`. + +## Verdict per claim + +| Claim | Verdict | Why | +|------|---------|-----| +| 1 · 15s poll latency → adaptive polling | **DROP** | Premature vs the documented `send→watch→bgtask` fire-and-forget workflow — no human/agent is staring at the screen, so 15s is invisible. The pro back-off half belongs to `260619_timeout_adaptive_scaling`, not here. | +| 2 · URL self-heal / watcher reuse existing tolerance | **KEEP** | Real asymmetry; fix is even smaller than stated (see below). The "already exists, just unused" win — same shape as skill-envelope's `--system`. | +| 3 · Per-provider conversation-id registry (gemini/grok) | **DROP** | Misdiagnosed: gemini/grok poll paths never call the `/c/{id}` regex; their guard is host + CDP target-id (`gemini-live.mjs:415`, `grok-live.mjs:243`). `extractConversationId` is chatgpt-local. Nothing broken → YAGNI. | +| 4 · Tier 2 CDP early-nudge | **DROP** | User already vetoed as risky; devlog itself made it opt-in default-off; it optimizes a latency claim 1 shows is invisible. Pure cost. | +| 5 · Reattach requires `--navigate` (#77) | **DOWNSCOPE** | Real in-repo P0 row (`docs/production-readiness.md:41`) but internal, not an external report. The watch-path half is fixed for free by claim 2. | + +## The decisive finding (fix is smaller than the plan said) + +The watcher **already routes through the self-healing resolver** — `watchSessionOnce` wraps `withSessionPage` (`watcher.mjs:176`) → `resolveSessionPage(... allowNavigate:true)` (`tab-recovery.mjs:392`), which self-heals root→`/c/` drift and returns the **updated** session (`tab-recovery.mjs:314-318`). But the watcher's callback destructures only `{ page, targetId }` (`watcher.mjs:176`), **discards the healed `session`**, then runs `ensureWatcherAttached(page, session, options)` (`:179`) against the **stale** `session` captured at `:116` using the strict hash-only `urlsEquivalentForWatch` (`:559-569`). So it re-introduces the very mismatch the resolver just healed and surfaces it as `status:'reattach-mismatch'` (`:183`). + +→ The fix is **deletion/consolidation, not new code.** + +## Already-exists-just-unused (the cheap real win) + +- `resolveSessionPage` self-heal (`tab-recovery.mjs:314-325`) runs **inside** the watcher already — then is overridden by a redundant strict re-check reading a stale session. +- `shouldPreferCurrentProviderUrl` + `urlsCompatible` (`tab-recovery.mjs:230-243,412-425`) already encode host-alias / trailing-slash / root-drift tolerance. The watcher's `urlsEquivalentForWatch` is a strictly-weaker duplicate to retire. + +## Minimal Viable Version + +Two micro-edits in `web-ai/watcher.mjs`: +1. Destructure and **use the resolved `session`** the resolver already healed (instead of closing over the stale outer `session` from `:116`). +2. Delete the redundant `ensureWatcherAttached` strict re-check (`:398-412`) / retire `urlsEquivalentForWatch`; keep `--navigate` governing only a genuinely *different* conversation (different target/host, no id match). + +Plus one table-driven unit test (drifted `/` → `/c/` yields no `reattach-mismatch` and rewrites `conversationUrl`) on the existing fake-provider harness. **Stop there.** ≈ 30-line consolidation PR. + +**Cut entirely:** adaptive interval (1A), conversation-id registry (1B — misdiagnosed), all of Tier 2 CDP nudge. + +## Recommendation + +DOWNSCOPE hard; close most of the folder to `_fin`. Keep only the ~30-line watcher consolidation (retires the watch-path half of #77). The "15s too slow" framing fails the user-visible test under the bgtask workflow; the only latency concern with teeth is the *opposite* (pro hammering), already owned by the timeout folder. `10_solution_plan.md` is **superseded by this verdict**; `01_root_cause.md` stays as background. diff --git a/devlog/_fin/260619_webai_stability/00_index.md b/devlog/_fin/260619_webai_stability/00_index.md new file mode 100644 index 00000000..a3703ec1 --- /dev/null +++ b/devlog/_fin/260619_webai_stability/00_index.md @@ -0,0 +1,72 @@ +# 260619 — Web-AI Stability & Concurrency: Index + Locked Decisions + +> Coordinating doc for the four 2026-06-19 devlog folders. This round is **documentation only** (root-cause + plan); no code is committed. Trigger: gallery feedback (쿠마방와) on agbrowse's web-AI driver — timeout granularity, tab-parallel instability, skill/attachment envelope, and a response-complete notification hook. + +## The four work folders + +| # | Folder | Theme | +|---|--------|-------| +| 1 | `260619_timeout_adaptive_scaling/` | Per-(model, effort) timeout table; intermediate-"thinking" false-completion; adaptive poll backoff | +| 2 | `260619_tab_parallel_stability/` | Multi-agent CDP sharing: TOCTOU, lease↔session race, send+poll mutex, crashed-lease leak | +| 3 | ~~`260619_skill_envelope_integration/`~~ → **closed** | Capability already exists (`--system` + `--file`). Reclassified code→docs-only, moved to `_fin/260620_skill_envelope_already_capable.md`. Fix = small SKILL.md + `--help` upgrade. | +| 4 | `260619_watch_notification_gaps/` | Lower completion-detection latency; format-agnostic URL; optional CDP early-nudge | + +**Scope update (2026-06-20):** Only areas 1, 2, 4 are real code work. Area 3 (skill envelope) was found to be already-capable — a discoverability/UX problem, not architecture — and is closed to `_fin`. + +## Pressure-test synthesis (2026-06-20) + +All four areas were adversarially pressure-tested ("is this REALLY needed?"). The skill-envelope pattern repeated everywhere: **existing mitigations under-credited, plans over-engineered.** Net — the four elaborate plans collapse to a handful of small, surgical fixes. Locked decisions (pro=3600s, vendor5/global12-16) still hold; only the *implementation surface* shrank. + +| Area | Original plan | Survived (MVV) | Verdict | +|------|---------------|----------------|---------| +| **timeout** | new `timeout-policy.mjs` + 20-row table + thinking-indicator state machine + adaptive backoff | 4-entry tier-default map + export `pro=3600` + 1 doc line + 1 test | DOWNSCOPE ~80%; claims 2/3/4 dropped (`--timeout` + action-button signal already exist) | +| **tab** | atomic-acquire + per-session mutex + heartbeat/TTL unification + admission/FIFO/priority | record-before-bind reorder + active-count cap (revert `maxTabs:Infinity`) + PID reaper + doc fix | DOWNSCOPE & split; **the real work for the multi-agent goal.** Claims 1/3 already mitigated by `active-command.target-owned` cross-process mutex; claim 5 out-of-scope | +| **skill envelope** | `buildEnvelope` refactor + trust-tier sections + manifest | docs: `--help` + SKILL.md (done, `e88a520`) | CLOSED → `_fin` | +| **watch** | adaptive interval + per-provider id registry + CDP nudge (Tier1+2) | ~30-line watcher consolidation (use the healed session it already discards) | DOWNSCOPE ~75%; 15s latency invisible under bgtask; registry misdiagnosed; Tier2 vetoed | + +**Bugs the pressure-test found in the plans themselves:** (a) timeout sketch reuses `pollTimeoutSec`, which is already the watcher's per-iteration timeout (name collision); (b) the watcher already routes through the self-healing resolver but discards the healed `session` and re-checks a stale one — so the watch fix is *deletion*, not new code. + +**Revised cross-cut:** the 3-TTL unification is now **deferred/out-of-scope** — `tab` needs only the single `pro=3600` constant exported from `timeout`, not the full TTL program. The dependency between areas 1 and 2 is now one constant, not a coupled rewrite. + +Each `260619_*` folder has a `20_pressure_test_verdict.md` with the per-claim KEEP/DOWNSCOPE/DROP table and evidence; the `10_solution_plan.md` files are retained as full background but **superseded for scope** by the verdicts. + +## Implementation status (2026-06-21) + +Per the user directive "patch everything except tab, close to _fin, push": + +| Area | Status | Where | +|------|--------|-------| +| **skill envelope** | docs patched (`e88a520`) | `_fin/260620_skill_envelope_already_capable.md` | +| **timeout** | code patched (`1d86985`) — tier-aware default, `pro=3600` | `_fin/260619_timeout_adaptive_scaling/` | +| **watch** | code patched (`1d86985`) — watcher reuses healed session (#77 watch-path) | `_fin/260619_watch_notification_gaps/` | +| **tab** | code patched — record-before-bind, active-session caps, PID reaper, stale TTL docs | `_fin/260619_tab_parallel_stability/` + `_fin/260621_tab_stability_mvv_closeout/` | + +Verification for the timeout+watch patch: vitest 837/837 unit + 14/14 affected + 6/6 integration; `npm run gate:all` 16/16. Verification for the tab MVV closeout is recorded in `_fin/260621_tab_stability_mvv_closeout/00_plan.md`. + +Each folder has: `00_overview.md` (problem framing), `01_root_cause.md` (verified, with `file:line` + real code), `10_solution_plan.md` (design + code sketch + test strategy + risks). + +## Locked decisions (2026-06-19 interview) + +1. **Scope** — all four areas are documented to devlog with *clear root-cause analysis*. No code implementation this round. +2. **Concurrency target** — **vendor ≤ 5, global 12–16** concurrent sessions on one Chrome (today: vendor 3 / global 8). Sized for "jaw 직원 3명 + boss" sharing one CDP. +3. **Timeout strategy** — **hardcoded mapping table** (no history-learning). Tiers: instant ≈ 120s, thinking/standard ≈ 600s, **pro / heavy / deep-research = 3600s** (one hour, explicit user choice). + +## ⚠️ Cross-cutting insight (independently confirmed by a sub-agent AND the Backend employee) + +The single biggest risk to the multi-agent goal is **TTL drift under a pro=3600s wait**. Today three independent TTLs govern a live session and none of them knows about pro=3600s: + +| TTL | Value today | Source | Failure at pro=3600s | +|-----|-------------|--------|----------------------| +| Session command lock | 35 min | `session-store.mjs:50` | Expires mid-pro-wait → another command can seize the session | +| Active-command heartbeat | 2 min | `active-command-store.mjs:37` | Expires mid-poll → cleanup stops protecting the tab | +| Active-session lease | **none** | `tab-lease-store.mjs:221-236` | Crashed session locks a tab forever | + +**Therefore decision #3 (timeout table) is a prerequisite for decision #2 (scale to 12–16):** the pro=3600s value must become the *single source of truth* for a model-aware deadline, and all three TTLs must derive from it + a 60s heartbeat, **before** raising the concurrency caps. Folders 1 and 2 are coupled — see `260619_timeout_adaptive_scaling/10_solution_plan.md` (§ deadline table) and `260619_tab_parallel_stability/10_solution_plan.md` (§ C lease TTL + § E scaling). + +## Test coverage (area 5) — woven, not separate + +The originally-listed "5. test coverage gaps" is **not** a standalone folder; each plan's `## Test Strategy` section absorbs the relevant cases (concurrency, mutex, lease TTL, mid-stream pause, URL drift, fallback-to-poll). Existing harnesses to extend: `test/integration/web-ai-fake-chatgpt.test.mjs`, `test/unit/web-ai-session-store.test.mjs`, `test/unit/web-ai-watcher.test.mjs`, `test/unit/web-ai-question.test.mjs`, `test/unit/content-boundary.test.mjs`. + +## Open reference + +- Issue **#77** (P0) — web-AI durable session recovery matrix, incl. "root-to-conversation URL drift". Confirmed in `docs/production-readiness.md:41`. Folder 4 closes the watch-path half of it. diff --git a/devlog/_fin/260620_skill_envelope_already_capable.md b/devlog/_fin/260620_skill_envelope_already_capable.md new file mode 100644 index 00000000..301cbf80 --- /dev/null +++ b/devlog/_fin/260620_skill_envelope_already_capable.md @@ -0,0 +1,43 @@ +# Skill Envelope — already capable; reclassified to docs-only (closed 2026-06-20) + +> Was `260619_skill_envelope_integration/` (a proposed code project). After verifying the code with the author, the conclusion is: **the capability already exists**. This is a discoverability/UX problem, not an architecture problem. Closed; the only action is a small `skills/web-ai/SKILL.md` + `agbrowse --help` upgrade (no `buildEnvelope` refactor). + +## What already works today (verified) + +| Need | Flag | Code | Trust | +|------|------|------|-------| +| Trusted operating instructions | `--system "..."` | `web-ai/question.mjs:145` → `[SYSTEM]` trusted section | ✅ honored | +| File attachment (dialog-free, native) | `--file ` (repeatable) | `cli.mjs:554` → `chatgpt.mjs:257` `attachLocalFilesLive()`; `attachmentPolicy:'upload'` `cli.mjs:645` | model sees it natively | + +So **`--system "이 PDF에서 breaking change 추출" --file spec.pdf` already works** and the model sees the file as a real ChatGPT attachment. "Instruction + attachment" is not a missing feature. + +## What the earlier devlog overstated (corrected) + +- ❌ "upload 시 모델이 파일 존재를 모름" — wrong. `attachLocalFilesLive` uploads a native composer attachment; the model sees it. The envelope text merely lacks a manifest *line*. +- ⚠️ "개발자 지침 무시됨" — overstated. Only true if instructions go in `--context` (untrusted, `question.mjs:153`). The correct channel `--system` is trusted. + +## Dropped code items (per author: "1,2,3,4는 딱히 필요가 없고") + +1. **Persist skill/tool selection to `session.envelopeSummary`** — on resume, agbrowse reattaches to the live ChatGPT tab whose tool state already lives server-side; agbrowse doesn't re-apply tools on resume, so this would only be agbrowse-side bookkeeping. Not needed for correctness. (If resume-reporting ever matters: a ~5-line `updateSession` after `chatgpt.mjs:224`. Revisit only if asked.) +2. **`buildEnvelope` unification / `[DEVELOPER INSTRUCTIONS]` + `[ATTACHMENT MANIFEST]` sections** — convenience refactor; the `--system`/`--file` primitives already cover the real need. +3. **Split `--context` into trusted/untrusted** — `--system` already is the trusted channel; just document it. A footgun, not a gap. +4. **Attachment manifest text on upload** — model sees plain `--file` natively; only marginally useful for ZIP context-packs. + +## The actual fix (docs-only — UX/discoverability) + +`skills/web-ai/SKILL.md` documents `--file` and context-packs well but **never mentions `--system` as the trusted-instruction channel, nor that `--context` is untrusted data**. Add a short "where do my instructions go" subsection + an `agbrowse --help` clarification: + +- Trusted operating instructions / skill guidance → `--system` (or the USER fields). **Not** `--context`. +- `--context` = untrusted reference *data* (treated as data-only; instructions inside are ignored by design). +- File for the model to read/analyze → `--file ` (native attachment; repeatable). +- Philosophy: if attachment intent is ambiguous, the agent can just **ask** rather than rely on envelope machinery. + +This matches the gallery feedback (쿠마방와): "에이전트가 스킬 오해한 거 제대로 이해시키니 낫더라" — the win was clarifying the skill, not rebuilding the pipeline. + +## Status + +- [x] Investigated, capability confirmed present +- [x] Reclassified code project → docs-only, moved to `_fin` +- [ ] SKILL.md + `--help` wording upgrade (pending author go-ahead) + +Sibling code work (genuinely real, devlogs in `devlog/_plan/260619_*`): timeout_adaptive_scaling, tab_parallel_stability, watch_notification_gaps. diff --git a/devlog/_fin/260621_mcp_wait_response_recovery/00_plan.md b/devlog/_fin/260621_mcp_wait_response_recovery/00_plan.md new file mode 100644 index 00000000..1e67a55c --- /dev/null +++ b/devlog/_fin/260621_mcp_wait_response_recovery/00_plan.md @@ -0,0 +1,222 @@ +# 00 — MCP Wait Response Recovery Plan + +Date: 2026-06-21 +Issue: https://github.com/lidge-jun/agbrowse/issues/79 +Scope: one PABCD pass + +## Summary + +`agbrowse@0.1.14` can submit a long ChatGPT Pro Extended run through MCP, but +`web_ai_wait_response` currently waits through the direct provider poll path. +That path does not reuse the CLI session-bound recovery path, does not register +an MCP active command, and can mark an already completed session as `timeout`. + +This plan fixes the local code-level MCP wait/resume contract while keeping the +CLI poll behavior compatible. It does not claim to prevent every MCP host from +enforcing its own request timeout, because a host-side `-32001 Request timed out` +can still terminate the client request before agbrowse returns a payload. The +goal of this pass is to make the agbrowse side session-bound, visible in +active-command evidence, recoverable when the provider poll returns a timeout, +and monotonic with respect to completed sessions. Docs will explicitly tell MCP +callers to preserve `sessionId` and use short repeated waits, resume, CLI poll, +or a sufficiently long host timeout for long Pro/Deep Research runs. + +## Root Cause Evidence + +- `web-ai/mcp-server.mjs` wraps `web_ai_submit_prompt` and `web_ai_copy_markdown` + in `withMcpActiveCommand(...)`, but `web_ai_wait_response` and + `web_ai_session_resume` call `pollByProvider(...)` directly. +- `agbrowse web-ai poll --session ` uses `withSessionCommandLock(...)`, + session page resolution, and active-command recording before provider poll. +- `web-ai/chatgpt.mjs`, `web-ai/gemini-live.mjs`, and `web-ai/grok-live.mjs` + set `updateSession(session.sessionId, { status: 'timeout' })` on poll timeout. +- `web-ai/session-store.mjs` applies patches as a plain merge, so a later timeout + can overwrite `status: 'complete'`. +- Existing MCP tests pass, but they do not cover long-running + `web_ai_wait_response`, session-bound recovery, or complete-to-timeout + downgrade prevention. + +## Requirements + +1. MCP `web_ai_wait_response` and `web_ai_session_resume` must use the same + session-bound lock/recovery semantics as CLI `poll --session`. +2. MCP wait/resume must register an active command so diagnostics can identify + which MCP tool owns a provider target. +3. Provider poll timeout must remain recoverable for incomplete sessions and + include enough session evidence for later poll/resume. +4. Completed sessions must not be downgraded to `timeout` by later polls. +5. MCP schema/help must communicate that long waits are recoverable and that + callers should preserve `sessionId`. +6. Add focused regression tests and update source-of-truth docs/counts. + +## Plan + +### 10.1 — Session Timeout Mutation Helper + +Modify: + +- `web-ai/session.mjs` +- `web-ai/chatgpt.mjs` +- `web-ai/gemini-live.mjs` +- `web-ai/grok-live.mjs` + +Add a helper in `session.mjs`: + +```js +export function markSessionTimeout(sessionId, patch = {}) {} +``` + +Behavior: + +- Re-read the current session before mutation. +- If no session exists, return `null`. +- If current status is `complete` or `completedAt`/`answer` is present, keep + `status: 'complete'` and append a warning such as + `timeout-after-complete-ignored`. +- Otherwise patch `status: 'timeout'`, `lastError`, warnings, and `updatedAt`. +- Merge warnings explicitly by reading the current `session.warnings`; the + session store patch operation is a plain merge and does not append arrays. + +Replace direct provider calls: + +```js +updateSession(session.sessionId, { status: 'timeout' }) +``` + +with: + +```js +markSessionTimeout(session.sessionId, { + lastError: { errorCode: 'provider.poll-timeout', message: 'timed out waiting for answer' }, +}) +``` + +Provider return objects should include: + +- `recoverable: true` +- `retryHint: 'poll-or-resume'` +- `deadlineAt` when a session exists +- `conversationUrl` when a session exists + +### 10.2 — MCP Session-bound Wait/Resume + +Modify: + +- `web-ai/mcp-server.mjs` + +Add imports: + +```js +import { withSessionCommandLock } from './session-store.mjs'; +import { withSessionPage } from './tab-recovery.mjs'; +``` + +Add a helper: + +```js +async function runMcpSessionPoll(name, args, deps) {} +``` + +Behavior: + +1. Resolve `sessionId` from `args.sessionId`. +2. Load the stored session and provider, failing fast if missing. +3. Run: + - `withSessionCommandLock(sessionId, ...)` + - `withSessionPage(deps, sessionId, ...)` + - `withMcpActiveCommand(name, provider, sessionDeps, argsWithSessionId, ...)` + - provider `pollByProvider(...)` +4. Pass a session-aware `deps` object whose `getPage`, `getTargetId`, and + `getCdpSession` resolve to the recovered session page. +5. Return provider poll result, preserving structured `timeout` evidence instead + of throwing a hard MCP error for normal provider timeout. +6. If provider poll returns a recoverable `tab-crashed` result, keep the result + structured and recoverable for MCP callers rather than throwing like the CLI + interactive path. MCP callers need the `sessionId` evidence more than a thrown + terminal shell error. + +This deliberately does not add a new cancellation protocol in this pass. It +does make active command ownership visible and prevents concurrent session +mutation while the MCP wait/resume is running. + +### 10.3 — MCP Schema and Source-of-truth Docs + +Modify: + +- `web-ai/tool-schema.mjs` +- `structure/commands.md` +- `structure/runtime_contracts.md` +- `structure/stability-upgrade/01_operational_weakness_register.md` +- `skills/web-ai/SKILL.md` +- `devlog/00_index.md` +- `structure/str_func.md` + +Schema/doc wording: + +- `web_ai_wait_response`: session-bound wait that can return recoverable + `timeout` while provider work continues. +- `web_ai_session_resume`: resume stored session through the same recovery path. +- Long Pro/Deep Research runs should preserve `sessionId` and retry/poll later. +- MCP clients with short host-level request timeouts should prefer repeated + bounded waits or CLI `web-ai poll --session` instead of a single long blocking + MCP request. + +Add register row: + +- `STAB-09`: MCP wait timeout/session recovery mismatch, closed by this pass. + +### 10.4 — Regression Tests + +Modify: + +- `test/integration/web-ai-mcp-server.test.mjs` +- `test/unit/web-ai-provider-session.test.mjs` +- `test/unit/web-ai-tool-schema.test.mjs` + +Tests: + +1. MCP `web_ai_wait_response` and `web_ai_session_resume` source path uses + `withSessionCommandLock`, `withSessionPage`, and `withMcpActiveCommand`. +2. Provider-level fake-page timeout tests verify structured timeout payloads + with `recoverable`, `retryHint`, `sessionId`, `deadlineAt`, and + `conversationUrl`. Do not make this a live-browser MCP integration test. + If a behavioral MCP test is needed, mock `tab-recovery.mjs` explicitly before + importing `mcp-server.mjs`; otherwise keep MCP wait coverage as source-string + contract plus provider fake-page timeout tests. +3. `markSessionTimeout` preserves completed sessions and appends a warning. +4. Existing provider source contracts expect `markSessionTimeout` instead of + raw `updateSession(... { status: 'timeout' })`. +5. MCP tool schema descriptions mention recoverable/session-bound wait. + +## Verification + +Run: + +```bash +npm run test:mcp +npx vitest run test/unit/web-ai-provider-session.test.mjs test/unit/web-ai-tool-schema.test.mjs +npm run test:release-gates +git diff --check +``` + +If source counts change, update `structure/str_func.md` and verify: + +```bash +bash structure/verify-counts.sh +``` + +## Acceptance Criteria + +- GitHub issue #79's local code-level root causes are addressed: MCP wait/resume + no longer bypasses session lock/recovery/active-command semantics, provider + timeout results are recoverable, and completed sessions cannot be downgraded + by later timeout polls. +- Host-side MCP request timeout (`-32001`) is documented as a client/runtime + timeout boundary; this pass does not implement a background cancellation or + lease protocol. +- MCP wait/resume no longer bypasses session lock/recovery/active-command + semantics. +- Completed session records remain complete after later timeout polls. +- Timeout result remains recoverable and points users back to session poll/resume. +- Docs and structure counts are consistent. +- Changes are committed locally and not pushed. diff --git a/devlog/_fin/260621_tab_stability_mvv_closeout/00_plan.md b/devlog/_fin/260621_tab_stability_mvv_closeout/00_plan.md new file mode 100644 index 00000000..4a92dcd2 --- /dev/null +++ b/devlog/_fin/260621_tab_stability_mvv_closeout/00_plan.md @@ -0,0 +1,154 @@ +# Tab Stability MVV Closeout Plan + +Date: 2026-06-21 +Branch: `dev-vision-upgrade` +Target: PR to `main`, stop before merge + +## Objective + +Close the remaining tab-stability work on `dev-vision-upgrade` with the +pressure-tested MVV only: + +1. Record the active tab lease before binding the session to the tab. +2. Enforce active-session capacity caps: vendor 5, global 14, hard tab cap 20. +3. Reap active leases owned by dead OS processes. +4. Fix stale provider pool TTL docs from 15m to 30m. + +Oracle follow-up stays out of scope. Issue `#79` is linked from the PR body with +`Closes #79`; the issue should close automatically only when the PR is merged. + +## Current Evidence + +- `devlog/_fin/260619_tab_parallel_stability/20_pressure_test_verdict.md` + supersedes the earlier broad plan and keeps only MVV work. +- `web-ai/chatgpt.mjs`, `web-ai/gemini-live.mjs`, and + `web-ai/grok-live.mjs` still bind session-to-tab before recording the durable + active lease. +- `web-ai/cli.mjs` still calls `cleanupIdleTabs` with + `maxTabs: Number.POSITIVE_INFINITY`. +- `web-ai/cli.mjs` and `skills/browser/browser.mjs` still document provider pool + TTL as 15m even though the store default is 30m. +- GitHub issue `#79` is open and there is no existing PR from + `dev-vision-upgrade` to `main`. + +## Implementation Plan + +### 1. Lease Store + +Modify `web-ai/tab-lease-store.mjs`: + +- Add `ownerPid` to `Lease` and `LeaseInput`. +- Add active cap defaults: + - `AGBROWSE_PROVIDER_ACTIVE_MAX_PER_KEY` default `5`. + - `AGBROWSE_PROVIDER_ACTIVE_GLOBAL_MAX` default `14`. +- Add `ProviderActiveCapacityError` with machine-readable fields. +- Make `recordActiveLease` fail fast when a new active lease would exceed the + per-key or global active cap. +- Preserve replacement semantics for the same target or same session before + counting, so rebinding an existing active lease does not self-fail. +- Persist `ownerPid`, defaulting to `process.pid`. +- Reuse the existing `isPidAlive` helper from `skills/browser/profile-lock.mjs` + and make `cleanupLeasedTabs` mark active-session leases for close only when + `ownerPid` is known and no longer alive. + +### 2. Provider Send Paths + +Modify: + +- `web-ai/chatgpt.mjs` +- `web-ai/gemini-live.mjs` +- `web-ai/grok-live.mjs` + +At each send/deep-research session creation site, record the active lease before +calling `bindSessionToTab`. + +If active capacity is exceeded, the send path should fail before the session is +bound to the target. The durable session row may exist for diagnostic purposes, +but it must not claim ownership of a tab. The thrown error should carry a +stable code so CLI/MCP JSON error handling can surface the cap reason. + +### 3. Tab Cap and Docs + +Modify `web-ai/cli.mjs`: + +- Replace `maxTabs: Number.POSITIVE_INFINITY` with the existing + `DEFAULT_MAX_TABS` from `skills/browser/tab-lifecycle.mjs`. +- Update provider pool help text only for the stale TTL default: + `TTL=15m` becomes `TTL=30m`. Pool `maxPerKey=3` and `globalMax=8` remain + pool-specific defaults unless changed by `AGBROWSE_PROVIDER_POOL_*`. +- Add or update help text for active caps separately: + `AGBROWSE_PROVIDER_ACTIVE_MAX_PER_KEY=5` and + `AGBROWSE_PROVIDER_ACTIVE_GLOBAL_MAX=14`. + +Modify `skills/browser/tab-lifecycle.mjs`: + +- Change the canonical `AGBROWSE_MAX_TABS` default from `10` to `20` so + browser cleanup and web-ai provider tab creation share one tab cap. + +Modify `skills/browser/browser.mjs`: + +- Update matching provider pool help text to keep `maxPerKey=3`, + `globalMax=8`, and `TTL=30m`. +- Update `AGBROWSE_MAX_TABS` help to default `20`. + +Modify user-facing docs that mention `AGBROWSE_MAX_TABS`: + +- `README.md` +- `skills/web-ai/SKILL.md` + +Also update `skills/web-ai/SKILL.md` if it still documents the stale provider +pool TTL as 15 minutes. + +### 4. Tests + +Modify `test/unit/tab-lifecycle.test.mjs`: + +- Verify active lease records include `ownerPid`. +- Verify per-key active cap rejects the next new active lease. +- Verify global active cap rejects the next new active lease. +- Verify replacing the same active session does not trip the cap. +- Verify cleanup reaps an active lease with a dead `ownerPid`. + +Add source-contract assertions where direct behavior is hard to isolate: + +- send paths record active leases before `bindSessionToTab`. +- `ensureProviderTab` no longer uses `Number.POSITIVE_INFINITY` for tab cleanup. +- TTL docs no longer mention the stale `TTL=15m` provider-pool default. +- `AGBROWSE_MAX_TABS` docs and runtime default agree on `20`. + +### 5. Devlog Move + +After code and gates pass, move completed branch-closeout planning folders to +`devlog/_fin`: + +- `devlog/_fin/260621_mcp_wait_response_recovery` +- `devlog/_fin/260621_tab_stability_mvv_closeout` +- `devlog/_fin/260619_tab_parallel_stability` + +Keep Oracle follow-up plans in `_plan`. +Move `260621_mcp_wait_response_recovery` only if MCP acceptance gates pass, +because `#79` closure depends on the already-committed `d647d58` timeout +recovery work plus this branch PR, not on the tab-stability MVV alone. + +## Verification Gates + +Run: + +```bash +npm run typecheck +npx vitest run test/unit/tab-lifecycle.test.mjs test/unit/web-ai-provider-session.test.mjs test/unit/web-ai-tool-schema.test.mjs +npm run test:mcp +npm run test:release-gates +git diff --check +``` + +Then run an independent read-only verification before commit/PR. + +## Git/PR Plan + +1. Commit implementation and devlog closeout as a stack commit on + `dev-vision-upgrade`. +2. Push `dev-vision-upgrade` to `origin`. +3. Create a PR targeting `main`. +4. Include `Closes #79` in the PR body. +5. Stop before merge. diff --git a/devlog/_plan/260513_oracle_zip_bundle_proposal.md b/devlog/_plan/260513_oracle_zip_bundle_proposal/00_proposal.md similarity index 99% rename from devlog/_plan/260513_oracle_zip_bundle_proposal.md rename to devlog/_plan/260513_oracle_zip_bundle_proposal/00_proposal.md index 98c01d53..64dbe496 100644 --- a/devlog/_plan/260513_oracle_zip_bundle_proposal.md +++ b/devlog/_plan/260513_oracle_zip_bundle_proposal/00_proposal.md @@ -18,7 +18,7 @@ Reference point: - Open PR #193 discusses browser max-file-size caps and skill ZIP guidance. This proposal is intentionally separate from the local agbrowse guardrail plan -in `devlog/_plan/260513_oracle_followup_guardrails_diff_plan.md`. +in `devlog/_fin/260513_oracle_followup_guardrails_diff_plan.md`. ## Part 1 — Easy Explanation diff --git a/devlog/_plan/260603_codebase_audit.md b/devlog/_plan/260603_codebase_audit/00_codebase_audit.md similarity index 100% rename from devlog/_plan/260603_codebase_audit.md rename to devlog/_plan/260603_codebase_audit/00_codebase_audit.md diff --git a/devlog/_plan/260603_issue_tracker.md b/devlog/_plan/260603_codebase_audit/01_issue_tracker.md similarity index 100% rename from devlog/_plan/260603_issue_tracker.md rename to devlog/_plan/260603_codebase_audit/01_issue_tracker.md diff --git a/devlog/_plan/260608_oracle_stability_gap/00_index.md b/devlog/_plan/260608_oracle_stability_gap/00_index.md index 0bd61ddc..841f96da 100644 --- a/devlog/_plan/260608_oracle_stability_gap/00_index.md +++ b/devlog/_plan/260608_oracle_stability_gap/00_index.md @@ -21,6 +21,9 @@ Oracle(steipete/oracle)은 ChatGPT Pro 브라우저 자동화에서 가장 성 | [06_dom_diagnostics.md](06_dom_diagnostics.md) | DOM 디버깅/진단 | **P2** | | [07_chrome_lifecycle.md](07_chrome_lifecycle.md) | Chrome 프로세스 생명주기 | **P2** | | [08_provider_abstraction.md](08_provider_abstraction.md) | 멀티 프로바이더 아키텍처 | **P3** | +| [30_oracle_0_15_delta_followup.md](30_oracle_0_15_delta_followup.md) | 0.11.1 이후 Oracle 0.15 델타와 agbrowse 추적 순서 | **P0/P1 follow-up** | +| [31_chatgpt_downloadable_artifacts_pabcd.md](31_chatgpt_downloadable_artifacts_pabcd.md) | ChatGPT 범용 다운로드 파일/ZIP 업로드 무결성 PABCD 계획 | **P0/P2** | +| [32_deep_research_session_followup_pabcd.md](32_deep_research_session_followup_pabcd.md) | Deep Research, model picker, later-session follow-up PABCD 계획 | **P0/P1/P2** | ## Severity Legend diff --git a/devlog/_plan/260608_oracle_stability_gap/30_oracle_0_15_delta_followup.md b/devlog/_plan/260608_oracle_stability_gap/30_oracle_0_15_delta_followup.md new file mode 100644 index 00000000..064c9d36 --- /dev/null +++ b/devlog/_plan/260608_oracle_stability_gap/30_oracle_0_15_delta_followup.md @@ -0,0 +1,228 @@ +# 30 — Oracle 0.15 Delta Follow-up + +Date: 2026-06-20 +Status: plan + +## Reference Pull + +Oracle reference repo: `/tmp/agbrowse-oracle-reference` + +Update command result: + +```text +git fetch --prune origin +git pull --ff-only +Already up to date. +``` + +Current Oracle head: + +```text +d5c5e954ecec856159d7eac62037ba19040c30e0 +2026-06-20 02:48:00 +0200 +build(deps): bump the dependencies group with 7 updates (#270) +``` + +Prior local Oracle review anchors: + +| Anchor | Date | Meaning | +| --- | --- | --- | +| `a1dbb13328dc75ef46a8b869618b4d5a8985c722` | 2026-05-12 | `0.11.1`-era guardrail follow-up basis | +| `1828e2b34b51d0888565bebd9f1245083b66601f` | 2026-05-13 | attachment-chip parity follow-up basis | + +Delta from `a1dbb133` to current Oracle head: + +```text +181 files changed, 27201 insertions(+), 4190 deletions(-) +``` + +Recent Oracle tags now visible in the reference clone: + +```text +v0.15.0 +v0.14.1 +v0.14.0 +v0.13.0 +v0.12.1 +v0.12.0 +v0.11.1 +``` + +## High-value Oracle changes since 0.11.1 + +### 0.12.x + +- `--perf-trace` startup timing and lazy loading. +- Multi-model partial success with classified provider failures. +- `--preflight`, `doctor --providers`, and redacted route diagnostics. +- Session lifecycle block with foreground/background/reattach metadata. +- `oracle docs check`. +- Opt-in ZIP browser bundle formatting. +- Timeout parsing and HTTP/stale timeout alignment. +- Model selection evidence and stricter Pro effort confirmation. + +### 0.13.x + +- Browser attachment timeout config. +- GPT-5.5 Instant picker row support. +- Project config defaults. +- Attachment readiness from the active ChatGPT composer. +- Scoped real picker-menu scanning. + +### 0.14.x + +- Later-session `--followup ` that reopens the exact saved ChatGPT conversation before sending a new prompt. +- `session --harvest` / `--live` recovery after Chrome was closed, using the saved conversation URL and manual-login profile. +- Persist ChatGPT downloadable files such as CSV, PDF, ZIP, wheel, and source distributions beside the transcript. +- MCP `chatgpt_image` and typed image output support. +- Deep Research report capture from out-of-process iframes, scoped to the active page/session. +- Login/auth probe hardening, exact saved-account selection, visible provider warning surfacing. +- Line-numbered prompt/text bundle context. +- Byte-preserving ZIP upload bundles for raw/archive/office/media files. + +### 0.15.x + +- `--copy-profile ` copies an active signed-in Chrome profile to a throwaway profile for login-free reuse, with cleanup and incompatible-mode rejection. +- Wait for the current ChatGPT Intelligence pill before selecting model/effort. +- Bounded retries for explicit model selection. +- Sequential generated-file downloads with browser-provided filenames and timeout attribution. +- Deep Research planning/status captures are rejected when ChatGPT returns a normal non-research answer. + +## agbrowse current coverage + +Already covered or partially covered: + +- ChatGPT code-mode ZIP retrieval exists in `web-ai/code-artifact.mjs`, including assistant/user filtering and zip validation. +- Generated ChatGPT image output exists in `web-ai/chatgpt-images.mjs`. +- Session artifacts exist in `web-ai/session-artifacts.mjs`. +- Deep Research exists in `web-ai/chatgpt-deep-research.mjs`, including iframe scan and report artifacts. +- Session resume/reattach/watch recovery exists in `web-ai/tab-recovery.mjs` and `web-ai/watcher.mjs`. +- Same-command ChatGPT batch follow-ups exist via `--follow-up`. +- Later-session prompt send already exists through `agbrowse web-ai query --session --prompt ` via `runBoundSendOrQuery()` and `withSessionPage()`. +- Capability truth table still treats later-session follow-up as deferred in the dedicated `--follow-up` capability row, so the remaining gap is parity labeling, discoverability, and stricter saved-conversation guards, not a greenfield feature. + +Still missing or worth auditing: + +- Generic ChatGPT downloadable file artifacts are not covered outside code ZIPs and generated images. Oracle now handles current-turn assistant files from known ChatGPT endpoints, including CSV/PDF/ZIP/wheel/source-dist. +- Deep Research extraction is still iframe-text oriented. It does not yet mirror Oracle's target-scoped OOPIF auto-attach/read-selection helper or incomplete-report rejection. +- Later-session follow-up needs an Oracle-parity audit: existing `query --session` can send a prompt into a saved session, but docs/truth-table wording and fail-closed conversation URL guards need to be reconciled. +- Profile-copy login reuse has no direct agbrowse equivalent. Existing `BROWSER_AGENT_HOME` and CDP reuse cover the normal flow but not "clone active user profile into throwaway profile". +- Sequential download attribution for generic generated files is not implemented because generic generated-file capture is not implemented. +- Byte-preserving ZIP upload should be audited against `web-ai/context-pack/` and `web-ai/chatgpt-attachments.mjs`; current context-pack zip behavior is not the same feature as Oracle's mixed raw/archive/office/media bundle writer. +- Oracle provider route diagnostics are mostly out of scope for agbrowse while agbrowse intentionally has no hosted API provider-routing layer. + +## Recommended follow-up order + +This delta is split into two PABCD-ready implementation plan documents: + +| File | Scope | Priority | +| --- | --- | --- | +| [31_chatgpt_downloadable_artifacts_pabcd.md](31_chatgpt_downloadable_artifacts_pabcd.md) | Generic ChatGPT downloadable files, session artifact descriptor expansion, sequential download attribution, byte-preserving upload audit | P0/P2 | +| [32_deep_research_session_followup_pabcd.md](32_deep_research_session_followup_pabcd.md) | Deep Research target-scoped capture, model picker/current-pill audit, existing `query --session` follow-up parity, profile-copy decision | P0/P1/P2 | + +### P0 — Generic ChatGPT downloadable file artifact capture + +Implement a ChatGPT generated-file artifact collector that is separate from code-mode ZIP retrieval. + +Required behavior: + +- Scope to the current assistant turn/run, not the whole visible conversation. +- Allow only known ChatGPT file endpoints and same-origin downloadable URLs. +- Reject path traversal and unsafe sandbox paths. +- Deduplicate aliases that point to the same file. +- Download sequentially and preserve browser-provided filenames when available. +- Stop attributing downloads after timeout so late downloads cannot be attached to the next file. +- Record descriptors through the existing session artifact mechanism. + +Test focus: + +- URL allowlist: `/backend-api/sandbox/download`, `/backend-api/files//download`, `/backend-api/files//content`, `/backend-api/estuary/content?id=file_...`. +- Non-ChatGPT host rejection. +- `/mnt/data/../` and encoded traversal rejection. +- Assistant-current-turn scoping, user text ignored. +- Duplicate alias dedupe. +- Timeout attribution. + +### P0/P1 — Deep Research target-scoped capture + +Upgrade `web-ai/chatgpt-deep-research.mjs` from "latest assistant or any research iframe text" to a scoped report-selection model. + +Required behavior: + +- Bind any iframe/OOPIF read to the active page session. +- Prefer completed Deep Research report targets over legacy frame fallback. +- Reject planning cards, progress/status pages, and normal non-research answers as final reports. +- Preserve the current artifact save contract through `session-artifacts.mjs`. + +Test focus: + +- Completed OOPIF report wins over incomplete frame text. +- Planning/status text fails clearly. +- Unrelated tab/iframe cannot be harvested. +- Legacy same-page iframe fallback still works. + +### P1 — Model picker/current-pill audit + +Compare agbrowse `web-ai/chatgpt-model.mjs` against Oracle 0.14.1/0.15.0 behavior: + +- current Intelligence pill wait +- explicit Thinking/Pro effort evidence +- bounded retries for model selection +- Instant row support and wrapper-row rejection + +This should be an audit before a patch, because agbrowse already has several model-selection guardrails from earlier parity work. + +### P1 — Later-session follow-up parity + +Audit and harden the existing later-session path: + +```text +agbrowse web-ai query --session --prompt +``` + +Required behavior: + +- Resolve the exact saved session and conversation URL. +- Fail closed on target mismatch unless navigation/recovery is explicitly authorized. +- Reuse existing `tab-recovery.mjs` and conversation URL checks. +- Do not send to provider root, a different thread, or an external URL. +- Update help/truth-table wording so users can distinguish same-command `--follow-up` from later-session `query --session`. + +### P2 — Profile-copy login reuse + +Treat this as optional unless users repeatedly hit "need active Chrome login but do not want to reuse the shared profile" flows. + +Questions before implementation: + +- Does `BROWSER_AGENT_HOME` plus headed CDP reuse already satisfy the agbrowse workflow? +- Is copying a real user Chrome profile acceptable for this project, including local keychain/cookie handling? +- Would a throwaway profile improve isolation enough to justify the added operational risk? + +### P2 — Byte-preserving ZIP bundle audit + +Audit context-package and attachment upload behavior for raw/archive/office/media file preservation. Do not assume this is solved by code-mode artifact ZIP verification; this is an upload-side bundle integrity issue. + +### Out of scope for now + +- Oracle API provider route diagnostics and `--allow-partial` provider failures remain out of scope unless agbrowse intentionally grows a hosted API provider-routing layer. +- MCP `chatgpt_image` parity is not urgent while generated-image output remains CLI-only in agbrowse's truth table. + +## Verification matrix for future patches + +| Area | Required verification | +| --- | --- | +| Generic generated files | unit tests for URL/path allowlist, current-turn scoping, dedupe, timeout attribution; fake ChatGPT download endpoint fixture | +| Deep Research | unit/fixture tests for completed report vs planning/status/normal answer; unrelated iframe rejection | +| Model picker | fixture tests for current Intelligence dialog, Instant row, wrapper rows, Pro effort evidence | +| Later-session follow-up | session resolver tests for exact URL, mismatch fail-closed behavior, `--navigate` recovery | +| Profile copy | platform-gated unit tests, temp-profile cleanup tests, incompatible-mode rejection | +| ZIP upload integrity | byte-level fixture comparing original raw/archive/office/media buffers after bundle write | + +After any implementation slice, run at minimum: + +```bash +npm run test:release-gates +git diff --check +``` + +Run `npm run gate:all` only when capability truth-table or release-claim surface changes. diff --git a/devlog/_plan/260608_oracle_stability_gap/31_chatgpt_downloadable_artifacts_pabcd.md b/devlog/_plan/260608_oracle_stability_gap/31_chatgpt_downloadable_artifacts_pabcd.md new file mode 100644 index 00000000..0b19e83a --- /dev/null +++ b/devlog/_plan/260608_oracle_stability_gap/31_chatgpt_downloadable_artifacts_pabcd.md @@ -0,0 +1,263 @@ +# 31 — ChatGPT Downloadable Artifacts PABCD + +Date: 2026-06-20 +Status: PABCD plan +Parent: [30_oracle_0_15_delta_followup.md](30_oracle_0_15_delta_followup.md) + +## Purpose + +Oracle 0.14+ added generic ChatGPT downloadable-file capture for CSV, PDF, +ZIP, wheel, source-distribution, and similar file outputs. agbrowse currently +has two narrower paths: + +- code-mode `/mnt/data/*.zip` retrieval in `web-ai/code-artifact.mjs` +- generated image retrieval in `web-ai/chatgpt-images.mjs` + +This plan keeps code-mode ZIP strictness intact and adds a separate generic +file artifact lane for normal ChatGPT answers that expose downloadable files. + +## Priority Map + +| ID | Priority | Outcome | +| --- | --- | --- | +| 31.1 | P0 | Generic assistant-turn downloadable file detection and safe URL/path allowlist | +| 31.2 | P0 | Sequential browser/download attribution and session artifact records | +| 31.3 | P1 | CLI/session visibility for saved generic file artifacts | +| 31.4 | P2 | Byte-preserving upload-side ZIP/context-pack audit | + +## P — Plan + +### Part 1 — Easy Explanation + +When ChatGPT creates a file that is not a code-mode ZIP or generated image, +agbrowse should save that file beside the session transcript. The feature must +only trust files from the current assistant turn, must reject unsafe URLs, and +must not accidentally attach a late download to the next file. Code-mode stays +separate because it has a stricter container-contract requirement. + +### Part 2 — Diff-level Precision + +#### NEW `web-ai/chatgpt-files.mjs` + +Create a new module instead of widening `web-ai/code-artifact.mjs`. + +Exports: + +```js +export function normalizeChatGptFileDownloadUrl(value) {} +export function normalizeChatGptSandboxUrl(value) {} +export async function readAssistantDownloadableFiles(page, { baselineAssistantCount } = {}) {} +export async function saveAssistantDownloadableFiles(page, deps, opts = {}) {} +``` + +Required behavior: + +- Accept only `https://chatgpt.com` and `https://chat.openai.com`. +- Accept only: + - `/backend-api/sandbox/download?path=/mnt/data/...` + - `/backend-api/files//download` + - `/backend-api/files//content` + - `/backend-api/estuary/content?id=file_...` +- Convert safe `sandbox:/mnt/data/...` URLs to `/backend-api/sandbox/download`. +- Reject `blob:`, external hosts, non-HTTPS URLs, explicit ports, backslashes, + null bytes, and any `..` path segment. +- Scan only assistant turns after `baselineAssistantCount`. +- Deduplicate aliases by `downloadUrl`, `sandboxUrl`, and original URL. +- Prefer filenames from `Content-Disposition`, then DOM `download`, then URL + basename, then `chatgpt-file-N.`. + +Suggested result shape: + +```js +{ + ok: true, + files: [{ + kind: 'file', + label: 'result.csv', + path: 'result.csv', + mimeType: 'text/csv', + sizeBytes: 1234, + sourceUrl: 'https://chatgpt.com/backend-api/files/.../download', + savedAt: '...' + }], + warnings: [] +} +``` + +#### MODIFY `web-ai/session-artifacts.mjs` + +Before: + +```js + * @property {'transcript'|'report'|'image'} kind +``` + +After: + +```js + * @property {'transcript'|'report'|'image'|'file'} kind +``` + +Add: + +```js +export function saveFileArtifact(sessionId, { filename, buffer, mimeType, sourceUrl }) {} +export function trySaveFileArtifact(sessionId, file) {} +``` + +Rules: + +- Reuse `resolveArtifactsDir(sessionId)`. +- Sanitize filename stem with the same path traversal protection used for + image artifacts. +- Preserve the extension from the resolved filename when present. +- Return `stage: 'artifact-file'` on save failure. +- Keep `appendArtifactRecord()` dedupe by `(kind, path)`. + +#### MODIFY `web-ai/chatgpt.mjs` + +Capture the assistant baseline at send time and pass it through poll/finalize. + +Existing signals to reuse: + +- `session.envelopeSummary?.assistantCount` as the already-recorded baseline +- private `countAssistantMessages(page)` only inside `web-ai/chatgpt.mjs` if a + new baseline must be captured in that same module +- `finalizeProviderTab(...)` +- `collectImages(...)` +- `resolveArtifactsDir(...)` + +Expected integration point: + +```js +const savedFiles = await saveAssistantDownloadableFiles(page, deps, { + sessionId: session.sessionId, + baselineAssistantCount: session.envelopeSummary?.assistantCount, +}); +``` + +Rules: + +- Run after a final assistant answer is detected and before archive. +- Do not run in `web-ai code` retrieval; code-mode continues to use + `web-ai/code-artifact.mjs`. +- Add warnings such as `file-artifact-save-failed:` without hiding the + answer. +- Append descriptors to the existing `session.artifacts` array. + +#### MODIFY `web-ai/tab-finalizer.mjs` + +If generic files are saved before archive, archive can proceed. If saving was +explicitly required by a future flag and failed, preserve the current +artifact-before-archive rule. + +No new archive policy should be introduced in this slice unless a caller adds a +required-file-artifact flag. + +#### MODIFY `web-ai/cli.mjs` + +No new top-level command in the first implementation slice. + +Human `sessions show` should already list session artifact descriptors through +the existing artifacts display. Add only the smallest text needed if generic +`kind: 'file'` descriptors render poorly. + +#### NEW `test/unit/chatgpt-files.test.mjs` + +Required tests: + +- Allows the four known ChatGPT file endpoint shapes. +- Rejects external hosts, non-HTTPS URLs, ports, `blob:`, unsafe sandbox paths, + encoded traversal, backslash, and null byte. +- Converts `sandbox:/mnt/data/result.csv` to a safe download URL. +- Scopes DOM scan to assistant turns after the baseline. +- Ignores user-turn links and stale assistant-turn links before the baseline. +- Deduplicates aliases for the same file. +- Derives filename from `Content-Disposition`. +- Stops sequential attribution after a timeout. + +#### MODIFY `test/unit/web-ai-session-artifacts.test.mjs` + +Add coverage for: + +- `trySaveFileArtifact()` success. +- `artifact-file` structured failure. +- `appendArtifactRecord()` dedupes `kind: 'file'` separately from `kind: 'image'` + even when paths match. + +#### OPTIONAL MODIFY `test/unit/web-ai-chatgpt*.test.mjs` + +Add only if the first implementation wires generic file capture directly into +`pollWebAi`/`queryWebAi` with testable fake pages. + +## A — Plan Audit Checklist + +Audit must verify: + +- The new module does not change code-mode ZIP behavior. +- `session-artifacts.mjs` remains under 500 lines after additions; split if not. +- `chatgpt.mjs` integration does not trigger downloads before final answer + completion. +- Existing generated-image collection remains independent. +- Artifacts are stored under `BROWSER_AGENT_HOME/sessions//artifacts`. +- No external URL or local path can escape the artifacts directory. + +## B — Build Slices + +1. Implement pure URL/path normalization helpers first. +2. Implement assistant-turn DOM scanner and dedupe. +3. Add `file` artifact save helpers. +4. Add sequential download/save behavior. +5. Wire into ChatGPT poll/finalization. +6. Add focused unit tests. +7. Run release gates and targeted tests. + +## C — Check + +Minimum verification: + +```bash +npm run test:release-gates +npx vitest run test/unit/chatgpt-files.test.mjs test/unit/web-ai-session-artifacts.test.mjs +git diff --check +``` + +Run broader tests if `chatgpt.mjs` finalization behavior changes: + +```bash +npx vitest run test/unit/web-ai-tab-finalizer.test.mjs test/unit/chatgpt-images.test.mjs +``` + +## D — Done Criteria + +- Generic ChatGPT file artifacts are saved and listed in session artifacts. +- Code-mode ZIP retrieval still uses `web-ai/code-artifact.mjs`. +- Generated images still use `web-ai/chatgpt-images.mjs`. +- Unsafe URLs and sandbox paths fail closed. +- Late downloads cannot be attributed to later file candidates. +- `structure/str_func.md` count snapshots are updated and + `bash structure/verify-counts.sh` passes if files are added. + +## 31.4 — Upload-side ZIP Integrity Audit + +This is not part of the P0 downloadable-file implementation. It is a P2 audit +because upload-side ZIP byte preservation is a different direction from +download-side artifact capture. + +Audit targets: + +- `web-ai/context-pack/` +- `web-ai/chatgpt-attachments.mjs` +- `web-ai/code-dev-context.mjs` + +Questions: + +- Does the current context-pack writer preserve bytes for raw/archive/office + media files? +- Are ZIP entries stored or deflated, and does that matter for ChatGPT upload + parsing? +- Are source filenames, extensions, and relative paths preserved? +- Does any text-bundle path still transform binary-ish input? + +If a patch is needed, create a separate `33_...` implementation plan rather +than mixing upload behavior into this P0 file-download slice. diff --git a/devlog/_plan/260608_oracle_stability_gap/32_deep_research_session_followup_pabcd.md b/devlog/_plan/260608_oracle_stability_gap/32_deep_research_session_followup_pabcd.md new file mode 100644 index 00000000..2d48f037 --- /dev/null +++ b/devlog/_plan/260608_oracle_stability_gap/32_deep_research_session_followup_pabcd.md @@ -0,0 +1,302 @@ +# 32 — Deep Research and Session Follow-up PABCD + +Date: 2026-06-20 +Status: PABCD plan +Parent: [30_oracle_0_15_delta_followup.md](30_oracle_0_15_delta_followup.md) + +## Purpose + +Oracle 0.14/0.15 hardened Deep Research, model selection, and session recovery. +agbrowse already has Deep Research, session reattach, watcher recovery, +same-command `--follow-up`, and a later-session send path via +`query --session --prompt `. It still lacks Oracle's newer +page-scoped OOPIF Deep Research report capture, and the existing later-session +path needs an Oracle-parity audit for discoverability, truth-table wording, and +saved-conversation fail-closed guards. + +This plan splits the work so the P0 capture correctness lands before more +visible command-surface changes. + +## Priority Map + +| ID | Priority | Outcome | +| --- | --- | --- | +| 32.1 | P0 | Deep Research target-scoped OOPIF/frame report capture and incomplete-result rejection | +| 32.2 | P1 | ChatGPT model picker/current Intelligence pill audit | +| 32.3 | P1 | Existing `query --session` later-session follow-up parity hardening | +| 32.4 | P2 | Profile-copy login reuse decision record | + +## P — Plan + +### Part 1 — Easy Explanation + +Deep Research should save the final report from the current run only. It should +not save a planning card, progress text, a normal answer that happened after a +failed tool selection, or a report from another tab. After that, agbrowse should +harden and document the existing `query --session` later-session path so it is +as clear and fail-closed as Oracle's `--followup ` behavior. + +### Part 2 — Diff-level Precision + +## 32.1 — P0 Deep Research Capture + +#### MODIFY `web-ai/chatgpt-deep-research.mjs` + +Current shape: + +- `extractResearchReport(page, deps)` reads latest assistant text first. +- If no text exists, it scans `page.frames()` for URLs containing + `deep-research` or `research`. +- Timeout path may save whatever final frame/text exists. + +Required change: + +- Add a report-selection helper that prefers a completed page-scoped target + read over legacy frame fallback. +- Reject incomplete report text before saving. +- Treat a normal assistant answer without observed research activity as + `deep-research-not-started`. +- Scope frame/target reads to turns after the submitted prompt baseline. + +Suggested exported helpers for tests: + +```js +export function isIncompleteDeepResearchText(text) {} +export function chooseDeepResearchReportRead(targetRead, frameRead) {} +export function normalizeDeepResearchReportText(text) {} +``` + +Suggested internal result shape: + +```js +{ + completed: true, + inProgress: false, + text: '...', + sources: [], + from: 'target' // 'target' | 'frame' | 'assistant' +} +``` + +Implementation notes: + +- Keep `autoConfirmPlan(page, timeoutMs = 70_000)` unchanged unless tests prove + it blocks the target-scoped capture. +- Preserve existing `trySaveReport()` and `appendArtifactRecord()` behavior. +- Do not introduce browser-wide target enumeration that can harvest another + tab's report. +- If CDP target attach is not available in the Playwright-only dependency + surface, implement the read-selection helpers first and leave target attach as + an explicit follow-up inside this same file. + +#### NEW `test/unit/web-ai-deep-research-report-selection.test.mjs` + +Required tests: + +- Completed target read wins over completed frame read. +- Completed frame read wins when target read is missing. +- Planning/status/progress text is not treated as completed. +- Normal non-research answer after a failed Deep Research activation fails with + `deep-research-not-started`. +- Stale frame before baseline is ignored. +- Existing `autoConfirmPlan()` tests still pass. + +#### MODIFY `test/unit/web-ai-chatgpt-deep-research.test.mjs` + +Keep current iframe Start-card tests. Add only small assertions for the exported +helpers if the new helper file is not enough. + +## 32.2 — P1 Model Picker Audit + +#### AUDIT `web-ai/chatgpt-model.mjs` + +Current agbrowse already has: + +- simplified Intelligence menu support +- Pro/Thinking effort verification +- legacy Pro row rejection via `isLegacyProModelLabel(text)` +- split-pill guardrails for standalone Heavy effort labels + +Audit against Oracle 0.14.1/0.15.0 before patching: + +- current Intelligence pill wait before defaulting +- bounded retries for explicit model selection +- Instant row support in simplified picker layouts +- wrapper-row rejection in Intelligence dialog +- failure envelope when requested effort cannot be verified + +Patch only if the audit finds a gap. + +#### IF PATCH NEEDED: MODIFY `web-ai/chatgpt-model.mjs` + +Expected patch direction: + +- Add bounded retry around `readCheckedModelEvidence()` after clicking. +- Add helper to wait for current Intelligence pill text to settle. +- Keep model-selection failures fail-closed when explicit model/effort was + requested. + +#### TESTS + +Target: + +```bash +npx vitest run test/unit/web-ai-chatgpt-model.test.mjs +``` + +Add cases only for newly found gaps. Do not duplicate already-covered split-pill +and effort tests. + +## 32.3 — P1 Existing Later-session Follow-up Parity + +#### MODIFY `web-ai/cli.mjs` + +Do not add a parallel command surface by default. The existing later-session +send path is: + +```text +agbrowse web-ai query --session --prompt [--navigate] +``` + +Existing implementation references: + +- help text distinguishes same-command `--follow-up` from later-session + `query --session` +- `runBoundSendOrQuery(command, deps, input)` binds `send`/`query` to a persisted + session +- `withSessionPage()` resolves the saved page/target before mutation + +Required changes: + +- Add or tighten validation so `query --session` rejects provider root URLs, + external URLs, and missing saved conversation URLs before prompt insertion. +- Improve errors so mismatch recovery points to the exact + `query --session --navigate --prompt ` shape when a follow-up prompt + is being sent. +- Update help/docs/truth-table wording if this path should no longer be treated + as deferred. +- Add a thin alias command only if user testing proves `query --session` remains + too hard to discover. + +#### MODIFY `web-ai/tab-recovery.mjs` + +Reuse, do not replace: + +- `resolveSessionPage(deps, sessionId, { allowNavigate })` +- `urlsCompatible(storedUrl, liveUrl)` +- `withSessionPage(deps, sessionId, fn)` + +If needed, add: + +```js +export function isSafeChatGptConversationUrl(url) {} +``` + +Rules: + +- Must resolve a persisted session by exact `sessionId`. +- Must require `conversationUrl` or recoverable `originalUrl`. +- Must reject provider root URLs for follow-up sends. +- Must reject external URLs. +- Must fail closed on target mismatch unless `--navigate` is provided. +- Must not send into a different thread. + +#### MODIFY `web-ai/chatgpt-multi-turn.mjs` + +Reuse `sendMultiTurn(page, deps, { followUps, session })`. + +Do not duplicate prompt insertion/polling logic in the CLI command. If +`query --session` already routes through the normal `queryWebAi()` path, only +add `sendMultiTurn()` reuse if the implementation chooses to model +later-session follow-up as an explicit single-turn follow-up rather than a +normal query in the existing conversation. + +#### MODIFY `test/unit/web-ai-sessions-command.test.mjs` + +Add parser/contract coverage: + +- `query --session --prompt ` remains documented. +- Missing `--prompt` fails before browser mutation for the later-session send + path. +- `--navigate` is passed into session page resolution where recovery is + authorized. +- Same-command `--follow-up` help remains clearly separate. + +#### NEW or MODIFY `test/unit/web-ai-follow-up-session.test.mjs` + +Add behavior coverage: + +- Exact saved conversation succeeds. +- Root ChatGPT URL is rejected. +- Different conversation URL fails without `--navigate`. +- Closed target with saved conversation URL can recover only when navigation is + authorized. + +## 32.4 — P2 Profile-copy Decision + +Oracle's `--copy-profile ` copies an active signed-in Chrome profile into a +throwaway profile. agbrowse should not implement this automatically. + +Decision record criteria: + +- Implement only if shared `BROWSER_AGENT_HOME` or existing headed CDP reuse + fails a real user workflow. +- Require platform-specific risk notes for Chrome Safe Storage/keychain/cookies. +- Reject remote/external CDP modes. +- Ensure cleanup on success, failure, and signal interruption. + +If approved later, create a separate plan file because this touches Chrome +profile state and should be reviewed as a higher-risk browser lifecycle slice. + +## A — Plan Audit Checklist + +Audit must verify: + +- Deep Research changes do not save stale reports from earlier turns. +- No browser-wide target scan is introduced without page-session scoping. +- Later-session follow-up reuses existing `query --session`, session recovery, + and command-lock code. +- CLI command naming does not conflict with existing `--follow-up`. +- Model-picker patches are only added after audit evidence, not by assumption. +- Profile-copy remains a decision record unless explicitly approved. + +## B — Build Slices + +1. Implement Deep Research report-selection helpers and tests. +2. Wire helper into completion and timeout save paths. +3. Audit model picker and patch only proven gaps. +4. Harden/document existing `query --session` later-session follow-up around + saved conversation guards and recovery guidance. +5. Add focused tests. +6. Update capability truth table only if public capability labels change. + +## C — Check + +Minimum verification: + +```bash +npx vitest run test/unit/web-ai-chatgpt-deep-research.test.mjs test/unit/web-ai-deep-research-report-selection.test.mjs +npx vitest run test/unit/web-ai-chatgpt-model.test.mjs +npx vitest run test/unit/web-ai-sessions-command.test.mjs +npm run test:release-gates +git diff --check +``` + +If CLI help or public capability status changes, also run: + +```bash +npm run gate:all +``` + +## D — Done Criteria + +- Deep Research saves only completed current-run reports. +- Planning/progress/normal-answer fallbacks fail clearly instead of producing + misleading report artifacts. +- Model picker changes, if any, are backed by tests. +- Existing `query --session` later-session follow-up sends only into exact saved + conversations or fails with explicit recovery guidance. +- Profile-copy remains explicitly deferred or receives a separate approved plan. +- `structure/str_func.md` count snapshots are updated and + `bash structure/verify-counts.sh` passes if implementation adds files or + changes CLI/help/truth-table surfaces. diff --git a/devlog/_plan/_plan_phase8_1_detailed.md b/devlog/_plan/legacy_mvp_phase_plans/08_1_detailed.md similarity index 99% rename from devlog/_plan/_plan_phase8_1_detailed.md rename to devlog/_plan/legacy_mvp_phase_plans/08_1_detailed.md index b12d8cba..3ec2414f 100644 --- a/devlog/_plan/_plan_phase8_1_detailed.md +++ b/devlog/_plan/legacy_mvp_phase_plans/08_1_detailed.md @@ -28,7 +28,7 @@ graph TD Prep --> F ``` -**Part 2 file:** `devlog/_plan/_plan_phase8_1_detailed.md` (this file, continued below) +**Part 2 file:** `devlog/_plan/legacy_mvp_phase_plans/08_1_detailed.md` (this file, continued below) --- diff --git a/devlog/_plan/_plan_phase9_1_multi_tab.md b/devlog/_plan/legacy_mvp_phase_plans/09_1_multi_tab.md similarity index 100% rename from devlog/_plan/_plan_phase9_1_multi_tab.md rename to devlog/_plan/legacy_mvp_phase_plans/09_1_multi_tab.md diff --git a/devlog/_plan/_plan_phase9_2_stabilization.md b/devlog/_plan/legacy_mvp_phase_plans/09_2_stabilization.md similarity index 100% rename from devlog/_plan/_plan_phase9_2_stabilization.md rename to devlog/_plan/legacy_mvp_phase_plans/09_2_stabilization.md diff --git a/skills/browser/browser.mjs b/skills/browser/browser.mjs index b770e307..2c9b9103 100755 --- a/skills/browser/browser.mjs +++ b/skills/browser/browser.mjs @@ -44,9 +44,9 @@ * tab-cleanup [--provider chatgpt --keep-provider-tabs 1] Close idle/overflow tabs * text [--format html] Get page text * get-dom [--selector CSS] [--max-chars N] Get current DOM - * console [--duration ms] [--clear] [--reload] [--expression js --unsafe-allow evaluate] Read buffered console logs + * console [--duration ms] [--clear] [--reload] [--expression js] Read buffered console logs * network [--duration ms] [--filter text] [--reload] Inspect network requests - * evaluate --unsafe-allow evaluate Execute JavaScript + * evaluate Execute JavaScript * scroll [--amount N] [--json] Scroll page * wait [--json] Wait fixed duration * wait-for-selector [--timeout ms] [--json] Wait for selector @@ -1410,7 +1410,14 @@ async function screenshotAction(port, opts = {}) { await page.screenshot({ path: filepath, fullPage: opts.fullPage, type }); } - return { path: filepath, dpr: viewport.dpr, viewport: { width: viewport.width, height: viewport.height }, clip }; + return { + path: filepath, + url: page.url(), + targetId: `cdp:${port}`, + dpr: viewport.dpr, + viewport: { width: viewport.width, height: viewport.height }, + clip, + }; } /** @@ -2408,6 +2415,7 @@ try { const maxTextChars = values['max-text-chars'] ? parseInt(/** @type {string} */ (values['max-text-chars'])) : undefined; const page = await getReadyPage(getPort()); const url = page.url(); + const targetId = `cdp:${getPort()}`; let title = ''; try { title = await page.title(); } catch { /* best-effort */ } const viewport = page.viewportSize() || { width: 0, height: 0 }; @@ -2429,7 +2437,7 @@ try { try { const cdp = await getCdpSession(getPort()); for (const n of nodes) { - if (!n.ref || n.ref === '...' || !n.ref.startsWith('@')) continue; + if (!n.ref || !/^@?e\d+$/.test(n.ref)) continue; try { const { root } = await cdp.send('DOM.getDocument', { depth: -1, pierce: true }); const sel = `[aria-label="${(n.name || '').replace(/"/g, '\\"')}"]`; @@ -2442,6 +2450,13 @@ try { boxes[n.ref] = { x: Math.round(c[0]), y: Math.round(c[1]), width: Math.round(model.width), height: Math.round(model.height) }; } } catch { /* best-effort per-node */ } + if (boxes[n.ref] || !n.role || !n.name) continue; + try { + const box = await page.getByRole(String(n.role), { name: String(n.name), exact: true }).first().boundingBox({ timeout: 500 }); + if (box) { + boxes[n.ref] = { x: Math.round(box.x), y: Math.round(box.y), width: Math.round(box.width), height: Math.round(box.height) }; + } + } catch { /* best-effort per-node */ } } await cdp.detach().catch(() => { }); } catch (err) { @@ -2456,6 +2471,7 @@ try { const bundle = buildObservationBundle({ url, title, + targetId, viewport, dpr, snapshotNodes: nodes, @@ -3299,7 +3315,7 @@ try { [--expression "console.log('hi')"] network Inspect requests [--duration ms] [--filter text] [--clear] [--reload] [--live-only] - evaluate Execute JavaScript only with --unsafe-allow evaluate + evaluate Execute JavaScript Web AI: Before agent-run Web AI automation: @@ -3352,8 +3368,10 @@ try { Tab lease policy: Completed provider tabs are runtime leases. Defaults: maxPerKey=3, - globalMax=8, TTL=15m. Override via AGBROWSE_PROVIDER_POOL_MAX_PER_KEY, + globalMax=8, TTL=30m. Override via AGBROWSE_PROVIDER_POOL_MAX_PER_KEY, AGBROWSE_PROVIDER_POOL_GLOBAL_MAX, AGBROWSE_PROVIDER_POOL_TTL. + Active session caps default to per-key=5 and global=14. Override via + AGBROWSE_PROVIDER_ACTIVE_MAX_PER_KEY and AGBROWSE_PROVIDER_ACTIVE_GLOBAL_MAX. Use --new-tab / --parallel to bypass pool reuse for a single call. Run tab-cleanup --json to inspect leaseClosedTabs. @@ -3405,7 +3423,7 @@ try { Holds web-ai-sessions.json (Phase 1 store) + web-ai-baselines.json (legacy) + browser profile. CDP_PORT Default CDP port (default: 9222) - AGBROWSE_MAX_TABS Max open tabs before cleanup closes oldest (default: 10) + AGBROWSE_MAX_TABS Max open tabs before cleanup closes oldest (default: 20) AGBROWSE_TAB_IDLE Idle threshold for cleanup (default: 30m) AGBROWSE_REUSE_TAB=1 Legacy web-ai behavior: reuse active tab AGBROWSE_WEB_AI_AUTO_START=0 diff --git a/skills/browser/tab-lifecycle.mjs b/skills/browser/tab-lifecycle.mjs index 7f376c31..46536ef0 100644 --- a/skills/browser/tab-lifecycle.mjs +++ b/skills/browser/tab-lifecycle.mjs @@ -14,7 +14,7 @@ import { activeCommandTargetIds } from '../../web-ai/active-command-store.mjs'; * @typedef {ManagedTabRow & { cleanupReason?: string, vendor?: string }} CleanupTab */ -const MAX_TABS = parseInt(process.env.AGBROWSE_MAX_TABS || '10', 10); +const MAX_TABS = parseInt(process.env.AGBROWSE_MAX_TABS || '20', 10); const IDLE_TIMEOUT_MS = parseDuration(process.env.AGBROWSE_TAB_IDLE || '30m'); export const DEFAULT_MAX_TABS = MAX_TABS; diff --git a/skills/vision-click/SKILL.md b/skills/vision-click/SKILL.md index 1b0ee397..81a2931d 100644 --- a/skills/vision-click/SKILL.md +++ b/skills/vision-click/SKILL.md @@ -37,9 +37,9 @@ Use when `agbrowse snapshot` returns **NO ref** for target: 3. If NO ref → vision-click fallback: a. agbrowse screenshot --json → { path, dpr, viewport } b. optional stable viewport / clip → more deterministic framing - c. codex exec -i --json → NDJSON events → { found, x, y } + c. codex exec -i --json → NDJSON events → vision bbox candidate d. optional verify crop → second-pass confirmation near center - e. DPR correction: x/dpr, y/dpr → CSS pixels + e. DPR correction + clip origin → CSS pixels f. agbrowse mouse-click → click g. agbrowse snapshot → verify ``` @@ -51,11 +51,11 @@ Use when `agbrowse snapshot` returns **NO ref** for target: ```jsonl {"type":"thread.started","thread_id":"..."} {"type":"turn.started"} -{"type":"item.completed","item":{"type":"agent_message","text":"{\"found\":true,\"x\":522,\"y\":82,\"description\":\"search button\"}"}} +{"type":"item.completed","item":{"type":"agent_message","text":"{\"found\":true,\"bbox\":{\"x\":500,\"y\":70,\"width\":44,\"height\":24},\"point\":{\"x\":522,\"y\":82},\"confidence\":0.88,\"description\":\"search button\"}"}} {"type":"turn.completed","usage":{"input_tokens":16964,"output_tokens":542}} ``` -The coordinate JSON is extracted from the `item.completed` event's `item.text` field. +The vision candidate JSON is extracted from the `item.completed` event's `item.text` field. Legacy `{found,x,y}` point-only JSON is still parsed, but it is marked lower confidence and must be verified before click. ## Examples @@ -75,6 +75,10 @@ agbrowse-vision-click "Menu" --browser-script /path/to/browser.mjs # Accuracy-first mode for dense UIs agbrowse-vision-click "first search result row" --prepare-stable --region left-panel --verify-before-click +# Reconcile a vision bbox against refs from observe-bundle +agbrowse observe-bundle --screenshot --boxes --json > /tmp/bundle.json +agbrowse-vision-click "Submit button" --bundle /tmp/bundle.json --verify-before-click + # Manual clip when you know the rough area agbrowse-vision-click "zoom button" --clip 980 120 220 220 ``` @@ -85,6 +89,8 @@ agbrowse-vision-click "zoom button" --clip 980 120 220 220 - Use `--region left-panel` for search result panels and `--region center-map` for map canvas targets. - Use `--verify-before-click` on dense UIs where a wrong click is expensive. - If you already know the rough target area, `--clip x y w h` is more reliable than full-screen analysis. +- Candidates below confidence `0.75` fail closed unless verification is explicitly requested. +- Prefer `agbrowse click ` whenever `snapshot --interactive` exposes a usable ref; coordinate click remains the last fallback. ## Environment Variables diff --git a/skills/vision-click/vision-candidate.mjs b/skills/vision-click/vision-candidate.mjs new file mode 100644 index 00000000..8c58d95d --- /dev/null +++ b/skills/vision-click/vision-candidate.mjs @@ -0,0 +1,249 @@ +// @ts-check + +/** + * @typedef {Object} Point + * @property {number} x + * @property {number} y + */ + +/** + * @typedef {Object} BBox + * @property {number} x + * @property {number} y + * @property {number} width + * @property {number} height + */ + +/** + * @typedef {Object} Viewport + * @property {number} width + * @property {number} height + */ + +/** + * @typedef {Object} Clip + * @property {number} x + * @property {number} y + * @property {number} width + * @property {number} height + */ + +/** + * @typedef {Object} VisionCandidate + * @property {'vision-candidate-v1'} schemaVersion + * @property {boolean} found + * @property {'vision_bbox'|'coordinate'|'not_found'} kind + * @property {BBox|null} bbox + * @property {Point} point + * @property {number} confidence + * @property {string} [description] + * @property {string} [reason] + * @property {string[]} riskFlags + */ + +/** + * @param {unknown} raw + * @returns {VisionCandidate|null} + */ +export function normalizeVisionCandidate(raw) { + if (!raw || typeof raw !== 'object') return null; + const obj = /** @type {any} */ (raw); + if (typeof obj.found !== 'boolean') return null; + if (!obj.found) { + return { + schemaVersion: 'vision-candidate-v1', + found: false, + kind: 'not_found', + bbox: null, + point: { x: 0, y: 0 }, + confidence: 0, + description: typeof obj.description === 'string' ? obj.description : 'not found', + reason: 'target_not_found', + riskFlags: [], + }; + } + const bbox = normalizeBBox(obj.bbox); + const rawPoint = normalizePoint(obj.point) || (typeof obj.x === 'number' && typeof obj.y === 'number' ? { x: obj.x, y: obj.y } : null); + if (!bbox && !rawPoint) return null; + const point = rawPoint || centerOfBBox(/** @type {BBox} */ (bbox)); + const pointOnly = !bbox; + const confidence = normalizeConfidence(obj.confidence, pointOnly ? 0.5 : 0.8); + return { + schemaVersion: 'vision-candidate-v1', + found: true, + kind: bbox ? 'vision_bbox' : 'coordinate', + bbox, + point, + confidence, + description: typeof obj.description === 'string' ? obj.description : undefined, + reason: typeof obj.reason === 'string' ? obj.reason : undefined, + riskFlags: [ + ...(Array.isArray(obj.riskFlags) ? obj.riskFlags.filter(isString) : []), + ...(pointOnly ? ['point_only'] : []), + ], + }; +} + +/** + * @param {string} text + * @returns {VisionCandidate|null} + */ +export function extractVisionCandidateJson(text) { + const candidates = extractJsonObjects(String(text || '')); + for (const candidate of candidates.reverse()) { + try { + const parsed = normalizeVisionCandidate(JSON.parse(candidate)); + if (parsed) return parsed; + } catch { + // Skip malformed JSON candidates and keep scanning. + } + } + return null; +} + +/** + * @param {VisionCandidate} candidate + * @param {{ viewport?: Viewport|null, dpr?: number, clip?: Clip|null }} observation + */ +export function validateVisionCandidate(candidate, observation = {}) { + if (!candidate.found) { + throw new Error(candidate.reason || 'target not found'); + } + if (!Number.isFinite(candidate.point.x) || !Number.isFinite(candidate.point.y)) { + throw new Error('invalid vision candidate point'); + } + if (candidate.bbox) { + const values = [candidate.bbox.x, candidate.bbox.y, candidate.bbox.width, candidate.bbox.height]; + if (!values.every((n) => Number.isFinite(n)) || candidate.bbox.width <= 0 || candidate.bbox.height <= 0) { + throw new Error('invalid vision candidate bbox'); + } + } + const dpr = typeof observation.dpr === 'number' && observation.dpr > 0 ? observation.dpr : 1; + const viewport = observation.viewport || null; + if (viewport) { + const clip = observation.clip || null; + const maxX = (clip ? clip.width : viewport.width) * dpr; + const maxY = (clip ? clip.height : viewport.height) * dpr; + if (candidate.point.x < 0 || candidate.point.y < 0 || candidate.point.x >= maxX || candidate.point.y >= maxY) { + throw new Error('vision candidate point is outside the captured image'); + } + } + return candidate; +} + +/** + * @param {VisionCandidate} candidate + * @returns {Point} + */ +export function candidateCenter(candidate) { + return candidate.bbox + ? { x: Math.round(candidate.bbox.x + candidate.bbox.width / 2), y: Math.round(candidate.bbox.y + candidate.bbox.height / 2) } + : candidate.point; +} + +/** + * @param {VisionCandidate} candidate + * @param {number} [threshold] + */ +export function isLowConfidence(candidate, threshold = 0.75) { + return !candidate.found || candidate.confidence < threshold; +} + +/** + * @param {string} text + * @returns {string[]} + */ +function extractJsonObjects(text) { + const objects = []; + let start = -1; + let depth = 0; + let inString = false; + let escaped = false; + + for (let i = 0; i < text.length; i++) { + const ch = text[i]; + if (escaped) { + escaped = false; + continue; + } + if (ch === '\\') { + escaped = true; + continue; + } + if (ch === '"') { + inString = !inString; + continue; + } + if (inString) continue; + if (ch === '{') { + if (depth === 0) start = i; + depth += 1; + } else if (ch === '}') { + if (depth === 0) continue; + depth -= 1; + if (depth === 0 && start !== -1) { + objects.push(text.slice(start, i + 1)); + start = -1; + } + } + } + + return objects; +} + +/** + * @param {BBox} bbox + * @returns {Point} + */ +function centerOfBBox(bbox) { + return { + x: Math.round(bbox.x + bbox.width / 2), + y: Math.round(bbox.y + bbox.height / 2), + }; +} + +/** + * @param {unknown} value + * @param {number} fallback + * @returns {number} + */ +function normalizeConfidence(value, fallback) { + return typeof value === 'number' && Number.isFinite(value) + ? Math.max(0, Math.min(1, value)) + : fallback; +} + +/** + * @param {unknown} value + * @returns {Point|null} + */ +function normalizePoint(value) { + if (!value || typeof value !== 'object') return null; + const point = /** @type {any} */ (value); + if (typeof point.x !== 'number' || typeof point.y !== 'number') return null; + return { x: point.x, y: point.y }; +} + +/** + * @param {unknown} value + * @returns {BBox|null} + */ +function normalizeBBox(value) { + if (!value || typeof value !== 'object') return null; + const box = /** @type {any} */ (value); + if ( + typeof box.x !== 'number' || + typeof box.y !== 'number' || + typeof box.width !== 'number' || + typeof box.height !== 'number' + ) return null; + return { x: box.x, y: box.y, width: box.width, height: box.height }; +} + +/** + * @param {unknown} value + * @returns {value is string} + */ +function isString(value) { + return typeof value === 'string'; +} diff --git a/skills/vision-click/vision-click.mjs b/skills/vision-click/vision-click.mjs index 1ff3ffd6..5fe0fbf2 100755 --- a/skills/vision-click/vision-click.mjs +++ b/skills/vision-click/vision-click.mjs @@ -14,18 +14,23 @@ */ import { execFileSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; import { join, dirname } from 'node:path'; import { fileURLToPath } from 'node:url'; import { buildCoordPrompt, - extractCoordJson, + candidateCenter, + extractVisionCandidateJson, assertCodexCli, applyDprCorrection, clipAroundPoint, describeRegion, + isLowConfidence, parseVisionClickCliArgs, resolveRegionClip, + validateVisionCandidate, } from './vision-core.mjs'; +import { assertFreshObservationBundle, reconcileVisionCandidate } from '../../web-ai/candidate-reconcile.mjs'; const __dirname = dirname(fileURLToPath(import.meta.url)); @@ -76,7 +81,7 @@ function browserCmd(args, opts = {}) { /** * @param {string} screenshotPath * @param {string} prompt - * @returns {{found:boolean, x:number, y:number, description?:string}} + * @returns {import('./vision-core.mjs').VisionCandidate} */ function codexVisionWithPrompt(screenshotPath, prompt) { const args = [ @@ -105,18 +110,18 @@ function codexVisionWithPrompt(screenshotPath, prompt) { const event = JSON.parse(line); const text = event.item?.text || event.item?.aggregated_output || ''; if (!text) continue; - const coords = extractCoordJson(text); - if (coords) return coords; + const candidate = extractVisionCandidateJson(text); + if (candidate) return candidate; } catch { /* skip non-JSON lines */ } } - throw new Error(`No coordinate JSON in codex NDJSON output (${lines.length} lines)`); + throw new Error(`No vision candidate JSON in codex NDJSON output (${lines.length} lines)`); } /** * @param {string} screenshotPath * @param {string} target * @param {any} [options] - * @returns {{found:boolean, x:number, y:number, description?:string}} + * @returns {import('./vision-core.mjs').VisionCandidate} */ function codexVision(screenshotPath, target, options = {}) { return codexVisionWithPrompt(screenshotPath, buildCoordPrompt(target, options)); @@ -148,6 +153,15 @@ function prepareStableViewport(opts = {}) { return viewport; } +/** + * @param {string|null|undefined} path + * @returns {any|null} + */ +function loadObservationBundle(path) { + if (!path) return null; + return JSON.parse(readFileSync(path, 'utf8')); +} + /** * @param {any} opts * @param {{width:number,height:number}} viewport @@ -177,12 +191,13 @@ function convertRawToCss(raw, dpr, clip = null) { /** * @param {string} target * @param {{dpr:number, viewport:{width:number,height:number}, clip:{x:number,y:number,width:number,height:number}|null}} capture - * @param {{x:number,y:number,description?:string}} initialResult + * @param {import('./vision-core.mjs').VisionCandidate} initialResult * @param {any} [opts] * @returns {{raw:{x:number,y:number}, css:{x:number,y:number}, clip:{x:number,y:number,width:number,height:number}, description?:string}} */ function verifyCandidate(target, capture, initialResult, opts = {}) { - const cssPoint = convertRawToCss({ x: initialResult.x, y: initialResult.y }, capture.dpr, capture.clip); + const center = candidateCenter(initialResult); + const cssPoint = convertRawToCss(center, capture.dpr, capture.clip); const verifyClip = clipAroundPoint(cssPoint, capture.viewport, { width: 280, height: 200 }); const verifyCapture = JSON.parse(browserCmd(screenshotJsonArgs(opts, verifyClip), opts)); const regionHint = describeRegion(opts.region); @@ -196,7 +211,9 @@ function verifyCandidate(target, capture, initialResult, opts = {}) { throw new Error('Verification crop did not contain the target'); } - const verifyCss = applyDprCorrection(verified.x, verified.y, verifyCapture.dpr || capture.dpr || 1); + validateVisionCandidate(verified, { viewport: { width: verifyClip.width, height: verifyClip.height }, dpr: verifyCapture.dpr || capture.dpr || 1 }); + const verifyCenter = candidateCenter(verified); + const verifyCss = applyDprCorrection(verifyCenter.x, verifyCenter.y, verifyCapture.dpr || capture.dpr || 1); const distanceX = Math.abs(verifyCss.x - verifyClip.width / 2); const distanceY = Math.abs(verifyCss.y - verifyClip.height / 2); if (distanceX > verifyClip.width * 0.45 || distanceY > verifyClip.height * 0.45) { @@ -224,7 +241,7 @@ function verifyCandidate(target, capture, initialResult, opts = {}) { /** * @param {string} target * @param {any} [opts] - * @returns {{success:boolean, reason?:string, clicked?:{x:number,y:number}, raw?:{x:number,y:number}, dpr?:number, description?:string, snap?:string|null, clip?:any, verified?:boolean}} + * @returns {{success:boolean, reason?:string, clicked?:{x:number,y:number}, raw?:{x:number,y:number}, dpr?:number, description?:string, candidate?:import('./vision-core.mjs').VisionCandidate, reconciliation?:string, snap?:string|null, clip?:any, verified?:boolean}} */ function visionClick(target, opts = {}) { const stableViewport = prepareStableViewport(opts); @@ -232,6 +249,10 @@ function visionClick(target, opts = {}) { // 1. Screenshot (get path + DPR via --json) console.error(`${c.dim}📸 Taking screenshot...${c.reset}`); const baseCapture = JSON.parse(browserCmd(['screenshot', '--json'], opts)); + const bundle = loadObservationBundle(opts.bundle); + if (bundle) { + assertFreshObservationBundle(bundle, { url: baseCapture.url, targetId: baseCapture.targetId }); + } const viewport = stableViewport || baseCapture.viewport; const clip = resolveInitialClip(opts, viewport); const ss = clip @@ -253,19 +274,79 @@ function visionClick(target, opts = {}) { if (!result.found) { return { success: false, reason: 'target not found' }; } + validateVisionCandidate(result, { viewport, dpr, clip }); + + const requiresVerification = + result.riskFlags.includes('point_only') || + (isLowConfidence(result) && result.confidence >= 0.5); + if (isLowConfidence(result) && !opts.verifyBeforeClick) { + return { + success: false, + reason: `vision candidate confidence ${result.confidence} is below 0.75; rerun with --verify-before-click`, + }; + } // 3. DPR correction: image pixels → CSS pixels - let finalRaw = { x: result.x, y: result.y }; + let finalRaw = candidateCenter(result); let finalCss = convertRawToCss(finalRaw, dpr, clip); let verification = null; + let reconciliation = 'unavailable'; + let shouldVerify = opts.verifyBeforeClick || requiresVerification; + + if (bundle) { + const decision = reconcileVisionCandidate({ candidate: candidateAtCssPoint(result, finalCss), bundle }); + reconciliation = decision.reason || decision.action; + if (decision.action === 'fail') { + throw new Error(`${decision.code || 'COMPUTER_TARGET_AMBIGUOUS'}: ${decision.reason}`); + } + if (decision.action === 'ref' && decision.ref) { + browserCmd(['click', decision.ref], opts); + return { + success: true, + clicked: finalCss, + raw: finalRaw, + dpr, + description: result.description, + candidate: result, + reconciliation, + snap: safeSnapshot(opts), + clip, + verified: false, + }; + } + shouldVerify = true; + } - if (opts.verifyBeforeClick) { + if (shouldVerify) { verification = verifyCandidate(target, { dpr, viewport, clip }, result, opts); finalRaw = verification.raw; finalCss = verification.css; console.error(`${c.dim} verified via crop: (${verification.clip.x}, ${verification.clip.y}, ${verification.clip.width}, ${verification.clip.height})${c.reset}`); } + if (bundle && verification) { + const decision = reconcileVisionCandidate({ candidate: candidateAtCssPoint(result, finalCss), bundle }); + reconciliation = decision.reason || decision.action; + if (decision.action === 'fail') { + throw new Error(`${decision.code || 'COMPUTER_TARGET_AMBIGUOUS'}: ${decision.reason}`); + } + if (decision.action === 'ref' && decision.ref) { + browserCmd(['click', decision.ref], opts); + return { + success: true, + clicked: finalCss, + raw: finalRaw, + dpr, + description: verification.description || result.description, + candidate: result, + reconciliation, + snap: safeSnapshot(opts), + clip, + verified: true, + }; + } + } + console.error(`${c.dim} raw: (${finalRaw.x}, ${finalRaw.y}) → css: (${finalCss.x}, ${finalCss.y}) [dpr=${dpr}]${c.reset}`); // 4. Click @@ -285,12 +366,38 @@ function visionClick(target, opts = {}) { raw: finalRaw, dpr, description: verification?.description || result.description, + candidate: result, + reconciliation, snap, clip, verified: Boolean(verification), }; } +/** + * @param {any} opts + * @returns {string|null} + */ +function safeSnapshot(opts) { + try { + return browserCmd(['snapshot', '--interactive'], opts); + } catch { + return null; + } +} + +/** + * @param {import('./vision-core.mjs').VisionCandidate} candidate + * @param {{x:number,y:number}} cssPoint + * @returns {import('./vision-core.mjs').VisionCandidate} + */ +function candidateAtCssPoint(candidate, cssPoint) { + return { + ...candidate, + point: cssPoint, + }; +} + // ═══════════════════════════════════════════════════ // CLI // ═══════════════════════════════════════════════════ @@ -315,10 +422,12 @@ if (opts.help || !target) { --viewport Custom viewport preset, e.g. 1440x900 --region Named crop: left-panel, center-map, top-bar --clip Manual crop in CSS pixels + --bundle ObservationBundle JSON from observe-bundle --json for ref reconciliation --verify-before-click Re-check a zoomed crop before clicking Pipeline: - screenshot → optional clip → codex exec (NDJSON) → optional verify crop → DPR correction → mouse click → verify + screenshot → optional clip → codex exec (bbox/confidence candidate) → optional verify crop → DPR correction → mouse click → verify + Ref clicks are preferred whenever snapshot --interactive exposes a usable ref. Coordinate click is the last fallback. Prerequisites: - agbrowse running Chrome (agbrowse start) diff --git a/skills/vision-click/vision-core.mjs b/skills/vision-click/vision-core.mjs index 0d884112..f26ea2ff 100644 --- a/skills/vision-click/vision-core.mjs +++ b/skills/vision-click/vision-core.mjs @@ -1,5 +1,14 @@ // @ts-check import { execFileSync } from 'node:child_process'; +import { extractVisionCandidateJson } from './vision-candidate.mjs'; + +export { + candidateCenter, + extractVisionCandidateJson, + isLowConfidence, + normalizeVisionCandidate, + validateVisionCandidate, +} from './vision-candidate.mjs'; /** * @typedef {Object} Viewport @@ -33,6 +42,7 @@ import { execFileSync } from 'node:child_process'; * @property {string|null} region * @property {Clip|null} clip * @property {boolean} help + * @property {string|null} bundle */ /** @@ -41,6 +51,27 @@ import { execFileSync } from 'node:child_process'; * @property {number} y */ +/** + * @typedef {Object} BBox + * @property {number} x + * @property {number} y + * @property {number} width + * @property {number} height + */ + +/** + * @typedef {Object} VisionCandidate + * @property {'vision-candidate-v1'} schemaVersion + * @property {boolean} found + * @property {'vision_bbox'|'coordinate'|'not_found'} kind + * @property {BBox|null} bbox + * @property {Point} point + * @property {number} confidence + * @property {string} [description] + * @property {string} [reason] + * @property {string[]} riskFlags + */ + /** * @typedef {Object} CoordPromptOptions * @property {string} [regionHint] @@ -54,55 +85,6 @@ import { execFileSync } from 'node:child_process'; * @property {string} [binary] */ -// ─── JSON extraction ───────────────────────────── - -/** - * @param {string} text - * @returns {string[]} - */ -function extractJsonObjects(text) { - const objects = []; - let start = -1; - let depth = 0; - let inString = false; - let escaped = false; - - for (let i = 0; i < text.length; i++) { - const ch = text[i]; - - if (escaped) { - escaped = false; - continue; - } - - if (ch === '\\') { - escaped = true; - continue; - } - - if (ch === '"') { - inString = !inString; - continue; - } - - if (inString) continue; - - if (ch === '{') { - if (depth === 0) start = i; - depth += 1; - } else if (ch === '}') { - if (depth === 0) continue; - depth -= 1; - if (depth === 0 && start !== -1) { - objects.push(text.slice(start, i + 1)); - start = -1; - } - } - } - - return objects; -} - // ─── CLI args ──────────────────────────────────── /** @@ -123,6 +105,7 @@ export function parseVisionClickCliArgs(args, defaults = {}) { region: null, clip: null, help: false, + bundle: null, }; for (let i = 0; i < args.length; i++) { @@ -177,6 +160,11 @@ export function parseVisionClickCliArgs(args, defaults = {}) { i += 4; continue; } + if (arg === '--bundle') { + opts.bundle = args[i + 1] || opts.bundle; + i += 1; + continue; + } if (arg.startsWith('--')) continue; positionals.push(arg); } @@ -282,10 +270,11 @@ export function clipAroundPoint(point, viewport, size = {}) { export function buildCoordPrompt(target, options = {}) { const prompt = [ 'Look at this screenshot image carefully.', - `Find the UI element "${target}" and return its center pixel coordinate.`, + `Find the UI element "${target}" and return its clickable bounding box plus center pixel coordinate.`, 'You MUST respond with ONLY this JSON format, nothing else:', - '{"found":true,"x":,"y":,"description":""}', - 'If not found: {"found":false,"x":0,"y":0,"description":"not found"}', + '{"found":true,"bbox":{"x":,"y":,"width":,"height":},"point":{"x":,"y":},"confidence":<0_to_1>,"description":""}', + 'If not found: {"found":false,"bbox":null,"point":{"x":0,"y":0},"confidence":0,"description":"not found"}', + 'Only if a bounding box is impossible, return the legacy point shape {"found":true,"x":,"y":,"confidence":0.5,"description":""}.', 'IMPORTANT: Do NOT run any commands. Just analyze the image visually and return the JSON.', ]; if (options.regionHint) { @@ -307,17 +296,14 @@ export function buildCoordPrompt(target, options = {}) { * @returns {{ found: boolean, x: number, y: number, description?: string } | null} */ export function extractCoordJson(text) { - const candidates = extractJsonObjects(String(text || '')); - for (const candidate of candidates.reverse()) { - try { - const coords = JSON.parse(candidate); - if (typeof coords?.found !== 'boolean') continue; - if (typeof coords.x === 'number' && typeof coords.y === 'number') return coords; - } catch { - // Skip malformed JSON candidates and keep scanning. - } - } - return null; + const candidate = extractVisionCandidateJson(text); + if (!candidate || !candidate.found) return candidate ? { found: false, x: 0, y: 0, description: candidate.description } : null; + return { + found: true, + x: candidate.point.x, + y: candidate.point.y, + description: candidate.description, + }; } // ─── Codex CLI check ───────────────────────────── diff --git a/skills/web-ai/SKILL.md b/skills/web-ai/SKILL.md index 82136538..26eb2849 100644 --- a/skills/web-ai/SKILL.md +++ b/skills/web-ai/SKILL.md @@ -120,6 +120,14 @@ Pass `--timeout 1800` (30 min) or higher for unusually long Pro/Deep Think runs. The provider tab and the agbrowse Chrome process stay open across a poll timeout — only the polling loop gives up. +MCP clients should preserve the `sessionId` returned by `web_ai_submit_prompt`. +`web_ai_wait_response` and `web_ai_session_resume` use the stored-session +recovery path and may return a recoverable timeout with `retryHint: +poll-or-resume`. A host-level MCP request timeout, such as `-32001 Request +timed out`, is a client/runtime boundary; after it, retry with the same +`sessionId` or use CLI `web-ai poll --session ` instead of sending a new +prompt immediately. + ## Long-Running / Background Sessions For responses that may take many minutes (ChatGPT Pro/Heavy, Gemini Deep @@ -134,7 +142,7 @@ agbrowse web-ai watch --session "$SID" --json --navigate ``` Key facts (verified 2026-06-11, details in -`devlog/_plan/260611_background_runtime_hook/02_agbrowse_sufficiency.md`): +`devlog/_fin/260611_background_runtime_hook/02_agbrowse_sufficiency.md`): - Sessions persist in `~/.browser-agent/web-ai-sessions.json` and survive process/machine restarts. `sessions show --json` is safe from any @@ -201,7 +209,7 @@ reducing tab creation overhead in batch scenarios. | Pool setting | Default | Env Var | | --- | --- | --- | -| TTL per pooled tab | 15 min | `AGBROWSE_PROVIDER_POOL_TTL` | +| TTL per pooled tab | 30 min | `AGBROWSE_PROVIDER_POOL_TTL` | | Max warm tabs per `(owner,vendor,sessionType,origin,profile)` | 3 | `AGBROWSE_PROVIDER_POOL_MAX_PER_KEY` | | Global cap on warm provider tabs | 8 | `AGBROWSE_PROVIDER_POOL_GLOBAL_MAX` | @@ -213,7 +221,7 @@ another in-flight Pro query without lease contention. | Setting | Default | Env Var | | --- | --- | --- | -| Max tabs | 10 | `AGBROWSE_MAX_TABS` | +| Max tabs | 20 | `AGBROWSE_MAX_TABS` | | Idle timeout | 30 min | `AGBROWSE_TAB_IDLE` | Idle tabs (inactive longer than the timeout) are auto-closed unless pinned or @@ -296,6 +304,33 @@ agbrowse web-ai query \ --prompt "Reply exactly GROK_OK" ``` +## Where instructions go — `--system` vs `--context` vs `--file` + +Attaching instructions *and* a file already works today; pick the right channel: + +- **Operating instructions / skill guidance** ("how to behave", "extract X from the + attached file") → `--system "..."`. Rendered in the trusted `[SYSTEM]` section and + honored. `--goal` / `--constraints` (USER fields) are also trusted. +- **Untrusted reference *data*** (scraped page text, provider output) → `--context "..."`. + Rendered as `[UNTRUSTED_CONTEXT]`; **instructions placed here are ignored by design** + (prompt-injection boundary). Never put operating instructions in `--context`. +- **A file for the model to read/analyze** → `--file ` (repeatable; native, + dialog-free attachment). The model sees it directly — you do **not** need to also + describe it in `--context`. + +Common mistake: putting instructions in `--context` and finding they are "ignored" → +move them to `--system`. Example of the working combo: + +```bash +agbrowse web-ai query --vendor chatgpt --model thinking \ + --system "Extract every breaking change from the attached PDF and group by module." \ + --file ./spec.pdf \ + --prompt "Summarize the breaking changes." +``` + +If you are unsure whether/how a file is attached, just **ask the user** rather than +relying on envelope guesswork. + ## File Upload ```bash diff --git a/structure/CAPABILITY_TRUTH_TABLE.md b/structure/CAPABILITY_TRUTH_TABLE.md index 56c3dd2a..9b1a0dfc 100644 --- a/structure/CAPABILITY_TRUTH_TABLE.md +++ b/structure/CAPABILITY_TRUTH_TABLE.md @@ -26,7 +26,7 @@ mirror entry of `n/a` means the capability is intentionally agbrowse-only. | --- | --- | --- | --- | --- | | Browser runtime cleanup / `doctor` | ready | `web-ai/doctor.mjs`, `skills/browser/browser.mjs` | `test/integration/web-ai-doctor*.test.mjs`, `test/unit/web-ai-doctor.test.mjs` | `src/browser/runtime/*` (cleanup); doctor surface re-exported via cli-jaw browser command. ready in cli-jaw. | | ChatGPT web-AI resolver | beta | `web-ai/chatgpt.mjs`, `web-ai/chatgpt-composer.mjs`, `web-ai/chatgpt-model.mjs` | `test/unit/web-ai-chatgpt*.test.mjs`, fixture evals under `test/fixtures/provider-dom/` | `src/browser/web-ai/chatgpt.ts` — beta in cli-jaw. | -| ChatGPT code mode (`web-ai code`) | beta | `web-ai/code-mode.mjs`, `web-ai/code-mode-prompt.mjs`, `web-ai/code-artifact.mjs`, `web-ai/code-dev-context.mjs`, `skills/web-ai/modules/gpt-dev-agent-context.*` | `test/unit/web-ai-code-mode*.test.mjs`, `test/unit/web-ai-code-artifact.test.mjs`, `test/unit/web-ai-code-dev-context.test.mjs` | saved GPT dev-agent context zip auto-upload → strict contract prompt → headless single/multi zip retrieval via in-page presigned fetch. New code artifacts must include `PLAN.md` or `00_plan.md`. cli-jaw independent mirror is tracked in `devlog/_plan/260611_code_mode_gpt_agent_context/00_plan.md`. | +| ChatGPT code mode (`web-ai code`) | beta | `web-ai/code-mode.mjs`, `web-ai/code-mode-prompt.mjs`, `web-ai/code-artifact.mjs`, `web-ai/code-dev-context.mjs`, `skills/web-ai/modules/gpt-dev-agent-context.*` | `test/unit/web-ai-code-mode*.test.mjs`, `test/unit/web-ai-code-artifact.test.mjs`, `test/unit/web-ai-code-dev-context.test.mjs` | saved GPT dev-agent context zip auto-upload → strict contract prompt → headless single/multi zip retrieval via in-page presigned fetch. New code artifacts must include `PLAN.md` or `00_plan.md`. cli-jaw independent mirror is tracked in `devlog/_fin/260611_code_mode_gpt_agent_context/00_plan.md`. | | ChatGPT code artifact extraction (`web-ai code-extract`) | beta | `web-ai/code-mode.mjs`, `web-ai/code-artifact.mjs`, CLI surface in `web-ai/cli.mjs` | `test/unit/web-ai-code-mode.test.mjs`, `test/unit/web-ai-code-artifact.test.mjs`, `test/integration/web-ai-cli-contract.test.mjs` | re-retrieves existing ChatGPT code-mode zip artifacts from an accessible conversation without sending a new prompt. ChatGPT-only; selectors are `--url`, `--conversation`, `--session`, or the current ChatGPT conversation tab. | | Gemini web-AI resolver | beta | `web-ai/gemini-live.mjs`, `web-ai/gemini-model.mjs` | `test/unit/web-ai-gemini*.test.mjs` | not mirrored; cli-jaw delegates via agbrowse. n/a in cli-jaw. | | Grok web-AI resolver | beta | `web-ai/grok-live.mjs`, `web-ai/grok-model.mjs` | `test/unit/web-ai-grok*.test.mjs` | not mirrored. n/a in cli-jaw. | diff --git a/structure/commands.md b/structure/commands.md index c124c457..ab05b45f 100644 --- a/structure/commands.md +++ b/structure/commands.md @@ -26,7 +26,7 @@ aliases: [agbrowse commands, agbrowse CLI 표면, web-ai commands] | Act | `click`, `type`, `press`, `hover`, `select`, `check`, `uncheck`, `drag`, `mouse-click`, `move-mouse`, `mouse-down`, `mouse-up` | ref 기반 또는 coordinate 기반 mutation | | Navigate | `navigate`, `reload`, `resize`, `tabs`, `active-tab`, `tab-switch`, `select-tab`, `new-tab`, `tab-close`, `tab-cleanup`, `scroll` | navigation, viewport, active target 조회, tab 관리 (multi-tab create/close 포함) | | Wait | `wait`, `wait-for-selector`, `wait-for-text`, `wait-for` | time, selector, text, legacy ref wait | -| Diagnostics | `console`, `network`, `evaluate` | console/network capture와 explicit unsafe JS evaluation | +| Diagnostics | `console`, `network`, `evaluate` | console/network capture와 page JS evaluation | | Web AI | `web-ai` | provider workflow subcommand | | Runway | `runway` | Runway Apps/Custom selector contract, current-tab status, read-only preflight | @@ -201,10 +201,10 @@ JSON 모드에서는 실패가 parseable envelope로 나온다. 이 shape는 MCP | `web_ai_snapshot` | compact accessibility snapshot과 `@eN` refs 반환 | | `web_ai_click_ref` | latest snapshot ref 클릭 | | `web_ai_submit_prompt` | provider web UI에 prompt 제출 | -| `web_ai_wait_response` | provider response completion 대기 | +| `web_ai_wait_response` | stored session response 대기; 긴 실행은 recoverable timeout으로 재시도 가능 | | `web_ai_copy_markdown` | 마지막 response를 markdown/text로 capture | | `web_ai_doctor` | provider diagnostics와 repair packet 반환 | -| `web_ai_session_resume` | stored session poll resume | +| `web_ai_session_resume` | stored session poll resume; session-bound recovery 경로 사용 | `web_ai_*` 입력은 strict schema로 검증한다. Runtime에서 쓰는 호환 alias (`vendor`, `policy`, submit의 `filePath`/`reasoningEffort` 등)만 명시적으로 @@ -231,7 +231,7 @@ tool description에도 그 제한을 명시한다. ## ObservationBundleV1 Schema (G06) -`agbrowse observe-bundle [--screenshot] [--boxes] [--json] [--max-text-chars N]`은 한 번의 호출로 URL, title, viewport, DPR, snapshot refs, optional bounding boxes, optional screenshot path, body innerText 요약을 단일 record로 묶는다. Vercel agent-browser/Playwright MCP `browser_observe`/VisualWebArena 류 multimodal 벤치가 요구하는 reproducible observation step을 만족시킨다. +`agbrowse observe-bundle [--screenshot] [--boxes] [--json] [--max-text-chars N]`은 한 번의 호출로 observation id, target id, URL, title, viewport, DPR, snapshot refs, optional bounding boxes, optional screenshot path, body innerText 요약을 단일 record로 묶는다. Vercel agent-browser/Playwright MCP `browser_observe`/VisualWebArena 류 multimodal 벤치가 요구하는 reproducible observation step을 만족시킨다. Browser action routing은 ref-first, coordinate-last 정책을 따른다. Vision coordinate fallback은 bbox/confidence candidate와 clip/DPR basis를 보존하고, confidence가 낮거나 point-only인 결과는 verification 없이 바로 클릭하지 않는다. ```json { diff --git a/structure/runtime_contracts.md b/structure/runtime_contracts.md index dd2aca01..ef599814 100644 --- a/structure/runtime_contracts.md +++ b/structure/runtime_contracts.md @@ -23,7 +23,7 @@ aliases: [agbrowse runtime contracts, agbrowse 런타임 계약, provider contra | Browser state | `browser-state.json`에 port, pid, target 상태 저장 | | Snapshots | latest snapshot과 per-target snapshot을 `BROWSER_AGENT_HOME` 아래 저장 | | Ref safety | navigation, reload, tab switch 이후 snapshot ref는 재관찰해야 한다 | -| Evaluation | `evaluate`는 `--unsafe-allow evaluate`가 있어야 실행된다 | +| Evaluation | `evaluate`는 기본 허용된다. custom policy에서 `allowEvaluate: false`로 끌 수 있고, `--unsafe-allow evaluate`는 explicit-deny 우회용 legacy opt-in으로 남긴다 | ## Runway Runtime @@ -94,6 +94,7 @@ Provider UI는 공식 API가 아니다. 따라서 live provider flow는 구현 | Server | `agbrowse web-ai mcp-server`는 stdio JSON-RPC MCP bridge다 | | Schema source | `web-ai/tool-schema.mjs`가 MCP와 AI SDK schema source다 | | Known tools | `browser_snapshot`, `browser_click_ref`, `web_ai_snapshot`, `web_ai_click_ref`, `web_ai_submit_prompt`, `web_ai_wait_response`, `web_ai_copy_markdown`, `web_ai_doctor`, `web_ai_session_resume` | +| MCP wait/resume | `web_ai_wait_response`와 `web_ai_session_resume`는 stored session lock/recovery를 사용한다. Provider poll timeout은 recoverable result로 반환될 수 있으며, host-side MCP request timeout은 client/runtime boundary로 남는다 | | Stale ref | snapshot mismatch나 unknown ref는 mutation하지 않는다 | | Snapshot scope | generic browser refs and provider web-AI refs use isolated latest snapshot state | | Policy | MCP mutation도 policy enforcement를 통과해야 한다 | diff --git a/structure/stability-upgrade/01_operational_weakness_register.md b/structure/stability-upgrade/01_operational_weakness_register.md index 0d638b6c..f75c0414 100644 --- a/structure/stability-upgrade/01_operational_weakness_register.md +++ b/structure/stability-upgrade/01_operational_weakness_register.md @@ -29,6 +29,7 @@ tags: [agbrowse, operational-weakness, web-ai] | STAB-06 | Deferred | Deep Research can run longer than normal poll/finalize assumptions | report may be partial or artifact warning may appear | marked experimental; report save uses structured artifact warnings and skips auto archive | Promote only after repeated live long-run smoke and report extraction fixtures | | STAB-07 | Closed | Temporary Chat could be archived despite being intentionally non-durable | archive claim would be misleading or no-op | Temporary Chat URL check wins even when archive is forced | None unless ChatGPT changes temporary URL shape | | STAB-08 | Closed | MCP `web_ai_*` accepted misspelled/unknown fields | agents could think a field worked when runtime ignored it | strict schema rejects unknown fields while preserving documented aliases | Keep deferred advanced surfaces documented in MCP descriptions | +| STAB-09 | Closed | MCP wait/resume bypassed session-bound recovery on long provider runs | `web_ai_wait_response` could time out before a long ChatGPT Pro response while CLI `poll --session` later recovered it; later timeout polls could downgrade completed sessions | MCP wait/resume uses session lock/recovery plus active-command ownership; provider timeouts are recoverable and completed sessions are monotonic | Add a future non-blocking MCP lease/cancel protocol only if host request timeout remains a practical blocker | ## Easy Read @@ -36,6 +37,7 @@ tags: [agbrowse, operational-weakness, web-ai] long-running browser state. - The newest Oracle parity gap, model evidence, is closed locally. - The biggest open item is recovery UX when the browser closes mid-generation. +- MCP wait/resume now shares the stored-session recovery path, but MCP host + request timeout remains a client/runtime boundary rather than a provider crash. - Deep Research and Project Sources should stay beta/experimental until live smoke evidence is stronger. - diff --git a/structure/str_func.md b/structure/str_func.md index 14eab4b5..a4c43641 100644 --- a/structure/str_func.md +++ b/structure/str_func.md @@ -16,26 +16,26 @@ aliases: [agbrowse source map, agbrowse str_func, agbrowse 파일 구조] ## 현재 구조 스냅샷 -마지막 측정: 2026-06-11. +마지막 측정: 2026-06-20. | 경로 | 파일 수 | 라인 수 | 역할 | | --- | ---: | ---: | --- | | `bin/` | 2 | 6 | published bin wrapper | -| `skills/browser/` | 45 | 13578 | Chrome lifecycle, CDP connection, refs, tabs, diagnostics, adaptive fetch v2, Runway task-runner preflight/poll | -| `skills/vision-click/` | 3 | 831 | screenshot to coordinate click helper | -| `skills/web-ai/` | 3 | 742 | bundled agent workflow skill | -| `web-ai/` | 94 | 21398 | provider automation, sessions, MCP, eval, policy, trace | +| `skills/browser/` | 45 | 13602 | Chrome lifecycle, CDP connection, refs, tabs, diagnostics, adaptive fetch v2, Runway task-runner preflight/poll | +| `skills/vision-click/` | 4 | 1182 | screenshot to coordinate click helper | +| `skills/web-ai/` | 3 | 850 | bundled agent workflow skill | +| `web-ai/` | 96 | 22206 | provider automation, sessions, MCP, eval, policy, trace | | `web-ai/context-pack/` | 8 | 858 | file selection, token budget, context rendering | | `web-ai/eval/` | 5 | 552 | offline provider DOM fixture harness | | `web-ai/policy/` | 4 | 238 | mutation and content-boundary guardrails | | `web-ai/trace/` | 5 | 444 | trace ID, redaction, report, writer helpers | | `scripts/` | 10 | 1514 | eval runner, release scripts, named release gates, strict-baseline / module-graph / bin smoke checks | -| `test/unit/` | 101 | 11824 | deterministic module tests | -| `test/integration/` | 17 | 2462 | CLI, MCP, policy, provider fixture tests | +| `test/unit/` | 105 | 12600 | deterministic module tests | +| `test/integration/` | 17 | 2468 | CLI, MCP, policy, provider fixture tests | | `test/e2e/` | 1 | 50 | browser smoke coverage | | `test/spec/` | 2 | 35 | high-level contract specs | | `docs/` | 41 | 2635 | adoption, trace, production-readiness, comparison, benchmark, EXTERNAL_CDP, migration docs, GitHub Pages developer docs | -| `devlog/` | 346 | 43240 | phased plan, research, implementation notes (incl. strict-migration phases) | +| `devlog/` | 370 | 46264 | phased plan, research, implementation notes (incl. strict-migration phases) | `structure/` 자체는 이 문서가 검증 대상으로 삼는 source tree 밖의 문서 허브라서 위 집계에서 제외한다. `verify-counts.sh`는 이 표의 경로별 파일 수와 라인 수를 live source 기준으로 비교한다. @@ -43,7 +43,7 @@ aliases: [agbrowse source map, agbrowse str_func, agbrowse 파일 구조] | 파일 | 라인 수 | 설명 | | --- | ---: | --- | -| `skills/browser/browser.mjs` | 3443 | root CLI parser, Chrome lifecycle, browser primitive commands | +| `skills/browser/browser.mjs` | 3467 | root CLI parser, Chrome lifecycle, browser primitive commands | | `skills/browser/runway.mjs` | 598 | Runway Apps/Custom selector contract, status/open/preflight command surface | | `skills/browser/runway-monitor.mjs` | 383 | Runway read-only queue/completion poller and progress signal extraction | | `skills/browser/tab-manager.mjs` | 446 | CDP target list, create, close, switch | @@ -66,12 +66,12 @@ aliases: [agbrowse source map, agbrowse str_func, agbrowse 파일 구조] | `skills/browser/adaptive-fetch/transforms.mjs` | 86 | URL transforms, HTML-to-text, content-type checks | | `skills/browser/adaptive-fetch/third-party-readers.mjs` | 46 | Jina Reader integration | | `skills/browser/adaptive-fetch/browser-runtime.mjs` | 38 | browser page acquisition and cleanup | -| `web-ai/cli.mjs` | 1839 | `web-ai` subcommand parser and command orchestration | +| `web-ai/cli.mjs` | 1884 | `web-ai` subcommand parser and command orchestration | | `web-ai/session-target-guard.mjs` | 151 | shared CDP session candidate selection, ambiguity errors, and target-mismatch recovery envelopes | -| `web-ai/chatgpt.mjs` | 927 | ChatGPT provider send/poll/query/status | -| `web-ai/gemini-live.mjs` | 784 | Gemini provider send/poll/query/status | -| `web-ai/grok-live.mjs` | 576 | Grok provider send/poll/query/status | -| `web-ai/mcp-server.mjs` | 354 | stdio JSON-RPC MCP bridge | +| `web-ai/chatgpt.mjs` | 965 | ChatGPT provider send/poll/query/status | +| `web-ai/gemini-live.mjs` | 801 | Gemini provider send/poll/query/status | +| `web-ai/grok-live.mjs` | 593 | Grok provider send/poll/query/status | +| `web-ai/mcp-server.mjs` | 384 | stdio JSON-RPC MCP bridge | | `web-ai/tool-schema.mjs` | 180 | MCP and AI SDK schema source | | `web-ai/answer-artifact.mjs` | 153 | provider poll result artifact normalization | | `web-ai/source-audit.mjs` | 183 | claim/source coverage audit helper | diff --git a/test/e2e/smoke.test.mjs b/test/e2e/smoke.test.mjs index e2fba088..8f83378e 100644 --- a/test/e2e/smoke.test.mjs +++ b/test/e2e/smoke.test.mjs @@ -40,7 +40,7 @@ describe.sequential('browser smoke e2e', () => { expect(existsSync(payload.path)).toBe(true); expect(payload.viewport.width).toBeGreaterThan(0); - const consoleLogs = await execBrowser(['console', '--clear', '--expression', 'console.log("smoke-console")', '--limit', '10', '--unsafe-allow', 'evaluate'], { env }); + const consoleLogs = await execBrowser(['console', '--clear', '--expression', 'console.log("smoke-console")', '--limit', '10'], { env }); expect(consoleLogs.stdout).toContain('smoke-console'); const network = await execBrowser(['network', '--clear', '--reload', '--duration', '700', '--filter', 'ping'], { env }); diff --git a/test/fixtures/browser-dpr-clip.json b/test/fixtures/browser-dpr-clip.json new file mode 100644 index 00000000..14aeffee --- /dev/null +++ b/test/fixtures/browser-dpr-clip.json @@ -0,0 +1,12 @@ +{ + "viewport": { "width": 1200, "height": 800 }, + "dpr": 2, + "clip": { "x": 300, "y": 100, "width": 400, "height": 300 }, + "candidate": { + "found": true, + "bbox": { "x": 80, "y": 40, "width": 120, "height": 60 }, + "point": { "x": 140, "y": 70 }, + "confidence": 0.84 + }, + "expectedCssPoint": { "x": 370, "y": 135 } +} diff --git a/test/fixtures/browser-observation-stale.json b/test/fixtures/browser-observation-stale.json new file mode 100644 index 00000000..b7df920c --- /dev/null +++ b/test/fixtures/browser-observation-stale.json @@ -0,0 +1,14 @@ +{ + "basis": { + "url": "https://example.test/a", + "targetId": "target-a", + "viewport": { "width": 1200, "height": 800 }, + "dpr": 2, + "capturedAt": "2026-06-17T00:00:00.000Z" + }, + "current": { + "url": "https://example.test/b", + "targetId": "target-b" + }, + "expected": { "stale": true } +} diff --git a/test/fixtures/browser-ref-vs-coordinate.json b/test/fixtures/browser-ref-vs-coordinate.json new file mode 100644 index 00000000..f6cac5bb --- /dev/null +++ b/test/fixtures/browser-ref-vs-coordinate.json @@ -0,0 +1,14 @@ +{ + "bundle": { + "refs": [ + { "ref": "@e1", "role": "button", "name": "Submit", "box": { "x": 100, "y": 100, "width": 120, "height": 40 } }, + { "ref": "@e2", "role": "button", "name": "Cancel", "box": { "x": 300, "y": 100, "width": 120, "height": 40 } } + ] + }, + "candidate": { + "point": { "x": 160, "y": 120 }, + "bbox": { "x": 110, "y": 105, "width": 100, "height": 30 }, + "confidence": 0.91 + }, + "expected": { "action": "ref", "ref": "@e1" } +} diff --git a/test/fixtures/vision-candidates.json b/test/fixtures/vision-candidates.json new file mode 100644 index 00000000..cf5480b0 --- /dev/null +++ b/test/fixtures/vision-candidates.json @@ -0,0 +1,34 @@ +[ + { + "id": "bbox-high-confidence", + "text": "Result: {\"found\":true,\"bbox\":{\"x\":80,\"y\":40,\"width\":120,\"height\":60},\"point\":{\"x\":140,\"y\":70},\"confidence\":0.84,\"description\":\"primary button\"}", + "expected": { + "found": true, + "kind": "vision_bbox", + "point": { "x": 140, "y": 70 }, + "confidence": 0.84, + "riskFlags": [] + } + }, + { + "id": "legacy-point", + "text": "{\"found\":true,\"x\":320,\"y\":160,\"description\":\"legacy point\"}", + "expected": { + "found": true, + "kind": "coordinate", + "point": { "x": 320, "y": 160 }, + "confidence": 0.5, + "riskFlags": ["point_only"] + } + }, + { + "id": "not-found", + "text": "{\"found\":false,\"bbox\":null,\"point\":{\"x\":0,\"y\":0},\"confidence\":0,\"description\":\"not found\"}", + "expected": { + "found": false, + "kind": "not_found", + "point": { "x": 0, "y": 0 }, + "confidence": 0 + } + } +] diff --git a/test/integration/cli-dom-commands.test.mjs b/test/integration/cli-dom-commands.test.mjs index 9c9946b2..a2100b51 100644 --- a/test/integration/cli-dom-commands.test.mjs +++ b/test/integration/cli-dom-commands.test.mjs @@ -47,11 +47,7 @@ describe.sequential('browser DOM commands', () => { const secondClick = await execBrowser(['click', buttonRefs[1]], { env }); expect(secondClick.code).toBe(0); - const deniedEvaluate = await execBrowser(['evaluate', 'document.body.dataset.lastProbe'], { env }); - expect(deniedEvaluate.code).not.toBe(0); - expect(deniedEvaluate.stderr).toContain('evaluate denied by policy'); - - const probeState = await execBrowser(['evaluate', 'document.body.dataset.lastProbe', '--unsafe-allow', 'evaluate'], { env }); + const probeState = await execBrowser(['evaluate', 'document.body.dataset.lastProbe'], { env }); expect(probeState.code).toBe(0); expect(probeState.stdout).toContain('"second"'); @@ -63,13 +59,13 @@ describe.sequential('browser DOM commands', () => { expect(select.code).toBe(0); expect(select.stdout).toContain(`selected "b"`); - const scrollBefore = await execBrowser(['evaluate', 'document.querySelector("[data-scroll-panel]").scrollTop', '--unsafe-allow', 'evaluate'], { env }); + const scrollBefore = await execBrowser(['evaluate', 'document.querySelector("[data-scroll-panel]").scrollTop'], { env }); const scroll = await execBrowser(['scroll', 'down', '--amount', '80', '--ref', scrollRef, '--json'], { env }); expect(scroll.code).toBe(0); const scrollPayload = JSON.parse(scroll.stdout); expect(scrollPayload).toMatchObject({ ok: true, direction: 'down', pixels: 80, ref: scrollRef }); - const scrollAfter = await execBrowser(['evaluate', 'document.querySelector("[data-scroll-panel]").scrollTop', '--unsafe-allow', 'evaluate'], { env }); + const scrollAfter = await execBrowser(['evaluate', 'document.querySelector("[data-scroll-panel]").scrollTop'], { env }); expect(Number(scrollAfter.stdout)).toBeGreaterThan(Number(scrollBefore.stdout)); const reload = await execBrowser(['reload'], { env }); diff --git a/test/integration/cli-network-console.test.mjs b/test/integration/cli-network-console.test.mjs index a0c6fe43..45082255 100644 --- a/test/integration/cli-network-console.test.mjs +++ b/test/integration/cli-network-console.test.mjs @@ -23,7 +23,7 @@ describe.sequential('browser console and network semantics', () => { }); it('reads buffered console logs from expressions and reloads', async () => { - const probe = await execBrowser(['console', '--clear', '--expression', 'console.log("probe-log")', '--limit', '10', '--unsafe-allow', 'evaluate'], { env }); + const probe = await execBrowser(['console', '--clear', '--expression', 'console.log("probe-log")', '--limit', '10'], { env }); expect(probe.code).toBe(0); expect(probe.stdout).toContain('probe-log'); diff --git a/test/integration/web-ai-mcp-server.test.mjs b/test/integration/web-ai-mcp-server.test.mjs index 5ae5aa65..d1d2169b 100644 --- a/test/integration/web-ai-mcp-server.test.mjs +++ b/test/integration/web-ai-mcp-server.test.mjs @@ -47,6 +47,9 @@ describe('web-ai MCP server', () => { expect(submitPrompt.inputSchema.properties.maxUploadFileSize.type).toBe('number'); expect(submitPrompt.description).toContain('generated image output'); expect(submitPrompt.description).toContain('CLI-only/deferred'); + const waitResponse = listed.result.tools.find(tool => tool.name === 'web_ai_wait_response'); + expect(waitResponse.description).toContain('recoverable timeout'); + expect(waitResponse.description).toContain('sessionId'); }); it('rejects unknown web_ai input fields before command execution', async () => { diff --git a/test/unit/browser-active-tab.test.mjs b/test/unit/browser-active-tab.test.mjs index 14a81717..d991d645 100644 --- a/test/unit/browser-active-tab.test.mjs +++ b/test/unit/browser-active-tab.test.mjs @@ -76,12 +76,12 @@ describe('active tab persistence contract', () => { expect(browserSrc).toContain("case 'uncheck'"); }); - it('keeps existing evaluate primitive policy-gated by default', () => { + it('keeps evaluate primitive policy-aware while defaulting to allowed', () => { expect(browserSrc).toContain("import { enforcePolicy } from '../../web-ai/policy/enforce.mjs'"); expect(browserSrc).toMatch(/async function evaluate\(port, expression, opts = \{\}\)/); expect(browserSrc).toMatch(/evaluate: true/); expect(browserSrc).toMatch(/unsafeAllow: opts\.unsafeAllow/); - expect(browserSrc).toContain('--unsafe-allow evaluate'); + expect(browserSrc).toContain("case 'evaluate'"); }); it('getActivePage resolves the persisted target before array-order fallback', () => { diff --git a/test/unit/candidate-reconcile.test.mjs b/test/unit/candidate-reconcile.test.mjs new file mode 100644 index 00000000..842d5190 --- /dev/null +++ b/test/unit/candidate-reconcile.test.mjs @@ -0,0 +1,48 @@ +// @ts-check +import { readFileSync } from 'node:fs'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { describe, expect, it } from 'vitest'; +import { + assertFreshObservationBundle, + reconcileVisionCandidate, +} from '../../web-ai/candidate-reconcile.mjs'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +describe('candidate reconciliation', () => { + it('prefers a ref when the vision candidate center is inside its box', () => { + const fixture = JSON.parse(readFileSync(join(__dirname, '..', 'fixtures', 'browser-ref-vs-coordinate.json'), 'utf8')); + expect(reconcileVisionCandidate({ candidate: fixture.candidate, bundle: fixture.bundle })).toMatchObject(fixture.expected); + }); + + it('fails when multiple boxes contain the candidate point', () => { + const result = reconcileVisionCandidate({ + candidate: { point: { x: 50, y: 50 } }, + bundle: { + refs: [ + { ref: '@e1', role: 'button', name: 'A', box: { x: 0, y: 0, width: 100, height: 100 } }, + { ref: '@e2', role: 'button', name: 'B', box: { x: 0, y: 0, width: 100, height: 100 } }, + ], + }, + }); + expect(result).toMatchObject({ action: 'fail', code: 'COMPUTER_TARGET_AMBIGUOUS' }); + }); + + it('falls back to coordinates when no ref box matches', () => { + const result = reconcileVisionCandidate({ + candidate: { point: { x: 500, y: 500 } }, + bundle: { refs: [{ ref: '@e1', role: 'button', name: 'A', box: { x: 0, y: 0, width: 100, height: 100 } }] }, + }); + expect(result).toMatchObject({ action: 'coordinate' }); + }); + + it('rejects stale observation bundles before coordinate fallback', () => { + const fixture = JSON.parse(readFileSync(join(__dirname, '..', 'fixtures', 'browser-observation-stale.json'), 'utf8')); + expect(() => assertFreshObservationBundle(fixture, fixture.current)).toThrow('COMPUTER_OBSERVATION_STALE'); + expect(() => assertFreshObservationBundle(fixture, { + url: fixture.basis.url, + targetId: fixture.basis.targetId, + })).not.toThrow(); + }); +}); diff --git a/test/unit/g06-observation-bundle.test.mjs b/test/unit/g06-observation-bundle.test.mjs index 6757e6ad..66d5de4d 100644 --- a/test/unit/g06-observation-bundle.test.mjs +++ b/test/unit/g06-observation-bundle.test.mjs @@ -12,11 +12,14 @@ const baseInput = { { ref: '@e2', role: 'textbox', name: 'Email', depth: 2 }, { ref: '@e3', role: 'textbox', name: 'Password', depth: 2 }, { ref: '@e4', role: 'button', name: 'Sign in', depth: 2 }, + { ref: 'e5', role: 'button', name: 'Browser ref', depth: 2 }, { ref: '...', role: 'note', name: '5 of 50 shown', depth: 0 }, + { ref: 'note-1', role: 'note', name: 'Non element ref', depth: 0 }, ], boxes: { '@e2': { x: 100, y: 200, width: 200, height: 30 }, '@e4': { x: 100, y: 400, width: 80, height: 32 }, + 'e5': { x: 220, y: 400, width: 90, height: 32 }, }, screenshotPath: '/tmp/screenshot.png', textSummary: 'Sign in to Example. Email Password Sign in', @@ -30,10 +33,11 @@ describe('G06 — observation-bundle ObservationBundleV1', () => { expect(OBSERVATION_BUNDLE_SCHEMA_VERSION).toBe('observation-bundle-v1'); }); - it('drops ellipsis/non-@ refs and preserves ordering', () => { + it('drops non-element refs and preserves @eN/eN ordering', () => { const b = buildObservationBundle(baseInput); - expect(b.refs.map((r) => r.ref)).toEqual(['@e1', '@e2', '@e3', '@e4']); + expect(b.refs.map((r) => r.ref)).toEqual(['@e1', '@e2', '@e3', '@e4', 'e5']); expect(b.refs.find((r) => r.ref === '...')).toBeUndefined(); + expect(b.refs.find((r) => r.ref === 'note-1')).toBeUndefined(); }); it('attaches boxes to refs that have them and leaves others without', () => { @@ -41,8 +45,9 @@ describe('G06 — observation-bundle ObservationBundleV1', () => { const map = Object.fromEntries(b.refs.map((r) => [r.ref, r])); expect(map['@e2'].box).toEqual({ x: 100, y: 200, width: 200, height: 30 }); expect(map['@e4'].box).toEqual({ x: 100, y: 400, width: 80, height: 32 }); + expect(map.e5.box).toEqual({ x: 220, y: 400, width: 90, height: 32 }); expect(map['@e1'].box).toBeUndefined(); - expect(b.stats.boxCount).toBe(2); + expect(b.stats.boxCount).toBe(3); }); it('clamps textSummary to maxTextChars', () => { @@ -55,12 +60,30 @@ describe('G06 — observation-bundle ObservationBundleV1', () => { it('reports stats correctly', () => { const b = buildObservationBundle(baseInput); - expect(b.stats.refCount).toBe(4); - expect(b.stats.boxCount).toBe(2); + expect(b.observationId).toBeTruthy(); + expect(b.basis).toMatchObject({ + url: baseInput.url, + viewport: baseInput.viewport, + dpr: 2, + capturedAt: baseInput.capturedAt, + }); + expect(b.stats.refCount).toBe(5); + expect(b.stats.boxCount).toBe(3); expect(b.stats.hasScreenshot).toBe(true); expect(b.screenshot).toBe('/tmp/screenshot.png'); }); + it('preserves explicit observationId and targetId', () => { + const b = buildObservationBundle({ + ...baseInput, + observationId: 'obs-explicit', + targetId: 'target-1', + }); + expect(b.observationId).toBe('obs-explicit'); + expect(b.targetId).toBe('target-1'); + expect(b.basis.targetId).toBe('target-1'); + }); + it('handles missing screenshot/boxes gracefully', () => { const b = buildObservationBundle({ url: 'https://x.test/', @@ -84,8 +107,9 @@ describe('G06 — observation-bundle ObservationBundleV1', () => { const b = buildObservationBundle(baseInput); const text = formatObservationBundle(b); expect(text).toMatch(/observation-bundle-v1/); - expect(text).toMatch(/refs=4/); - expect(text).toMatch(/boxes=2/); + expect(text).toMatch(/refs=5/); + expect(text).toMatch(/boxes=3/); expect(text).toMatch(/@e2.*box=100,200,200x30/); + expect(text).toMatch(/e5.*box=220,400,90x32/); }); }); diff --git a/test/unit/tab-lifecycle.test.mjs b/test/unit/tab-lifecycle.test.mjs index d09decea..cd5aecc3 100644 --- a/test/unit/tab-lifecycle.test.mjs +++ b/test/unit/tab-lifecycle.test.mjs @@ -3,7 +3,7 @@ import { readFileSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { parseDuration, selectProviderTabsForCleanup, selectTabsForCleanup } from '../../skills/browser/tab-lifecycle.mjs'; import { createTempBrowserEnv } from '../helpers/temp-env.mjs'; -import { checkoutPooledLease, cleanupLeasedTabs, listLeases, recordActiveLease, releaseCompletedLease } from '../../web-ai/tab-lease-store.mjs'; +import { checkoutPooledLease, cleanupLeasedTabs, listLeases, ProviderActiveCapacityError, recordActiveLease, releaseCompletedLease } from '../../web-ai/tab-lease-store.mjs'; describe('tab lifecycle cleanup selection', () => { it('parses duration strings used by tab-cleanup UX', () => { @@ -187,12 +187,45 @@ describe('tab lifecycle cleanup selection', () => { expect(lifecycleSource).toContain('!activeCommandTargets.has(tab.targetId)'); expect(leaseSource).toContain('activeCommandTargetIds'); expect(leaseSource).toContain('!activeTargets.has(lease.targetId)'); + expect(leaseSource).toContain('isPidAlive'); + expect(leaseSource).toContain('owner-pid-dead'); expect(lifecycleSource).not.toContain('activeCommandTargetIds({ browserProfileKey: String(port) }).catch'); expect(leaseSource).not.toContain('activeCommandTargetIds({ browserProfileKey }).catch'); expect(lifecycleSource).toContain('selectProviderTabsForCleanup'); expect(lifecycleSource).toContain('providerClosed'); }); + it('records provider leases before binding sessions to tabs', () => { + const chatgptSource = readFileSync(new URL('../../web-ai/chatgpt.mjs', import.meta.url), 'utf8'); + const geminiSource = readFileSync(new URL('../../web-ai/gemini-live.mjs', import.meta.url), 'utf8'); + const grokSource = readFileSync(new URL('../../web-ai/grok-live.mjs', import.meta.url), 'utf8'); + + expectOrder(chatgptSource, "sessionType: 'send-poll'", 'recordActiveLease', 'bindSessionToTab'); + expectOrder(chatgptSource, "sessionType: 'deep-research'", 'recordActiveLease', 'bindSessionToTab'); + expectOrder(geminiSource, "sessionType: 'send-poll'", 'recordActiveLease', 'bindSessionToTab'); + expectOrder(grokSource, "sessionType: 'send-poll'", 'recordActiveLease', 'bindSessionToTab'); + }); + + it('uses finite tab caps and current provider lease docs', () => { + const cliSource = readFileSync(new URL('../../web-ai/cli.mjs', import.meta.url), 'utf8'); + const lifecycleSource = readFileSync(new URL('../../skills/browser/tab-lifecycle.mjs', import.meta.url), 'utf8'); + const browserSource = readFileSync(new URL('../../skills/browser/browser.mjs', import.meta.url), 'utf8'); + const skillSource = readFileSync(new URL('../../skills/web-ai/SKILL.md', import.meta.url), 'utf8'); + const readmeSource = readFileSync(new URL('../../README.md', import.meta.url), 'utf8'); + + expect(cliSource).not.toContain('maxTabs: Number.POSITIVE_INFINITY'); + expect(cliSource).toContain('maxTabs: DEFAULT_MAX_TABS'); + expect(lifecycleSource).toContain("process.env.AGBROWSE_MAX_TABS || '20'"); + for (const source of [cliSource, browserSource]) { + expect(source).not.toContain('TTL=15m'); + expect(source).toContain('TTL=30m'); + expect(source).toContain('AGBROWSE_PROVIDER_ACTIVE_MAX_PER_KEY'); + } + expect(skillSource).toContain('| TTL per pooled tab | 30 min |'); + expect(skillSource).toContain('| Max tabs | 20 |'); + expect(readmeSource).toContain('| Max tabs | 20 |'); + }); + it('removes dead pooled lease metadata during checkout', async () => { const temp = createTempBrowserEnv('agbrowse-lease-dead-'); const previousHome = process.env.BROWSER_AGENT_HOME; @@ -229,6 +262,107 @@ describe('tab lifecycle cleanup selection', () => { } }); + it('records owner pid on active leases', async () => { + const temp = createTempBrowserEnv('agbrowse-owner-pid-'); + const previousHome = process.env.BROWSER_AGENT_HOME; + process.env.BROWSER_AGENT_HOME = temp.homeDir; + try { + await recordActiveLease({ + port: 65_532, + vendor: 'chatgpt', + targetId: 'active-with-owner', + sessionId: 'session-owner', + url: 'https://chatgpt.com/c/owner', + }); + + const [lease] = await listLeases(); + expect(lease.ownerPid).toBe(process.pid); + } finally { + if (previousHome === undefined) delete process.env.BROWSER_AGENT_HOME; + else process.env.BROWSER_AGENT_HOME = previousHome; + temp.cleanup(); + } + }); + + it('rejects active leases above the per-key cap but allows same-session replacement', async () => { + const temp = createTempBrowserEnv('agbrowse-active-per-key-'); + const previousHome = process.env.BROWSER_AGENT_HOME; + process.env.BROWSER_AGENT_HOME = temp.homeDir; + try { + for (const id of ['one', 'two']) { + await recordActiveLease({ + port: 65_533, + vendor: 'chatgpt', + targetId: `target-${id}`, + sessionId: `session-${id}`, + url: `https://chatgpt.com/c/${id}`, + activeMaxPerKey: 2, + activeGlobalMax: 10, + }); + } + + await expect(recordActiveLease({ + port: 65_533, + vendor: 'chatgpt', + targetId: 'target-three', + sessionId: 'session-three', + url: 'https://chatgpt.com/c/three', + activeMaxPerKey: 2, + activeGlobalMax: 10, + })).rejects.toBeInstanceOf(ProviderActiveCapacityError); + + await recordActiveLease({ + port: 65_533, + vendor: 'chatgpt', + targetId: 'target-one-rebound', + sessionId: 'session-one', + url: 'https://chatgpt.com/c/one-rebound', + activeMaxPerKey: 2, + activeGlobalMax: 10, + }); + + expect((await listLeases()).map(lease => lease.targetId).sort()).toEqual(['target-one-rebound', 'target-two']); + } finally { + if (previousHome === undefined) delete process.env.BROWSER_AGENT_HOME; + else process.env.BROWSER_AGENT_HOME = previousHome; + temp.cleanup(); + } + }); + + it('rejects active leases above the browser-profile global cap', async () => { + const temp = createTempBrowserEnv('agbrowse-active-global-'); + const previousHome = process.env.BROWSER_AGENT_HOME; + process.env.BROWSER_AGENT_HOME = temp.homeDir; + try { + await recordActiveLease({ + port: 65_534, + vendor: 'chatgpt', + targetId: 'target-chatgpt', + sessionId: 'session-chatgpt', + url: 'https://chatgpt.com/c/global', + activeMaxPerKey: 10, + activeGlobalMax: 1, + }); + + await expect(recordActiveLease({ + port: 65_534, + vendor: 'gemini', + targetId: 'target-gemini', + sessionId: 'session-gemini', + url: 'https://gemini.google.com/app/global', + activeMaxPerKey: 10, + activeGlobalMax: 1, + })).rejects.toMatchObject({ + errorCode: 'provider.active-capacity', + stage: 'provider-capacity', + }); + } finally { + if (previousHome === undefined) delete process.env.BROWSER_AGENT_HOME; + else process.env.BROWSER_AGENT_HOME = previousHome; + temp.cleanup(); + } + }); + it('completed-session cleanup is scoped to the current browser profile', async () => { const temp = createTempBrowserEnv('agbrowse-lease-profile-'); const previousHome = process.env.BROWSER_AGENT_HOME; @@ -307,3 +441,12 @@ function completedLease(targetId, browserProfileKey) { leaseKey: `web-ai:chatgpt:send-poll:https://chatgpt.com:${browserProfileKey}`, }; } + +function expectOrder(source, anchor, first, second) { + const anchorIndex = source.indexOf(anchor); + expect(anchorIndex).toBeGreaterThanOrEqual(0); + const snippet = source.slice(Math.max(0, anchorIndex - 220), anchorIndex + 420); + expect(snippet.indexOf(first)).toBeGreaterThanOrEqual(0); + expect(snippet.indexOf(second)).toBeGreaterThanOrEqual(0); + expect(snippet.indexOf(first)).toBeLessThan(snippet.indexOf(second)); +} diff --git a/test/unit/vision-core.test.mjs b/test/unit/vision-core.test.mjs index dedade3f..88de4a00 100644 --- a/test/unit/vision-core.test.mjs +++ b/test/unit/vision-core.test.mjs @@ -4,26 +4,38 @@ import { fileURLToPath } from 'node:url'; import { describe, expect, it } from 'vitest'; import { buildCoordPrompt, + candidateCenter, clipAroundPoint, describeRegion, extractCoordJson, + extractVisionCandidateJson, assertCodexCli, applyDprCorrection, + isLowConfidence, parseVisionClickCliArgs, parseViewportSpec, resolveRegionClip, + validateVisionCandidate, } from '../../skills/vision-click/vision-core.mjs'; const __dirname = dirname(fileURLToPath(import.meta.url)); const coordFixtures = JSON.parse( readFileSync(join(__dirname, '..', 'fixtures', 'coord-responses.json'), 'utf8') ); +const visionFixtures = JSON.parse( + readFileSync(join(__dirname, '..', 'fixtures', 'vision-candidates.json'), 'utf8') +); +const dprClipFixture = JSON.parse( + readFileSync(join(__dirname, '..', 'fixtures', 'browser-dpr-clip.json'), 'utf8') +); describe('vision-core', () => { it('buildCoordPrompt includes the target and JSON contract', () => { const prompt = buildCoordPrompt('Submit button'); expect(prompt).toContain('Submit button'); expect(prompt).toContain('"found"'); + expect(prompt).toContain('"bbox"'); + expect(prompt).toContain('"confidence"'); }); it('buildCoordPrompt includes region and verification hints when provided', () => { @@ -67,6 +79,44 @@ describe('vision-core', () => { expect(parsed).toBeNull(); }); + it('extractVisionCandidateJson parses bbox candidates and legacy points', () => { + for (const fixture of visionFixtures) { + const parsed = extractVisionCandidateJson(fixture.text); + expect(parsed).toMatchObject(fixture.expected); + } + }); + + it('candidateCenter derives the bbox center', () => { + const candidate = extractVisionCandidateJson(visionFixtures[0].text); + expect(candidateCenter(candidate)).toEqual({ x: 140, y: 70 }); + }); + + it('validateVisionCandidate rejects invalid bbox and out-of-bounds point', () => { + expect(() => validateVisionCandidate({ + schemaVersion: 'vision-candidate-v1', + found: true, + kind: 'vision_bbox', + bbox: { x: 1, y: 1, width: 0, height: 10 }, + point: { x: 1, y: 1 }, + confidence: 0.8, + riskFlags: [], + }, { viewport: { width: 100, height: 100 }, dpr: 1 })).toThrow('invalid vision candidate bbox'); + expect(() => validateVisionCandidate({ + schemaVersion: 'vision-candidate-v1', + found: true, + kind: 'coordinate', + bbox: null, + point: { x: 200, y: 1 }, + confidence: 0.8, + riskFlags: [], + }, { viewport: { width: 100, height: 100 }, dpr: 1 })).toThrow('outside'); + }); + + it('isLowConfidence applies the shared threshold', () => { + const candidate = extractVisionCandidateJson(visionFixtures[1].text); + expect(isLowConfidence(candidate)).toBe(true); + }); + it('assertCodexCli accepts an injected exec implementation', () => { const execFn = () => 'codex'; expect(() => assertCodexCli({ execFn, binary: 'which' })).not.toThrow(); @@ -83,6 +133,15 @@ describe('vision-core', () => { expect(applyDprCorrection(400, 276, 2)).toEqual({ x: 200, y: 138 }); }); + it('applyDprCorrection preserves clip origin evidence', () => { + const center = candidateCenter(dprClipFixture.candidate); + const css = applyDprCorrection(center.x, center.y, dprClipFixture.dpr); + expect({ + x: dprClipFixture.clip.x + css.x, + y: dprClipFixture.clip.y + css.y, + }).toEqual(dprClipFixture.expectedCssPoint); + }); + it('parseVisionClickCliArgs keeps option values out of the target', () => { const parsed = parseVisionClickCliArgs([ 'Submit', @@ -94,6 +153,7 @@ describe('vision-core', () => { '--viewport', '1440x900', '--region', 'left-panel', '--clip', '10', '20', '300', '180', + '--bundle', '/tmp/bundle.json', ], { port: '9222', browserScript: '/tmp/browser.mjs', @@ -109,7 +169,16 @@ describe('vision-core', () => { region: 'left-panel', clip: { x: 10, y: 20, width: 300, height: 180 }, help: false, + bundle: '/tmp/bundle.json', + }); + }); + + it('parseVisionClickCliArgs defaults bundle to null', () => { + const parsed = parseVisionClickCliArgs(['Target'], { + port: '9222', + browserScript: '/tmp/default-browser.mjs', }); + expect(parsed.opts.bundle).toBeNull(); }); it('parseVisionClickCliArgs reads browser-script explicitly', () => { diff --git a/test/unit/web-ai-policy.test.mjs b/test/unit/web-ai-policy.test.mjs index 99453b79..112947a0 100644 --- a/test/unit/web-ai-policy.test.mjs +++ b/test/unit/web-ai-policy.test.mjs @@ -25,7 +25,8 @@ describe('web-ai policy', () => { it('denies risky actions before mutation', () => { expect(() => enforcePolicy({}, { clipboardWriteIntercept: true })).toThrow(/provider copy capture denied/); - expect(() => enforcePolicy({}, { evaluate: true })).toThrow(/evaluate denied/); + expect(enforcePolicy({}, { evaluate: true }).ok).toBe(true); + expect(() => enforcePolicy({ allowEvaluate: false }, { evaluate: true })).toThrow(/evaluate denied/); }); it('exposes ruleId in policy errors', () => { diff --git a/test/unit/web-ai-provider-session.test.mjs b/test/unit/web-ai-provider-session.test.mjs index 252091db..9a1544f1 100644 --- a/test/unit/web-ai-provider-session.test.mjs +++ b/test/unit/web-ai-provider-session.test.mjs @@ -65,6 +65,78 @@ describe('web-ai summarizeEnvelope', () => { }); }); +describe('web-ai session timeout monotonicity', () => { + it('does not downgrade completed sessions when a later poll times out', async () => { + const { createSession, markSessionTimeout, updateSession, getSession } = await import('../../web-ai/session.mjs'); + const session = createSession({ vendor: 'chatgpt', prompt: 'x', attachmentPolicy: 'inline-only' }); + updateSession(session.sessionId, { + status: 'complete', + answer: 'done', + completedAt: '2026-06-21T00:00:00.000Z', + warnings: ['kept'], + }); + + const updated = markSessionTimeout(session.sessionId, { + lastError: { errorCode: 'provider.poll-timeout', message: 'late timeout' }, + }); + + expect(updated.status).toBe('complete'); + expect(updated.answer).toBe('done'); + expect(updated.warnings).toEqual(['kept', 'timeout-after-complete-ignored']); + expect(getSession(session.sessionId).status).toBe('complete'); + }); + + it('marks incomplete sessions as timeout with merged warning and lastError', async () => { + const { createSession, markSessionTimeout } = await import('../../web-ai/session.mjs'); + const session = createSession({ vendor: 'chatgpt', prompt: 'x', attachmentPolicy: 'inline-only' }); + + const updated = markSessionTimeout(session.sessionId, { + warning: 'poll-timeout', + lastError: { errorCode: 'provider.poll-timeout', message: 'timeout' }, + }); + + expect(updated.status).toBe('timeout'); + expect(updated.warnings).toEqual(['poll-timeout']); + expect(updated.lastError).toMatchObject({ errorCode: 'provider.poll-timeout' }); + }); +}); + +describe('web-ai provider timeout envelope', () => { + it('ChatGPT poll timeout remains recoverable and keeps session evidence', async () => { + const { createSession } = await import('../../web-ai/session.mjs'); + const { pollWebAi } = await import('../../web-ai/chatgpt.mjs'); + const session = createSession( + { vendor: 'chatgpt', prompt: 'slow', attachmentPolicy: 'inline-only' }, + { + targetId: 'target-1', + conversationUrl: 'https://chatgpt.com/c/slow', + deadlineAt: '2026-06-21T01:00:00.000Z', + envelopeSummary: { assistantCount: 0 }, + }, + ); + + const result = await pollWebAi({ + getPage: async () => createTimeoutChatGptPage(), + getTargetId: async () => 'target-1', + }, { + vendor: 'chatgpt', + session: session.sessionId, + timeout: 1, + }); + + expect(result).toMatchObject({ + ok: false, + vendor: 'chatgpt', + status: 'timeout', + sessionId: session.sessionId, + recoverable: true, + retryHint: 'poll-or-resume', + deadlineAt: '2026-06-21T01:00:00.000Z', + conversationUrl: 'https://chatgpt.com/c/slow', + }); + }); +}); + describe('web-ai provider integration (source-string contracts)', () => { const root = process.cwd(); const chatgptSrc = readFileSync(join(root, 'web-ai/chatgpt.mjs'), 'utf8'); @@ -88,10 +160,12 @@ describe('web-ai provider integration (source-string contracts)', () => { } }); - it('all three providers finalize completion and updateSession on timeout', () => { + it('all three providers finalize completion and markSessionTimeout on timeout', () => { for (const src of [chatgptSrc, geminiSrc, grokSrc]) { expect(src).toMatch(/finalizeProviderTab\(deps, \{[\s\S]*?session[\s\S]*?answerText/); - expect(src).toMatch(/updateSession\(session\.sessionId, \{ status: 'timeout' \}\)/); + expect(src).toMatch(/markSessionTimeout\(session\.sessionId/); + expect(src).toContain("retryHint: 'poll-or-resume'"); + expect(src).toContain('recoverable: true'); } expect(finalizerSrc).toMatch(/updateSession\(session\.sessionId, \{[\s\S]*?status: 'complete'/); expect(finalizerSrc).toMatch(/completedAt: new Date\(\)\.toISOString\(\)/); @@ -157,6 +231,17 @@ describe('web-ai cli session flags', () => { expect(cliSrc).toMatch(/case 'query': return withWebAiActiveCommand/); }); + it('wraps MCP wait/resume in session command lock, session page recovery, and MCP active command', () => { + const mcpSrc = readFileSync(join(process.cwd(), 'web-ai/mcp-server.mjs'), 'utf8'); + expect(mcpSrc).toContain("import { withSessionCommandLock } from './session-store.mjs'"); + expect(mcpSrc).toContain("import { withSessionPage } from './tab-recovery.mjs'"); + expect(mcpSrc).toMatch(/if \(name === 'web_ai_wait_response' \|\| name === 'web_ai_session_resume'\) \{[\s\S]*?return runMcpSessionPoll\(name, args, deps\)/); + expect(mcpSrc).toMatch(/async function runMcpSessionPoll\(name, args, deps\)/); + expect(mcpSrc).toMatch(/withSessionCommandLock\(sessionId/); + expect(mcpSrc).toMatch(/withSessionPage\(deps, sessionId/); + expect(mcpSrc).toMatch(/withMcpActiveCommand\(name, provider, sessionDeps, sessionArgs/); + }); + it('reuses inactive provider tabs before creating another ChatGPT tab', () => { expect(cliSrc).toContain("import { createTab, listManagedTabs, waitForPageByTargetId }"); expect(cliSrc).toMatch(/async function findReusableProviderTab\(port, vendor, targetUrl\)/); @@ -184,3 +269,17 @@ describe('web-ai cli session flags', () => { expect(recoverySrc).toContain('export async function resolveSessionPage'); }); }); + +function createTimeoutChatGptPage() { + return { + url: () => 'https://chatgpt.com/c/slow', + evaluate: async () => [], + waitForTimeout: async () => undefined, + locator: () => ({ + first: () => ({ + isVisible: async () => false, + }), + all: async () => [], + }), + }; +} diff --git a/test/unit/web-ai-timeout-default.test.mjs b/test/unit/web-ai-timeout-default.test.mjs new file mode 100644 index 00000000..912e4c5b --- /dev/null +++ b/test/unit/web-ai-timeout-default.test.mjs @@ -0,0 +1,67 @@ +import { describe, expect, it } from 'vitest'; +import { + TIER_DEFAULT_TIMEOUT_SEC, + PRO_TIMEOUT_SEC, + tierDefaultTimeoutSec, + deriveTimeoutTier, + resolveTimeoutDefaultSec, +} from '../../web-ai/session.mjs'; + +describe('web-ai tier-aware default poll timeout', () => { + it('exposes the locked tier table (pro = 3600)', () => { + expect({ ...TIER_DEFAULT_TIMEOUT_SEC }).toEqual({ + instant: 120, + thinking: 600, + pro: 3600, + 'deep-research': 3600, + }); + expect(PRO_TIMEOUT_SEC).toBe(3600); + }); + + it('derives the ChatGPT tier from model + research', () => { + expect(deriveTimeoutTier('chatgpt', 'instant')).toBe('instant'); + expect(deriveTimeoutTier('chatgpt', 'thinking')).toBe('thinking'); + expect(deriveTimeoutTier('chatgpt', 'pro')).toBe('pro'); + expect(deriveTimeoutTier('chatgpt', 'pro', 'deep')).toBe('deep-research'); + expect(deriveTimeoutTier('chatgpt', 'thinking', 'deep')).toBe('deep-research'); + expect(deriveTimeoutTier('chatgpt', undefined)).toBe(null); + }); + + it('derives the Grok tier so heavy != expert', () => { + expect(deriveTimeoutTier('grok', 'heavy')).toBe('pro'); + expect(deriveTimeoutTier('grok', 'expert')).toBe('thinking'); + expect(deriveTimeoutTier('grok', 'thinking')).toBe('thinking'); // alias -> expert + expect(deriveTimeoutTier('grok', 'fast')).toBe('instant'); + expect(deriveTimeoutTier('grok', 'auto')).toBe('thinking'); + }); + + it('derives the Gemini tier including deep-think', () => { + expect(deriveTimeoutTier('gemini', 'deepthink')).toBe('deep-research'); + expect(deriveTimeoutTier('gemini', 'flash-lite')).toBe('instant'); + expect(deriveTimeoutTier('gemini', 'flash')).toBe('thinking'); + expect(deriveTimeoutTier('gemini', 'pro')).toBe('thinking'); + }); + + it('maps tiers to seconds with vendor fallback for unknown models', () => { + expect(tierDefaultTimeoutSec('pro')).toBe(3600); + expect(tierDefaultTimeoutSec('instant')).toBe(120); + expect(tierDefaultTimeoutSec('thinking')).toBe(600); + expect(tierDefaultTimeoutSec('deep-research')).toBe(3600); + expect(tierDefaultTimeoutSec(null, 'chatgpt')).toBe(1200); + expect(tierDefaultTimeoutSec(null, 'grok')).toBe(600); + expect(tierDefaultTimeoutSec(null, 'gemini')).toBe(1200); + }); + + it('resolves end-to-end defaults: pro=3600, grok heavy(3600) != grok expert(600)', () => { + expect(resolveTimeoutDefaultSec({ model: 'pro' }, 'chatgpt')).toBe(3600); + expect(resolveTimeoutDefaultSec({ model: 'instant' }, 'chatgpt')).toBe(120); + expect(resolveTimeoutDefaultSec({ model: 'thinking' }, 'chatgpt')).toBe(600); + expect(resolveTimeoutDefaultSec({ model: 'pro', research: 'deep' }, 'chatgpt')).toBe(3600); + expect(resolveTimeoutDefaultSec({ model: 'heavy' }, 'grok')).toBe(3600); + expect(resolveTimeoutDefaultSec({ model: 'expert' }, 'grok')).toBe(600); + expect(resolveTimeoutDefaultSec({ model: 'deepthink' }, 'gemini')).toBe(3600); + // unknown / resume without model -> vendor default (no regression below today's value) + expect(resolveTimeoutDefaultSec({}, 'chatgpt')).toBe(1200); + expect(resolveTimeoutDefaultSec({}, 'grok')).toBe(600); + }); +}); diff --git a/test/unit/web-ai-tool-schema.test.mjs b/test/unit/web-ai-tool-schema.test.mjs index abc41689..57ac0413 100644 --- a/test/unit/web-ai-tool-schema.test.mjs +++ b/test/unit/web-ai-tool-schema.test.mjs @@ -32,6 +32,9 @@ describe('web-ai MCP tool schema', () => { const submit = toolSchemaForMcp('web_ai_submit_prompt'); expect(submit.description).toContain('CLI-only/deferred'); expect(submit.inputSchema.properties.maxUploadFileSize).toMatchObject({ type: 'number', minimum: 1 }); + expect(toolSchemaForMcp('web_ai_wait_response').description).toContain('recoverable timeout'); + expect(toolSchemaForMcp('web_ai_wait_response').description).toContain('sessionId'); + expect(toolSchemaForMcp('web_ai_session_resume').description).toContain('session-bound recovery'); }); it('exposes Phase 18 browser tools from the shared schema source', () => { diff --git a/test/unit/web-ai-watcher.test.mjs b/test/unit/web-ai-watcher.test.mjs index c9f16876..f8d258c2 100644 --- a/test/unit/web-ai-watcher.test.mjs +++ b/test/unit/web-ai-watcher.test.mjs @@ -34,3 +34,24 @@ describe('web-ai watcher transient-timeout promotion (source-string contract)', expect(watcherSrc).toContain('watcher-resumed-transient-timeout'); }); }); + +describe('web-ai watcher self-heals drifted conversation URL (source-string contract)', () => { + it('destructures the resolver-healed session from the withSessionPage callback', () => { + expect(watcherSrc).toMatch( + /withSessionPage\(deps, options\.sessionId, async \(\{ page, targetId, session: resolvedSession \}\)/, + ); + }); + + it('feeds the healed session (not the stale outer one) to the attach check', () => { + expect(watcherSrc).toContain('ensureWatcherAttached(page, resolvedSession || session, options)'); + }); + + it('uses the canonical tolerant urlsCompatible predicate imported from tab-recovery', () => { + expect(watcherSrc).toContain("import { withSessionPage, urlsCompatible } from './tab-recovery.mjs'"); + expect(watcherSrc).toContain('if (urlsCompatible(targetUrl, currentUrl))'); + }); + + it('retires the strict urlsEquivalentForWatch helper', () => { + expect(watcherSrc).not.toContain('urlsEquivalentForWatch'); + }); +}); diff --git a/tsconfig.checkjs.json b/tsconfig.checkjs.json index 69af4a55..21583b4a 100644 --- a/tsconfig.checkjs.json +++ b/tsconfig.checkjs.json @@ -34,6 +34,8 @@ "web-ai/trace/writer.mjs", "web-ai/action-trace.mjs", "web-ai/capability.mjs", + "web-ai/observation-bundle.mjs", + "web-ai/candidate-reconcile.mjs", "web-ai/answer-artifact.mjs", "web-ai/ref-registry.mjs", "web-ai/tool-schema.mjs", diff --git a/web-ai/candidate-reconcile.mjs b/web-ai/candidate-reconcile.mjs new file mode 100644 index 00000000..293267f8 --- /dev/null +++ b/web-ai/candidate-reconcile.mjs @@ -0,0 +1,74 @@ +// @ts-check + +/** + * @typedef {{x:number,y:number,width:number,height:number}} Box + * @typedef {{ref:string, role:string, name:string, box?:Box}} BundleRef + * @typedef {{refs:BundleRef[]}} ObservationBundle + * @typedef {{point:{x:number,y:number}, bbox?:Box|null, confidence?:number}} VisionCandidate + */ + +/** + * @param {{candidate: VisionCandidate, bundle: ObservationBundle, maxDistance?: number}} input + */ +export function reconcileVisionCandidate(input) { + const maxDistance = input.maxDistance ?? 32; + const refs = Array.isArray(input.bundle?.refs) ? input.bundle.refs.filter((r) => r.box) : []; + const point = input.candidate.point; + const containing = refs.filter((r) => contains(r.box, point)); + if (containing.length === 1) { + return { action: 'ref', ref: containing[0].ref, reason: 'candidate_center_inside_ref_box' }; + } + if (containing.length > 1) { + return { action: 'fail', code: 'COMPUTER_TARGET_AMBIGUOUS', reason: 'multiple_ref_boxes_contain_candidate' }; + } + const nearby = refs + .map((r) => ({ ref: r.ref, distance: distanceToBoxCenter(point, r.box) })) + .filter((r) => r.distance <= maxDistance) + .sort((a, b) => a.distance - b.distance); + if (nearby.length === 1 || (nearby.length > 1 && nearby[0].distance + 8 < nearby[1].distance)) { + return { action: 'ref', ref: nearby[0].ref, reason: 'candidate_center_near_ref_box' }; + } + if (nearby.length > 1) { + return { action: 'fail', code: 'COMPUTER_TARGET_AMBIGUOUS', reason: 'multiple_nearby_ref_boxes' }; + } + return { action: 'coordinate', reason: 'no_matching_ref_box' }; +} + +/** + * @param {{basis?:{url?:string,targetId?:string}, url?:string, targetId?:string}} bundle + * @param {{url?:string,targetId?:string}} current + */ +export function assertFreshObservationBundle(bundle, current) { + const basis = bundle.basis || bundle; + if (basis.url && current.url && basis.url !== current.url) { + throw new Error('COMPUTER_OBSERVATION_STALE: observation URL does not match current page'); + } + if (basis.targetId && current.targetId && basis.targetId !== current.targetId) { + throw new Error('COMPUTER_OBSERVATION_STALE: observation targetId does not match current page'); + } +} + +/** + * @param {Box|undefined} box + * @param {{x:number,y:number}} point + */ +function contains(box, point) { + if (!box) return false; + return ( + point.x >= box.x && + point.y >= box.y && + point.x <= box.x + box.width && + point.y <= box.y + box.height + ); +} + +/** + * @param {{x:number,y:number}} point + * @param {Box|undefined} box + */ +function distanceToBoxCenter(point, box) { + if (!box) return Number.POSITIVE_INFINITY; + const cx = box.x + box.width / 2; + const cy = box.y + box.height / 2; + return Math.hypot(point.x - cx, point.y - cy); +} diff --git a/web-ai/chatgpt.mjs b/web-ai/chatgpt.mjs index 1416260f..b30bc617 100644 --- a/web-ai/chatgpt.mjs +++ b/web-ai/chatgpt.mjs @@ -14,6 +14,7 @@ import { getBaseline, getLatestBaseline, getSession, + markSessionTimeout, resolveDeadlineAt, saveBaseline, sessionToBaseline, @@ -197,7 +198,6 @@ export async function sendWebAi(deps, input = {}) { if (selectedModel?.modelSelection) { updateSession(session.sessionId, { modelSelection: selectedModel.modelSelection }); } - if (targetId) bindSessionToTab(session.sessionId, targetId); if (targetId) await recordActiveLease({ owner: 'web-ai', vendor: envelope.vendor, @@ -207,6 +207,7 @@ export async function sendWebAi(deps, input = {}) { url: page.url(), port: deps.getPort?.() || 9222, }); + if (targetId) bindSessionToTab(session.sessionId, targetId); const editorOptions = { insertText: async (/** @type {any} */ text) => { @@ -523,22 +524,44 @@ export async function pollWebAi(deps, input = {}) { responseStableMs: stableSince ? Date.now() - stableSince : 0, }); } - if (session) updateSession(session.sessionId, { status: 'timeout' }); + const timedOutSession = session ? markSessionTimeout(session.sessionId, { + lastError: { errorCode: 'provider.poll-timeout', message: 'timed out waiting for answer' }, + }) : null; return { ok: false, vendor, status: 'timeout', url: page.url(), ...(session ? { sessionId: session.sessionId } : {}), + ...(timedOutSession?.deadlineAt ? { deadlineAt: timedOutSession.deadlineAt } : {}), + ...(timedOutSession?.conversationUrl ? { conversationUrl: timedOutSession.conversationUrl } : {}), baseline, ...(traceSummary ? { traceSummary } : {}), warnings: [`copy-markdown-fallback-unavailable:${(/** @type {any} */ (copied)).status || 'unknown'}`], usedFallbacks: [], + recoverable: true, + retryHint: 'poll-or-resume', error: 'timed out waiting for answer', }; } - if (session) updateSession(session.sessionId, { status: 'timeout' }); - return { ok: false, vendor, status: 'timeout', url: page.url(), ...(session ? { sessionId: session.sessionId } : {}), baseline, warnings: [], usedFallbacks: [], error: 'timed out waiting for answer' }; + const timedOutSession = session ? markSessionTimeout(session.sessionId, { + lastError: { errorCode: 'provider.poll-timeout', message: 'timed out waiting for answer' }, + }) : null; + return { + ok: false, + vendor, + status: 'timeout', + url: page.url(), + ...(session ? { sessionId: session.sessionId } : {}), + ...(timedOutSession?.deadlineAt ? { deadlineAt: timedOutSession.deadlineAt } : {}), + ...(timedOutSession?.conversationUrl ? { conversationUrl: timedOutSession.conversationUrl } : {}), + baseline, + warnings: [], + usedFallbacks: [], + recoverable: true, + retryHint: 'poll-or-resume', + error: 'timed out waiting for answer', + }; } /** @@ -622,7 +645,6 @@ export async function deepResearchWebAi(deps, input = {}) { deadlineAt: resolveDeadlineAt(input, 'chatgpt'), envelopeSummary: { ...summarizeEnvelope(input), assistantCount }, }); - if (targetId) bindSessionToTab(session.sessionId, targetId); if (targetId) await recordActiveLease({ owner: 'web-ai', vendor: envelope.vendor, @@ -632,6 +654,7 @@ export async function deepResearchWebAi(deps, input = {}) { url: page.url(), port: deps.getPort?.() || 9222, }); + if (targetId) bindSessionToTab(session.sessionId, targetId); const timeoutMs = Math.max(1, Number(input.timeout || 1200)) * 1000; const selectedTools = await selectChatGptComposerTools(page, { ...input, research: 'deep' }); const result = await sendDeepResearch(page, deps, { diff --git a/web-ai/cli.mjs b/web-ai/cli.mjs index 4e8fddd0..7fdb0d92 100644 --- a/web-ai/cli.mjs +++ b/web-ai/cli.mjs @@ -17,10 +17,10 @@ import { watchSession } from './watcher.mjs'; import { buildWebAiSnapshot } from './ax-snapshot.mjs'; import { runSessionsCommand, printSessionsHuman, parseDurationToMs } from './cli-sessions.mjs'; import { createTab, listManagedTabs, waitForPageByTargetId } from '../skills/browser/tab-manager.mjs'; -import { cleanupIdleTabs, isPinned } from '../skills/browser/tab-lifecycle.mjs'; +import { cleanupIdleTabs, isPinned, DEFAULT_MAX_TABS } from '../skills/browser/tab-lifecycle.mjs'; import { resolveSessionPage, withSessionPage } from './tab-recovery.mjs'; import { withSessionCommandLock } from './session-store.mjs'; -import { listSessions, getSession } from './session.mjs'; +import { listSessions, getSession, resolveTimeoutDefaultSec } from './session.mjs'; import { resolveImplicitSessionSelection } from './session-target-guard.mjs'; import { listLeases } from './tab-lease-store.mjs'; import { cleanupPoolTabs, getPooledTab } from './tab-pool.mjs'; @@ -121,15 +121,21 @@ Provider: Thinking: light, standard, extended, heavy --reasoning-effort Alias for --effort - --timeout Polling timeout. Defaults: ChatGPT 1200, Gemini 1200, Grok 600. + --timeout Polling timeout. When omitted, the default scales by model tier: + instant 120s, thinking 600s, pro/deep-research 3600s (vendor + default 1200/1200/600 for unknown models). --timeout overrides. Prompt envelope (every prompt also gets a [INSTRUCTIONS] block telling the model to use web search and cite sources inline): --prompt Main user prompt/question (required) - --system System / role instruction + --system Trusted operating/role instructions — the channel for skill + guidance and "how to behave". Honored, not treated as data. + Put instructions HERE, not in --context. --project Project name --goal Task goal - --context Inline context + --context UNTRUSTED reference data only (scraped text, provider output). + Rendered as [UNTRUSTED_CONTEXT]; instructions placed here are + ignored by design. For a file the model should read, use --file. --question Alias for prompt detail --output Output preference --constraints Constraints to include in the prompt @@ -210,10 +216,12 @@ Browser: Tab lease policy: Completed provider tabs are runtime leases, not history storage. - Defaults: maxPerKey=3, globalMax=8, TTL=15m. Per-key limit is the + Pool defaults: maxPerKey=3, globalMax=8, TTL=30m. Per-key limit is the number of warm pooled tabs allowed per (owner,vendor,sessionType,origin,profile). Override via AGBROWSE_PROVIDER_POOL_MAX_PER_KEY / _GLOBAL_MAX / _TTL. + Active session caps default to per-key=5 and global=14. Override via + AGBROWSE_PROVIDER_ACTIVE_MAX_PER_KEY / _GLOBAL_MAX. Expired or overflow pooled tabs are closed with CDP. Use --new-tab / --parallel to bypass pool reuse for a single call. Use "agbrowse tab-cleanup --json" to inspect leaseClosedTabs. @@ -638,7 +646,12 @@ async function runWebAiCliInner(argv = [], deps) { question: values.question, output: values.output, constraints: values.constraints, - timeout: values.timeout, + // When --timeout is omitted, default scales by model tier (instant 120s, + // thinking 600s, pro/deep-research 3600s) so a long pro run is not capped + // at the legacy 1200s. An explicit --timeout still wins. + timeout: values.timeout != null + ? values.timeout + : resolveTimeoutDefaultSec({ model: values.model, research: values.research }, values.vendor || 'chatgpt'), deadline: values.deadline, session: values.session, navigate: values.navigate === true, @@ -975,7 +988,7 @@ async function ensureProviderTab(deps, input) { const port = deps.getPort?.() || 9222; await cleanupPoolTabs(port); - await cleanupIdleTabs(port, { maxTabs: Number.POSITIVE_INFINITY }); + await cleanupIdleTabs(port, { maxTabs: DEFAULT_MAX_TABS }); if (input.forceNewTab !== true) { // Phase 9.2: try tab pool first diff --git a/web-ai/code-artifact.mjs b/web-ai/code-artifact.mjs index 2f6a2c5a..624274aa 100644 --- a/web-ai/code-artifact.mjs +++ b/web-ai/code-artifact.mjs @@ -1,5 +1,5 @@ // @ts-check -// Code-mode artifact retrieval (Phase 10 of devlog/_plan/260611_webai_gpt_code_mode). +// Code-mode artifact retrieval (Phase 10 of devlog/_fin/260611_webai_gpt_code_mode). // Retrieves a /mnt/data/*.zip built by ChatGPT's container tools without any // button click: conversation JSON → sandbox path scan → interpreter/download // presigned URL → in-page credentialed fetch (estuary URLs are cookie-bound; diff --git a/web-ai/code-mode-prompt.mjs b/web-ai/code-mode-prompt.mjs index bed8595f..445ec2c2 100644 --- a/web-ai/code-mode-prompt.mjs +++ b/web-ai/code-mode-prompt.mjs @@ -1,5 +1,5 @@ // @ts-check -// Code-mode prompt contract (Phase 11 of devlog/_plan/260611_webai_gpt_code_mode). +// Code-mode prompt contract (Phase 11 of devlog/_fin/260611_webai_gpt_code_mode). // The contract text was settled by interrogating GPT (thinking/standard) and // verified end-to-end: GPT used container.exec, built exactly one // /mnt/data/result.zip, and answered with a machine-readable path. The hardening diff --git a/web-ai/code-mode.mjs b/web-ai/code-mode.mjs index 3017e457..8fef1119 100644 --- a/web-ai/code-mode.mjs +++ b/web-ai/code-mode.mjs @@ -1,5 +1,5 @@ // @ts-check -// Code-mode orchestration (Phase 12 of devlog/_plan/260611_webai_gpt_code_mode): +// Code-mode orchestration (Phase 12 of devlog/_fin/260611_webai_gpt_code_mode): // strict contract prompt → ChatGPT query → conversation-id resolution → // in-page artifact retrieval → zip verification. ChatGPT-only: the contract // depends on container tools that Gemini/Grok do not expose the same way. diff --git a/web-ai/errors.mjs b/web-ai/errors.mjs index ce30e601..9e249cdb 100644 --- a/web-ai/errors.mjs +++ b/web-ai/errors.mjs @@ -95,6 +95,23 @@ export class WebAiError extends Error { */ export function wrapError(err, fallback = {}) { if (err instanceof WebAiError) return err; + if (err && typeof err === 'object' && typeof /** @type {{ errorCode?: unknown }} */ (err).errorCode === 'string') { + const e = /** @type {{ message?: string, errorCode?: string, stage?: string, retryHint?: string, vendor?: string, mutationAllowed?: boolean, selectorsTried?: string[], evidence?: unknown, traceId?: string, ruleId?: string }} */ (err); + return new WebAiError({ + errorCode: e.errorCode, + stage: e.stage, + retryHint: e.retryHint, + vendor: e.vendor, + mutationAllowed: e.mutationAllowed, + selectorsTried: e.selectorsTried, + evidence: e.evidence, + traceId: e.traceId, + ruleId: e.ruleId, + message: e.message, + ...fallback, + cause: err, + }); + } const e = /** @type {{ message?: string }} */ (err); return new WebAiError({ errorCode: 'internal.unhandled', diff --git a/web-ai/gemini-live.mjs b/web-ai/gemini-live.mjs index 62017360..84d3771c 100644 --- a/web-ai/gemini-live.mjs +++ b/web-ai/gemini-live.mjs @@ -14,6 +14,7 @@ import { getBaseline, getLatestBaseline, getSession, + markSessionTimeout, resolveDeadlineAt, saveBaseline, sessionToBaseline, @@ -266,7 +267,6 @@ export async function geminiSendWebAi(deps, input = {}) { deadlineAt: resolveDeadlineAt(input, 'gemini'), envelopeSummary: { ...summarizeEnvelope(input, contextPack), assistantCount: turnsBefore }, }); - if (targetId) bindSessionToTab(session.sessionId, targetId); if (targetId) await recordActiveLease({ owner: 'web-ai', vendor: 'gemini', @@ -276,6 +276,7 @@ export async function geminiSendWebAi(deps, input = {}) { url: page.url(), port: deps.getPort?.() || 9222, }); + if (targetId) bindSessionToTab(session.sessionId, targetId); return { ok: true, vendor: 'gemini', @@ -492,8 +493,24 @@ export async function geminiPollWebAi(deps, input = {}) { throw pollErr; } } - if (session) updateSession(session.sessionId, { status: 'timeout' }); - return { ok: false, vendor: 'gemini', status: 'timeout', url: page.url(), ...(session ? { sessionId: session.sessionId } : {}), baseline, warnings: [], usedFallbacks: [], error: 'timed out waiting for gemini response' }; + const timedOutSession = session ? markSessionTimeout(session.sessionId, { + lastError: { errorCode: 'provider.poll-timeout', message: 'timed out waiting for gemini response' }, + }) : null; + return { + ok: false, + vendor: 'gemini', + status: 'timeout', + url: page.url(), + ...(session ? { sessionId: session.sessionId } : {}), + ...(timedOutSession?.deadlineAt ? { deadlineAt: timedOutSession.deadlineAt } : {}), + ...(timedOutSession?.conversationUrl ? { conversationUrl: timedOutSession.conversationUrl } : {}), + baseline, + warnings: [], + usedFallbacks: [], + recoverable: true, + retryHint: 'poll-or-resume', + error: 'timed out waiting for gemini response', + }; } /** diff --git a/web-ai/grok-live.mjs b/web-ai/grok-live.mjs index 2e33eb33..707ef1ff 100644 --- a/web-ai/grok-live.mjs +++ b/web-ai/grok-live.mjs @@ -12,6 +12,7 @@ import { getBaseline, getLatestBaseline, getSession, + markSessionTimeout, resolveDeadlineAt, saveBaseline, sessionToBaseline, @@ -206,7 +207,6 @@ export async function grokSendWebAi(deps, input = {}) { deadlineAt: resolveDeadlineAt(input, 'grok'), envelopeSummary: { ...summarizeEnvelope(input, contextPack), assistantCount }, }); - if (targetId) bindSessionToTab(session.sessionId, targetId); if (targetId) await recordActiveLease({ owner: 'web-ai', vendor: 'grok', @@ -216,6 +216,7 @@ export async function grokSendWebAi(deps, input = {}) { url: page.url(), port: deps.getPort?.() || 9222, }); + if (targetId) bindSessionToTab(session.sessionId, targetId); return { ok: true, vendor: 'grok', @@ -329,8 +330,24 @@ export async function grokPollWebAi(deps, input = {}) { throw pollErr; } } - if (session) updateSession(session.sessionId, { status: 'timeout' }); - return { ok: false, vendor: 'grok', status: 'timeout', url: page.url(), ...(session ? { sessionId: session.sessionId } : {}), baseline, warnings: [], usedFallbacks: [], error: 'timed out waiting for grok response' }; + const timedOutSession = session ? markSessionTimeout(session.sessionId, { + lastError: { errorCode: 'provider.poll-timeout', message: 'timed out waiting for grok response' }, + }) : null; + return { + ok: false, + vendor: 'grok', + status: 'timeout', + url: page.url(), + ...(session ? { sessionId: session.sessionId } : {}), + ...(timedOutSession?.deadlineAt ? { deadlineAt: timedOutSession.deadlineAt } : {}), + ...(timedOutSession?.conversationUrl ? { conversationUrl: timedOutSession.conversationUrl } : {}), + baseline, + warnings: [], + usedFallbacks: [], + recoverable: true, + retryHint: 'poll-or-resume', + error: 'timed out waiting for grok response', + }; } /** diff --git a/web-ai/mcp-server.mjs b/web-ai/mcp-server.mjs index 69e980ff..101dd0c2 100644 --- a/web-ai/mcp-server.mjs +++ b/web-ai/mcp-server.mjs @@ -23,6 +23,8 @@ import { isKnownBrowserTool, validateBrowserToolInput, getDeferredBrowserToolMet import { enforcePolicy } from './policy/enforce.mjs'; import { applyProviderDefaults } from './policy/default-policy.mjs'; import { withActiveCommand } from './active-command-store.mjs'; +import { withSessionCommandLock } from './session-store.mjs'; +import { withSessionPage } from './tab-recovery.mjs'; import { requireLatestSnapshot, setLatestSnapshot } from './mcp-state.mjs'; const MCP_PROTOCOL_VERSION = '2025-06-18'; @@ -209,14 +211,7 @@ async function callMcpTool(name, args, deps, state) { ); } if (name === 'web_ai_wait_response' || name === 'web_ai_session_resume') { - const session = getSession(args.sessionId); - const provider = args.provider || args.vendor || session?.vendor || 'chatgpt'; - return pollByProvider(providerFromArgs({ provider }), deps, { - ...args, - vendor: provider, - session: args.sessionId, - timeout: args.timeout, - }); + return runMcpSessionPoll(name, args, deps); } if (name === 'web_ai_copy_markdown') { const provider = providerFromArgs(args); @@ -238,6 +233,41 @@ async function callMcpTool(name, args, deps, state) { throw new Error(`unhandled tool: ${name}`); } +/** + * @param {any} name + * @param {any} args + * @param {any} deps + */ +async function runMcpSessionPoll(name, args, deps) { + const sessionId = args.sessionId; + const stored = getSession(sessionId); + if (!stored) throw new Error(`no session record for ${sessionId}`); + providerFromArgs({ provider: args.provider || args.vendor || stored.vendor || 'chatgpt' }); + return withSessionCommandLock(sessionId, () => + withSessionPage(deps, sessionId, async ({ page, targetId, session }) => { + const provider = providerFromArgs({ provider: session.vendor || stored.vendor || args.provider || args.vendor || 'chatgpt' }); + const sessionDeps = { + ...deps, + getPage: async () => page, + getTargetId: async () => targetId, + getCdpSession: async () => /** @type {any} */ (page).context?.().newCDPSession?.(page), + }; + const sessionArgs = { + ...args, + sessionId, + }; + return withMcpActiveCommand(name, provider, sessionDeps, sessionArgs, () => + pollByProvider(provider, sessionDeps, { + ...args, + vendor: session.vendor || provider, + session: session.sessionId, + timeout: args.timeout, + }), + ); + }), + ); +} + /** * @param {any} name * @param {any} provider diff --git a/web-ai/observation-bundle.mjs b/web-ai/observation-bundle.mjs index e8aa4426..5c45db4a 100644 --- a/web-ai/observation-bundle.mjs +++ b/web-ai/observation-bundle.mjs @@ -19,6 +19,8 @@ * @typedef {Object} ObservationBundleInput * @property {string} url * @property {string} [title] + * @property {string} [observationId] + * @property {string} [targetId] * @property {{width:number,height:number}} viewport * @property {number} [dpr] * @property {Array<{ref:string,role:string,name?:string,depth?:number,occurrenceIndex?:number}>} snapshotNodes @@ -32,6 +34,8 @@ /** * @typedef {Object} ObservationBundleV1 * @property {'observation-bundle-v1'} schemaVersion + * @property {string} observationId + * @property {string} targetId * @property {string} url * @property {string} title * @property {{width:number,height:number}} viewport @@ -40,6 +44,7 @@ * @property {Array<{ref:string,role:string,name:string,depth:number,occurrenceIndex?:number,box?:{x:number,y:number,width:number,height:number}}>} refs * @property {string|null} screenshot * @property {string} textSummary + * @property {{url:string,targetId:string,viewport:{width:number,height:number},dpr:number,capturedAt:string}} basis * @property {{refCount:number,boxCount:number,textChars:number,hasScreenshot:boolean}} stats */ @@ -79,7 +84,7 @@ export function buildObservationBundle(input) { const refs = []; for (const node of input.snapshotNodes) { if (!node || typeof node.ref !== 'string') continue; - if (node.ref === '...' || !node.ref.startsWith('@')) continue; + if (!isElementRef(node.ref)) continue; const row = { ref: node.ref, role: String(node.role || ''), @@ -99,10 +104,14 @@ export function buildObservationBundle(input) { const textSummary = clampText(input.textSummary || '', input.maxTextChars); const screenshot = input.screenshotPath || null; const capturedAt = input.capturedAt || new Date().toISOString(); + const observationId = input.observationId || `obs-${hashBasis(input.url, capturedAt, refs.length)}`; + const targetId = input.targetId || ''; let boxCount = 0; for (const r of refs) if (r.box) boxCount += 1; return { schemaVersion: SCHEMA_VERSION, + observationId, + targetId, url: input.url, title: String(input.title || ''), viewport: { width: input.viewport.width, height: input.viewport.height }, @@ -111,6 +120,13 @@ export function buildObservationBundle(input) { refs, screenshot, textSummary, + basis: { + url: input.url, + targetId, + viewport: { width: input.viewport.width, height: input.viewport.height }, + dpr, + capturedAt, + }, stats: { refCount: refs.length, boxCount, @@ -120,6 +136,28 @@ export function buildObservationBundle(input) { }; } +/** + * @param {string} ref + */ +function isElementRef(ref) { + return /^@?e\d+$/.test(ref); +} + +/** + * @param {string} url + * @param {string} capturedAt + * @param {number} refCount + */ +function hashBasis(url, capturedAt, refCount) { + let hash = 2166136261; + const input = `${url}|${capturedAt}|${refCount}`; + for (let i = 0; i < input.length; i++) { + hash ^= input.charCodeAt(i); + hash = Math.imul(hash, 16777619); + } + return (hash >>> 0).toString(16); +} + /** * @param {ObservationBundleV1} bundle * @returns {string} diff --git a/web-ai/policy/default-policy.mjs b/web-ai/policy/default-policy.mjs index a5f9f208..394efdef 100644 --- a/web-ai/policy/default-policy.mjs +++ b/web-ai/policy/default-policy.mjs @@ -7,7 +7,7 @@ export const DEFAULT_WEB_AI_POLICY = Object.freeze({ allowUploads: 'explicit-only', allowClipboardRead: false, allowClipboardWrite: 'explicit-only', - allowEvaluate: false, + allowEvaluate: true, allowFileAccess: false, allowCrossOriginNavigation: 'confirm', destructiveFormPolicy: 'deny', diff --git a/web-ai/session.mjs b/web-ai/session.mjs index a59fd4af..ec761d5c 100644 --- a/web-ai/session.mjs +++ b/web-ai/session.mjs @@ -10,6 +10,9 @@ import { patchSession, pruneSessions, } from './session-store.mjs'; +import { normalizeChatGptModelChoice } from './chatgpt-model.mjs'; +import { normalizeGrokModelChoice } from './grok-model.mjs'; +import { normalizeGeminiModelChoice, isGeminiDeepThinkChoice } from './gemini-model.mjs'; /** * @typedef {import('./session-store.mjs').WebAiSession} WebAiSession @@ -222,6 +225,53 @@ export function updateSession(sessionId, patch = {}) { return patchSession(sessionId, { ...patch, updatedAt: new Date().toISOString() }); } +/** + * Mark an incomplete session as timed out without downgrading completed work. + * + * @param {string} sessionId + * @param {Partial & { warnings?: unknown[], warning?: unknown, lastError?: unknown }} [patch] + * @returns {WebAiSession|null} + */ +export function markSessionTimeout(sessionId, patch = {}) { + const session = getSession(sessionId); + if (!session) return null; + const { warning, warnings: patchWarnings, ...sessionPatch } = patch; + const warnings = mergeWarnings(session.warnings || [], patchWarnings || [], warning); + const hasCompletedEvidence = session.status === 'complete' || + session.status === 'completed' || + Boolean(session.completedAt) || + Boolean(session.answer); + if (hasCompletedEvidence) { + return updateSession(sessionId, { + warnings: mergeWarnings(warnings, ['timeout-after-complete-ignored']), + status: session.status === 'completed' ? 'completed' : 'complete', + }); + } + return updateSession(sessionId, { + ...sessionPatch, + status: 'timeout', + warnings, + }); +} + +/** + * @param {unknown[]} base + * @param {unknown[]} extra + * @param {unknown} [single] + * @returns {unknown[]} + */ +function mergeWarnings(base, extra, single) { + const out = Array.isArray(base) ? [...base] : []; + for (const warning of [...(Array.isArray(extra) ? extra : []), single]) { + if (warning == null) continue; + const key = typeof warning === 'string' ? warning : JSON.stringify(warning); + if (!out.some((existing) => (typeof existing === 'string' ? existing : JSON.stringify(existing)) === key)) { + out.push(warning); + } + } + return out; +} + /** * @param {string|null|undefined} sessionId * @returns {WebAiSession|null} @@ -335,6 +385,73 @@ export function resolveDeadlineAt(input = {}, vendor = 'chatgpt') { return new Date(Date.now() + seconds * 1000).toISOString(); } +/** + * Hardcoded default poll timeout (seconds) per normalized model tier. + * Long-reasoning tiers (pro / deep-research) get an hour; shorter tiers scale down. + * An explicit --timeout / --deadline always overrides these defaults. + * @type {Readonly>} + */ +export const TIER_DEFAULT_TIMEOUT_SEC = Object.freeze({ + instant: 120, + thinking: 600, + pro: 3600, + 'deep-research': 3600, +}); + +/** Long-reasoning ceiling (seconds), exported for cross-module reuse (e.g. lease TTLs). */ +export const PRO_TIMEOUT_SEC = TIER_DEFAULT_TIMEOUT_SEC.pro; + +/** + * Resolve a tier name to a default timeout (seconds), falling back to the vendor + * default and finally 1200s when the tier is unknown. + * @param {string|null} tier + * @param {string} [vendor] + * @returns {number} + */ +export function tierDefaultTimeoutSec(tier, vendor = 'chatgpt') { + if (tier && TIER_DEFAULT_TIMEOUT_SEC[tier] != null) return TIER_DEFAULT_TIMEOUT_SEC[tier]; + return VENDOR_DEFAULT_TIMEOUT_SEC[vendor] || 1200; +} + +/** + * Map (vendor, model, research) to a normalized timeout tier, or null when unknown. + * Reuses the existing per-vendor model normalizers; deep-research is signalled by + * the separate `research` flag (chatgpt) or the deep-think alias (gemini). + * @param {string} vendor + * @param {unknown} model + * @param {unknown} [research] + * @returns {string|null} + */ +export function deriveTimeoutTier(vendor, model, research) { + if (vendor === 'gemini') { + if (isGeminiDeepThinkChoice(model)) return 'deep-research'; + const m = normalizeGeminiModelChoice(model); + if (m === 'flash-lite') return 'instant'; + if (m === 'flash' || m === 'pro') return 'thinking'; + return null; + } + if (vendor === 'grok') { + const m = normalizeGrokModelChoice(model); + if (m === 'heavy') return 'pro'; + if (m === 'fast') return 'instant'; + return m ? 'thinking' : null; + } + // chatgpt (default vendor) + if (String(research || '').trim().toLowerCase() === 'deep') return 'deep-research'; + return normalizeChatGptModelChoice(model); +} + +/** + * Tier-aware default poll timeout (seconds), applied when no explicit --timeout is given. + * @param {{ model?: unknown, research?: unknown }} [input] + * @param {string} [vendor] + * @returns {number} + */ +export function resolveTimeoutDefaultSec(input = {}, vendor = 'chatgpt') { + const tier = deriveTimeoutTier(vendor, input.model, input.research); + return tierDefaultTimeoutSec(tier, vendor); +} + /** * @param {WebAiEnvelope} [input] * @param {{ files?: unknown[], transport?: string } | null} [contextPack] diff --git a/web-ai/tab-lease-store.mjs b/web-ai/tab-lease-store.mjs index 43f9a756..ecbd0903 100644 --- a/web-ai/tab-lease-store.mjs +++ b/web-ai/tab-lease-store.mjs @@ -3,6 +3,7 @@ import { existsSync, mkdirSync, openSync, closeSync, readFileSync, renameSync, u import { dirname, join } from 'node:path'; import { homedir } from 'node:os'; import { closeTab, isTabAlive } from '../skills/browser/tab-manager.mjs'; +import { isPidAlive } from '../skills/browser/profile-lock.mjs'; import { activeCommandTargetIds } from './active-command-store.mjs'; /** @@ -26,6 +27,7 @@ import { activeCommandTargetIds } from './active-command-store.mjs'; * closePreviousState?: string, * cleanupReason?: string, * closeFailedAt?: string, + * ownerPid?: number|null, * }} Lease * * @typedef {{ @@ -51,6 +53,9 @@ import { activeCommandTargetIds } from './active-command-store.mjs'; * globalMax?: number, * completedSessions?: boolean, * now?: number, + * ownerPid?: number|null, + * activeMaxPerKey?: number, + * activeGlobalMax?: number, * }} LeaseInput * * @typedef {{ targetId?: string, vendor?: string, owner?: string, state?: string }} ListLeasesFilter @@ -64,6 +69,23 @@ const STALE_LOCK_MS = 30_000; const DEFAULT_POOL_TTL_MS = parseDuration(process.env.AGBROWSE_PROVIDER_POOL_TTL || '30m'); const DEFAULT_POOL_MAX_PER_KEY = parseInt(process.env.AGBROWSE_PROVIDER_POOL_MAX_PER_KEY || '3', 10); const DEFAULT_POOL_GLOBAL_MAX = parseInt(process.env.AGBROWSE_PROVIDER_POOL_GLOBAL_MAX || '8', 10); +const DEFAULT_ACTIVE_MAX_PER_KEY = parseInt(process.env.AGBROWSE_PROVIDER_ACTIVE_MAX_PER_KEY || '5', 10); +const DEFAULT_ACTIVE_GLOBAL_MAX = parseInt(process.env.AGBROWSE_PROVIDER_ACTIVE_GLOBAL_MAX || '14', 10); + +export class ProviderActiveCapacityError extends Error { + /** + * @param {{ reason: string, limit: number, current: number, leaseKey: string, vendor: string, browserProfileKey: string }} details + */ + constructor(details) { + super(`provider active tab capacity exceeded: ${details.reason} ${details.current}/${details.limit}`); + this.name = 'ProviderActiveCapacityError'; + this.errorCode = 'provider.active-capacity'; + this.stage = 'provider-capacity'; + this.retryHint = 'wait-or-retry-later'; + this.mutationAllowed = false; + this.evidence = details; + } +} function home() { return process.env.BROWSER_AGENT_HOME || join(homedir(), '.browser-agent'); @@ -228,7 +250,12 @@ export async function recordActiveLease(input = {}) { leasedAt: input.leasedAt || now, updatedAt: now, }); - store.leases = store.leases.filter(row => !sameTargetScope(row, lease) && !sameSessionScope(row, lease)); + const retained = store.leases.filter(row => !sameTargetScope(row, lease) && !sameSessionScope(row, lease)); + assertActiveCapacity(retained, lease, { + maxPerKey: input.activeMaxPerKey ?? DEFAULT_ACTIVE_MAX_PER_KEY, + globalMax: input.activeGlobalMax ?? DEFAULT_ACTIVE_GLOBAL_MAX, + }); + store.leases = retained; store.leases.push(lease); writeStore(store); return lease; @@ -353,6 +380,9 @@ export async function cleanupLeasedTabs(port, input = {}) { } store.leases = store.leases.filter(lease => lease.browserProfileKey !== browserProfileKey || !dead.includes(lease.targetId)); const closeableLeases = store.leases.filter(lease => lease.browserProfileKey === browserProfileKey && !activeTargets.has(lease.targetId)); + toClose.push(...closeableLeases + .filter(lease => lease.state === 'active-session' && Number.isFinite(lease.ownerPid) && !isPidAlive(Number(lease.ownerPid))) + .map(lease => ({ ...lease, cleanupReason: 'owner-pid-dead' }))); toClose.push(...selectOverflowAndExpired(closeableLeases, { nowMs, maxPerKey: input.maxPerKey ?? DEFAULT_POOL_MAX_PER_KEY, @@ -418,6 +448,7 @@ function normalizeLease(input = {}) { leaseDisposition: input.leaseDisposition || null, updatedAt: input.updatedAt || new Date().toISOString(), leaseKey: '', + ownerPid: input.ownerPid === null ? null : Number(input.ownerPid || process.pid), }; lease.leaseKey = input.leaseKey || buildLeaseKey(lease); return lease; @@ -454,6 +485,53 @@ function sameBrowserProfile(a, b) { return a?.owner === b?.owner && a?.vendor === b?.vendor && a?.sessionType === b?.sessionType && a?.browserProfileKey === b?.browserProfileKey; } +/** + * @param {Lease[]} retained + * @param {Lease} nextLease + * @param {{ maxPerKey: number, globalMax: number }} limits + */ +function assertActiveCapacity(retained, nextLease, limits) { + const active = retained.filter(lease => + lease.state === 'active-session' && + lease.owner === nextLease.owner && + lease.browserProfileKey === nextLease.browserProfileKey + ); + const maxPerKey = normalizeLimit(limits.maxPerKey); + if (maxPerKey >= 0) { + const perKeyCount = active.filter(lease => lease.leaseKey === nextLease.leaseKey).length; + if (perKeyCount >= maxPerKey) { + throw new ProviderActiveCapacityError({ + reason: 'active-max-per-key', + limit: maxPerKey, + current: perKeyCount, + leaseKey: nextLease.leaseKey, + vendor: nextLease.vendor, + browserProfileKey: nextLease.browserProfileKey, + }); + } + } + const globalMax = normalizeLimit(limits.globalMax); + if (globalMax >= 0 && active.length >= globalMax) { + throw new ProviderActiveCapacityError({ + reason: 'active-global-max', + limit: globalMax, + current: active.length, + leaseKey: nextLease.leaseKey, + vendor: nextLease.vendor, + browserProfileKey: nextLease.browserProfileKey, + }); + } +} + +/** + * @param {unknown} value + */ +function normalizeLimit(value) { + const parsed = Number(value); + if (!Number.isFinite(parsed)) return -1; + return Math.max(0, Math.floor(parsed)); +} + /** * @param {Lease[]} leases * @param {{ nowMs: number, maxPerKey: number, globalMax: number }} options diff --git a/web-ai/tab-recovery.mjs b/web-ai/tab-recovery.mjs index 0ad69f36..d8c05ea8 100644 --- a/web-ai/tab-recovery.mjs +++ b/web-ai/tab-recovery.mjs @@ -227,7 +227,7 @@ export function isPageDeathError(err) { * @param {string|null|undefined} storedUrl * @param {string|null|undefined} liveUrl */ -function urlsCompatible(storedUrl, liveUrl) { +export function urlsCompatible(storedUrl, liveUrl) { if (!storedUrl || !liveUrl) return false; if (storedUrl === liveUrl) return true; try { diff --git a/web-ai/tool-schema.mjs b/web-ai/tool-schema.mjs index 7ff51be2..0e5333fb 100644 --- a/web-ai/tool-schema.mjs +++ b/web-ai/tool-schema.mjs @@ -66,7 +66,7 @@ export const WEB_AI_TOOLS = { }, ['prompt']), }, web_ai_wait_response: { - description: 'Wait for provider response completion.', + description: 'Wait for a stored provider session response. Long runs may return a recoverable timeout; preserve sessionId and retry wait/resume or CLI poll.', inputSchema: objectSchema({ sessionId: { type: 'string' }, provider: providerSchema, @@ -93,7 +93,7 @@ export const WEB_AI_TOOLS = { }), }, web_ai_session_resume: { - description: 'Resume a stored session by ID.', + description: 'Resume a stored provider session by ID through session-bound recovery. Long runs may need repeated waits or CLI poll.', inputSchema: objectSchema({ sessionId: { type: 'string' }, provider: providerSchema, diff --git a/web-ai/watcher.mjs b/web-ai/watcher.mjs index f387f2e2..05d5346e 100644 --- a/web-ai/watcher.mjs +++ b/web-ai/watcher.mjs @@ -12,7 +12,7 @@ import { pollWebAi } from './chatgpt.mjs'; import { geminiPollWebAi } from './gemini-live.mjs'; import { grokPollWebAi } from './grok-live.mjs'; import { getSession, updateSession } from './session.mjs'; -import { withSessionPage } from './tab-recovery.mjs'; +import { withSessionPage, urlsCompatible } from './tab-recovery.mjs'; import { withSessionCommandLock } from './session-store.mjs'; import { WebAiError, wrapError } from './errors.mjs'; import { @@ -172,11 +172,15 @@ export async function watchSessionOnce(deps, input = {}) { }; } - // Phase 9.1: Use withSessionPage to resolve session's specific page, not active tab - return withSessionPage(deps, options.sessionId, async ({ page, targetId }) => { + // Phase 9.1: Use withSessionPage to resolve session's specific page, not active tab. + // resolveSessionPage (allowNavigate) already self-heals root->/c/ URL drift and + // returns the updated session; use that healed copy for the attach check instead + // of the stale outer `session`, which previously re-introduced a false + // reattach-mismatch (issue #77 watch-path). + return withSessionPage(deps, options.sessionId, async ({ page, targetId, session: resolvedSession }) => { const profileLockSummary = await readProfileLockSummary() .catch(err => ({ state: 'unknown', error: err?.message || String(err) })); - const reattach = await ensureWatcherAttached(page, session, options); + const reattach = await ensureWatcherAttached(page, resolvedSession || session, options); if (!reattach.ok) { return { ok: false, sessionId: session.sessionId, vendor, @@ -399,7 +403,11 @@ async function ensureWatcherAttached(page, session, options) { const targetUrl = session.conversationUrl || session.originalUrl; if (!targetUrl) return { ok: true, warnings: ['session-has-no-conversation-url'] }; const currentUrl = page.url?.() || ''; - if (urlsEquivalentForWatch(currentUrl, targetUrl)) return { ok: true, url: currentUrl, warnings: [] }; + // Use the canonical tolerant predicate (shared with resolveSessionPage) instead + // of a stricter hash-only compare: same-conversation root->/c/ drift and trailing + // slashes are compatible, while a genuinely different conversation or a + // non-provider landing still mismatches and (with --navigate) re-navigates. + if (urlsCompatible(targetUrl, currentUrl)) return { ok: true, url: currentUrl, warnings: [] }; if (options.navigate) { await page.goto(targetUrl, { waitUntil: 'domcontentloaded', timeout: options.navigateTimeoutMs }); return { ok: true, url: targetUrl, warnings: [`reattached:navigated-from=${currentUrl}`] }; @@ -552,22 +560,6 @@ function pidAlive(pid) { try { process.kill(pid, 0); return true; } catch (err) { return (/** @type {any} */ (err))?.code === 'EPERM'; } } -/** - * @param {any} a - * @param {any} b - */ -function urlsEquivalentForWatch(a, b) { - try { - const ua = new URL(a); - const ub = new URL(b); - ua.hash = ''; - ub.hash = ''; - return ua.toString() === ub.toString(); - } catch { - return String(a || '') === String(b || ''); - } -} - /** * @param {any} value */