Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
a74a3d5
docs: plan computer-use contract hardening
lidge-jun Jun 17, 2026
6634e7e
docs: plan vision fallback upgrade
lidge-jun Jun 17, 2026
e7ec9b3
docs: tighten vision upgrade plan
lidge-jun Jun 17, 2026
3392adc
feat: harden vision coordinate fallback
lidge-jun Jun 17, 2026
e055739
docs: record vision upgrade verification
lidge-jun Jun 17, 2026
2b03e74
fix: wire vision candidate reconciliation
lidge-jun Jun 17, 2026
58e022e
docs: update vision upgrade verification
lidge-jun Jun 17, 2026
f7e8cd0
refactor: split vision candidate helpers
lidge-jun Jun 17, 2026
816a54a
docs: record vision helper split verification
lidge-jun Jun 17, 2026
6f3b5ea
fix: reconcile live vision bundles
lidge-jun Jun 17, 2026
df723c5
fix: allow browser evaluate by default
lidge-jun Jun 17, 2026
f35c3ea
devlog(webai): root-cause + plan for timeout/tab/skill-envelope/watch…
lidge-jun Jun 20, 2026
8a5731e
devlog(webai): close skill-envelope as already-capable, move to _fin
lidge-jun Jun 20, 2026
e88a520
docs(web-ai): clarify --system (trusted) vs --context (untrusted) vs …
lidge-jun Jun 20, 2026
2a423e3
devlog(webai): adversarial pressure-test of timeout/tab/watch — downs…
lidge-jun Jun 20, 2026
1d86985
feat(web-ai): tier-aware default poll timeout + watcher self-heal reuse
lidge-jun Jun 20, 2026
ba896c9
devlog(webai): close timeout + watch to _fin (implemented in 1d86985)
lidge-jun Jun 20, 2026
c90c626
refactor(web-ai): retire urlsEquivalentForWatch, reuse tolerant urlsC…
lidge-jun Jun 20, 2026
8a49ae4
devlog: move completed plans to fin
lidge-jun Jun 20, 2026
5b0a226
devlog: folderize active plans and capture oracle delta
lidge-jun Jun 20, 2026
53ed885
devlog: split oracle delta into PABCD followups
lidge-jun Jun 20, 2026
d647d58
fix(web-ai): recover MCP wait sessions after timeout
lidge-jun Jun 20, 2026
562a8ad
fix(web-ai): cap active provider tabs
lidge-jun Jun 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@ Tab limits:

| Setting | Default | Env var |
| --- | --- | --- |
| Max tabs | 10 | `AGBROWSE_MAX_TABS` |
| Max tabs | 20 | `AGBROWSE_MAX_TABS` |
| Idle timeout | 30 min | `AGBROWSE_TAB_IDLE` |

`send` and `query` run tab cleanup before opening another tab. Cleanup never
Expand All @@ -454,14 +454,20 @@ navigation, reload, tab switch, or any major page mutation.

Use `vision-click` only when a target is visible in a screenshot but has no
usable DOM/ref target, such as canvas/WebGL-heavy UIs.
The normal order is ref click first, coordinate click last. Vision results are
treated as bbox candidates with confidence; low-confidence or legacy point-only
results require verification instead of clicking directly.

```bash
agbrowse screenshot --json
agbrowse-vision-click "the visible Submit button"

agbrowse observe-bundle --screenshot --boxes --json > /tmp/bundle.json
agbrowse-vision-click "Submit button" --bundle /tmp/bundle.json --verify-before-click
```

The vision path handles device-pixel-ratio correction before sending
`page.mouse.click()` coordinates.
The vision path handles device-pixel-ratio correction and clip-origin evidence
before sending `page.mouse.click()` coordinates.

## Web AI

Expand Down
44 changes: 38 additions & 6 deletions devlog/00_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,53 @@ must be treated as historical do not edit them after release.evidence

| Folder | Purpose |
| --- | --- |
| `_fin/` | Completed non-MVP closeouts, research outcomes, and shipped implementation plans. |
| `_fin/mvp/` | Shipped MVP phases, grouped by topic (read-only history). |
| `_plan/` | Active or deferred plans not yet shipped. |
| `_legacy/` | Pre-rewrite changelogs / plans / research dumps. |
| `context/` | Verbatim Pro / Grok peer reviews and gap audits. |

## `_plan/` active research
## `_plan/` active or deferred work

| Topic | Folder | Status |
| --- | --- | --- |
| Post-MVP competitive gap closeout | `_plan/260506_post_mvp_gap_closeout/` | Deferred backlog; not fully shipped. |
| Oracle ZIP browser bundle proposal | `_plan/260513_oracle_zip_bundle_proposal/` | External upstream proposal draft; no local agbrowse implementation authority. |
| Adaptive Fetch v2 hardening | `_plan/260515_adaptive_fetch_v2_hardening/` | Follow-up planning after v2; not closed. |
| Provider expansion | `_plan/260519_provider_expansion/` | Deferred provider roadmap for Claude, Perplexity, NotebookLM. |
| Codebase audit backlog | `_plan/260603_codebase_audit/` | Historical audit + issue tracker; keep grouped until separately closed. |
| K-BrowseComp search gap analysis | `_plan/260608_kbrowsecomp_search_gap/` | Research/spec complete; implementation deferred. |
| web-ai GPT Code Mode (artifact zip 회수) | `_plan/260611_webai_gpt_code_mode/` | Implemented as ChatGPT-only beta: single zip, multi-zip, and JS-only artifact retrieval verified. |
| web-ai code mode GPT dev-agent context | `_plan/260611_code_mode_gpt_agent_context/` | Active: auto-upload saved GPT dev-agent context zip, require `PLAN.md`/`00_plan.md`, and mirror independent runtime into cli-jaw. |
| web-ai skill + cli-jaw mirror closeout | `_plan/260611_webai_skill_cli_jaw_mirror/` | Active: update agent-facing skill docs and mirror the simplified ChatGPT Intelligence picker into cli-jaw. |
| Background runtime hook research | `_plan/260611_background_runtime_hook/` | Research complete; findings reflected in `skills/web-ai/SKILL.md` (Long-Running section). Hook design relocated to cli-jaw `devlog/_plan/260611_bgtask_background_runtime/` for implementation planning. |
| ChatGPT composer tool selection live probe | `_plan/260615_chatgpt_composer_tools_live_probe.md` | Active: PR #78 live DOM/UX evidence for explicit-only tool/plugin/model gating. |
| Oracle stability gap analysis | `_plan/260608_oracle_stability_gap/` | Partial follow-up backlog; includes 2026-06-20 Oracle 0.15 delta and 31/32 PABCD follow-up plans. |
| Legacy MVP phase plans | `_plan/legacy_mvp_phase_plans/` | Pre-closeout phase 8.1/9 planning references; grouped for audit hygiene. |
| Strict migration | `_plan/strict-migration/` | Deferred migration planning and arbitration notes. |

Other grouped planning folders under `_plan/` remain until they receive a
separate closeout audit.

## Recent `_fin/` closeouts

| Topic | Folder | Closeout signal |
| --- | --- | --- |
| UX blocker fixes | `_fin/260507_ux-blockers-p0p1/` | README maps fixes to implemented commits `ccb7051`, `1a4743b`, and `f7b0e97`. |
| Oracle parity feature batch | `_fin/260508_oracle_parity/` | Implemented by `fe359a9` and follow-up commits. |
| web-ai session rebinding hardening | `_fin/260510_webai_session_rebind_diff_plan.md` | Implemented by `276aeac`. |
| Oracle follow-up guardrails | `_fin/260513_oracle_followup_guardrails_diff_plan.md` | Implemented by `085cc83`. |
| Adaptive Fetch v1 / Insane Search mirror | `_fin/260514_insane_search_adaptive_fetch/` | README status `implemented-v1`; shipped by `39708a3` and follow-ups. |
| Adaptive Fetch v2 | `_fin/260515_adaptive_fetch_v2/` | Index status `implemented`; hardening follow-up remains in `_plan/`. |
| Shared web-ai target lock | `_fin/260525_shared_web_ai_target_lock/` | Implemented by `602a700` and `e28f66e`. |
| Runway MCP parity expansion | `_fin/260528_runway_mcp_parity_expansion/` | Implemented by `7458f64` and Runway continuity follow-ups. |
| Defuddle reader candidate | `_fin/260610_defuddle_reader/` | Implementation result recorded; shipped by `631615d`. |
| Background runtime hook research | `_fin/260611_background_runtime_hook/` | Research complete; cli-jaw implementation planning relocated. |
| web-ai GPT Code Mode | `_fin/260611_webai_gpt_code_mode/` | ChatGPT-only beta implemented: single zip, multi-zip, JS-only retrieval. |
| Code Mode GPT dev-agent context | `_fin/260611_code_mode_gpt_agent_context/` | Implemented by `7ef4955`, `81af74f`, and `864ae41`. |
| web-ai multi/mixed attachments | `_fin/260611_webai_multi_attach/` | Implemented and live verified by `ef01881`. |
| web-ai skill + cli-jaw mirror | `_fin/260611_webai_skill_cli_jaw_mirror/` | Agent-facing docs and simplified picker mirror closed. |
| Docs Pages and code-mode overhaul | `_fin/260611_docs_pages_overhaul/` | Final goal audit proves local gates, push, and live Pages deployment. |
| ChatGPT composer tool selection probe | `_fin/260615_chatgpt_composer_tools_live_probe.md` | PR #78 evidence and follow-up patches applied. |
| Computer-use contract hardening / vision upgrade | `_fin/260617_computer_use_contract_hardening/` | `dev-vision-upgrade` verification report records implementation and live smoke evidence. |
| Web-AI stability and concurrency closeout | `_fin/260619_webai_stability/`, `_fin/260619_tab_parallel_stability/` | Timeout/watch/skill-envelope closed earlier; tab MVV closed by active lease cap + PID reaper + record-before-bind. |
| MCP wait response recovery | `_fin/260621_mcp_wait_response_recovery/` | GitHub #79 PABCD: session-bound MCP wait/resume recovery and monotonic timeout handling. |
| Tab stability MVV closeout | `_fin/260621_tab_stability_mvv_closeout/` | Final branch closeout plan for tab MVV, verification, push, and PR body `Closes #79`. |

## `_fin/mvp/` topics

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Out of scope for this first slice:
- Direct `--file` browser max-size cap.
- Nested current-UI attachment-chip hardening from Oracle PR #192.
- Oracle ZIP proposal. That is tracked separately in
`devlog/_plan/260513_oracle_zip_bundle_proposal.md`.
`devlog/_plan/260513_oracle_zip_bundle_proposal/00_proposal.md`.

## Part 1 — Easy Explanation

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ cookies unless explicitly requested.
npm test -- browser-fetch-command
bash structure/verify-counts.sh
bash structure/check-doc-drift.sh
git diff --check HEAD -- skills/browser README.md structure devlog/_plan/260514_insane_search_adaptive_fetch
git diff --check HEAD -- skills/browser README.md structure devlog/_fin/260514_insane_search_adaptive_fetch
```

## Phase 07 — Integrated Gates And Mirror Readiness
Expand All @@ -174,9 +174,9 @@ git diff --check HEAD -- skills/browser README.md structure devlog/_plan/260514_
MODIFY:

```text
devlog/_plan/260514_insane_search_adaptive_fetch/README.md
devlog/_plan/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md
devlog/_plan/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md
devlog/_fin/260514_insane_search_adaptive_fetch/README.md
devlog/_fin/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md
devlog/_fin/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md
```

Optional new closeout note:
Expand Down
6 changes: 3 additions & 3 deletions devlog/_fin/260515_adaptive_fetch_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,9 @@ Updated source-of-truth surfaces:
- `skills/browser/SKILL.md`
- `structure/commands.md`
- `structure/CAPABILITY_TRUTH_TABLE.md`
- `devlog/_plan/260514_insane_search_adaptive_fetch/README.md`
- `devlog/_plan/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md`
- `devlog/_plan/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md`
- `devlog/_fin/260514_insane_search_adaptive_fetch/README.md`
- `devlog/_fin/260514_insane_search_adaptive_fetch/13_cli_jaw_implementation_plan_baseline.md`
- `devlog/_fin/260514_insane_search_adaptive_fetch/15_skill_frontmatter_routing_baseline.md`

The documented product boundary is:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,10 @@ All research-only — no source code modifications.

| Path | Purpose |
|------|---------|
| `devlog/_plan/260611_background_runtime_hook/00_plan.md` | This plan (you're reading it) |
| `devlog/_plan/260611_background_runtime_hook/01_runtime_survey.md` | Task 1 결과: 런타임별 비교표 |
| `devlog/_plan/260611_background_runtime_hook/02_agbrowse_sufficiency.md` | Task 2 결과: agbrowse 현재 구현 평가 |
| `devlog/_plan/260611_background_runtime_hook/03_hook_design.md` | Task 3 결과: cli-jaw background hook 설계 초안 |
| `devlog/_fin/260611_background_runtime_hook/00_plan.md` | This plan (you're reading it) |
| `devlog/_fin/260611_background_runtime_hook/01_runtime_survey.md` | Task 1 결과: 런타임별 비교표 |
| `devlog/_fin/260611_background_runtime_hook/02_agbrowse_sufficiency.md` | Task 2 결과: agbrowse 현재 구현 평가 |
| `devlog/_fin/260611_background_runtime_hook/03_hook_design.md` | Task 3 결과: cli-jaw background hook 설계 초안 |

### MODIFY files

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ agbrowse/
│ ├── web-ai-code-mode-prompt.test.mjs
│ ├── web-ai-code-mode.test.mjs
│ └── web-ai-code-dev-context.test.mjs NEW
└── devlog/_plan/260611_code_mode_gpt_agent_context/
└── devlog/_fin/260611_code_mode_gpt_agent_context/
└── 00_plan.md

cli-jaw/
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Browser Computer Control Alignment

Date: 2026-06-17
Status: P-phase plan

## Objective

Align agbrowse browser control with the shared computer-use contract while preserving its existing ref-first design.

Shared master spec:

- `/Users/jun/Developer/codex/23_computer_use/devlog/_plan/260617_computer_use_contract_hardening/00_shared_contract_spec.md`

## Scope

Target repository:

- `/Users/jun/Developer/new/700_projects/agbrowse`

Primary surfaces:

- `agbrowse snapshot --interactive`
- `agbrowse observe-bundle --screenshot --boxes --json`
- `agbrowse observe-actions <instruction> --json`
- `agbrowse-vision-click`
- `agbrowse mouse-click`

## Current State

agbrowse already has the right top-level preference order:

1. accessibility snapshot refs
2. locator/ref actions
3. observe-bundle boxes
4. screenshot/coordinate fallback
5. vision-click for no-ref targets

The hardening work should keep that order and strengthen the fallback path.

## Contract Mapping

### Observation

Existing equivalents:

- `snapshot --interactive` -> refs
- `observe-bundle` -> refs, boxes, optional screenshot, viewport, DPR
- `screenshot --json` -> screenshot path, viewport, DPR, optional clip

Required additions:

- stable observation/capture id inside vision-click flow
- explicit target id/url freshness check for coordinate fallback
- screenshot clip metadata preserved through candidate resolution

### Candidate

Existing equivalents:

- snapshot ref
- observe-action candidate
- vision-click coordinate result

Required additions:

- `vision_bbox` candidate with bbox, point, confidence, and reason
- candidate can reconcile to a nearby ref when possible
- low confidence becomes a failure, not a click

### Action

Existing actions:

- `click <ref>`
- `mouse-click <x> <y>`

Required policy:

- prefer ref click
- use coordinate click only when no usable ref exists
- require vision bbox verification before coordinate click

### Verification

Existing verification:

- optional verify crop in vision-click
- post-click snapshot best effort

Required additions:

- pre-click viewport/clip bounds validation
- bbox center must remain inside verified crop
- optional `elementFromPoint` or nearby-ref reconciliation before click
- post-click snapshot or caller-provided assertion for risky actions

## Non-Goals

- Do not replace Playwright locator/ref actions.
- Do not make vision-click the default browser action path.
- Do not add browser-level desktop permissions.
- Do not modify JWC or jawcode.

## Open Design Defaults

Adopt these defaults unless audit finds a blocker:

- bbox confidence threshold: 0.75 for normal targets
- require verify crop for coordinate fallback in dense UI
- reject ambiguous target descriptions instead of clicking
- keep existing `--clip` and `--region` flags, but preserve their coordinate basis explicitly
Loading
Loading