Skip to content

feat: automate three steps the skills told the agent to do by hand - #984

Closed
latekvo wants to merge 2 commits into
mainfrom
feat/automate-mandated-agent-steps
Closed

latekvo wants to merge 2 commits into
mainfrom
feat/automate-mandated-agent-steps

Conversation

@latekvo

@latekvo latekvo commented Aug 31, 2026 •

Copy link
Copy Markdown
Member

Why

A fellow pointed out that we mandate a describe after every gesture, when the tool-server could just return it. That was one instance of a class, and #958 fixed exactly that instance. I audited every skill, rule file and tool description for the same shape - guidance that mandates a call whose inputs already live server-side - and found nine more. Three are fixed here; the rest are listed at the bottom with why they are not.

What

tv-remote takes the auto-screenshot

A TV is focus-driven and tv-remote is its only navigation primitive, but it was in neither auto-capture set. argent-tv-interact therefore prescribed describe -> tv-remote -> "describe again to confirm": two mandated calls per D-pad move, on the one platform that never got #958's benefit. shake was missing from both sets too, and joins them.

tv-remote deliberately does not get the element tree. I booted an Apple TV 4K (tvOS 18.5) to check, because that was the load-bearing unknown:

call result
describe ×3 tv-control request timed out after 10.1s, 11.2s, 39.8s
screenshot 0.6s, fine
await-screen-idle (1500ms cap) 1.50s, respects its cap, settled:false
tv-remote 0.19s, fine

classifyDevice keys off id shape, so an Apple TV UDID and an iPhone UDID are indistinguishable at the MCP layer - the tree cannot be appended for TVs without risking that wait on every press. The screenshot carries focus visually instead, and the skill now records the measurement.

launch-app / restart-app assert the Android Metro reverse

adb reverse tcp:8081 tcp:8081 was mandated in six skill files and automated in zero (grep -rn reverse packages/tool-server/src returned nothing). The mapping dies on a device reboot or an adb restart, and argent-metro-debugger documented it as the most common cause of a failed Metro connection - a footgun the code could remove. Both tools now assert it before starting the process (adb reverse is idempotent). Best-effort: a device that refuses still launches, since most launches are of apps that never talk to Metro.

ARGENT_METRO_PORT overrides the port; disable-metro-reverse opts out. The skills keep the manual command for the one path that still needs it - npx react-native run-android and Android Studio start the app outside argent.

react-profiler-analyze accepts a gesture's own timestampMs

The skill asked the agent to save startedAtEpochMs, compute offsetMs = timestampMs - startedAtEpochMs after every tap and swipe, and carry the list across the session. Both operands are already server-side: react-profiler-start stores profileStartWallMs with the profile. Annotations now take { timestampMs, label } and the tool does the subtraction. { offsetMs, label } still works; exactly one of the two per annotation. A timestamp with no stored anchor fails with REACT_PROFILER_ANALYZE_ANNOTATION_UNANCHORED rather than silently landing every action on the first commit.

One stale claim corrected

The skills said to ToolSearch every gesture tool before first use. gesture-tap, gesture-swipe, gesture-scroll and gesture-drag are alwaysLoad: true, so the mandated ceremony was already a no-op for four of the seven. Now names only the three that are actually deferred.

Verification

  • Mutation-tested every new assertion. Dropping tv-remote from the describe set, adding it back, adding a tool with no settle budget, removing the offsetMs subtraction, ignoring disable-metro-reverse, trusting ARGENT_METRO_PORT blindly, letting an adb failure escape, and moving the reverse to after am start each fail the suite. The tvOS numbers above are from a real booted simulator, not inference.
  • The adb reverse test redirects both flag scopes ($HOME and cwd) into a temp dir - it passes with a hostile disable-metro-reverse: true planted at the repo root, which is the exact condition that broke it while I was writing it.
  • npm run build, lint --max-warnings 0, knip (ceiling 0), format, typecheck:tests -w @argent/tool-server, npx docusaurus build.
  • Full suites: tool-server 4,848 across 6 shards, argent-mcp 89, configuration-core 122, registry 120. All green.

Docs

reference/configuration.mdx gains the disable-metro-reverse row; features/interacting-with-apps.mdx records that a tv-remote press returns the screenshot only, and why.

Audited but deliberately not changed

  • list-devices before everything (rules, both setup skills, argent-test-ui-flow, argent-lens) - udid is required on 25 tool schemas and optional on none. Worth noting for whoever picks this up: resolveFlowDevice in flows/flow-device.ts already implements the exact fallback (explicit id -> the single booted device of platform -> the single booted device overall -> throw, enumerating what is available), and flow-run exposes it as auto-detection today. So this is extending an existing mechanism to the interaction tools, not inventing one - but it changes the contract on 25 tools, so it wants its own PR.
  • flow-add-step cannot take a selector - the recorder reverse-derives one from the pre-tap tree in id -> text -> role order, and two of the three outcomes are wrong (one silently). The agent had the selector before it computed the frame centre. Touches recording semantics, so out of scope here.
  • "Tap the field before typing" - typing with no focused field is a documented silent no-op. fix(flow): refuse to type into a field that never took focus #726 is already fixing this for flow replay.
  • screenshot-diff baseline bookkeeping, Lens screenshot hand-off (propose_variant already describes the device at propose time), stop-all-simulator-servers default scope - all real, all better as separate changes.

I have not filed these as issues yet; say the word and I will, or I can take any of them next.

The tapping rule was one instance of a class: guidance that mandates a call
whose inputs already live server-side. #958 retired the pre-tap `describe`;
this does the same for three more.

- tv-remote takes the auto-screenshot. It is the only navigation primitive
  on a TV, and was in neither auto-capture set, so every D-pad press cost a
  separate look at the screen. It stays out of the auto-describe set: on a
  booted Apple TV 4K (tvOS 18.5) a home-screen `describe` returned
  `tv-control request timed out` after 10.1s, 11.2s and 39.8s where
  `screenshot` answered in 0.6s, and a UUID-shaped id cannot be told apart
  from an iPhone's at the MCP layer. `shake` joins both sets.

- launch-app / restart-app assert the Android Metro reverse before starting
  the process. Six skill files told the agent to run `adb reverse` by hand
  and nothing in the tool-server ever did; the mapping dies on a device
  reboot or an adb restart, and `argent-metro-debugger` documented that as
  the most common cause of a failed Metro connection. `ARGENT_METRO_PORT`
  overrides the port, `disable-metro-reverse` opts out.

- react-profiler-analyze accepts a gesture's own `timestampMs`. The skill
  asked the agent to save `startedAtEpochMs`, subtract it after every tap
  and swipe, and carry the offsets across the session; the anchor is
  already stored with the profile. `offsetMs` still works. An annotation
  that gives a timestamp with no stored anchor fails loudly rather than
  landing every action on the first commit.

Also corrects a stale claim: the skills told the agent to ToolSearch every
gesture tool before first use, but gesture-tap/swipe/scroll/drag are
alwaysLoad, so the ceremony was already a no-op for four of the seven.
@coderabbitai

This comment has been minimized.

@latekvo

latekvo commented Sep 28, 2026

Copy link
Copy Markdown
Member Author

Still needed as of 2026-09-28 (main 133b1dfcd). With the PR's tests applied, main fails 19 of them. tv-remote/shake are missing from the auto-capture set, launch-app/restart-app never run adb reverse, and react-profiler-analyze still only accepts a hand-computed offsetMs.

@latekvo

latekvo commented Oct 12, 2026

Copy link
Copy Markdown
Member Author

Closing: the tv-remote change goes the opposite way from open #1262, which gives tv-remote the element tree as well as the screenshot and conflicts with this PR in argent-tv-interact/SKILL.md. The adb reverse runs without --no-rebind, so it overwrites any existing device:8081 mapping on every Android launch. The profiler timestampMs, shake capture and ToolSearch skill changes are unrelated to each other and should each come back as a separate PR.

@latekvo latekvo closed this Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant