Skip to content

Point outside agents to ask_app when tool-search matches an action they can't call - #9445

Open
3mdistal wants to merge 15 commits into
mainfrom
alice/mcp-discoverability
Open

3mdistal wants to merge 15 commits into
mainfrom
alice/mcp-discoverability

Conversation

@3mdistal

Copy link
Copy Markdown
Contributor

Why

Each action in an app now carries its own decision about outside agents (Claude, ChatGPT, Cursor and other MCP clients). mcpTool: true lists it as a tool, "search" serves it through tool-search and call-tool (#8921), and false keeps it in the app. An outside agent reaches a kept action only by asking the app's own agent through ask_app.

tool-search searched only what the connection serves. An agent looking for something the app keeps for its own agent found nothing, and told the person the app can't do it, when one ask_app call would have done it.

Separately, externalAgents.writes: "ask_app_only" still keeps every write in the app at once. Now that each action carries its own decision (#8888), that blanket policy overrides the writes an app chose to serve.

What changes

  • When a search's query, names or task match actions this connection can't call, MCP tool-search adds a notServed field. It holds up to three of them, each with its name and a one-line description, and a message telling the caller to describe the task to ask_app. Results the connection can call come back as before:

    "notServed": {
      "message": "These match your search, but this connection can't call them. The app's own agent can: describe the task to ask_app. When a tool in results does the job, call it instead.",
      "tools": [{ "name": "send-report-now", "description": "Email a saved report to its subscribers now." }]
    }
  • Only a signed-in person whose connection can call ask_app gets it. Service principals (automated callers limited to a list of actions), callers whose grant leaves out ask_app, read-only tokens and apps that deny ask_app see no change. Neither does a menu search (no query, names or task) or a readOnlyOnly one, since ask_app isn't read-only.

  • "Can't call" is decided when the search runs, against the tools this connection can actually call, so an action the app serves but whose input MCP refuses (a raw sql string, say) is named too.

  • It never names actions the operator denies, framework tools every app carries (such as navigate), actions the app's own agent can't run (agentTool: false), actions that end the in-app turn (endsTurn), UI-only actions, names the framework never serves over MCP (db-*, provider-api-* and the like), or anything the caller's OAuth scope or the app's discovery checks hide from this caller. (A discovery check, agentDiscoveryAvailable, is an action's own test of whether a given caller should see it, such as whether a provider it needs is connected.)

  • Those discovery checks now run for actions kept in the app, under an MCP caller, which they never did before. So the lookup is advisory: if a check throws or hasn't answered by the 1.3-second deadline task searches already use (counted from the start of the search), the search returns its results without notServed. Each check now starts inside its promise, so one that throws can't leave another's rejection unhandled.

  • writes: "ask_app_only", set or implied by authenticatedReads: "auto", logs a deprecation warning once at startup. It still works. The warning says to decide each write on its action instead: mcpTool: false to keep it in the app.

  • The access-control and external-agents docs, the external-agents skill (synced to the template copies) and a changeset describe both.

Risk

notServed tells a caller an action exists without letting it call the action, so the danger is naming something the caller shouldn't learn about. It goes only to a signed-in person who can already ask the app's own agent to run that action, applies the caller's OAuth scope, and leaves out what that agent can't run or what the app hides from this caller. It returns names and one-line descriptions, never parameters or schemas.

The other new cost is running apps' discovery checks for actions they keep in the app. Only targeted searches run them, each check runs once per search, and a failing or slow check costs the search its notServed field, never its results.

The warning changes no behavior. An app that relies on ask_app_only keeps it until it moves its writes to per-action decisions.

Verification

  • build-server.not-served.spec.ts: 26 tests through an in-memory MCP client. They cover what gets named: closest matches, exact names, a task's text, at most three, served writes the app sends through ask_app, and an action whose input MCP refuses. They cover what doesn't: served tools, denied, agentTool: false, UI-only and turn-ending actions, actions a discovery check hides, framework tools, db-* names, a denied ask_app, a grant without ask_app, a read-only token, a service principal, a readOnlyOnly search, and a read action hidden from a write-only token through an app's own tool-search. And they cover the lookup itself: skipped for a menu search, given the caller's identity, and dropped (keeping the results) when a check throws, when one check rejects and a later one throws, and when a check never answers. The file is named after build-server.ts because the mutation check runs only the specs named after a changed file.
  • tool-search.spec.ts: 8 tests of closestToolMatches (ranking and limit, several queries plus a task, exact names in any case, what it skips, one-line descriptions) and namesUnservedTools.
  • mcp-options.spec.ts: 3 tests of the warning: set explicitly, implied by authenticatedReads: "auto", and quiet for allowlisted writes or no policy.
  • At this head, on a Linux machine: pnpm typecheck, oxfmt and oxlint on the changed files, the three specs and pnpm guards pass.
  • Two commits earlier (before a one-line change that starts each discovery check inside its promise, and two added assertions): core's src/mcp, src/agent, src/server/agent-chat and action.spec.ts pass (364 files, 9,514 tests), and pnpm mutation:changed scores 87.8% against its 50% threshold. The two added assertions cover the surviving mutants that changed behavior; the rest leave output unchanged, such as fields of a search candidate that never reach the response.

🤖 Generated with Claude Code

@builder-io-integration builder-io-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builder reviewed your changes and found 2 potential issues 🟡

Review Details

Code Review Summary

This PR adds MCP tool-search hints for app actions the connection cannot call directly, while directing eligible signed-in users to ask_app. It also deprecates the blanket ask_app_only writes policy and updates the core access-control documentation, translations, template skills, and tests. The approach keeps suggestions separate from callable search results and adds targeted discovery checks; the added test coverage exercises OAuth scopes, denials, grants, and discovery failures. I classified this as high risk because it changes authorization-adjacent MCP discovery behavior.

Key Findings

  • 🟡 notServed candidates are not filtered by the caller’s service-principal action grant, which can expose names/descriptions outside the grant.
  • 🟡 The writes-policy deprecation warning recommends action flags that do not serve writes while ask_app_only remains active.

🧪 Browser testing: Skipped — PR only modifies backend/config/docs/tests, no browser-facing UI impact.

filterActionsAvailableForDiscovery(
Object.fromEntries(
Object.entries(mergedActions).filter(
([name, entry]) =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Apply the service-principal grant to unserved matches

callable is grant-scoped, but this candidate filter uses all of mergedActions without checking the service-principal grant. A user-assurance principal with a restricted grant that includes tool-search and ask_app can therefore see names and descriptions for actions outside its grant; apply the same isActionGranted filtering used for served actions here.

Fix in Builder

console.warn(
`[agent-native] ${setting} sends every write through ask_app, which is deprecated. ` +
`Decide each write on its action instead: \`mcpTool: false\` keeps it in the app, where outside ` +
`agents reach it through ask_app, and \`mcpTool: true\` or \`"search"\` serves it to them directly.`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Clarify the required write-policy migration

With writes: "ask_app_only" (or its implied automatic-read default), setting mcpTool: true or "search" still does not serve a write; the policy filters it out. The warning’s suggested action-level change therefore has no effect unless the app also moves to writes: "allowlisted"—please make that prerequisite explicit.

Fix in Builder

@builder-io-integration builder-io-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builder reviewed your changes and has a few items to flag 🟡

Review Details

Incremental Code Review Summary

I reviewed the current PR head and rechecked the two open comments against the updated implementation. Both issues remain unchanged: notServedActions still does not apply service-principal action grants, and the ask_app_only deprecation warning still implies action flags alone can serve writes. I left those comments open and did not repost them. The update does not introduce another confirmed issue in the reviewed MCP discovery, authorization, matching, or policy paths. The implementation remains focused on advisory search results and preserves the direct-call surface; however, the open authorization and migration-guidance findings still need attention. Risk remains high because the change affects authorization-adjacent MCP behavior.

🧪 Browser testing: Skipped — PR only modifies backend/config/docs/tests, no browser-facing UI impact.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants