Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 58 additions & 16 deletions docs/adding-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,13 +274,20 @@ pnpm eval your-tool

## Agent-in-Tool Pattern

Some tools (`search_events` and `search_issues`) embed AI agents to handle complex natural language translation. This pattern is used when:
Some tools (`search_events`, `search_issue_events`, and `search_issues`) embed
AI agents to normalize search parameters before the handler calls Sentry. Treat
the agent as a repair step for a structured request, not only as a natural
language query translator. The agent may rewrite the query string, but it may
also correct or fill related parameters such as dataset, fields, sort, and time
range when the provided combination would fail or produce the wrong result.

### When to Use This Pattern

1. **Complex query translation** - Converting natural language to domain-specific query languages
2. **Dynamic field discovery** - When available fields vary by project/context
3. **Semantic understanding** - When the tool needs to understand intent, not just parameters
1. **Parameter repair** - Fixing mismatched or incomplete search parameters
2. **Query normalization** - Converting natural language or loose syntax to
valid Sentry search syntax
3. **Dynamic field discovery** - When available fields vary by project/context
4. **Semantic understanding** - When the tool needs to understand intent across multiple parameters

### When NOT to Use This Pattern

Expand All @@ -293,21 +300,53 @@ Some tools (`search_events` and `search_issues`) embed AI agents to handle compl
```typescript
// Tool handler delegates to embedded agent
async handler(params, context) {
// 1. Embedded agent translates natural language
const translated = await translateQuery(params.naturalLanguageQuery, ...);
const request = hasAgentProvider()
? await repairSearchParams({
query: params.query,
dataset: params.dataset,
fields: params.fields,
sort: params.sort,
statsPeriod: params.statsPeriod,
})
: {
query: params.query,
dataset: params.dataset,
fields: params.fields,
sort: params.sort,
statsPeriod: params.statsPeriod,
};

// 2. Tool executes the translated query
const results = await apiService.searchEvents(translated.query, ...);
// Tool executes either the repaired request or the direct parameters.
const results = await apiService.searchEvents({
query: request.query,
dataset: request.dataset,
fields: request.fields,
sort: request.sort,
statsPeriod: request.statsPeriod,
});

// 3. Format and return results
return formatResults(results);
}
```

### Provider Availability

Direct-capable tools should still work when no embedded agent provider is
available. Use `hasAgentProvider()` to decide whether to run the repair step.
If it returns false because API keys are missing, both OpenAI and Anthropic keys
are set without an explicit provider, or Azure OpenAI is missing a supported
base URL, execute the direct parameters as provided.

Do not silently fall back after a provider has been selected and the provider API
call fails. Invalid keys, deactivated accounts, rate limits, and other provider
4xx responses should become user-facing `LLMProviderError`s from
`callEmbeddedAgent()`. That makes configuration/account problems visible instead
of hiding them behind an un-repaired direct search.

### Error Handling Philosophy

**DO NOT retry internally**. When the embedded agent fails:
1. Throw a clear `UserInputError` with specific guidance
1. Throw a clear `UserInputError` or `LLMProviderError` with specific guidance
2. Let the calling agent (Claude/Cursor) see the error
3. The calling agent can retry with corrections if needed

Expand All @@ -323,21 +362,24 @@ if (previousError) {
// GOOD: Static prompt with clear error boundaries
const systemPrompt = STATIC_SYSTEM_PROMPT;
try {
return await translateQuery(...);
return await repairSearchParams(...);
} catch (error) {
throw new UserInputError(`Could not translate query: ${error.message}`);
throw new UserInputError(
`Could not repair search parameters: ${error.message}`,
);
}
```

### Tool Boundaries

1. **Embedded Agent Responsibilities**:
- Translate natural language to structured queries
- Repair or normalize structured search parameters
- Convert natural language to Sentry search syntax when needed
- Discover available fields/attributes
- Validate query syntax
- Validate query syntax and parameter combinations

2. **Tool Handler Responsibilities**:
- Execute the translated query
- Execute the repaired request
- Handle API errors
- Format results for the calling agent

Expand All @@ -350,7 +392,7 @@ try {

1. **Create an AGENTS.md file** in the tool directory documenting:
- The embedded agent's prompt and behavior
- Common translation patterns
- Common repair and normalization patterns
- Known limitations

2. **Keep agent prompts focused** - Don't duplicate general MCP knowledge
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ Some tools (`search_events` and `search_issues`) implement a two-tier agent patt
```
1. User: "Show me errors from yesterday"
↓
2. Claude: Calls search_events(naturalLanguageQuery="errors from yesterday")
2. Claude: Calls search_events(query="errors from yesterday")
↓
3. MCP Tool Handler: Receives request
↓
Expand Down
12 changes: 6 additions & 6 deletions docs/specs/search-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ A unified search tool that accepts natural language queries and translates them
```typescript
interface SearchEventsParams {
organizationSlug: string; // Required
naturalLanguageQuery: string; // Natural language search description
query: string; // Natural language search description
dataset?: "spans" | "errors" | "logs" | "metrics"; // Dataset to search (default: "errors")
projectSlug?: string; // Optional - limit to specific project
regionUrl?: string;
Expand All @@ -30,28 +30,28 @@ interface SearchEventsParams {
// Find errors (errors dataset is default)
search_events({
organizationSlug: "my-org",
naturalLanguageQuery: "database timeouts in checkout flow from last hour"
query: "database timeouts in checkout flow from last hour"
})

// Find slow transactions
search_events({
organizationSlug: "my-org",
naturalLanguageQuery: "API calls taking over 5 seconds",
query: "API calls taking over 5 seconds",
projectSlug: "backend",
dataset: "spans"
})

// Find logs
search_events({
organizationSlug: "my-org",
naturalLanguageQuery: "warning logs about memory usage",
query: "warning logs about memory usage",
dataset: "logs"
})

// Find request duration metrics
search_events({
organizationSlug: "my-org",
naturalLanguageQuery: "p95 request duration by transaction this week",
query: "p95 request duration by transaction this week",
dataset: "metrics"
})
```
Expand Down Expand Up @@ -125,7 +125,7 @@ find_errors({
// After
search_events({
organizationSlug: "sentry",
naturalLanguageQuery: "unresolved errors in checkout.js"
query: "unresolved errors in checkout.js"
})
```

Expand Down
2 changes: 1 addition & 1 deletion docs/testing-remote.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ Opens at `http://localhost:6274`
// Test search_events with AI
{
"organizationSlug": "your-org",
"naturalLanguageQuery": "errors in the last hour",
"query": "errors in the last hour",
"dataset": "errors"
}
```
Expand Down
8 changes: 4 additions & 4 deletions docs/testing-stdio.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ This opens the MCP Inspector at `http://localhost:6274`
1. **List Tools** - Verify expected tools appear
2. **Call a tool** - Start with `whoami` (no parameters required)
3. **Test with parameters** - Try `find_organizations()`
4. **Test complex operations** - Try `search_events(naturalLanguageQuery="errors in the last hour")`
4. **Test complex operations** - Try `search_events(query="errors in the last hour")`

**Example test sequence:**
```
Expand All @@ -207,7 +207,7 @@ This opens the MCP Inspector at `http://localhost:6274`
3. find_projects(organizationSlug="your-org")
4. search_events(
organizationSlug="your-org",
naturalLanguageQuery="errors from yesterday"
query="errors from yesterday"
)
```

Expand Down Expand Up @@ -485,8 +485,8 @@ OPENAI_API_KEY=your-key pnpm start --access-token=TOKEN

# Test search_events and search_issues work
# In MCP Inspector:
# - Call search_events(naturalLanguageQuery="errors in production")
# - Call search_issues(naturalLanguageQuery="unresolved crashes")
# - Call search_events(query="errors in production")
# - Call search_issues(query="unresolved crashes")
```

### 5. Test Agent Mode
Expand Down
2 changes: 1 addition & 1 deletion packages/mcp-core/src/internal/formatting.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1929,7 +1929,7 @@ export function formatIssueOutput({
output += `- You can reference the IssueID in commit messages (e.g. \`Fixes ${issue.shortId}\`) to automatically close the issue when the commit is merged.\n`;
output +=
"- The stacktrace includes both first-party application code as well as third-party code, its important to triage to first-party code.\n";
output += `- To search for specific occurrences or filter events within this issue, use \`search_issue_events(organizationSlug='${organizationSlug}', issueId='${issue.shortId}', naturalLanguageQuery='your query')\`\n`;
output += `- To search for specific occurrences or filter events within this issue, use \`search_issue_events(organizationSlug='${organizationSlug}', issueId='${issue.shortId}', query='your query')\`\n`;
if (experimentalMode) {
output += `- To see the trail of events leading up to this error, use \`get_sentry_resource(url='${apiService.getIssueUrl(organizationSlug, issue.shortId)}', resourceType='breadcrumbs')\`\n`;
}
Expand Down
5 changes: 0 additions & 5 deletions packages/mcp-core/src/server.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,6 @@ vi.mock("@sentry/core", () => ({
wrapMcpServerWithSentry: vi.fn((server) => server),
}));

// Mock the agent provider factory
vi.mock("./internal/agents/provider-factory", () => ({
hasAgentProvider: vi.fn(() => false),
}));

/**
* Helper to get registered tool names from an McpServer.
* Uses the internal _registeredTools object which exists directly on McpServer instances.
Expand Down
21 changes: 1 addition & 20 deletions packages/mcp-core/src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,7 @@ import type {
ServerRequest,
ServerNotification,
} from "@modelcontextprotocol/sdk/types.js";
import tools, {
AGENT_DEPENDENT_TOOLS,
SIMPLE_REPLACEMENT_TOOLS,
} from "./tools/index";
import tools from "./tools/index";
import {
type ToolConfig,
resolveDescription,
Expand All @@ -51,7 +48,6 @@ import {
getConstraintParametersToInject,
getConstraintKeysToFilter,
} from "./internal/constraint-helpers";
import { hasAgentProvider } from "./internal/agents/provider-factory";

/**
* Creates and configures a complete MCP server with Sentry instrumentation.
Expand Down Expand Up @@ -167,21 +163,6 @@ function configureServer({
? { use_sentry: tools.use_sentry }
: (customTools ?? tools);

// Filter tools based on agent provider availability
// Skip filtering in agent mode (use_sentry handles all tools internally) or when custom tools are provided
if (!agentMode && !customTools) {
const hasAgent = hasAgentProvider();
const toolsToExclude = new Set<string>(
hasAgent ? SIMPLE_REPLACEMENT_TOOLS : AGENT_DEPENDENT_TOOLS,
);

toolsToRegister = Object.fromEntries(
Object.entries(toolsToRegister).filter(
([key]) => !toolsToExclude.has(key),
),
) as typeof toolsToRegister;
}

// Filter tools based on public visibility and experimental mode
// (applies to all tools, including custom)
// Skip in agent mode (use_sentry handles filtering internally)
Expand Down
Loading
Loading