Skip to content

Same-turn query embedded up to 3x separately (RL rerank, MCP tool discovery, tool schema filter) #6267

Description

@bug-ops

Finding

Within a single call to Agent::rebuild_system_prompt (crates/zeph-core/src/agent/context/assembly.rs), the identical query: &str string can be sent to self.embedding_provider.embed(query) (or an equivalently-configured provider, since discovery_provider/pruning_provider default to self.embedding_provider when unset) up to three separate times, each independently awaited and none reused:

  1. match_and_rank_skills — RL-head rerank branch: self.embedding_provider.embed(query) (line 1182), gated on services.skill.rl_head.is_some().
  2. discover_mcp_tools_for_turn — semantic tool discovery: embed_provider.embed(query) (line 1681), gated on discovery_strategy == Embedding and tools non-empty; embed_provider defaults to self.embedding_provider when discovery_provider is unset.
  3. filter_tool_schemas_for_turn — dynamic tool schema filter: self.embedding_provider.embed(query) (line 1794), gated on tool_schema_filter being configured.

All three embed the exact same query text (not the possibly-rewritten effective_query used for skill matching itself at line 1101). When RL rerank, MCP embedding-based discovery, and the tool schema filter are all enabled — a realistic default-ish configuration — this issues three separate embedding network/inference calls per turn for identical input, each with its own timeout and its own sequential .await, on the hot per-turn context-assembly path.

Line 1777-1778's doc comment already acknowledges one instance of this as "accepted as MVP duplication" for the schema-filter vs. MCP-discovery pair, but the RL-rerank embed (line 1182) is a third, previously undocumented instance of the same redundant computation.

Location

  • crates/zeph-core/src/agent/context/assembly.rs:1182 (RL rerank query embed)
  • crates/zeph-core/src/agent/context/assembly.rs:1681 (MCP semantic tool discovery query embed)
  • crates/zeph-core/src/agent/context/assembly.rs:1794 (tool schema filter query embed)

Before

Three independent, sequential embed(query) calls scattered across match_and_rank_skills, discover_mcp_tools_for_turn, and filter_tool_schemas_for_turn, each awaited in turn with its own timeout.

After

Compute the query embedding once per turn (e.g. cache it alongside cached_prompt_tokens-style per-turn state, or thread an Option<Vec<f32>> through rebuild_system_prompt's call chain) and reuse it across the RL rerank, MCP tool discovery, and tool schema filter steps whenever they resolve to the same provider. Fall back to a fresh per-consumer embed only when a consumer configures a distinct discovery_provider/pruning_provider.

Why

Same text, same (in the common unconfigured case) provider, computed three times sequentially on the per-turn hot path — wasted latency and, for remote embedding APIs, wasted cost. This is the same class of redundant-I/O issue as #6266 (duplicate load_skill_outcome_stats call), just for the embedding provider instead of SQLite.

Metadata

Metadata

Assignees

Labels

P2High value, medium complexityarchitectureArchitecture improvementsperformancePerformance improvements

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions