Develop Metrics to Evaluate Agent Path Efficiency in Agentic Hierarchical RAG #9
Description
Your evaluation framework is excellent for assessing the final answer, but it does not address the efficiency of the path the agent took to reach that answer. An agent might eventually produce the correct result but follow a long, inefficient, or costly sequence of actions.
This issue proposes the development of specific metrics to measure and optimize the agent's decision-making process itself:
- Define metrics such as number of retrieval steps, LLM calls, computation time, and resource usage per query.
- Consider metrics for unnecessary detours, redundant retrievals, or suboptimal strategy choices.
- Explore visualization and logging methods to trace the agent's route and support diagnostic analysis.
- Discuss how to incorporate these metrics into the evaluation framework for benchmarking and continuous improvement.
Further discussion is encouraged to refine the scope and prioritize actionable metrics.