-
Notifications
You must be signed in to change notification settings - Fork 5.6k
Feature/qa automation agent #6840
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Ashraf-Khabar
wants to merge
2
commits into
aden-hive:main
Choose a base branch
from
Ashraf-Khabar:feature/qa-automation-agent
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| """QA Engineer Agent package.""" | ||
|
|
||
| # On importe les variables depuis agent.py et config.py pour que Hive les trouve | ||
| from .agent import default_agent, edges, goal, nodes | ||
| from .config import metadata | ||
|
|
||
| # On indique à Python ce qui est exposé publiquement | ||
| __all__ = [ | ||
| "default_agent", | ||
| "edges", | ||
| "goal", | ||
| "metadata", | ||
| "nodes", | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| """Entry point for testing the QA Engineer Agent directly.""" | ||
|
|
||
| import asyncio | ||
| import logging | ||
| from .agent import default_agent | ||
|
|
||
| # Configure logging to see what the agent is doing | ||
| logging.basicConfig(level=logging.INFO) | ||
|
|
||
| async def main(): | ||
| print("=== Running QA Engineer Agent ===") | ||
|
|
||
| # Test message simulating a user request | ||
| test_context = { | ||
| "user_request": "Please run the automated tests in the 'test_project/' directory using 'pytest' and tell me if there are any bugs." | ||
| } | ||
|
|
||
| # Run the agent | ||
| result = await default_agent.run(test_context) | ||
|
|
||
| print("\n=== Results of execution ===") | ||
| if result.success: | ||
| print("Success!") | ||
| else: | ||
| print(f"Error: {result.error}") | ||
|
|
||
| if __name__ == "__main__": | ||
| asyncio.run(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,185 @@ | ||
| """Agent graph construction for Advanced QA Engineer Agent.""" | ||
|
|
||
| from pathlib import Path | ||
|
|
||
| from framework.graph import EdgeSpec, EdgeCondition, Goal, SuccessCriterion, Constraint | ||
| from framework.graph.edge import GraphSpec | ||
| from framework.graph.executor import ExecutionResult | ||
| from framework.graph.checkpoint_config import CheckpointConfig | ||
| from framework.llm import LiteLLMProvider | ||
| from framework.runner.tool_registry import ToolRegistry | ||
| from framework.runtime.agent_runtime import AgentRuntime, create_agent_runtime | ||
| from framework.runtime.execution_stream import EntryPointSpec | ||
|
|
||
| from .config import default_config | ||
| from .nodes import planning_node, execution_node, ui_testing_node, reporting_node | ||
|
|
||
| # Define the overarching goal of the QA Agent | ||
| goal = Goal( | ||
| id="comprehensive-qa-testing", | ||
| name="Comprehensive QA Testing", | ||
| description=( | ||
| "Execute full-project QA testing including automated script execution " | ||
| "and exploratory browser testing, culminating in a detailed bug report." | ||
| ), | ||
| success_criteria=[ | ||
| SuccessCriterion( | ||
| id="test-execution", | ||
| description="Successfully execute automated tests and parse logs.", | ||
| metric="tests_run", | ||
| target=">0", | ||
| weight=0.4, | ||
| ), | ||
| SuccessCriterion( | ||
| id="report-generation", | ||
| description="Generate a detailed report with passing/failing tests.", | ||
| metric="report_created", | ||
| target="true", | ||
| weight=0.6, | ||
| ), | ||
| ], | ||
| constraints=[ | ||
| Constraint( | ||
| id="safe-execution", | ||
| description="Only execute tests inside the designated workspace sandbox.", | ||
| constraint_type="security", | ||
| category="safety", | ||
| ), | ||
| ], | ||
| ) | ||
|
|
||
| # Register nodes (Subagents must still be registered in the graph) | ||
| nodes = [ | ||
| planning_node, | ||
| execution_node, | ||
| ui_testing_node, | ||
| reporting_node, | ||
| ] | ||
|
|
||
| # Define the workflow (Edges) - Linear flow, UI testing happens INSIDE test_execution | ||
| edges = [ | ||
| EdgeSpec( | ||
| id="plan-to-exec", | ||
| source="planning", | ||
| target="test_execution", | ||
| condition=EdgeCondition.ON_SUCCESS, | ||
| priority=1, | ||
| ), | ||
| EdgeSpec( | ||
| id="exec-to-report", | ||
| source="test_execution", | ||
| target="reporting", | ||
| condition=EdgeCondition.ON_SUCCESS, | ||
| priority=1, | ||
| ), | ||
| EdgeSpec( | ||
| id="report-to-plan", | ||
| source="reporting", | ||
| target="planning", | ||
| condition=EdgeCondition.CONDITIONAL, | ||
| condition_expr="needs_more_testing == True", | ||
| priority=1, | ||
| ), | ||
| ] | ||
|
|
||
| class QaEngineerAgent: | ||
| """ | ||
| Advanced QA Engineer Agent. | ||
| Flow: Planning -> Test Execution (w/ UI Subagent) -> Reporting -> Planning (optional loop) | ||
| """ | ||
|
|
||
| def __init__(self, config=None): | ||
| self.config = config or default_config | ||
| self.goal = goal | ||
| self.nodes = nodes | ||
| self.edges = edges | ||
| self.entry_node = "planning" | ||
| self.entry_points = {"start": "planning"} | ||
| self.terminal_nodes = ["reporting"] | ||
| self._graph: GraphSpec | None = None | ||
| self._agent_runtime: AgentRuntime | None = None | ||
| self._tool_registry: ToolRegistry | None = None | ||
| self._storage_path: Path | None = None | ||
|
|
||
| def _build_graph(self) -> GraphSpec: | ||
| return GraphSpec( | ||
| id="qa-engineer-agent-graph", | ||
| goal_id=self.goal.id, | ||
| version="1.0.0", | ||
| entry_node=self.entry_node, | ||
| entry_points=self.entry_points, | ||
| terminal_nodes=self.terminal_nodes, | ||
| pause_nodes=[], | ||
| nodes=self.nodes, | ||
| edges=self.edges, | ||
| default_model=self.config.model, | ||
| max_tokens=self.config.max_tokens, | ||
| loop_config={ | ||
| "max_iterations": 50, | ||
| "max_tool_calls_per_turn": 30, | ||
| }, | ||
| ) | ||
|
|
||
| def _setup(self, mock_mode: bool = False) -> None: | ||
| self._storage_path = Path.home() / ".hive" / "agents" / "qa_engineer_agent" | ||
| self._storage_path.mkdir(parents=True, exist_ok=True) | ||
|
|
||
| self._tool_registry = ToolRegistry() | ||
|
|
||
| # Load external tools like browser and execute_command via MCP | ||
| mcp_config_path = Path(__file__).parent / "mcp_servers.json" | ||
| if mcp_config_path.exists(): | ||
| self._tool_registry.load_mcp_config(mcp_config_path) | ||
|
|
||
| llm = None | ||
| if not mock_mode: | ||
| llm = LiteLLMProvider( | ||
| model=self.config.model, | ||
| api_key=self.config.api_key, | ||
| api_base=self.config.api_base, | ||
| ) | ||
|
|
||
| self._graph = self._build_graph() | ||
|
|
||
| checkpoint_config = CheckpointConfig(enabled=True) | ||
|
|
||
| entry_point_specs = [ | ||
| EntryPointSpec( | ||
| id="default", | ||
| name="Default", | ||
| entry_node=self.entry_node, | ||
| trigger_type="manual", | ||
| ) | ||
| ] | ||
|
|
||
| self._agent_runtime = create_agent_runtime( | ||
| graph=self._graph, | ||
| goal=self.goal, | ||
| storage_path=self._storage_path, | ||
| entry_points=entry_point_specs, | ||
| llm=llm, | ||
| tools=list(self._tool_registry.get_tools().values()), | ||
| tool_executor=self._tool_registry.get_executor(), | ||
| checkpoint_config=checkpoint_config, | ||
| ) | ||
|
|
||
| async def start(self, mock_mode=False) -> None: | ||
| if self._agent_runtime is None: | ||
| self._setup(mock_mode=mock_mode) | ||
| if not self._agent_runtime.is_running: | ||
| await self._agent_runtime.start() | ||
|
|
||
| async def stop(self) -> None: | ||
| if self._agent_runtime and self._agent_runtime.is_running: | ||
| await self._agent_runtime.stop() | ||
|
|
||
| async def run(self, context: dict, mock_mode=False, session_state=None) -> ExecutionResult: | ||
| await self.start(mock_mode=mock_mode) | ||
| try: | ||
| result = await self._agent_runtime.trigger_and_wait("default", context, session_state=session_state) | ||
| return result or ExecutionResult(success=False, error="Execution timeout") | ||
| finally: | ||
| await self.stop() | ||
|
|
||
| # Default instance to be exported in __init__.py | ||
| default_agent = QaEngineerAgent() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| """Runtime configuration for the QA Engineer Agent.""" | ||
|
|
||
| from dataclasses import dataclass | ||
| from framework.config import RuntimeConfig | ||
|
|
||
| default_config = RuntimeConfig() | ||
|
|
||
| @dataclass | ||
| class AgentMetadata: | ||
| name: str = "QA Engineer Agent" | ||
| version: str = "1.0.0" | ||
| description: str = ( | ||
| "Advanced Quality Assurance agent capable of analyzing requirements, " | ||
| "executing automated test suites (Robot Framework, Java, Playwright, etc.), " | ||
| "performing exploratory UI testing via browser tools, and reporting bugs." | ||
| ) | ||
| intro_message: str = ( | ||
| "Hello! I am your QA Engineer Agent. Please provide the repository path, " | ||
| "the test commands to run (e.g., 'robot tests/' or 'mvn test'), or a URL " | ||
| "for exploratory UI testing. I will execute the tests, analyze the logs, " | ||
| "and provide a detailed bug report." | ||
| ) | ||
|
|
||
| metadata = AgentMetadata() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| { | ||
| "original_draft": { | ||
| "agent_name": "qa_engineer_agent", | ||
| "goal": "Execute full-project QA testing including automated script execution and exploratory browser testing.", | ||
| "nodes": [ | ||
| { | ||
| "id": "planning", | ||
| "name": "Test Planning", | ||
| "node_type": "event_loop", | ||
| "flowchart_type": "start", | ||
| "flowchart_shape": "stadium", | ||
| "flowchart_color": "#8aad3f" | ||
| }, | ||
| { | ||
| "id": "test_execution", | ||
| "name": "Test Execution", | ||
| "node_type": "event_loop", | ||
| "sub_agents": ["ui_testing"], | ||
| "flowchart_type": "process", | ||
| "flowchart_shape": "rectangle", | ||
| "flowchart_color": "#508878" | ||
| }, | ||
| { | ||
| "id": "ui_testing", | ||
| "name": "UI/Browser Subagent", | ||
| "node_type": "gcu", | ||
| "flowchart_type": "browser", | ||
| "flowchart_shape": "hexagon", | ||
| "flowchart_color": "#cc8850" | ||
| }, | ||
| { | ||
| "id": "reporting", | ||
| "name": "QA Report", | ||
| "node_type": "event_loop", | ||
| "flowchart_type": "terminal", | ||
| "flowchart_shape": "stadium", | ||
| "flowchart_color": "#b5453a" | ||
| } | ||
| ], | ||
| "edges": [ | ||
| { "source": "planning", "target": "test_execution", "condition": "on_success" }, | ||
| { "source": "test_execution", "target": "reporting", "condition": "on_success" }, | ||
| { "source": "reporting", "target": "planning", "condition": "conditional" } | ||
| ], | ||
| "entry_node": "planning", | ||
| "terminal_nodes": ["reporting"] | ||
| }, | ||
| "flowchart_map": { | ||
| "planning": ["planning"], | ||
| "test_execution": ["test_execution"], | ||
| "ui_testing": ["ui_testing"], | ||
| "reporting": ["reporting"] | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| { | ||
| "mcpServers": { | ||
| "filesystem": { | ||
| "command": "python", | ||
| "args": ["-m", "aden_tools", "serve", "filesystem"] | ||
| }, | ||
| "bash_executor": { | ||
| "command": "python", | ||
| "args": ["-m", "aden_tools", "serve", "bash"] | ||
| }, | ||
| "browser": { | ||
| "command": "python", | ||
| "args": ["-m", "gcu", "serve", "browser"] | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| """Node definitions for the QA Engineer Agent.""" | ||
|
|
||
| from framework.graph.node import NodeSpec | ||
|
|
||
| # Node 1: Plan the testing strategy | ||
| planning_node = NodeSpec( | ||
| id="planning", | ||
| name="Test Planning", | ||
| description="Analyze the user request, identify test suites, and plan the execution strategy.", | ||
| node_type="event_loop", | ||
| system_prompt="Analyze the user request. Identify what needs to be tested and formulate a clear test plan.", | ||
| client_facing=True, | ||
| ) | ||
|
|
||
| # Node 2: Exploratory UI Testing (GCU Subagent) | ||
| # Notice this is NOT connected via edges, it is a subagent. | ||
| ui_testing_node = NodeSpec( | ||
| id="ui_testing", | ||
| name="Exploratory UI Testing", | ||
| description="Use browser tools to navigate the application and assert UI correctness.", | ||
| node_type="gcu", | ||
| system_prompt="Perform exploratory visual testing. Use browser tools to navigate to the target URLs and verify the UI state. Report findings back to the parent.", | ||
| client_facing=False, | ||
| ) | ||
|
|
||
| # Node 3: Execution Coordinator (Runs CLI tests and delegates UI tests) | ||
| execution_node = NodeSpec( | ||
| id="test_execution", | ||
| name="Test Execution", | ||
| description="Execute CLI test scripts and delegate UI testing to the browser subagent.", | ||
| node_type="event_loop", | ||
| system_prompt=( | ||
| "Execute the test plan. Use the execute_command_tool to run scripts (e.g., pytest, robot). " | ||
| "If UI testing is required, use the delegate_to_sub_agent tool to call the 'ui_testing' agent." | ||
| ), | ||
| client_facing=False, | ||
| sub_agents=["ui_testing"], # CRITICAL FIX: Declaring the GCU node as a subagent | ||
| ) | ||
|
|
||
| # Node 4: Compile results and create a report | ||
| reporting_node = NodeSpec( | ||
| id="reporting", | ||
| name="QA Report Generation", | ||
| description="Compile logs, test results, and browser snapshots into a final QA report.", | ||
| node_type="event_loop", | ||
| system_prompt="Review all test outputs and visual findings. Create a comprehensive QA report summarizing bugs, passed tests, and recommendations.", | ||
| client_facing=True, | ||
| ) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: aden-hive/hive
Length of output: 10702
Fix incorrect GCU MCP server invocation in browser configuration.
The browser server uses an incorrect invocation pattern. The correct pattern, as documented in
core/framework/graph/gcu.pyandcore/framework/agents/queen/reference/gcu_guide.md, is:Replace
["-m", "gcu", "serve", "browser"]with the canonical invocation above to ensure proper MCP server communication.🤖 Prompt for AI Agents