|
| 1 | +--- |
| 2 | +name: run-e2e-tests |
| 3 | +description: Run and debug kagent end-to-end tests. Use when the user asks to run e2e tests, debug e2e test failures, or understand the e2e test setup. |
| 4 | +allowed-tools: Bash, Read, Grep, Glob |
| 5 | +argument-hint: "[test-name-pattern]" |
| 6 | +--- |
| 7 | + |
| 8 | +# Running Kagent E2E Tests |
| 9 | + |
| 10 | +## Prerequisites |
| 11 | + |
| 12 | +1. A running Kind cluster with kagent deployed: |
| 13 | + ```bash |
| 14 | + make create-kind-cluster |
| 15 | + make helm-install |
| 16 | + ``` |
| 17 | + |
| 18 | +2. Test agent images built and pushed to the local registry (for BYO agent tests): |
| 19 | + - `localhost:5001/basic-openai:latest` |
| 20 | + - `localhost:5001/poem-flow:latest` |
| 21 | + - `kind-registry:5000/kebab-maker:latest` (for skill tests) |
| 22 | + |
| 23 | +3. The `kagent-tool-server` RemoteMCPServer must exist in the `kagent` namespace. |
| 24 | + |
| 25 | +4. The `kebab-agent` must be deployed in the `kagent` namespace (used by `TestE2EInvokeExternalAgent`). |
| 26 | + |
| 27 | +## Environment Variables |
| 28 | + |
| 29 | +Set `KAGENT_URL` using the controller's load balancer IP (no port-forward needed): |
| 30 | +```bash |
| 31 | +export KAGENT_URL="http://$(kubectl get svc -n kagent kagent-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'):8083" |
| 32 | +``` |
| 33 | + |
| 34 | +| Variable | Default | Description | |
| 35 | +|----------|---------|-------------| |
| 36 | +| `KAGENT_URL` | `http://localhost:8083` | Base URL of the kagent controller. Use the load balancer command above rather than port-forwarding. | |
| 37 | +| `KAGENT_LOCAL_HOST` | Auto-detected (`host.docker.internal` on macOS, `172.17.0.1` on Linux) | Host IP accessible from inside Kind pods (for mock LLM server) | |
| 38 | +| `SKIP_CLEANUP` | (unset) | If set and test fails, skip deleting test resources for debugging | |
| 39 | + |
| 40 | +## Running Tests |
| 41 | + |
| 42 | +If the user provides a test name pattern via `$ARGUMENTS`, run that specific test. Otherwise run all e2e tests. |
| 43 | + |
| 44 | +All commands must be run from the `go/` directory. |
| 45 | + |
| 46 | +Set `KAGENT_URL` before every test command: |
| 47 | +```bash |
| 48 | +export KAGENT_URL="http://$(kubectl get svc -n kagent kagent-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'):8083" |
| 49 | +``` |
| 50 | + |
| 51 | +Run all e2e tests: |
| 52 | +```bash |
| 53 | +KAGENT_URL="$KAGENT_URL" go test -v -count=1 ./test/e2e/ -failfast |
| 54 | +``` |
| 55 | + |
| 56 | +Run a single test (using `$ARGUMENTS` as the pattern): |
| 57 | +```bash |
| 58 | +KAGENT_URL="$KAGENT_URL" go test -v -count=1 -run "$ARGUMENTS" ./test/e2e/ |
| 59 | +``` |
| 60 | + |
| 61 | +Run with shuffle (to detect ordering dependencies): |
| 62 | +```bash |
| 63 | +KAGENT_URL="$KAGENT_URL" go test -v -count=1 ./test/e2e/ -failfast -shuffle=on |
| 64 | +``` |
| 65 | + |
| 66 | +## Test Structure |
| 67 | + |
| 68 | +Tests live in `go/test/e2e/invoke_api_test.go`. Each test: |
| 69 | +1. Starts a mock LLM server (via `mockllm`) that listens on the host |
| 70 | +2. Creates K8s resources (ModelConfig, Agent, optionally MCPServer) |
| 71 | +3. Waits for `condition=Ready` on the Agent CR |
| 72 | +4. Polls the A2A endpoint via `waitForEndpoint()` until it returns non-5xx |
| 73 | +5. Sends A2A messages (sync and/or streaming) and checks responses |
| 74 | +6. Cleans up resources (or logs debug info on failure) |
| 75 | + |
| 76 | +The mock LLM server runs on the test host machine and must be reachable from inside Kind pods. The `KAGENT_LOCAL_HOST` variable controls what IP is used — auto-detected as `172.17.0.1` on Linux and `host.docker.internal` on macOS. |
| 77 | + |
| 78 | +## Debugging Failures |
| 79 | + |
| 80 | +When a test fails, the cleanup function automatically logs: |
| 81 | +- Agent CR YAML |
| 82 | +- Pod logs for all pods matching the agent |
| 83 | +- Pod describe output |
| 84 | +- Deployment describe output |
| 85 | +- Service describe output |
| 86 | + |
| 87 | +To manually investigate after a failed test (run with `SKIP_CLEANUP=1` to preserve resources): |
| 88 | + |
| 89 | +```bash |
| 90 | +SKIP_CLEANUP=1 KAGENT_URL="$KAGENT_URL" go test -v -count=1 -run TestName ./test/e2e/ |
| 91 | +``` |
| 92 | + |
| 93 | +Then inspect: |
| 94 | +```bash |
| 95 | +kubectl get agents -n kagent |
| 96 | +kubectl describe agent <agent-name> -n kagent |
| 97 | +kubectl get pods -n kagent -l app.kubernetes.io/managed-by=kagent |
| 98 | +kubectl logs -n kagent -l app.kubernetes.io/name=<agent-name> |
| 99 | +kubectl get events -n kagent --sort-by=.lastTimestamp |
| 100 | +kubectl logs -n kagent deployments/kagent-controller |
| 101 | +kubectl get endpoints <agent-name> -n kagent |
| 102 | +``` |
| 103 | + |
| 104 | +## Common Failure Modes |
| 105 | + |
| 106 | +- **Empty streaming response / "does not contain"**: The stream connected but the agent returned 0 events (not fully warmed up). The retry logic handles this — if it still fails after retries, check if the agent pod is crash-looping or if the mock LLM server is unreachable from inside the cluster. |
| 107 | +- **"connection refused" to agent pod**: The agent deployment/service hasn't propagated yet. `waitForEndpoint()` should handle this. If persistent, check `kubectl get endpoints <agent-name> -n kagent`. |
| 108 | +- **"serviceaccount not found"**: The controller hasn't finished creating all child resources for the agent. This is a timing issue handled by `waitForEndpoint()`. |
| 109 | +- **"All connection attempts failed" (MCP server)**: The MCPServer's pod isn't ready or its service isn't resolvable from the agent pod. Check `kubectl get mcpservers -n kagent` and `kubectl describe mcpserver <name> -n kagent`. |
| 110 | +- **Mock LLM server unreachable from pods**: The `KAGENT_LOCAL_HOST` is wrong for your environment. On Linux it defaults to `172.17.0.1` (the docker bridge IP). Set it explicitly if your setup differs. |
| 111 | +- **`TestE2EInvokeExternalAgent` fails**: The `kebab-agent` must be pre-deployed. This agent is not created by the test — it must exist beforehand. |
| 112 | +- **`TestE2EInvokeOpenAIAgent` or `TestE2EInvokeCrewAIAgent` fails with image pull error**: The BYO agent images must be built and pushed to the local registry (`localhost:5001`) before running these tests. |
0 commit comments