Cross-links: API.md, DATA_MODEL.md, GENERATORS.md, AI_AGENT.md, ARCHITECTURE.md, DEPLOYMENT.md.
⚠️ PARTIAL. Bundled providers built:playwright,api-http,postgres,graphql,mysql,mongo,kubernetes, andgrpc, plus lifecycle-localsuitest.whitebox.v1adapters for pytest and Vitest/Jest. M14 adds the desktop set:slint-mcpandtauri-mcpare built and bundled in-process;computer-use-mcpandelectron-mcpare configs only — the binaries they name do not exist in this repo (see DESKTOP_TESTING.md). Documented-but-unbuilt:browser-use/appium/jira/github(later milestones). See ROADMAP.md.
MCP (Model Context Protocol) is the Anthropic-originated open standard for connecting LLMs and runtimes to typed external tools. It is multi-provider, multi-transport, and language-agnostic. In Suitest, MCP is not a browser-automation shim — it is the first-class plugin layer for all testing.
Suitest starts MCP only when the workspace has a validated active LLM. The client
holds only SUITEST_API_URL and SUITEST_API_KEY; model credentials stay encrypted
on the server and completions use /api/v1/llm/complete. MCP sampling is unsupported.
Key implications:
- Every step that touches an external system goes through an MCP server. Browser clicks, API requests, DB queries, K8s assertions, gRPC calls — all flow through a typed tool.
- MCP servers are first-class citizens. They appear in the UI (Integrations → MCP Servers), in the DB (
mcp_providerstable), in the API (/mcp/providers/*), and in every step row (step.mcp_provider). - Each MCP server exposes typed tools discoverable at runtime via the standard
tools/listMCP method. Suitest persists the discovered catalog so generators, the agent, and the UI tool-browser can introspect what is available. - Per-step
mcp_providerfield — a single test case can mix browser, API, DB, gRPC, K8s, and infra steps. This is the defining capability that makes Suitest fundamentally beyond TestSprite, which is locked to browser-and-API surfaces. - MCP is provider-agnostic. Suitest uses the official
mcpPython SDK so any MCP server (Anthropic-built, community, or user-authored) works without code changes.
Tagline (from the memo): "MCP-native testing platform." Universal MCP plugins are the signature differentiator from every competitor in the matrix.
The MCP layer lives in its own Python package and is consumed by both the agent (packages/agent) and the deterministic runner (packages/runner). See ARCHITECTURE.md for the larger picture.
packages/mcp/
├── pyproject.toml
└── suitest_mcp/
├── __init__.py
├── registry.py # provider catalog, health, discovery cache
├── client.py # session manager, transport (stdio/sse/ws)
├── routing.py # default routing table per target_kind
├── invoker.py # invoke tool with retry, telemetry, artifacts
├── pool.py # connection pool, lifecycle, idle TTL eviction
├── providers/ # bundled provider configs (built into image)
│ ├── playwright.py
│ ├── browser_use.py
│ ├── api_http.py # generic HTTP MCP (built-in, in-process)
│ ├── postgres.py
│ ├── mysql.py
│ ├── mongo.py
│ ├── graphql.py
│ ├── grpc.py
│ ├── kubernetes.py
│ ├── appium.py
│ └── computer_use.py
└── plugins/ # user-registered providers
# (loaded from mcp_providers table at startup)
| Module | Responsibility |
|---|---|
registry.py |
Loads built-in + DB-backed providers. Exposes get_provider(name), list_providers(), discover_tools(provider). Caches discovered tool catalogs. |
client.py |
Wraps mcp Python SDK. Owns transports: stdio (subprocess), SSE (httpx), WebSocket (websockets). Handles handshake + initialize. |
routing.py |
Resolves (target_kind, workspace) → provider. Consults workspace overrides first, then global default table. |
invoker.py |
Single entry point invoke(provider, tool, args, ctx). Retry policy, timeout, span emission, artifact capture, Redis pub/sub streaming. |
pool.py |
Per-provider connection pool. Lazy spawn, reuse for N requests, idle TTL, kill. Enforces per-workspace concurrency cap. |
providers/*.py |
Declarative config for built-in MCP servers (command, args, env, default tool catalog hint, auth shape). |
plugins/*.py |
Hot-loaded at startup from mcp_providers table. User-registered MCPs become first-class here. |
┌─────────────────────────────────────────────────────────────┐
│ invoker.invoke(provider="postgres-mcp", tool="query", ...) │
└──────────────┬──────────────────────────────────────────────┘
│
┌───────▼────────┐
│ pool.acquire() │
└───────┬────────┘
│
existing idle session?
┌─────┴─────┐
yes no
│ │
│ ┌──────▼───────┐
│ │ spawn process │ (transport-dependent)
│ │ + handshake │
│ └──────┬───────┘
│ │
└─────┬──────┘
│
┌───────▼────────┐
│ session.call() │
└───────┬────────┘
│
┌───────▼────────┐
│ pool.release() │
└───────┬────────┘
│
idle_ttl exceeded?
┌─────┴─────┐
yes no
│ │
┌─────▼─────┐ └─→ keep alive
│ terminate │
└───────────┘
Defaults: lazy spawn, reuse up to max_sessions=4, idle TTL 60s, hard spawn_timeout=10s.
| Transport | When used | Mechanism |
|---|---|---|
| stdio | Most providers (CLI-based MCP servers). Default. | Subprocess pipes; JSON-RPC framed over stdin/stdout. |
| SSE | Long-running hosted MCP web servers. | HTTP + Server-Sent Events; httpx client. |
| WebSocket | Interactive / bidirectional (computer-use, custom). | websockets client, full duplex. |
Transport is per-provider config; the same provider can declare a fallback transport.
These providers ship inside the main Suitest image (see DEPLOYMENT.md § air-gapped). They appear pre-registered in every workspace.
| Name | Kind | Transport | Target | Use case | Auth |
|---|---|---|---|---|---|
playwright-mcp |
browser | stdio (npx @playwright/mcp) |
FE_WEB | E2E web tests, browser recorder backend | none |
browser-use-mcp |
browser | stdio (python -m browser_use.mcp) |
FE_WEB | AI-driven web exploration & semantic crawling | none (LLM uses workspace LLMConfig) |
api-http-mcp |
http | in-process | BE_REST | Generic HTTP test calls (REST, OpenAPI) | per-test (header / OAuth) |
postgres-mcp |
db | stdio | DATA | DB invariants, seed, query, transactional probes | connection string |
mysql-mcp |
db | stdio | DATA | MySQL invariants & seeding | connection string |
mongo-mcp |
db | stdio | DATA | Mongo invariants & seeding | connection string |
graphql-mcp |
api | stdio | BE_GRAPHQL | GraphQL queries / mutations / subscriptions | bearer |
grpc-mcp |
api | stdio | BE_GRPC | gRPC unary + streaming tests | per-test (mTLS / metadata) |
kubernetes-mcp |
infra | stdio | INFRA | K8s resource assertions, port-forward, log tail | kubeconfig (mounted) |
appium-mcp |
mobile | stdio | FE_MOBILE | iOS / Android UI tests | appium server URL |
computer-use-mcp |
desktop | stdio (Anthropic computer-use) | FE_DESKTOP (v1.x preview, v2 stable) | Desktop app tests — screen-level default | OS-level (VNC / X11 / Wayland) |
electron-mcp |
desktop | stdio | FE_DESKTOP | Electron app tests via CDP DOM (Playwright _electron) |
none |
slint-mcp |
desktop | bundled, in-process | FE_DESKTOP | Slint app tests via accessible tree (headless-capable) | none |
tauri-mcp |
desktop | bundled, in-process | FE_DESKTOP | Tauri 2 app tests over W3C WebDriver served inside the app; screenshots + MP4 recording | ffmpeg (video only) |
jirac-mcp |
issue-tracker | stdio | EXTERNAL_TOOL | Jira issue tracker (file defect, transition, comment, JQL search) | cloud-token / DC PAT |
github-mcp |
issue-tracker | stdio | EXTERNAL_TOOL | GitHub Issues + labels + comments | github-app-installation-token / PAT |
api-http-mcpalso asserts on binary responses: a response whoseContent-Typeis not textual carriesbody_base64in the envelope, andhttp.assert_pdf_textextracts the PDF's text (all pages, or one 1-basedpage) to assertcontainssubstrings / amatchesregex. Whitespace is normalized before thecontainscheck — PDF text extraction breaks lines where the layout does. Scanned/image-only PDFs have no text layer; assert those visually instead.
api-http-mcpruns in-process (same Python process as the runner). It is implemented as an MCP server with stdio transport but launched via an internal channel for zero subprocess overhead. All other built-ins spawn out-of-process for isolation.
jirac-mcpbundled frommulhamna/jira-commands@jira-mcp-v2.0.1;github-mcpbundled fromgithub/github-mcp-server@v1.1.2. SeeDEPLOYMENT.md §15for binary bundling Dockerfile pattern.
Each target_kind resolves to a primary MCP plus an optional fallback. The runner consults this when step.mcp_provider is left blank.
target_kind |
Primary | Fallback |
|---|---|---|
BE_REST |
api-http-mcp |
none |
BE_GRAPHQL |
graphql-mcp |
api-http-mcp (raw POST) |
BE_GRPC |
grpc-mcp |
none |
FE_WEB |
playwright-mcp |
browser-use-mcp |
FE_MOBILE |
appium-mcp |
none |
FE_DESKTOP |
computer-use-mcp |
electron-mcp / slint-mcp / tauri-mcp (per step) |
DATA |
postgres-mcp |
user must override per workspace |
INFRA |
kubernetes-mcp |
none |
CUSTOM |
user must set per step or workspace | none |
A workspace can override the default routing table. Stored as JSONB in the workspaces.mcp_routing_overrides column (see DATA_MODEL.md).
{
"FE_WEB": {
"primary": "browser-use-mcp",
"fallback": "playwright-mcp"
},
"DATA": {
"primary": "mysql-mcp",
"fallback": null
},
"BE_REST": {
"primary": "api-http-mcp",
"fallback": "my-team-rest-mcp"
}
}Empty / missing keys fall back to the global table above. Validated on save: each named provider must exist and be enabled=true.
Users register their own MCP servers without code changes.
Integrations → MCP Servers → "Add Custom MCP" opens a modal with:
| Field | Type | Notes |
|---|---|---|
name |
text | URL-safe slug, unique per workspace |
kind |
text | Free-form label (e.g. payments, ml-eval, kafka) — informational |
endpoint |
text | URL (for SSE/WS) or shell command (for stdio) |
transport |
enum | stdio / sse / ws |
config_json |
JSON | Free-form provider config (env vars, args, headers) |
secrets_json |
JSON | Encrypted at rest, write-only field |
is_default_for_targets |
list | Optional list of target_kind values to auto-route to this provider |
Mirrored over REST — see API.md § MCP for full schemas.
GET /mcp/providers
POST /mcp/providers
GET /mcp/providers/:id
PATCH /mcp/providers/:id
DELETE /mcp/providers/:id
POST /mcp/providers/:id/health # manual health re-check
POST /mcp/providers/:id/discover # re-run tools/list
POST /mcp/providers/:id/invoke # dev-mode only, role-gated
When a custom MCP is registered or edited, Suitest:
- Spawns / connects with the supplied transport.
- Performs the MCP
initializehandshake. - Invokes the standard
tools/listmethod. - Persists the discovered tool catalog in
mcp_providers.config_json.tools(name + JSON schema per tool). - Marks
health_status = okand records the version string returned by the server.
If any step fails, the registration is rejected with a structured error. Secrets are never written to logs.
When the runner (deterministic step) or the agent (LangGraph node) needs to invoke a tool, the flow is:
1. Resolve provider P:
- if step.mcp_provider is set → use it
- else → routing.resolve(workspace, step.target_kind) → primary
2. Acquire session S for P from pool (spawn if needed, see § 2.3).
3. If P.secrets is set:
decrypt AES-GCM → inject into session env / handshake context
4. Build call:
name = step.tool (or tool inferred from step.code AST)
args = step.arguments (resolved against runtime context)
5. invoker.invoke(P, S, name, args, ctx):
emit OTel span, start timer
S.call_tool(name=name, arguments=args)
→ stream stdout/stderr → Redis pub/sub channel run:{run_id}:log
→ run:* channel fanned out to WebSocket subscribers
6. Capture artifacts:
screenshots, HAR, logs → MinIO under runs/{run_id}/steps/{step_id}/
7. Return typed result (matches tool's JSON schema).
8. Telemetry:
span attrs: provider, tool, duration_ms, outcome, error_code?
metrics: mcp_invocations_total{provider,tool,outcome}
audit row: actor, provider, tool, arg_hash, result_summary
9. Pool: mark session idle (or evict if exceeded ttl).
Pseudocode (Python):
# packages/mcp/suitest_mcp/invoker.py
async def invoke(provider_name, tool, args, ctx) -> ToolResult:
provider = registry.get(provider_name)
if provider.health_status == "down":
provider = routing.fallback_for(provider_name)
if provider is None:
raise McpProviderUnavailable(provider_name)
async with pool.acquire(provider) as session:
secrets = await secrets_store.decrypt(provider.secrets_json_encrypted)
await session.ensure_context(secrets)
with tracer.start_span("mcp.invoke") as span:
span.set_attributes({
"mcp.provider": provider.name, "mcp.tool": tool,
"suitest.run_id": ctx.run_id, "suitest.step_id": ctx.step_id,
})
try:
result = await session.call_tool(
name=tool, arguments=args,
on_progress=lambda c: redis.publish(f"run:{ctx.run_id}:log", c),
)
await artifacts.capture(ctx, result.artifacts)
metrics.mcp_invocations_total.labels(
provider=provider.name, tool=tool, outcome="ok").inc()
return result
except McpError as e:
span.record_exception(e)
metrics.mcp_invocations_total.labels(
provider=provider.name, tool=tool, outcome="error").inc()
raise
finally:
audit.log(actor=ctx.actor, provider=provider.name, tool=tool,
arg_hash=hash_args(args),
outcome=span.status.status_code.name)Every provider exposes a health-check method (uniform contract: invoke tools/list and assert ≥1 tool returned within timeout).
| Aspect | Default |
|---|---|
| Probe interval | 60s background task (one ARQ scheduled job per provider) |
| Probe timeout | 5s |
| Persisted to | mcp_providers.health_status + health_checked_at |
| Status values | ok / degraded (>1s latency or partial tools) / down |
| Auto-disable threshold | down for >5 minutes → removed from active routing, fallback elevated |
| Broadcast | WebSocket event mcp.provider.health with {provider, status, latency_ms} |
| Recovery | Probe still runs while disabled; first ok restores routing |
Operators can also subscribe to the Prometheus /metrics endpoint: mcp_provider_up{provider=…}, mcp_provider_probe_latency_ms{provider=…}.
Per-provider tunables (set in mcp_providers.config_json.pool):
| Key | Default | Notes |
|---|---|---|
max_sessions |
4 |
Max concurrent live sessions for this provider |
idle_ttl |
60s |
Idle session evicted after this duration |
spawn_timeout |
10s |
Cap on handshake time |
max_requests_per_session |
1000 |
Recycle after N invocations (memory hygiene) |
kill_grace |
3s |
SIGTERM grace before SIGKILL on shutdown |
Workspace-wide cap (defaults set per Helm values / .env):
SUITEST_MCP_MAX_SESSIONS_PER_WORKSPACE=16
SUITEST_MCP_QUEUE_TIMEOUT_SECONDS=30
When the workspace cap is reached, new invocations block on a fair FIFO queue serialised by an asyncio.Condition (see packages/mcp/src/suitest_mcp/workspace_cap.py). Waiters that exceed SUITEST_MCP_QUEUE_TIMEOUT_SECONDS are raised as McpPoolExhausted (code MCP_POOL_EXHAUSTED) and surfaced by the runner as a step error with reason=POOL_EXHAUSTED.
The cap is enforced above the per-provider pool — cross-provider bursts inside one workspace (e.g. seed pg → call api → drive browser) count against a single ceiling, so a runaway suite can't pin every provider's pool at the same time.
See DEPLOYMENT.md § 4.5 for the full concurrency env matrix and Helm overrides.
MCP plugins extend Suitest's attack surface. The security model is layered.
- Built-in MCP servers run as separate subprocesses under the Suitest service account. The single in-process exception (
api-http-mcp) carries no user-supplied code. - User-provided MCP commands run sandboxed where feasible: gVisor in Helm production (
mcp.sandbox.runtimeClass: gvisor),firejailprofile in docker-compose. This is best-effort. The OSS docs warn: trust your MCP servers — they execute as code. - In Kubernetes, MCP processes can be placed in a separate pod via the
mcp-runnerdeployment, with NetworkPolicy enforced egress allowlist per provider.
- Per-provider secrets stored AES-GCM encrypted inline on the provider row (
mcp_providers.secrets_json_encrypted, LargeBinary blob). No separateencrypted_secretstable — the encrypted blob lives directly onmcp_providers. Matches DATA_MODEL.md canonical schema. - Master key from
SUITEST_SECRET_KEYenv var (or KMS-derived in Helm viaextraEnv). - Secrets decrypted only at invocation time, only in memory, never logged.
- UI write-only — once saved, the cleartext is unrecoverable through the app.
Every tool invocation writes a row to audit_log:
actor_user_id | workspace_id | provider | tool | arg_hash (sha256) | outcome | duration_ms | run_id?
Argument hashes (not raw args) are stored to balance traceability and PII. Operators can enable SUITEST_AUDIT_MCP_RAW=true for environments where compliance requires full payloads — clearly flagged as PII-sensitive.
Per-provider network_policy block in config:
{
"network_policy": {
"egress_allow": ["api.stripe.com:443", "10.0.0.0/8:*"],
"egress_default": "deny"
}
}Rendered into Kubernetes NetworkPolicy in Helm. In docker-compose, enforced via iptables init container (opt-in).
| Threat | Mitigation |
|---|---|
| Malicious user-provided MCP exfiltrates secrets | Sandbox + per-provider egress allowlist + secrets scoped per provider (no cross-provider secret access) |
| MCP command injection via test args | All args serialized as JSON-RPC params; no shell interpolation by Suitest. Provider responsibility to validate args. |
| MCP impersonation (DNS / hostname spoof for SSE/WS) | Pin TLS fingerprints in config_json.tls.pinned_sha256 when supplied |
| Resource exhaustion (fork bomb, OOM) | max_sessions + cgroups limits in Helm + spawn_timeout |
| Long-lived sessions accumulating PII in memory | max_requests_per_session recycle |
| Privilege escalation via container escape | gVisor runtimeClass + non-root UID + read-only rootfs |
The single-cell step.mcp_provider field unlocks heterogeneous cases. All YAML examples below correspond directly to test_steps rows (see DATA_MODEL.md).
case: User can sign in
steps:
- { name: Open login, target_kind: FE_WEB, mcp_provider: playwright-mcp, tool: browser.navigate, arguments: { url: "{{base_url}}/login" } }
- { name: Fill creds, mcp_provider: playwright-mcp, tool: browser.fill_form, arguments: { fields: { "#email": "{{user.email}}", "#password": "{{user.password}}" } } }
- { name: Submit, mcp_provider: playwright-mcp, tool: browser.click, arguments: { selector: "button[type=submit]" } }
- { name: Assert dashboard, mcp_provider: playwright-mcp, tool: browser.assert_text, arguments: { selector: "h1", contains: "Dashboard" } }case: Create order returns 201 with order id
steps:
- name: POST /orders
target_kind: BE_REST
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: POST, url: "{{api_base}}/orders", headers: { Authorization: "Bearer {{token}}" }, json: { sku: "BOOK-01", qty: 2 } }
assertions:
- status_eq: 201
- { jsonpath: "$.order_id", matches: "^ord_[a-z0-9]{12}$" }case: Checkout creates persisted order
steps:
- name: Seed inventory
target_kind: DATA
mcp_provider: postgres-mcp
tool: db.exec
arguments: { sql: "INSERT INTO inventory (sku, qty) VALUES ('BOOK-01', 10) ON CONFLICT (sku) DO UPDATE SET qty=10;" }
- name: Login (API)
target_kind: BE_REST
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: POST, url: "{{api_base}}/auth/login", json: { email: "{{user.email}}", password: "{{user.password}}" } }
capture: { token: "$.access_token" }
- name: Walk through checkout (browser)
target_kind: FE_WEB
mcp_provider: playwright-mcp
tool: browser.script
arguments:
script: |
await page.goto('{{base_url}}/shop');
await page.click('text=BOOK-01');
await page.click('text=Checkout');
await page.fill('#card', '4242424242424242');
await page.click('text=Pay');
- name: Verify order via API
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: GET, url: "{{api_base}}/orders?latest=true", headers: { Authorization: "Bearer {{token}}" } }
assertions: [ { jsonpath: "$.items[0].sku", equals: "BOOK-01" } ]
- name: Verify DB state
mcp_provider: postgres-mcp
tool: db.query
arguments: { sql: "SELECT qty FROM inventory WHERE sku='BOOK-01';" }
assertions: [ { row_eq: { qty: 9 } } ]case: Production deployment serves traffic
steps:
- name: Assert deployment replicas
target_kind: INFRA
mcp_provider: kubernetes-mcp
tool: k8s.get
arguments: { kind: Deployment, namespace: prod, name: api }
assertions: [ { jsonpath: "$.status.readyReplicas", gte: 3 } ]
- name: Hit canary endpoint
target_kind: BE_REST
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: GET, url: "https://api.example.com/_health" }
assertions: [ { status_eq: 200 } ]case: Mobile login + server-side session
steps:
- { name: Launch app, target_kind: FE_MOBILE, mcp_provider: appium-mcp, tool: mobile.launch_app, arguments: { bundle_id: "com.example.app" } }
- { name: Sign in, mcp_provider: appium-mcp, tool: mobile.type_text, arguments: { selector: "id=emailField", text: "{{user.email}}" } }
- name: Verify server session
target_kind: BE_REST
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: GET, url: "{{api_base}}/me", headers: { Authorization: "Bearer {{token}}" } }
assertions: [ { jsonpath: "$.email", equals: "{{user.email}}" } ]Verifying a generated document needs no browser: request it and assert its text
layer. The runner feeds the http.request envelope into each assertion as
result, so a single step covers status, content type and body.
case: Report prints with the expected content
steps:
- name: Preview carries the draft watermark
target_kind: BE_REST
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: GET, url: "{{api_base}}/reports/{{report_id}}/print?draft=true", headers: { Authorization: "Bearer {{token}}" } }
assertions:
- { tool: http.assert_status, arguments: { equals: 200 } }
- { tool: http.assert_header, arguments: { name: content-type, equals: "application/pdf" } }
- { tool: http.assert_pdf_text, arguments: { contains: ["DRAFT"] } }
- name: Final print carries the report body, not the watermark
mcp_provider: api-http-mcp
tool: http.request
arguments: { method: GET, url: "{{api_base}}/reports/{{report_id}}/print", headers: { Authorization: "Bearer {{token}}" } }
assertions:
- { tool: http.assert_header, arguments: { name: content-disposition, equals: "attachment; filename=report.pdf" } }
- { tool: http.assert_pdf_text, arguments: { page: 1, contains: ["Report No.", "Facility Data"], matches: "No\\.\\s*\\d+/\\d{4}" } }The persisted TestStep.code for the first step is the same thing as a JSON
envelope (DATA_MODEL.md §3.4) — this is what the step editor stores:
{
"tool": "http.request",
"arguments": { "method": "GET", "url": "{{api_base}}/reports/{{report_id}}/print?draft=true" },
"assertions": [
{ "tool": "http.assert_status", "arguments": { "equals": 200 } },
{ "tool": "http.assert_pdf_text", "arguments": { "contains": ["DRAFT"] } }
]
}Pair it with a postgres-mcp step asserting the same values in the database
when the printed content must match stored data.
The MCP tool browser is the integration hub for debugging and exploring providers.
Navigation: Integrations → MCP Servers → <provider> → Tools tab.
| Element | Behavior |
|---|---|
| Provider list | Status dot (ok / degraded / down), pinned built-ins on top, custom below |
| Tools tab | Lists every tool discovered via tools/list. Expandable rows show the JSON schema (Zod-like rendering). |
| Try it form | Auto-generated form from the tool's input schema. User fills, clicks "Invoke". Output appears below: result panel, stderr panel, span trace link. |
| History sub-tab | Last 50 invocations from this provider across the workspace, filterable. |
| Config tab | Read-only view of config_json; secrets fields show •••• only. |
API endpoint backing the Try-it form:
POST /mcp/providers/:id/invoke
{
"tool": "db.query",
"arguments": { "sql": "SELECT 1;" }
}
Restrictions on /invoke:
- Available only when
SUITEST_DEV_MODE=trueor caller has rolemcp_admin. - Rate-limited per user (10/min default).
- Every invocation is audit-logged with
actor.invocation_source=tool_browser. - Never executed against
production-tagged workspaces unlessdev_invoke_in_prod=trueflag set.
Precedence (highest first):
step.mcp_provider— explicit per-step.- Suite-level override on the parent suite (
suites.mcp_routing_overrides). - Workspace-level override (
workspaces.mcp_routing_overrides). - Global default table from § 4.
The test case editor exposes a "Re-route steps" action:
- Select steps (filter by current
mcp_providerortarget_kind). - Pick new provider from a typeahead.
- Preview affected step count.
- Confirm → bulk UPDATE.
When a new MCP provider is registered, a background job (mcp_migration_suggest) scans existing cases and:
- For each step whose current provider matches the new provider's
kindandtarget_kind, generates a suggestion. - Suggestions surface in a banner: "12 existing steps could use
payments-mcp. Review →". - No automatic mutation — always human-approved.
Trust hinges on reproducibility. Each provider registration pins a version.
| Transport | Version pin column | Example |
|---|---|---|
| stdio (npm) | mcp_providers.command_pin |
npx @playwright/mcp@1.42.0 |
| stdio (image) | mcp_providers.image_pin |
ghcr.io/suitest/postgres-mcp:0.7.1 |
| stdio (git) | mcp_providers.git_ref |
https://github.com/acme/mcp@v0.3.4 |
| SSE / WS | version_header (recorded into mcp_providers.version_pin) |
Server returns X-MCP-Version: 2.1.0 during handshake; recorded |
command_pin, image_pin, version_pin, and git_ref are first-class columns on mcp_providers (added in DATA_MODEL.md §4.3 fix). Previously these were stored inside config_json; that is no longer the source of truth — readers MUST consult the dedicated columns. Surfaced in every run's metadata so traces are reproducible across upgrades.
Upgrade flow:
- Clone provider into
stagingworkspace, bump pin. - Re-run smoke suite tagged
mcp:<provider>. - Diff discovered tool catalog (
tools/list) for breaking changes. - Promote pin to production workspaces.
Test cases can be exported to standalone frameworks (Playwright, Cypress, Selenium). Non-browser MCP steps don't have a direct equivalent in those frameworks, so Suitest emits adapter helpers.
Mapping table:
| MCP provider | Playwright export | Cypress export | Selenium (JUnit) export |
|---|---|---|---|
playwright-mcp |
Native page.* calls |
cy.* equivalents |
WebDriver API |
browser-use-mcp |
Falls back to recorded steps (Playwright) | Falls back to recorded steps (Cypress) | Falls back to recorded WebDriver script |
api-http-mcp |
await request.fetch(...) (Playwright APIRequestContext) |
cy.request(...) |
OkHttpClient helper |
postgres-mcp |
import { Client } from 'pg' setup/teardown helper |
cy.task('db:exec', ...) + Node task plugin |
@BeforeAll JDBC helper |
mysql-mcp |
mysql2 helper |
cy.task('db:exec', ...) |
JDBC helper |
mongo-mcp |
mongodb helper |
cy.task('mongo:exec', ...) |
Mongo Java Driver helper |
graphql-mcp |
graphql-request |
cy.request with GraphQL body |
OkHttp + manual query |
grpc-mcp |
@grpc/grpc-js helper |
cy.task('grpc:call', ...) |
grpc-java helper |
kubernetes-mcp |
@kubernetes/client-node |
cy.task('kubectl', ...) |
kubernetes-client/java |
appium-mcp |
n/a (export blocked) | n/a | Appium Java client |
computer-use-mcp |
n/a (v2 only) | n/a | n/a |
The exporter writes a helpers/ directory alongside the generated test files. Step-level comments preserve the original MCP provider name to aid round-tripping.
See GENERATORS.md § export for the full exporter pipeline.
Air-gap is a first-class constraint (see DEPLOYMENT.md).
- Bundled binaries ship inside the main Suitest container image:
playwright-mcp(vianpxcache pre-warmed in build)browser-use-mcp(Python package vendored)api-http-mcp(in-process, no external deps)postgres-mcp,mysql-mcp,mongo-mcp(binaries pinned in image)
- Custom MCPs in air-gapped mode must come from a local OCI registry / local npm registry referenced by the registration's
image_pinorcommand_pin. - No outbound calls during normal operation. Discovery (
tools/list) is local. Workspace must explicitly setair_gapped=falseto allow MCPs that phone home (e.g.browser-use-mcpwith cloud LLM). - Helm value
mcp.airGapped: true(defaultfalse) enforces a NetworkPolicy that drops all egress from MCP pods except DNS + Suitest internal services.
suitest.whitebox.v1 is the framework-neutral capability contract used by
the local lifecycle. Its MCP tools are:
| Tool | Purpose |
|---|---|
whitebox_discover_tests |
Detect pytest, Vitest, or Jest; return native targets, command, and coverage path. |
whitebox_run_tests |
Execute the native targets and emit the normal strategy/plan/TC/results/report/TCM artifacts. |
Commands are argument arrays and never pass through a shell. Test discovery
excludes VCS, virtualenv, dependency, build, and suitest-output directories.
Native test files are retained as TCxxx automation source, while coverage.py
and Istanbul JSON are normalized for the existing publish path. See
TESTING_APPROACHES.md.
- Plugin SDK — Python entrypoint
suitest.plugins.mcpfor programmatic provider registration. Distributable as a pip package; auto-discovered at startup. - MCP marketplace — curated registry hosted at
mcp.suitest.dev, fully optional, served as static manifest JSON. Self-hosters can mirror. - On-demand MCP via npx — currently we require pre-install for safety. v1.x adds an opt-in
allow_npx_on_demand=trueflag with signature verification. - MCP-as-code — declare providers in a YAML/TOML file checked into the user's repo, synced into the DB on push (Git source of truth instead of UI clicks).
# suitest.mcp.yaml
version: 1
providers:
- name: payments-mcp
kind: payments
transport: stdio
command: "node ./mcp/payments/index.js"
is_default_for_targets: []
network_policy:
egress_allow: ["api.stripe.com:443"]- Desktop testing bundle —
computer-use-mcp(screen default),electron-mcp(CDP DOM),slint-mcp(accessible tree, headless). Selector-driven steps; binaries resolved viacommand_pin, not shipped in the image. See DESKTOP_TESTING.md.
computer-use-mcpfull support — desktop app testing GA, with replay + screen recording artifacts.- MCP swarm / multi-MCP per step — a step can fan out to multiple MCPs (e.g. assert DB state across 3 shards in parallel).
- MCP capability negotiation — Suitest classifier picks MCP based on advertised capabilities, not just registered
kind. - MCP tracing across providers — distributed trace continuity when one MCP invokes another (e.g.
kubernetes-mcptriggers a probe viaapi-http-mcp).
End of MCP_PLUGINS.md. For agent-side consumption of these providers, see AI_AGENT.md § tool wiring. For the relevant DB tables, see DATA_MODEL.md § mcp_providers. For HTTP contract details, see API.md § MCP endpoints.