[Feat]: Preproduction QA Validation of UCP Implementations

[RyanLamanskyInsite]

Is your feature request related to a problem? Please describe.
No documented strategy for pre-production QA validation, or at least it's not apparent on the site.

Describe the solution you'd like
The documentation basically says if you do everything right, it will work, but it implies that a real site with real products and real payments are all required.

We need an officially documented way to connect a real agent to a pre-production site to get the "all-clear" for production deployment.

Describe alternatives you've considered
Internet research suggests that testing must be done with a tool like Postman and validated manually by cross-checking results with the official spec. This could lead to both false positives and false negatives if the spec is misunderstood by either the original developer or QA personnel.

Additional context
My company's production deployment process has rigorous QA steps and I need to make sure UCP is testable.

[Nolpak14]

This is a real gap. I've spent the past couple of months building validation tooling for UCP (UCPtools), so I can share what we've learned about the testing landscape and what's actually available today.

What exists right now:

The official conformance test suite is the closest thing to what you're looking for. It's a Python-based integration test suite with 11 test files covering: protocol discovery, the full checkout lifecycle (create → update → complete → cancel + state machine enforcement), input validation (out-of-stock, inventory limits, 402 payment failures), fulfillment flows, order operations, webhook delivery, AP2 payment mandates, idempotency keys, and security controls.

However, it has two significant limitations for your use case:

1. It requires a running server, not a static profile. The tests exercise runtime behavior via HTTP — they're integration tests, not validation. If your QA workflow needs to gate deployments on "is our UCP profile correct before we expose endpoints," the conformance suite doesn't help until you have a fully running implementation.

2. It's hardcoded to sample data. The tests currently reference specific product IDs, shipping codes, and customer data from a "Flower Shop" fixture (test_data/flower_shop/). Issue #13 tracks decoupling this — the maintainer acknowledged it and said the fix involves moving all data to persona-specific config files — but it's not resolved yet. So running it against your catalog means adapting the fixtures. The suite also has 15 open issues, several of which are bugs in the tests themselves (#18, #24, #25).

The @ucp-js/sdk (v0.1.0) provides auto-generated Zod schemas for every UCP type — UcpDiscoveryProfileSchema, CheckoutCreateRequestSchema, CheckoutResponseSchema, etc. You can use .safeParse() to validate payloads against the official schemas. But it's a schema library, not a testing tool — no HTTP client, no test runner, no CLI. You build on top of it.

What's missing:

• No official static profile validator (check your /.well-known/ucp JSON before deploying)
• No hosted sandbox environment (you run your own server against the conformance tests)
• No CI/CD integration guide
• No agent-side testing (testing discovery + checkout from the agent's perspective)
• Conformance tests are Python-only — no JS/TS, Java, or Go frameworks

What we built to fill this gap:

At UCPtools, we run four validation layers that map to different QA stages:

1. Structural — JSON Schema validation against the UCP spec (required fields, types, version format YYYY-MM-DD). Catches "looks right but isn't" errors.
2. Rules — UCP-specific compliance: namespace/origin binding (your capability namespaces must match your domain), extension chain integrity, HTTPS enforcement on all endpoints, signing key presence. These are spec requirements that aren't obvious from the JSON Schema alone.
3. Network — Fetches your profile from /.well-known/ucp and verifies the live response matches what the schema declares. Catches deployment issues (CORS, CDN caching stale profiles, reverse proxy path stripping).
4. SDK — Runs your profile through the official @ucp-js/sdk Zod schemas to check compliance from the consumer's perspective.

This is available as an API (POST /v1/profiles/validate) so you can integrate it into your CI/CD pipeline — POST your profile JSON, get back a structured report with error codes, JSON paths, severity levels, and fix hints. If any errors come back, fail the build.

That said, this only covers the discovery profile side. For the full checkout lifecycle testing you're describing — connecting a real agent to your staging server — the conformance suite is still the only option, and it needs the decoupling work from issue #13 to be practical for custom implementations.

Would be curious what your deployment pipeline looks like — are you looking for pre-merge validation (CI gate), pre-release staging checks, or both?

[chelevich]

I managed to wire up conformance tests into build pipelines with a couple docker containers.
And I love how the tests cover entire flow from the user perspective and business logic.

However, my biggest problem as of now is that conformance tests are built for schemas 2026-01-11 whereas API docs make us build APIs for schemas 2026-01-23.
And the changes in schemas are breaking even in the discovery profile, like array-of-objects changed to key-value objects with arrays inside, like removing "fulfillment" in favor of "context".

Waiting for updates of the conformance tests so much...