Skip to content

feat: add executable production acceptance suite - #10

Merged
YangYuS8 merged 34 commits into
mainfrom
agent/production-acceptance-suite
Jul 16, 2026
Merged

feat: add executable production acceptance suite#10
YangYuS8 merged 34 commits into
mainfrom
agent/production-acceptance-suite

Conversation

@YangYuS8

Copy link
Copy Markdown
Contributor

What changed

Deterministic security probes

  • add nivora-probe with JSONL scenarios for service authentication, tenant isolation, anonymous Scope restrictions, Provider context requirements, unknown request fields, and history-role injection
  • avoid storing response bodies or configured secrets in probe result artifacts
  • add server-side acceptance tests for the same critical boundaries

Load and recovery tooling

  • add nivora-load for bounded concurrent SSE testing
  • record first-token and completion p50/p95/p99, success rate, and error distribution
  • expose goroutine, heap allocation, and heap object metrics for post-load recovery checks
  • add a synthetic Provider with deterministic knowledge, failed resources, matching charge/refund records, idempotent cases, latency injection, and first-N 429/5xx failures

Privacy-safe shadow comparison

  • add nivora-shadow for baseline/candidate comparison
  • evaluate the candidate against the existing deterministic regression cases
  • store answer SHA-256, byte length, Tool sets, completion/error state, and latency instead of answer text
  • replace request error details with generic codes so Provider responses or URLs are not persisted

Staging workflow and documentation

  • add a manual Production Acceptance GitHub Actions workflow for protected nivora-staging environments
  • add the executable production gate covering probes, regression, load/recovery, Shadow comparison, CozeLoop review, release metadata, and rollback evidence
  • update English and Chinese README files and Make targets

Important boundary

This PR completes the repository-side acceptance tooling. It does not itself approve Lumio production traffic. Real acceptance still requires an isolated company staging environment, approved Ark/CozeLoop/VikingDB credentials, production-like PostgreSQL and infrastructure, synthetic or consented-redacted traffic, measured SLO/cost baselines, and a successful rollback drill.

Validation

CI will run:

  • clean go mod tidy
  • gofmt
  • go vet ./...
  • go test -race ./...
  • go build -trimpath ./...

This builds Nivora and all evaluation, knowledge, probe, load, Shadow, and synthetic Provider commands.

YangYuS8 added 30 commits July 16, 2026 19:27
@YangYuS8
YangYuS8 marked this pull request as ready for review July 16, 2026 11:49
@YangYuS8
YangYuS8 merged commit 9d51c96 into main Jul 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant