test: probe ats GitOps bundle testing under Flux (do not merge)#199
test: probe ats GitOps bundle testing under Flux (do not merge)#199QuentinBisson wants to merge 10 commits into
Conversation
Experimental branch to exercise app-test-suite's new GitOps bundle testing (app-test-suite #664-#667) against this meta-package. Un-skips the smoke step, selects the Flux engine, and pins the ats dev image carrying the feature. ats installs Flux on the kind cluster, deploys the bundle into agentic-platform-flux, and waits for every rendered HelmRelease/OCIRepository to become Ready. Short bundle-ready timeout so a non-converging component set fails fast. Not for merge: this is a convergence probe, not a permanent CI gate.
Local verification (prod image, kind v1.34)Ran the same ats build locally against a fresh kind cluster to get full logs. The bundle converges under Flux and the smoke leg is green end to end:
So my earlier non-convergence worry was wrong: the trimmed Two findings that block the CI run (not the feature)
Feature verdict: works against the real reference bundle chart. Remaining redness is test-harness plumbing, tracked above. |
The new ats runs `uv sync` for the pytest executor; the devctl-generated Pipfile is pipenv-only and can't be consumed. Convert tests/ats to a uv project (pyproject.toml + uv.lock) and drop the Pipfile and pytest.ini (markers move to [tool.pytest.ini_options]). Verified locally against a kind cluster with the pinned ats dev image: the bundle reaches Ready under Flux (muster + agentic-platform-connectivity) and the smoke test passes.
Replicated the muster#854 migrationReplaced the earlier ad-hoc
Re-verified locally against kind with the pinned dev image: bundle Ready under Flux ( The uv migration is the part worth keeping; the Flux engine + image pin in |
✅ Green in CI
So end to end, on real CI: ats installs Flux on the kind cluster, deploys the agentic-platform bundle, waits for the rendered HelmReleases/OCIRepositories to become Ready (muster + agentic-platform-connectivity), and runs the smoke test. The GitOps bundle testing feature (app-test-suite #664-#667) works against the real reference bundle chart. Keep: the |
tests/ats had only a placeholder smoke test. Add real assertions for the GitOps bundle: the expected component HelmReleases (muster, agentic-platform-connectivity) are present and Ready, and the OCIRepository sources are Ready. Gated to the Flux engine leg so a plain Helm run skips them. .ats runs smoke/functional/upgrade under Flux. The upgrade baseline is the latest published stable deployed with the same trimmed test-values, so the stable release renders the 2-component bundle and converges on kind. Verified locally on kind: all three scenarios green, upgrade exercises the two-phase deploy (stable Ready -> upgrade -> Ready) under a live engine.
Real tests added + all three scenarios green in CI
Verified locally (all three green; upgrade exercises stable Ready → upgrade → Ready under Flux) and now green in CI ( |
Add test_muster_component_running (the muster Deployment reaches all-replicas-ready, flaky-rerun guarded, runs in every scenario incl. before/after upgrade) and test_connectivity_httproute_present (the connectivity component renders its HTTPRoute wiring). HelmRelease Ready only means the child chart installed; these assert the produced workload/wiring is up. Verified locally on kind: smoke 3/3, functional 5/5, upgrade 3+3 (pre+post).
Extend the Flux experiment to run every scenario under both engines. ats stacks
ci/gitops-values-<engine>.yaml to switch the meta-package's render layer between
Flux HelmReleases and Argo Applications, so the bundle is validated identically
on each.
- ci/gitops-values-{flux,argo}.yaml: engine-selecting overlays (argo project and
destination server default to what ats provisions on the test cluster)
- .ats/main.yaml: *-tests-gitops-engines flux -> flux,argo
- tests/ats: engine-aware bundle assertions. Component readiness now covers both
HelmRelease Ready and Application Healthy+Synced; the workload and HTTPRoute
checks run on both legs; the OCIRepository-source check stays Flux-only (an Argo
Application embeds its source, with no separate CR)
1 participant
Experimental. Do not merge. Convergence probe, not a permanent CI gate.
Purpose
Exercise app-test-suite's new GitOps bundle testing (giantswarm/app-test-suite #664-#667) against this meta-package, the reference bundle chart. The feature was already proven end-to-end on the
flux-bundle-appexample (all scenarios green on kind, including the upgrade two-phase). This probes whether the trimmedtests/test-values.yamlrender converges under a live Flux engine on kind.What changed
.ats/main.yaml: un-skipsmoke, setsmoke-tests-gitops-engines: flux, shortsmoke-tests-gitops-bundle-ready-timeout: 5m..circleci/workflows.yml: pinexecute-chart-teststo the ats dev image0.15.0-84cda623836b7c03358837fdcdbcecdfbad8370c(the #664-#667 stack tip). Edited the generated file directly because thecustom.ymldeep-merge appends list items rather than overriding a generated job's params.ats installs Flux, deploys the bundle into
agentic-platform-flux, and waits for every renderedHelmRelease/OCIRepositoryto report Ready.Expected outcome
Uncertain, and that is the point. This meta-package renders
HelmReleases for ~9 components that pull child charts from OCI and need their own CRDs. On a bare kind cluster the bundle-ready fixpoint may never be reached, which is exactly why.ats/main.yamlskips ATS today and defers real validation to a management cluster. If it times out, that confirms the meta-package is not a kind-testable bundle as-is; if it converges, we have a real CI smoke for the bundle. Either result is informative.