Add database upgrade and rollback compatibility tests by iplay88keys · Pull Request #2084 · kagent-dev/kagent

iplay88keys · 2026-06-24T21:51:41Z

Description

Closes: #1637

Adds automated coverage that a Helm upgrade from a prior release to the current build is safe: migrations apply cleanly, existing data survives, the resulting schema matches a fresh install, and the migrations reverse cleanly. Also covers the rolling-deploy window where prior-release pods serve against an already-migrated schema, and — at each point in the upgrade/rollback cycle — drives a real agent invocation through the controller so kagent's own query paths are exercised, not just raw SQL.

Tests

Upgrade round-trip (TestUpgrade, gated by RUN_UPGRADE_TESTS): installs the prior release, seeds representative data, upgrades to the current build, then asserts the controller rolls out without crashing, migrations reach the target version cleanly, the seeded data survives, the upgraded schema is identical to a clean install, and reversing the migrations restores the prior schema with data intact. It then redeploys the previous release over the migrated schema to confirm the old binary boots and serves rather than crash-looping (ahead-schema tolerance).
Rolling compatibility (TestRollingUpgradeCompatibility, gated by RUN_ROLLING_UPGRADE_TESTS): runs the upgrade with two controller replicas and verifies a prior-release pod can still read and write while the new image rolls out. It runs regardless of whether migrations changed — the image swap alone can break a rolling deploy — and additionally exercises the old-code/new-schema window when the build adds migrations.
Version-matched invocation: at each state — post-upgrade, old-code-against-the-new-schema, and post-rollback — a representative invoke e2e slice (TestE2EInvokeInlineAgent) runs from the tree whose version matches the serving controller (the previous-release slices from a git worktree at that release's tag), so real deploy/invoke/read paths are exercised end to end. Self-skips when the harness (mock-LLM host / previous-release worktree) isn't set up, so a bare go test still runs the DB round-trip.

Targets

Upgrade-from targets are chosen per branch, mirroring how releases ship (a CI matrix; a leg is skipped when it doesn't apply):

on main (building the next minor): the previous minor line's latest patch.
on a release branch (building a patch): the same line's last patch, and the previous minor's latest.

The resolved upgrade-from version is printed before the build. Override locally with UPGRADE_FROM_VERSION.

Running locally

Requires a kind cluster, then:

make create-kind-cluster
make run-upgrade-tests

or

make create-kind-cluster
make run-rolling-upgrade-tests

A run leaves the database migrated, so recreate the cluster between runs.

Notes

The previous install pins the bundled Postgres image to whatever the upgrade-from release's own install target shipped, so the baseline matches how that release actually runs rather than a hardcoded guess.
Query-level backward compatibility (a prior release's queries against the new schema) is covered separately by a static check in Add query-contraction check against the previous release #2085.

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Copilot

Pull request overview

Adds end-to-end upgrade/rollback coverage for the database migration system by exercising real Helm upgrades from prior releases (including a rolling-upgrade compatibility window) and validating schema/data invariants across forward and reverse migrations.

Changes:

Add Go e2e tests to validate upgrade round-trip (seed → upgrade → schema equivalence → rollback) and rolling upgrade compatibility (old pods against new schema).
Add scripts/Makefile targets and CI jobs to resolve upgrade-from versions and run the new tests in a version matrix.
Update database-migrations reference docs to reflect that the round-trip upgrade testing is now enforced.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/upgrade-from-version.sh	Resolves the adjacent prior release tag reachable from HEAD for upgrade testing.
scripts/prev-stable-version.sh	Resolves the latest patch tag from the previous stable release line for rollback-floor coverage.
Makefile	Adds targets to install the prior release and run upgrade/rolling-upgrade e2e suites.
go/core/test/e2e/upgrade/upgrade_test.go	Implements the upgrade round-trip test (seed, upgrade, schema equivalence, rollback assertions).
go/core/test/e2e/upgrade/roundtrip_test.go	Adds shared helpers for schema dumping, clean-install schema construction, port-forward, and migration driving.
go/core/test/e2e/upgrade/rolling_upgrade_test.go	Implements rolling-upgrade compatibility checks (old-code/new-schema window).
.github/workflows/ci.yaml	Adds CI jobs to run upgrade tests across adjacent + previous-stable version targets.
.github/actions/upgrade-test-setup/action.yaml	Composite action to resolve target versions, dedupe legs, and set up the toolchain/cluster.
.claude/skills/kagent-dev/references/database-migrations.md	Updates migration policy docs to reflect enforced upgrade/rollback testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

EItanya

My main comment about these tests is that they sidestep the actual kagent DB usage. They interact directly with the DB rather than testing how the system behaves. For example they pre-seed the DB with data by directly inserting it rather than performing kagent operations. How much more work would it be to access the data that way?

EItanya · 2026-06-26T18:15:20Z

+func TestRollingUpgradeCompatibility(t *testing.T) {
+	if os.Getenv("RUN_ROLLING_UPGRADE_TESTS") != "true" {
+		t.Skip("set RUN_ROLLING_UPGRADE_TESTS=true to run rolling upgrade tests")
+	}


This feels weird, can't we have this in a separate package or something?

The concern here is that these upgrade/downgrade Kagent, which breaks the other tests. I mostly didn't want them running on a go test ./... or similar call without explicitly invoking them. I am looking into alternative approaches.

I am moving the tests to go/core/test/upgrade instead of being within e2e so it's a bit more clear that these tests are not run against the same cluster that the e2e tests are run against. I think that the env var is the cleanest approach to skipping these specific tests, though I am open to other options.

EItanya · 2026-06-26T18:19:49Z

+UPGRADE_FROM_VERSION ?= $(shell ./scripts/upgrade-from-version.sh)
+
+# The bundled-Postgres image is selected by the install target's --set flags, not
+# by the chart defaults (the chart ships a non-vector image). So the previous
+# install must use the exact pins the upgrade-from release shipped — otherwise the
+# baseline DB would differ from how that release actually runs, and the upgrade
+# would conflate a DB swap with the migration change under test. Read those flags
+# straight from that release's own helm-install-provider target (via its tagged
+# Makefile) instead of hardcoding values that drift as the bundled image changes.
+# Assumes the flags are literal (no make/env variables); the guard in
+# install-previous-release fails loudly if they can't be read.
+PREV_DB_SET_FLAGS = $(shell git show v$(UPGRADE_FROM_VERSION):Makefile 2>/dev/null | \


This means that these will be called every time make is used. Should we instead add this specifically to the targets that need it?

UPGRADE_FROM_VERSION ?= is a recursive assignment, so it stays the literal string $(shell ./scripts/upgrade-from-version.sh) until something expands it (docs). It doesn't run on every make, only when an upgrade target references it. I moved the other one into the recipe.

I kept this one as a variable for a single source of truth and so CI can override it and have it flow through to the targets (including the install-previous-release prerequisite).

## Description Adds a static check that a migration on this branch doesn't break the queries a prior release still runs — the backward-compatibility property that makes a one-step rollback (and a rolling deploy) safe. It compiles a prior release's sqlc queries against the current schema (derived from the migration files) and fails if a migration dropped, renamed, or retyped a column or table an older query still references. Runs in normal CI on every PR — no database, no cluster. ## Targets Checks two prior versions (deduplicated when they coincide): - the latest release reachable from HEAD — the adjacent previous release, and - the previous stable line's latest patch (the `release/vX.Y.x` tip) — the rollback-window floor. ## Running ``` make -C go check-query-contraction ``` Override the targets with `TARGET_VERSIONS` (space-separated, no leading `v`). ## Notes - Verified it passes today and fails on a simulated column drop. - Catches column/table/type-shape contractions; it does not catch data rewrites (no DDL to inspect) or semantic changes a query still compiles against. - Complements the contraction-declaration check in #2067: that one requires destructive DDL to be acknowledged; this one verifies a prior release's queries actually survive the new schema. - Also documents the existing `sqlc generate` gate (schema derived from migrations) as the forward-direction equivalent for current-release queries. - This branch carries its own copies of the helper scripts, which are shared with #2084, so that will require a trivial merge conflict resolution. --------- Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys · 2026-06-29T15:26:25Z

My main comment about these tests is that they sidestep the actual kagent DB usage. They interact directly with the DB rather than testing how the system behaves. For example they pre-seed the DB with data by directly inserting it rather than performing kagent operations. How much more work would it be to access the data that way?

@EItanya I agree these tests sidestep kagent's real query paths. The scope I'm going for in this PR is migration correctness: that the migration files preserve schema and data forward and back. Using Postgres directly lets us seed every table and diff the upgraded schema against a clean install, which there's no API for.

For testing actual system behavior across an upgrade, I've been thinking through our options, but I haven't come up with a great solution. Building out a new test suite which exercises Kagent to test it's behavior and keeping it up-to-date with the old and new versions seems unnecessary when the version-matched e2e suite already is that code, written against each version's API and CRDs. However, running the entire e2e test suite at each state (post-upgrade, old-code-against-the-new-schema, post-rollback) is slow.

The static side of this (that an older release's queries still compile against the new schema) is covered by #2085. So I'd land this PR as the migration-correctness layer and consider doing the version-matched e2e work as a follow-up.

So the question is: do we want to take on the extra cost of of running the entire e2e test suite multiple times for safety right now and optimize it later or start with the subset I have in this PR and expand it over time?

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

…e-compatibility-tests Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

…e-compatibility-tests Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys · 2026-07-01T17:08:40Z

I have updated this to run the TestE2EInvokeInlineAgent E2E test after each step of the upgrade/rollback cycle.

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys added 2 commits June 24, 2026 14:50

Add db upgrade and rollback compatibility tests

a28a05e

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Remove agent-sandbox for now

273e39b

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys mentioned this pull request Jun 24, 2026

Add query-contraction check against the previous release #2085

Merged

iplay88keys added 5 commits June 24, 2026 15:10

Fix linting error

4cd602b

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Updates now that we have an lts branch

1065890

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Update kagent-dev skill

f449f5f

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Fixes based on claude review

b5930fc

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Fix ci issues

bf230a8

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys marked this pull request as ready for review June 25, 2026 20:28

iplay88keys requested review from jmhbh and supreme-gg-gg as code owners June 25, 2026 20:28

Copilot AI review requested due to automatic review settings June 25, 2026 20:28

iplay88keys requested review from EItanya, ilackarms, peterj and yuval-k as code owners June 25, 2026 20:28

Copilot started reviewing on behalf of iplay88keys June 25, 2026 20:28 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread go/core/test/upgrade/roundtrip_test.go

iplay88keys mentioned this pull request Jun 25, 2026

Add check for schema-agnostic sql #2067

Merged

mesutoezdil reviewed Jun 26, 2026

View reviewed changes

Comment thread go/core/test/e2e/upgrade/rolling_upgrade_test.go Outdated

mesutoezdil reviewed Jun 26, 2026

View reviewed changes

Comment thread go/core/test/e2e/upgrade/upgrade_test.go Outdated

mesutoezdil reviewed Jun 26, 2026

View reviewed changes

Comment thread go/core/test/e2e/upgrade/upgrade_test.go Outdated

mesutoezdil reviewed Jun 26, 2026

View reviewed changes

Comment thread Makefile Outdated

EItanya and others added 2 commits June 26, 2026 11:40

Merge branch 'main' into iplay88keys/db-upgrade-compatibility-tests

c686f0b

Review comment fixes

6e24c50

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

EItanya reviewed Jun 26, 2026

View reviewed changes

iplay88keys added 2 commits June 29, 2026 08:26

Fix review comments

7a02cba

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Merge remote-tracking branch 'origin/main' into iplay88keys/db-upgrad…

c262d09

…e-compatibility-tests Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys added 3 commits June 29, 2026 10:57

Add Kagent application rollback step to upgrade tests

d64380f

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Test adding in a build cache

984b059

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Revert build cache

d2fd15b

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys requested a review from EItanya June 30, 2026 15:33

iplay88keys added 3 commits July 1, 2026 09:33

Add single e2e test between upgrade test phases

9043978

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Fix upgrade tests to be branch aware

ca0398c

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Merge remote-tracking branch 'origin/main' into iplay88keys/db-upgrad…

688404a

…e-compatibility-tests Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

iplay88keys added 2 commits July 1, 2026 10:42

run upgrade tests even if there are no new migrations

2a6ef69

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

Print more info during upgrade tests

66d691b

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>

EItanya approved these changes Jul 2, 2026

View reviewed changes

EItanya merged commit 6ee77b7 into main Jul 2, 2026
29 checks passed

EItanya deleted the iplay88keys/db-upgrade-compatibility-tests branch July 2, 2026 20:57

Uh oh!

Conversation

iplay88keys commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests

Targets

Running locally

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EItanya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

EItanya Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

iplay88keys Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

iplay88keys Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

EItanya Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

iplay88keys Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iplay88keys commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iplay88keys commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iplay88keys commented Jun 24, 2026 •

edited

Loading

iplay88keys Jun 26, 2026 •

edited

Loading

iplay88keys commented Jun 29, 2026 •

edited

Loading