Add database upgrade and rollback compatibility tests

.claude/skills/kagent-dev/references/database-migrations.md

@@ -133,7 +133,7 @@ Files must follow `NNNNNN_description.up.sql` / `NNNNNN_description.down.sql` wi
 
 Every `.up.sql` must have a corresponding `.down.sql` that exactly reverses it. Down migrations are used for rollbacks and by automatic rollback on migration failure. They must be **idempotent** — the two-track rollback logic (roll back core if vector fails) may call them more than once in failure scenarios.
 
-A down file that never runs is a down file you cannot trust. There are no up-only migrations — a working down has shipped with every migration since the golang-migrate adoption. Exercising every migration up → down → up against the real migration set, to prove the reversal rather than assume it, is a *Target — not yet enforced* (see [Upgrade and rollback testing](#upgrade-and-rollback-testing)).
+A down file that never runs is a down file you cannot trust. There are no up-only migrations — a working down has shipped with every migration since the golang-migrate adoption. The reversal is proven, not assumed: the upgrade round-trip applies `HEAD`'s migrations over a prior release and then reverses them back, asserting the reverted schema matches a clean install of that release and that seeded data survives (see [Upgrade and rollback testing](#upgrade-and-rollback-testing)).
 
 ## One Linear History
 
@@ -205,9 +205,9 @@ These tests catch policy violations at PR time without needing a running databas
 
 ## Upgrade and rollback testing
 
-Static analysis covers file *content*; round-trip tests cover *behavior* against a real Postgres. Beyond `runner_test.go` (rollback and concurrency), release-to-release coverage makes the rollback promise real.
+Static analysis covers file *content*; round-trip tests cover *behavior* against a real Postgres. Beyond `runner_test.go` (rollback and concurrency), release-to-release tests make the rollback promise real.
 
-**Previous-minor round-trip** (*Target — not yet enforced*). Seed a database at the previous minor's latest release with representative data, apply migrations up to `HEAD`, and assert the schema matches a clean `HEAD` install and the data survives; then reverse to the previous minor and assert the schema matches a clean previous-minor install and the data survives. This exercises every changed down file rather than only reviewing it.
+**Previous-release round-trip** (enforced by `TestUpgrade`, run by the `upgrade-tests` CI job). Seed a database at a prior release with representative data, apply migrations up to `HEAD`, and assert the controller rolls out without crashing, the schema matches a clean `HEAD` install, and the data survives; then reverse the migrations back to the prior release and assert the schema matches a clean install of that release and the data survives. It runs against two prior versions — the latest release reachable from `HEAD` and the previous stable line's latest patch (the `release/vX.Y.x` tip) — and `TestRollingUpgradeCompatibility` (the `rolling-upgrade-tests` job) additionally exercises the old-code/new-schema window, with the prior release's controller serving while `HEAD`'s migrations are applied.
 
 **Query-level backward compatibility.** A static check — `scripts/check-query-contraction.sh`, run by the `query-contraction-check` CI job — compiles a previous release's sqlc queries against the `HEAD` schema and fails if a migration dropped, renamed, or retyped a column or table an older query still reads. It catches column/table/type-shape contraction with no database, against two prior versions: the latest release reachable from `HEAD` and the previous stable line's latest patch (the `release/vX.Y.x` tip, via `scripts/prev-stable-version.sh`). The fuller property — running the previous minor's whole database *test suite* against a `HEAD`-migrated schema, which also covers semantic breaks a query still compiles against — remains a *Target — not yet enforced*.
 

.github/actions/upgrade-test-setup/action.yaml

@@ -0,0 +1,112 @@
+name: Upgrade Test Setup
+
+description: >-
+  Shared prelude for the upgrade-tests and rolling-upgrade-tests jobs: resolve
+  the upgrade-from version (with the prev-stable == adjacent skip) and bring up
+  the build/cluster toolchain. Exposes the resolved version and skip flag so the
+  caller can gate its test step. The caller MUST run actions/checkout (with
+  fetch-depth: 0 + fetch-tags) before this action — a local action is loaded
+  from the checked-out workspace and the version resolvers need full history.
+
+inputs:
+  upgrade-from:
+    description: 'Which release to upgrade from: "adjacent" or "prev-stable".'
+    required: true
+
+outputs:
+  skip:
+    description: '"true" when this leg is redundant (prev-stable == adjacent) and the caller should skip its test step.'
+    value: ${{ steps.resolve.outputs.skip }}
+  version:
+    description: The resolved upgrade-from version (empty when skip is true).
+    value: ${{ steps.resolve.outputs.version }}
+
+runs:
+  using: "composite"
+  steps:
+    # The caller must run actions/checkout (fetch-depth: 0 + fetch-tags) before
+    # this action: a local action is loaded from the checked-out workspace, so
+    # its files don't exist on the runner until checkout has run.
+    - name: Resolve upgrade-from version
+      id: resolve
+      shell: bash
+      run: |
+        ADJ="$(./scripts/upgrade-from-version.sh)"
+        base_ref="${GITHUB_BASE_REF:-${GITHUB_REF_NAME:-}}"
+        is_release_line=false
+        if [[ "$base_ref" =~ ^release/v[0-9]+\.[0-9]+\.x$ ]]; then
+          is_release_line=true
+        fi
+
+        if [ "${{ inputs.upgrade-from }}" = "prev-stable" ]; then
+          # prev-stable = the previous minor line's latest patch (the rollback
+          # floor). On main this resolves to the newest release line overall
+          # (e.g. the 0.9.x tip); on a release branch, to the line below it
+          # (skipped when none exists).
+          V="$(./scripts/prev-stable-version.sh)"
+          if [ -z "$V" ]; then
+            echo "no stable line below the current line; skipping prev-stable leg."
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          if [ "$V" = "$ADJ" ]; then
+            echo "prev-stable ($V) == adjacent; skipping (covered by the adjacent leg)."
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+        else
+          # adjacent = the same-line previous patch, i.e. the release just before
+          # the one being built on the SAME line. It is only meaningful when
+          # building a patch on a release branch. On main we are building the next
+          # MINOR, which has no same-line predecessor (git-describe would report a
+          # stale patch nobody upgrades from — the newest patch lives on the
+          # release branch, covered by prev-stable), so skip this leg there.
+          if [ "$is_release_line" != "true" ]; then
+            echo "base '$base_ref' is not a release line (building the next minor); skipping the adjacent leg (only prev-stable is meaningful)."
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          V="$ADJ"
+        fi
+        echo "version=$V" >> "$GITHUB_OUTPUT"
+        echo "=== Upgrade test (${{ inputs.upgrade-from }} leg): will upgrade FROM v$V TO the current build — building images next ==="
+        echo "::notice title=Upgrade from::v$V (${{ inputs.upgrade-from }} leg) -> current build"
+    - name: Initialize Environment
+      if: steps.resolve.outputs.skip != 'true'
+      uses: ./.github/actions/initialize-environment
+    - name: Allow unprivileged user namespaces
+      if: steps.resolve.outputs.skip != 'true'
+      shell: bash
+      run: |
+        sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
+    - name: Set up QEMU
+      if: steps.resolve.outputs.skip != 'true'
+      uses: docker/setup-qemu-action@v4
+      with:
+        platforms: linux/amd64,linux/arm64
+    - name: Set up Docker Buildx
+      if: steps.resolve.outputs.skip != 'true'
+      uses: docker/setup-buildx-action@v4
+      with:
+        # Builder name/version match the Makefile (BUILDKIT_VERSION) and the other
+        # workflows (e.g. tag.yaml); kept as literals for consistency.
+        name: kagent-builder-v0.23.0
+        version: v0.23.0
+        platforms: linux/amd64,linux/arm64
+        use: "true"
+        driver-opts: network=host
+    - name: Set up Helm
+      if: steps.resolve.outputs.skip != 'true'
+      uses: azure/setup-helm@v5.0.0
+      with:
+        version: v3.18.0
+    - name: Install Kind
+      if: steps.resolve.outputs.skip != 'true'
+      uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc
+      with:
+        install_only: true
+    - name: Create Kind cluster
+      if: steps.resolve.outputs.skip != 'true'
+      shell: bash
+      run: |
+        make create-kind-cluster

.github/workflows/ci.yaml

@@ -167,6 +167,104 @@ jobs:
       - name: Previous-release queries vs current schema
         run: make -C go check-query-contraction
 
+  upgrade-tests:
+    needs:
+      - setup
+    env:
+      VERSION: v0.0.1-test
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # prev-stable: the previous minor line's latest patch — runs on every base
+        #   (on main it resolves to the newest release line, e.g. the 0.9.x tip).
+        # adjacent: the same-line previous patch — only meaningful on a release
+        #   branch (building a patch); skipped on main, which builds a new minor
+        #   with no same-line predecessor. See upgrade-test-setup resolve step.
+        upgrade-from: [adjacent, prev-stable]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          # Full history + tags so the version resolvers can derive the
+          # upgrade-from release, and so the local action below is on disk.
+          fetch-depth: 0
+          fetch-tags: true
+      - name: Prepare upgrade test environment
+        id: prep
+        uses: ./.github/actions/upgrade-test-setup
+        with:
+          upgrade-from: ${{ matrix.upgrade-from }}
+      - name: Run upgrade tests
+        if: steps.prep.outputs.skip != 'true'
+        env:
+          OPENAI_API_KEY: fake
+          BUILDX_BUILDER_NAME: ${{ env.BUILDX_BUILDER_NAME }}
+          KAGENT_HELM_EXTRA_ARGS: --cleanup-on-fail=false
+          DOCKER_BUILD_ARGS: >-
+            --cache-from=type=gha,scope=${{ needs.setup.outputs.cache-key }}-e2e
+            --cache-from=type=gha,scope=${{ env.CACHE_KEY_PREFIX }}-main-e2e
+            --platform=linux/amd64
+            --push
+        run: |
+          make run-upgrade-tests UPGRADE_FROM_VERSION="${{ steps.prep.outputs.version }}"
+      - name: fail print info
+        if: failure() && steps.prep.outputs.skip != 'true'
+        run: |
+          echo "::error::Failed to run upgrade tests"
+          kubectl describe pods -n kagent
+          kubectl get events -n kagent
+          kubectl logs -n kagent deployment/kagent-controller || true
+
+  rolling-upgrade-tests:
+    needs:
+      - setup
+    env:
+      VERSION: v0.0.1-test
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # prev-stable: the previous minor line's latest patch — runs on every base
+        #   (on main it resolves to the newest release line, e.g. the 0.9.x tip).
+        # adjacent: the same-line previous patch — only meaningful on a release
+        #   branch (building a patch); skipped on main, which builds a new minor
+        #   with no same-line predecessor. See upgrade-test-setup resolve step.
+        upgrade-from: [adjacent, prev-stable]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          # Full history + tags so the version resolvers can derive the
+          # upgrade-from release, and so the local action below is on disk.
+          fetch-depth: 0
+          fetch-tags: true
+      - name: Prepare upgrade test environment
+        id: prep
+        uses: ./.github/actions/upgrade-test-setup
+        with:
+          upgrade-from: ${{ matrix.upgrade-from }}
+      - name: Run rolling upgrade tests
+        if: steps.prep.outputs.skip != 'true'
+        env:
+          OPENAI_API_KEY: fake
+          BUILDX_BUILDER_NAME: ${{ env.BUILDX_BUILDER_NAME }}
+          KAGENT_HELM_EXTRA_ARGS: --cleanup-on-fail=false
+          DOCKER_BUILD_ARGS: >-
+            --cache-from=type=gha,scope=${{ needs.setup.outputs.cache-key }}-e2e
+            --cache-from=type=gha,scope=${{ env.CACHE_KEY_PREFIX }}-main-e2e
+            --platform=linux/amd64
+            --push
+        run: |
+          make run-rolling-upgrade-tests UPGRADE_FROM_VERSION="${{ steps.prep.outputs.version }}"
+      - name: fail print info
+        if: failure() && steps.prep.outputs.skip != 'true'
+        run: |
+          echo "::error::Failed to run rolling upgrade tests"
+          kubectl describe pods -n kagent
+          kubectl get events -n kagent
+          kubectl logs -n kagent deployment/kagent-controller || true
+
   go-unit-tests:
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -213,4 +213,7 @@ file::memory:*
 
 ## Test certificates
 python/packages/kagent-adk/tests/fixtures/certs/*.pem
-python/packages/kagent-adk/tests/fixtures/certs/*.srl
+python/packages/kagent-adk/tests/fixtures/certs/*.srl
+
+## Upgrade test: previous-release worktree (created by `make run-upgrade-tests`)
+.upgrade-prev/