ResourceSet dynamic inputs for preview environments automation

[stefanprodan]

With the introduction of the ResourceSet API, the Flux Operator can generate a set of Kubernetes resources based on a static input list defined by users in the ResourceSet spec. There are use-cases when the inputs are not known ahead of time and can't be set in a static manner. One such use-case is deploying app code and/or config changes made in a GitHub Pull Request or GitLab Merge Request. To accommodate this use-case, the Flux Operator should allow users to define inputs in a dynamic manner using a CRD named ResourceSetInputProvider. Based on this CRD the operator would call an external system (e.g. GitHub, GitLab, AzureDevOps APIs) and transform the response for use as ResouceSet inputs.

Example

We'll use GitHub and the podinfo demo app to demonstrate how the new API should work, but the ResourceSetInputProvider could support GitLab, Azure DevOps, Gitea and any other Git service.

Preview environments workflow

• A developer makes code changes to the podinfo source code and its Helm chart and opens a PR in GitHub
• The CI builds the new app container image and pushes a tag to GHCR with the Git commit SHA
• Another developer reviews the changes and adds a label to the PR deploy/flux-preview
• Flux Operator running in the preview cluster scans the repository and finds the new PR using the label filter
• Flux Operator installs a Helm release using the PR number and the commit SHA inputs to deploy the app and chart changes in the cluster
• The developers iterate over changes, with each push to the PR branch triggering a Helm release upgrade in the cluster
• Once the PR is approved and merged, the Flux Operator uninstalls the Helm release form the cluster

Flux Operator Config

To enable the above workflow, we'll define a series of Flux Operator custom resources in the preview cluster.

First we'll create a dedicated namespace called podinfo-preview where all the app instances generated from PRs will be deployed. In that namespace, we'll create a Kubernetes Secret containing a GitHub token that grants read access to the podinfo repository and PRs.

In the podinfo-preview namespace, we'll create a ResourceSetInputProvider that tells Flux Operator to scan the repository for PRs labeled with deploy/flux-preview:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: podinfo-pull-requests
  namespace: podinfo-preview
  annotations:
    fluxcd.controlplane.io/reconcileEvery: "10m"
spec:
  type: GitHubPullRequest
  url: https://github.com/stefanprodan/podinfo
  secretRef:
    name: github-token-readonly
  filter:
    labels:
      - "deploy/flux-preview"

To not wait 10 minutes for the operator to detect changes in PRs, we can trigger the scan every time there is any activity on Pull Requests by creating a Flux Receiver for GitHub to call:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-receiver
  namespace: podinfo-preview
spec:
  type: github
  secretRef:
    name: receiver-token
  resources:
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSetInputProvider
      name: podinfo-pull-requests

Finally, to deploy podinfo from PRs we'll create a ResouceSet that takes its inputs from the ResourceSetInputProvider:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: podinfo
  namespace: podinfo-preview
spec:
  inputsFrom:
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSetInputProvider
      name: podinfo-pull-requests
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      metadata:
        name: podinfo-<< inputs.id >>
        namespace: podinfo-preview
      spec:
        interval: 1h
        url: https://github.com/stefanprodan/podinfo
        ref:
          commit: << inputs.sha >>
        secretRef:
          name: github-token-readonly
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: podinfo-<< inputs.id >>
        namespace: podinfo-preview
        annotations:
          event.toolkit.fluxcd.io/preview-url: "https://podinfo-<< inputs.id >>.example.com"
          event.toolkit.fluxcd.io/branch: << inputs.branch | slugify >>
          event.toolkit.fluxcd.io/author: << inputs.author | quote >>
      spec:
        interval: 1h
        releaseName: podinfo-<< inputs.id >>
        serviceAccountName: flux-restricted
        chart:
          spec:
            chart: ./charts/podinfo
            reconcileStrategy: Revision
            sourceRef:
              kind: GitRepository
              name: podinfo-<< inputs.id >>
        values:
          image:
            tag: << inputs.sha >>
          ingress:
            hosts:
              - host: podinfo-<< inputs.id >>.example.com

The above ResouceSet will generate a Flux GitRepository and a HelmRelease for each opened PR. The PR number passed as << inputs.id >> is used as the name suffix for the Flux objects, and is also used to compose the Ingress host name where the app can be accessed. The latest commit SHA pushed to the PR HEAD is passed as << inputs.sha >>, the SHA is used to set the app image tag in the Helm release values. The preview URL, branch name and author are set as annotations on the HelmRelease object to enrich the Flux notifications that the dev team receives.

To receive notifications when a PR triggers a Helm release install, upgrade and uninstall (including any deploy erorrs), a Flux Alert can be created in the podinfo-preview namespace:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack-bot
  namespace: podinfo-preview
spec:
  type: slack
  channel: general
  address: https://slack.com/api/chat.postMessage
  secretRef:
    name: slack-bot-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: slack
  namespace: podinfo-preview
spec:
  providerRef:
    name: slack-bot
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: HelmRelease
      name: '*'
  eventMetadata:
    cluster: "preview-cluster-1"
    region: "us-east-1"

Note that all the Kubernetes manifests part of this workflow should be stored in the Git repository used to define the preview cluster desired state.