Incident report: Docker Hub container tagged as "v3.3.1-0" contained grocy version 3.3.0

[jayaddison]

Timeline:

• 2022-06-10 - 🚀 grocy v3.3.1 released
• 2022-06-13 - 🔧 container upgrade to v3.3.1-0 prepared - code diff here
• 2022-06-13 - 🔬 QA/testing passes, based on Makefile-built images
• 2022-06-13 - 🚀 container release v3.3.1-0 tagged and released to Docker Hub (GHA job log here)
• 2022-06-13 - ⚠️ We now have a container image tagged as v3.3.1-0 on Docker Hub that contains application version 3.3.0
• 2022-06-25 - 🚨 Update Grocy version in GitHub Actions #167 is received, fixing the incorrect version number
• 2022-06-25 - 🚀 container release v3.3.1-1 tagged and released with the fix in place

What went wrong:

• Human error: not all version numbers in the grocy-docker.git repository were updated before the v3.3.1-0 release took place
• Inconsistency: Testing was performed using a different build process from the container upload build process
• Inconsistency: Our GHA build and upload process proceeds and passes even if the GROCY_VERSION is not a prefix of the git tag being published

Recommended improvements:

• Reduce the number of places where the version number has to be updated - this should lower the risk of human error during version number updates (Reduce the number of files in which the GROCY_IMAGE_TAG value has to be updated during version upgrades #170)
• Investigate using identical build processes for QA and container upload
  • We have three build options: docker, buildah, and GitHub Actions
  • It could be challenging to cater to all workflows and environments, but it is worth seeing if we can do better than we do today
• Ensure that the image build process uses the same grocy version contained in the grocy-docker git tag (GitHub Actions: Determine the grocy application version to bundle based on the image tag to publish #169)
  • This could either be a check that fails if the application version doesn't match the tag, or we could attempt to parse the tag and infer the version from there (edit: we chose the latter approach)

Related: #109 (release automation)

cc @berrnd @manheraz

I also wonder whether there is a communication / participation problem. If I were a more active maintainer, I would probably participate on the grocy subreddit and reply to the release threads to inform people about container updates, perhaps making them more popular and helping to get feedback about issue reports more quickly.

That said: my sense at the moment is that the linuxserver images are more popular, and I'm personally fine with that - it seems likely that they have a more automated release process.

[jayaddison]

Ensure that the image build process uses the same grocy version contained in the grocy-docker git tag

I think we can use GHA scripts to do this by writing to the GITHUB_ENV environment file.

We'd read ${{ github.event.release.tag_name }} as our input value -- in this case, v3.3.1-0 -- and aim to safely output GROCY_VERSION=v3.3.1 into the file.

The buildah-build action that we're using doesn't populate build arguments (like ARG GROCY_VERSION) from environment variables from what I understand, so we need to pass the version to it as an argument.

We can do that, I think, by editing the build-args to supply GROCY_VERSION=${{ env.GROCY_VERSION }}

[jayaddison]

With #169 adding logic to determine the grocy app version from the container git tag, I don't think we should see this happen again.

About this item in the recommendation list:

Investigate using identical build processes for QA and container upload

Using the same build process for QA as publication looks tricky, for various reasons. It may make sense to spend more time on that in future, but I think that the immediate problem here has been resolved.