Evaluate Kaniko and other solutions for multi-platform builds

[Sawthis]

• Check if the Docker container is certified one.
• Evaluate Kaniko and other certified solutions such as Docker for multi-platform builds (docker buildx build --platform=linux/amd64,linux/arm64 .).
• Determine the best tool for implementation in the image builder hosted on Azure DevOps.
• Check how BDBA deals with the multi-architecture images.
• Check how Signify deals with the multi-architecture images.
• Document the findings and rationale behind the chosen tool.

Estimate: 3 days

[Sawthis]

Hi Wojciech, 
for multi-arch images the tricky part is that you have to sign (and later verify) the whole manifest-list.json
 
Universal command for getting the manifest of an image (standard or multiplatform) as a raw json:
 
docker buildx imagetools inspect --raw europe-docker.pkg.dev/kyma-project/prod/keda-manager:1.3.0
 
docker buildx imagetools inspect --raw docker.io/library/ubuntu:22.04
 
For multi-arch builds, e.g.
 
docker manifest inspect docker.io/library/ubuntu:22.04
 
returns a list of manifests instead of a single manifest… in that case you can use
 
docker buildx imagetools inspect --raw docker.io/library/ubuntu:22.04 > Desktop\manifest-list.json
 
and then
 
certutil -hashfile Desktop\manifest-list.json SHA256
 
to find the checksum of the raw manifest list… this is also the checksum of the image digest returned in the output of the command
 
docker image pull docker.io/library/ubuntu:22.04
 
Note: docker.io/library/ubuntu:22.04 is the fully qualified image name of the shorter ubuntu:22.04

CC: @kyma-project/otters

[KacperMalachowski]

Using the buildx, we need to switch to containerd image store, see: https://docs.docker.com/build/building/multi-platform/#prerequisites

docker buildx build --file ../../../github.com/test-infra/cmd/image-detector/Dockerfile --metadata-file=metadata.txt --iidfile=iid.txt --platform=linux/arm64,linux/amd64 ../../../github.com/test-infra
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                                                                                                                                                                                               docker:default
ERROR: Multi-platform build is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/build-multi-platform/

[KacperMalachowski]

Success scan of image: https://bdba.tools.sap/#/product/2375602/details/analysis-settings

[KacperMalachowski]

POC Summary

Choosen Tool

Docker Buildx

FIndings

• Signify client implementation from image-builder client signs only the first manifest from manifests-list. We need to extend that to support manifests-list.
• Docker Buildx requires creation of new docker buidler instance using docker buildx create --use, see: https://docs.docker.com/reference/cli/docker/buildx/create/
• Docker Buildx requires preauthentication to docker registry using gcloud
• We have sudo access to runners, registry mirror can be configured directly on daemon level, see the tests pipeline

Rationale

Docker Buildx is certified tool to build docker images in SAP. It's also native builder, provided by Docker.
It supports exporting image digest using --iidfile <path>, which is what we currently using to provide image build report for SRE team. It can also directly push built image into docker registry using --push flag.

Implementing builds using docker buildx also simplify passing platforms as we do not need anymore to split it into separate flags. See https://github.tools.sap/kyma/oci-image-builder/pull/192/files#diff-04f131daa1262d0246e779a526a6fb48df09743a6c3abfc70de5baa694e597daL248-R248

You can see required changes in https://github.tools.sap/kyma/oci-image-builder/pull/192/files, which was used to test the buildx implementation.

Custom buildxpacks requires additional pack CLI to use, kaniko requires hosting of custom agents as we do not have currently any with arm architetcures.