Support OCI Helm charts

[janeczku]

Storing Helm charts in OCI-compliant container registries is becoming the standard. Fleet needs to support downloading these chart artefacts when referenced in fleet.yaml.

Details: rancher/rancher#29105

[Martin-Weiss]

Just had an other customer using harbor where we store the helm charts using OCI.. so are there any plans for this feature??

[ferhimedamine]

@deniseschannon any updates regarding this feature ?

[SimonAlling]

I decided to give this a shot and tried to upgrade the helm.sh/helm/v3 dependency for starters, after removing the helm.sh/helm/v3 => github.com/rancher/helm/v3 replacement.

📜 What I did

$ sed -i '/rancher\/helm\/v3/d' go.mod

$ go get helm.sh/helm/v3
go: upgraded github.com/evanphx/json-patch v4.9.0+incompatible => v4.12.0+incompatible
go: upgraded github.com/google/go-containerregistry v0.1.1 => v0.5.1
go: upgraded github.com/hashicorp/go-multierror v1.1.0 => v1.1.1
go: upgraded github.com/spf13/cobra v1.1.3 => v1.3.0
go: upgraded golang.org/x/sync v0.0.0-20201207232520-09787c993a3a => v0.0.0-20210220032951-036812b2e83c
go: upgraded helm.sh/helm/v3 v3.5.1 => v3.8.2
go: upgraded k8s.io/api v0.21.3 => v0.23.5
go: upgraded k8s.io/apimachinery v0.21.3 => v0.23.5
go: upgraded k8s.io/cli-runtime v0.21.3 => v0.23.5
go: upgraded sigs.k8s.io/yaml v1.2.0 => v1.3.0

$ make
Building for linux-amd64 ...
go: github.com/deislabs/[email protected] requires
	github.com/Microsoft/[email protected]: missing go.sum entry; to add it:
	go mod download github.com/Microsoft/hcsshim
FATA[0006] exit status 1
make: *** [Makefile:17: default] Error 1

$ go mod download github.com/Microsoft/hcsshim

$ make

At this point, I got literally 100+ missing go.sum entry errors, so I ran all the corresponding suggested go get commands.

$ go get github.com/hashicorp/[email protected]
$ … (~100 go get commands)
$ go get github.com/rancher/fleet/pkg/bundle

$ make
note: module requires Go 1.17
FATA[0006] exit status 2

$ sed -i 's/golang:1\.16\.10/golang:1.17.9/' Dockerfile.dapper

$ sed -i 's/go 1\.16/go 1.17/' go.mod

$ make
go: updates to go.mod needed; to update it:
	go mod tidy
FATA[0008] exit status 1

$ go mod tidy
go: downloading github.com/google/martian v2.1.0+incompatible

$ make
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:34:9: cannot use &logrusr{...} (type *logrusr) as type logr.Logger in return argument
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:69:32: undefined: logr.InfoLogger
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:84:2: cannot use l (type *logrusr) as type logr.Logger in return argument
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:96:2: cannot use l (type *logrusr) as type logr.Logger in return argument
FATA[0011] exit status 2

$ go get github.com/bombsimon/logrusr
go: upgraded github.com/bombsimon/logrusr v1.0.0 => v1.1.0

$ make
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:34:9: cannot use &logrusr{...} (type *logrusr) as type logr.Logger in return argument
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:69:32: undefined: logr.InfoLogger
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:86:2: cannot use newLogger (type *logrusr) as type logr.Logger in return argument
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:99:2: cannot use newLogger (type *logrusr) as type logr.Logger in return argument
FATA[0009] exit status 2

$ sed -i 's/golang:1\.17\.9/golang:1.18.1/' Dockerfile.dapper

$ sed -i 's/go 1\.17/go 1.18/' go.mod

$ make
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:34:9: cannot use &logrusr{…} (value of type *logrusr) as type logr.Logger in return statement
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:69:37: undefined: logr.InfoLogger
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:86:9: cannot use newLogger (variable of type *logrusr) as type logr.Logger in return statement
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:99:9: cannot use newLogger (variable of type *logrusr) as type logr.Logger in return statement
FATA[0189] exit status 2

$ go get -u github.com/bombsimon/logrusr
go: upgraded github.com/go-logr/logr v1.2.2 => v1.2.3
go: upgraded golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e => v0.0.0-20220422013727-9388b58f7150

$ make
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:34:9: cannot use &logrusr{…} (value of type *logrusr) as type logr.Logger in return statement
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:69:37: undefined: logr.InfoLogger
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:86:9: cannot use newLogger (variable of type *logrusr) as type logr.Logger in return statement
/go/pkg/mod/github.com/bombsimon/[email protected]/logrusr.go:99:9: cannot use newLogger (variable of type *logrusr) as type logr.Logger in return statement
FATA[0039] exit status 2

[ghost]

We would need this feature too. Are there any plans?

[manno]

We could use a chart similar to https://github.com/rancher/fleet-examples/tree/master/single-cluster/helm to check if this already works.

[manno]

Just found #187 which describes potentially missing "go-getter" functionality around the helm.repo/helm.chart fields in gitrepo.

[thardeck]

As mentioned in #187 the GitRepo helm.chart value is a go-getter url as described in https://fleet.rancher.io/gitrepo-structure/#reference. The go-getter URL format is described here: https://github.com/hashicorp/go-getter#url-format.

  # Use a custom location for the Helm chart. This can refer to any go-getter URL.
  # This allows one to download charts from most any location.  Also know that
  # go-getter URL supports adding a digest to validate the download. If repo
  # is set below this field is the name of the chart to lookup
  chart: ./chart

OCI is not supported yet in the latest version of go-getter, but there is a feature request for it hashicorp/go-getter#271 and even a PoC pr mentioned.

Helm seems to use the oras library for their oci support.

[deniseschannon]

Where/how was this tested by QA?

[thardeck]

Where/how was this tested by QA?

I think it was not tested yet by QA, the issue got closed automatically after merging the pr. Reopening for now.

I have added an example to fleet-examples, which is also used for our CI test and updated the Repo Structure documentation of Fleet.

Authentication for OCI based registries is not supported yet but will be delivered in a separate pr to not block the release.

Let me know if anything else is missing.

[thardeck]

Authentication for OCI based registries are now supported. The structure of the fleet.yaml is the same, the credentials are provided as Kubernetes secret as described in our Repo Structure documentation in the box Private Helm Repo.

The name of the created Kubernetes secret is specified in the gitRepo.spec.helmSecretName value.

We also have an additional CI test for the OCI registry authentication using GitHub secrets.

Both tests are using GHCR hosted packages. There is a great howto which describes how to use GitHub Container Registry (GHCR) to host your Helm Charts.

I am closing this issue now since it is fixed in master, we have integration tests for both use cases and if issues are found in the implementation by QA or somebody else we should create new issues to address them.

[thardeck]

@rancher/qa

Additionnal QA

Problem

Helm charts provided by OCI registries were not supported.

Solution

We are using the helm library to support OCI registries including basic authentication.

Testing

Reference an OCI based helm chart, like this CI example, and check if the app comes up.

Automated Testing

Our CI runs tests against a public OCI registry example and a private OCI registry example.

[MSpencer87]

Verified on Rancher v2.7.0-rc6 with fleet:101.0.0+up0.5.0-rc3

Steps to reproduce:

1. Bring up HA Rancher on v2.6.8 with fleet:100.0.5+up0.3.11
2. Provision a downstream cluster
3. Navigate to Continuous Delivery -> GitRepos and select Add Repository
4. Add the Fleet Examples repo https://github.com/rancher/fleet-examples.git and set the path for an OCI installation /single-cluster/helm-oci
5. Select the downstream cluster as your target and select Create
6. Repo is showing Git Updating label with error message and 0/0 Clusters Ready

Time="2022-10-25T23:55:08Z" level=fatal msg="download not supported for scheme 'oci'"

Steps for Validation:

1. Bring up HA Rancher on v2.7.0 with fleet:101.0.0+up0.5.0-rc3
2. Provision a downstream cluster
3. Navigate to Continuous Delivery -> GitRepos and select Add Repository
4. Add the Fleet Examples repo https://github.com/rancher/fleet-examples.git and set the path for an OCI installation /single-cluster/helm-oci
5. Select the downstream cluster as your target and select Create
6. The guestbook example is reachable inside the downstream clusters via Rancher UI under Service Discovery->Services

This was also tested successfully via the Rancher upgrade scenario v2.6.8->v2.7.0

1. Fleet is upgraded fleet:100.0.5+up0.3.11->fleet:101.0.0+up0.5.0-rc3 during upgrade
2. The repo showing the Git Updating label with the error message below changes to the Active label with 2/2 Clusters Ready

Time="2022-10-25T23:55:08Z" level=fatal msg="download not supported for scheme 'oci'"

3. The guestbook example is reachable inside the downstream clusters via Rancher UI under Service Discovery->Services