Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor issues with the k8s template #2992

Closed
afbjorklund opened this issue Dec 7, 2024 · 6 comments
Closed

Minor issues with the k8s template #2992

afbjorklund opened this issue Dec 7, 2024 · 6 comments

Comments

@afbjorklund
Copy link
Member

afbjorklund commented Dec 7, 2024

Description

Trying out k8s 1.32 RC, found some issues:

  • Flannel was a bit outdated (PR added)

#2991

  • kubeadm config version was updated
[  191.575331] cloud-init[1077]: + kubeadm init --config kubeadm-config.yaml
[  191.605187] cloud-init[1077]: W1207 10:05:49.428211    3159 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3"

#2996

  • sandbox image is not configured properly
  • still using the old containerd config version (v2)
[  194.231682] cloud-init[1077]: [preflight] Running pre-flight checks
[  194.360972] cloud-init[1077]: [preflight] Pulling images required for setting up a Kubernetes cluster
[  194.361128] cloud-init[1077]: [preflight] This might take a minute or two, depending on the speed of your internet connection
[  194.361354] cloud-init[1077]: [preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[  194.365795] cloud-init[1077]: W1207 10:05:52.188987    3159 checks.go:846] detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.

detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm

containerd/containerd#11114
kubernetes/website#48976


EDIT: It did work OK, just complained in the logs.

Switching k8s versions involves hacking the yaml.

@afbjorklund
Copy link
Member Author

afbjorklund commented Dec 7, 2024

Actual the sandbox was configured, so the problem seems to be with containerd 2.0 CRI and kubernetes 1.31

crictl info

// SandboxImage returns the sandbox image used by the container runtime
func (runtime *CRIRuntime) SandboxImage() (string, error) {
        ctx, cancel := defaultContext()
        defer cancel()
        status, err := runtime.impl.Status(ctx, runtime.runtimeService, true)
        if err != nil {
                return "", errors.Wrap(err, "failed to get runtime status")
        }

        infoConfig, ok := status.GetInfo()["config"]
        if !ok {
                return "", errors.Errorf("no 'config' field in CRI info: %+v", status)
        }

        type config struct {
                SandboxImage string `json:"sandboxImage,omitempty"`
        }
        c := config{}

        if err := json.Unmarshal([]byte(infoConfig), &c); err != nil {
                return "", errors.Wrap(err, "failed to unmarshal CRI info config")
        }

        return c.SandboxImage, nil
}

Maybe they shouldn't have used the semi-internal "config", but a proper CRI method to communicate it?

containerd/containerd@3baf5ed

@afbjorklund
Copy link
Member Author

afbjorklund commented Dec 8, 2024

The new config format (v3) is not documented upstream yet, they only do v2:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

https://github.com/containerd/containerd/blob/main/docs/cri/config.md

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Nice touch of documenting an ancient version.

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = 'registry.k8s.io/pause:3.10'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

Since it has been moved to "images", it is no longer visible in the "runtime" status.

                configByt, err := json.Marshal(c.config)
...
                resp.Info = make(map[string]string)
                resp.Info["config"] = string(configByt)

@afbjorklund
Copy link
Member Author

@afbjorklund
Copy link
Member Author

Probably these things are never seen upstream, since kind disables the preflight <sigh>

@afbjorklund
Copy link
Member Author

Issues have been opened in the upstream projects, so nothing more to do in Lima...

And it is only a couple of lines of log warnings now, who looks at those anyway right?

Configuration migrated from version 2, use containerd config migrate to avoid migration
WARN[0000] Ignoring unknown key in TOML for plugin error="strict mode: fields in the document are missing in the target struct" key=sandbox_image plugin=io.containerd.grpc.v1.cri

detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.


It does work fine, and the configuration is the correct one (despite all the false complaints)

[  155.739859] cloud-init[1103]: [init] Using Kubernetes version: v1.32.0
[  155.740445] cloud-init[1103]: [preflight] Running pre-flight checks
[  155.819598] cloud-init[1103]: [preflight] Pulling images required for setting up a Kubernetes cluster
[  155.820668] cloud-init[1103]: [preflight] This might take a minute or two, depending on the speed of your internet connection
[  155.822049] cloud-init[1103]: [preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[  155.823408] cloud-init[1103]: W1214 10:00:30.977242    3115 checks.go:846] detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.
anders@lima-k8s:~$ sudo containerd config dump
...
    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.k8s.io/pause:3.10'
...
CONTAINER ID    IMAGE                                              COMMAND                   CREATED          STATUS    PORTS    NAMES
7d4ce40061bc    registry.k8s.io/pause:3.10                         "/pause"                  5 minutes ago    Up                 k8s://kube-system/coredns-668d6bf9bc-bdpdb
cb9846518450    registry.k8s.io/pause:3.10                         "/pause"                  5 minutes ago    Up                 k8s://kube-system/coredns-668d6bf9bc-vthwq
2d5d56cf0a6d    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-flannel/kube-flannel-ds-9m4gp
bbd8907d1685    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-proxy-sqc7t
ed7594cd9743    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-scheduler-lima-k8s
c7de7bee144f    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-controller-manager-lima-k8s
4b179d0c0b6c    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-apiserver-lima-k8s
833a01c3900a    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/etcd-lima-k8s

There are other known issues at startup anyway, like the kubelet crash-looping (because it is started too soon, before being configured and before the cluster networking has been set up - which is supposed to pushed from CNI to CRI)

@afbjorklund
Copy link
Member Author

This output seems redundant as well, ideally kubeadm should stay quiet - since we already did that step.

[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'

Again this is something for upstream, one would add a exist-only step (depending on the pull policy)...

        if data.DryRun() {
                fmt.Println("[preflight] Would pull the required images (like 'kubeadm config images pull')")
                return nil
        }

        // new code goes here:
        // if preflight.RunImagesExistCheck(utilsexec.New(), data.Cfg(), data.IgnorePreflightErrors()) == nil {
        //        return nil
        //}

        fmt.Println("[preflight] Pulling images required for setting up a Kubernetes cluster")
        fmt.Println("[preflight] This might take a minute or two, depending on the speed of your internet connection")
        fmt.Println("[preflight] You can also perform this action beforehand using 'kubeadm config images pull'")
        return preflight.RunPullImagesCheck(utilsexec.New(), data.Cfg(), data.IgnorePreflightErrors())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant