runc create failed: unable to start container process: exec: \"/cluster/kube-apiserver-operator\": executable file not found in $PATH

[Syx5290]

Description

Hi, I've encountered a weird issue during cluster creation where some nodes frequently experience the following situations:

1. level=error msg="Container creation error: runc create failed: unable to start container process: exec: \"/bin/prometheus-config-reloader\": permission denied"

2. level=error msg="Container creation error: runc create failed: unable to start container process: exec: \"/cluster/kube-apiserver-operator\": executable file not found in $PATH"

3. level=error msg="Container creation error: runc create failed: unable to start container process: exec: \"/usr/bin/pod\": stat /usr/bin/pod: no such file or directory"

These container creation failures occur when non-root users execute the files, which have executable permissions set to 755.

What puzzles me most is that after restarting the pod, the containers are created successfully. This issue only occurs repeatedly during the cluster creation process.

Steps to reproduce the issue

The problem occurs frequently after upgrading cri-o from version 1.25.4 to 1.29.13

crio version 1.25.4
Version:        1.25.4
GitCommit:      unknown
GitCommitDate:  unknown
GitTreeState:   clean
GoVersion:      go1.18.4

crio version 1.29.13
Version:        1.29.13
GitCommit:      unknown
GitCommitDate:  unknown
GitTreeState:   clean 
GoVersion:      go1.21.11

Describe the results you received and expected

Container creation should normal.

What version of runc are you using?

runc --version
runc version 1.1.12
spec: 1.0.2-dev
go: go1.18.4
libseccomp: 2.5.2

Host OS information

No response

Host kernel information

uname -a

5.15.131-18.cl9.x86_64

[Syx5290]

I've found a similar issue #3715 where the container runs as a non-root user. The executable there has 744 permissions and retains the default CAP_DAC_OVERRIDE capability, and that problem is reproducible every time.

In my case, the container also runs as a non-root user, but the executable has 755 permissions, and the capabilities are set to drop_all. My issue occurs frequently but intermittently.

[Syx5290]

This is the runc state

# runc state 2eb6e9c6a847b1b27162428dd59c4889be74550609b0167e8bac67fd638d8f56

{
  "ociVersion": "1.0.2-dev",
  "id": "2eb6e9c6a847b1b27162428dd59c4889be74550609b0167e8bac67fd638d8f56",
  "pid": 0,
  "status": "stopped",
  "bundle": "/run/containers/storage/overlay-containers/2eb6e9c6a847b1b27162428dd59c4889be74550609b0167e8bac67fd638d8f56/userdata",
  "rootfs": "/data/ceake/imagefs/overlay/26b9505d448a853995b768712269f9ac27140cf7874514c3d5c8822281405ee6/merged",
  "created": "2025-04-03T07:34:15.742786039Z",
  "annotations": {
    "io.container.manager": "cri-o",
    "io.kubernetes.container.hash": "1763c6e3",
    "io.kubernetes.container.name": "kube-apiserver-operator",
    "io.kubernetes.container.ports": "[{\"name\":\"metrics\",\"containerPort\":8443,\"protocol\":\"TCP\"}]",
    "io.kubernetes.container.restartCount": "5",
    "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
    "io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
    "io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"1763c6e3\",\"io.kubernetes.container.ports\":\"[{\\\"name\\\":\\\"metrics\\\",\\\"containerPort\\\":8443,\\\"protocol\\\":\\\"TCP\\\"}]\",\"io.kubernetes.container.restartCount\":\"5\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"30\"}",
    "io.kubernetes.cri-o.ContainerID": "2eb6e9c6a847b1b27162428dd59c4889be74550609b0167e8bac67fd638d8f56",
    "io.kubernetes.cri-o.ContainerType": "container",
    "io.kubernetes.cri-o.Created": "2025-04-03T15:34:15.622503908+08:00",
    "io.kubernetes.cri-o.IP.0": "21.100.166.2",
    "io.kubernetes.cri-o.Image": "e17e7fba7ae1ad615bad5fece01109fac726bc3b5737993e5f33a2dccc26a01a",
    "io.kubernetes.cri-o.ImageName": "image.test.io/ceake/ceake@sha256:0a93f13da1adab6b708342469585c7380ce364b874a2a2b99294d4a9876db166",
    "io.kubernetes.cri-o.ImageRef": "e17e7fba7ae1ad615bad5fece01109fac726bc3b5737993e5f33a2dccc26a01a",
    "io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"kube-apiserver-operator\",\"io.kubernetes.pod.name\":\"kube-apiserver-operator-6bd4fb55db-vpj97\",\"io.kubernetes.pod.namespace\":\"test-kube-apiserver-operator\",\"io.kubernetes.pod.uid\":\"a65dde94-ada0-4bfa-847b-a114d7b69782\"}",
    "io.kubernetes.cri-o.LogPath": "/var/log/pods/test-kube-apiserver-operator_kube-apiserver-operator-6bd4fb55db-vpj97_a65dde94-ada0-4bfa-847b-a114d7b69782/kube-apiserver-operator/5.log",
    "io.kubernetes.cri-o.Metadata": "{\"name\":\"kube-apiserver-operator\",\"attempt\":5}",
    "io.kubernetes.cri-o.MountPoint": "/data/ceake/imagefs/overlay/26b9505d448a853995b768712269f9ac27140cf7874514c3d5c8822281405ee6/merged",
    "io.kubernetes.cri-o.Name": "k8s_kube-apiserver-operator_kube-apiserver-operator-6bd4fb55db-vpj97_test-kube-apiserver-operator_a65dde94-ada0-4bfa-847b-a114d7b69782_5",
    "io.kubernetes.cri-o.PlatformRuntimePath": "",
    "io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/7a4192fa4525782c87dc9a02ef8f7fc6f5b79faad7480e96b602304ed29a7bae/userdata/resolv.conf",
    "io.kubernetes.cri-o.SandboxID": "7a4192fa4525782c87dc9a02ef8f7fc6f5b79faad7480e96b602304ed29a7bae",
    "io.kubernetes.cri-o.SandboxName": "k8s_kube-apiserver-operator-6bd4fb55db-vpj97_test-kube-apiserver-operator_a65dde94-ada0-4bfa-847b-a114d7b69782_0",
    "io.kubernetes.cri-o.SeccompProfilePath": "RuntimeDefault",
    "io.kubernetes.cri-o.Stdin": "false",
    "io.kubernetes.cri-o.StdinOnce": "false",
    "io.kubernetes.cri-o.TTY": "false",
    "io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/a65dde94-ada0-4bfa-847b-a114d7b69782/etc-hosts\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/a65dde94-ada0-4bfa-847b-a114d7b69782/containers/kube-apiserver-operator/6839de36\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/var/run/configmaps/config\",\"host_path\":\"/var/lib/kubelet/pods/a65dde94-ada0-4bfa-847b-a114d7b69782/volumes/kubernetes.io~configmap/config\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/var/run/secrets/serving-cert\",\"host_path\":\"/var/lib/kubelet/pods/a65dde94-ada0-4bfa-847b-a114d7b69782/volumes/kubernetes.io~secret/serving-cert\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/a65dde94-ada0-4bfa-847b-a114d7b69782/volumes/kubernetes.io~projected/kube-api-access\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true}]",
    "io.kubernetes.pod.name": "kube-apiserver-operator-6bd4fb55db-vpj97",
    "io.kubernetes.pod.namespace": "test-kube-apiserver-operator",
    "io.kubernetes.pod.terminationGracePeriod": "30",
    "io.kubernetes.pod.uid": "a65dde94-ada0-4bfa-847b-a114d7b69782",
    "kubernetes.io/config.seen": "2025-04-03T15:09:29.819544366+08:00",
    "kubernetes.io/config.source": "api",
    "org.systemd.property.After": "['crio.service']",
    "org.systemd.property.DefaultDependencies": "true",
    "org.systemd.property.TimeoutStopUSec": "uint64 30000000"
  },
  "owner": ""
}

[rata]

Interesting.

I understand you can repro there. I can think of the following things to try:

1. Try with runc latest 1.1, latest 1.2, 1.3.0-rc.2 (will be cut soon, but you can compile the release-1.3 branch if you want) and main. If it's not too hard to repro, this info can be valuable for debugging and nailing it down. Otherwise, just latest 1.1.x, latest 1.2.x and main are also enough :)
2. If you can create a repro (even if it is running this container in a loop until it fails), that would be great. You can follow these steps to create a runc container directly. Then, you can change permissions, the config.json, etc. to make it match to your and see if you can copy enough until you can repro the issue

To know what config.json crio is currently doing, you can do a hack like: create a bash script that checks the params and when it is a create, you copy the config.json and then call runc:

echo "Getting param ${9}"  >> /tmp/rata.log
if [ "${9}" = "--bundle" ]; then
       echo "Getting config.json"  >> /tmp/rata.log
       mkdir -p /tmp/rata-debug-k8s/
       cp -ar "${10}" "/tmp/rata-debug-k8s/$$/"
       echo "Getting bundle: ${10}" >> /tmp/rata.log
fi

exec <path-to>/runc "$@"

IIRC crio might use this order. But maybe the bundle param is not 9. This is the gist, it might need adjustments.

If you can do both things, it would be great! :)

[rata]

@kolyshkin You were dealing with the linked "similar" issue, do you have any idea in mind?

[ningmingxiao]

exec:bin/prometheus-config is a hook? you can print config.json. @xuegege5290

[Syx5290]

Interesting.

I understand you can repro there. I can think of the following things to try:

1. Try with runc latest 1.1, latest 1.2, 1.3.0-rc.2 (will be cut soon, but you can compile the release-1.3 branch if you want) and main. If it's not too hard to repro, this info can be valuable for debugging and nailing it down. Otherwise, just latest 1.1.x, latest 1.2.x and main are also enough :)
2. If you can create a repro (even if it is running this container in a loop until it fails), that would be great. You can follow these steps to create a runc container directly. Then, you can change permissions, the config.json, etc. to make it match to your and see if you can copy enough until you can repro the issue

Thanks for your suggestions. I'm currently validating the first method, and I'll try the second approach as well.
I'm not sure if it's related to the ociVersion. I noticed that the ociVersion range for runc is 1.0.0-1.0.2-dev.
However, the config.json created by crio 1.29 shows ociVersion 1.1.0-dev.

[Syx5290]

It happened again yesterday with lots of container failures. I grabbed the config.json from two containers. @rata @ningmingxiao

[root@spheresp20-node-3 userdata]# cat config.json |jq
{
  "ociVersion": "1.1.0+dev",
  "process": {
    "user": {
      "uid": 65532,
      "gid": 0,
      "additionalGids": [
        0
      ]
    },
    "args": [
      "/account-operator",
      "--leader-elect"
    ],
    "env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "TERM=xterm",
      "HOSTNAME=account-operator-controller-manager-6964f8b87f-4vjbg",
      "NSS_SDB_USE_CACHE=no",
      "GlobalBaseDomain=apps.spheresp20.spheresp20.io",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT=8443",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT=tcp://21.101.240.77:8443",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO=tcp",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR=21.101.240.77",
      "KUBERNETES_PORT_443_TCP_PROTO=tcp",
      "KUBERNETES_PORT_443_TCP_ADDR=21.101.0.1",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP=tcp://21.101.240.77:8443",
      "KUBERNETES_SERVICE_PORT_HTTPS=443",
      "KUBERNETES_PORT_443_TCP=tcp://21.101.0.1:443",
      "KUBERNETES_PORT_443_TCP_PORT=443",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_HOST=21.101.240.77",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT_HTTPS=8443",
      "ACCOUNT_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PORT=8443",
      "KUBERNETES_SERVICE_HOST=21.101.0.1",
      "KUBERNETES_SERVICE_PORT=443",
      "KUBERNETES_PORT=tcp://21.101.0.1:443",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "cwd": "/",
    "capabilities": {
      "bounding": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FSETID",
        "CAP_FOWNER",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETPCAP",
        "CAP_NET_BIND_SERVICE",
        "CAP_KILL"
      ],
      "effective": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FSETID",
        "CAP_FOWNER",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETPCAP",
        "CAP_NET_BIND_SERVICE",
        "CAP_KILL"
      ],
      "permitted": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FSETID",
        "CAP_FOWNER",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETPCAP",
        "CAP_NET_BIND_SERVICE",
        "CAP_KILL"
      ]
    },
    "noNewPrivileges": true,
    "oomScoreAdj": 999
  },
  "root": {
    "path": "/var/lib/containers/storage/overlay/90a64202a302fb68a8dc9bd06be6faa7d01815279ee774f7406d9d88486af542/merged"
  },
  "hostname": "account-operator-controller-manager-6964f8b87f-4vjbg",
  "mounts": [
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/sys",
      "type": "sysfs",
      "source": "sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/sys/fs/cgroup",
      "type": "cgroup",
      "source": "cgroup",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "relatime",
        "ro"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/2d3909fc46db37015bf790adf46d071998c5f291da318ce49624d8788595c6e6/userdata/shm",
      "options": [
        "rw",
        "bind"
      ]
    },
    {
      "destination": "/etc/resolv.conf",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/2d3909fc46db37015bf790adf46d071998c5f291da318ce49624d8788595c6e6/userdata/resolv.conf",
      "options": [
        "rw",
        "bind",
        "nodev",
        "nosuid",
        "noexec"
      ]
    },
    {
      "destination": "/etc/hostname",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/2d3909fc46db37015bf790adf46d071998c5f291da318ce49624d8788595c6e6/userdata/hostname",
      "options": [
        "rw",
        "bind"
      ]
    },
    {
      "destination": "/run/.containerenv",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/2d3909fc46db37015bf790adf46d071998c5f291da318ce49624d8788595c6e6/userdata/.containerenv",
      "options": [
        "rw",
        "bind"
      ]
    },
    {
      "destination": "/etc/passwd",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/51b1777092afb9d7d1e86a9777012bdee11bcfb888c7c2ba4ddfc8b22c6f817b/userdata/passwd",
      "options": [
        "rw",
        "bind",
        "nodev",
        "nosuid",
        "noexec"
      ]
    },
    {
      "destination": "/etc/hosts",
      "type": "bind",
      "source": "/var/lib/kubelet/pods/500bc8c8-0936-4e13-ae64-219f7e07b702/etc-hosts",
      "options": [
        "rw",
        "rbind",
        "rprivate",
        "bind"
      ]
    },
    {
      "destination": "/dev/termination-log",
      "type": "bind",
      "source": "/var/lib/kubelet/pods/500bc8c8-0936-4e13-ae64-219f7e07b702/containers/manager/ff157364",
      "options": [
        "rw",
        "rbind",
        "rprivate",
        "bind"
      ]
    },
    {
      "destination": "/run/secrets",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/51b1777092afb9d7d1e86a9777012bdee11bcfb888c7c2ba4ddfc8b22c6f817b/userdata/run/secrets",
      "options": [
        "bind",
        "rprivate",
        "bind"
      ]
    },
    {
      "destination": "/var/run/secrets/kubernetes.io/serviceaccount",
      "type": "bind",
      "source": "/var/lib/kubelet/pods/500bc8c8-0936-4e13-ae64-219f7e07b702/volumes/kubernetes.io~projected/kube-api-access-l7tbv",
      "options": [
        "ro",
        "rbind",
        "rprivate",
        "bind"
      ]
    }
  ],
  "annotations": {
    "test.io/scc": "anyuid",
    "cni.projectcalico.org/containerID": "77b0b0954b8fa698fad621831f3aaf578cc732bf83a5bfac99b22f43d43aeb63",
    "cni.projectcalico.org/podIP": "21.100.19.25/32",
    "cni.projectcalico.org/podIPs": "21.100.19.25/32",
    "io.container.manager": "cri-o",
    "io.kubernetes.container.hash": "b07cae3e",
    "io.kubernetes.container.name": "manager",
    "io.kubernetes.container.restartCount": "1",
    "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
    "io.kubernetes.container.terminationMessagePolicy": "File",
    "io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"b07cae3e\",\"io.kubernetes.container.restartCount\":\"1\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"File\",\"io.kubernetes.pod.terminationGracePeriod\":\"10\"}",
    "io.kubernetes.cri-o.ContainerID": "51b1777092afb9d7d1e86a9777012bdee11bcfb888c7c2ba4ddfc8b22c6f817b",
    "io.kubernetes.cri-o.ContainerType": "container",
    "io.kubernetes.cri-o.Created": "2025-04-08T13:12:09.649392347+08:00",
    "io.kubernetes.cri-o.IP.0": "21.100.18.1",
    "io.kubernetes.cri-o.Image": "20ab258d430b3bfec05c273e4df8516ca193fcc8ce00fe5720dc1fa4a2836e7a",
    "io.kubernetes.cri-o.ImageName": "image.test.io/test-ceasphere/ceasphere@sha256:b0c3e00ada1489ad9840df6e9150ba0bfc20927125bd5f6fcf814b8baf10e25b",
    "io.kubernetes.cri-o.ImageRef": "20ab258d430b3bfec05c273e4df8516ca193fcc8ce00fe5720dc1fa4a2836e7a",
    "io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"manager\",\"io.kubernetes.pod.name\":\"account-operator-controller-manager-6964f8b87f-4vjbg\",\"io.kubernetes.pod.namespace\":\"account-operator\",\"io.kubernetes.pod.uid\":\"500bc8c8-0936-4e13-ae64-219f7e07b702\"}",
    "io.kubernetes.cri-o.LogPath": "/var/log/pods/account-operator_account-operator-controller-manager-6964f8b87f-4vjbg_500bc8c8-0936-4e13-ae64-219f7e07b702/manager/1.log",
    "io.kubernetes.cri-o.Metadata": "{\"name\":\"manager\",\"attempt\":1}",
    "io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/90a64202a302fb68a8dc9bd06be6faa7d01815279ee774f7406d9d88486af542/merged",
    "io.kubernetes.cri-o.Name": "k8s_manager_account-operator-controller-manager-6964f8b87f-4vjbg_account-operator_500bc8c8-0936-4e13-ae64-219f7e07b702_1",
    "io.kubernetes.cri-o.PlatformRuntimePath": "",
    "io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/2d3909fc46db37015bf790adf46d071998c5f291da318ce49624d8788595c6e6/userdata/resolv.conf",
    "io.kubernetes.cri-o.SandboxID": "2d3909fc46db37015bf790adf46d071998c5f291da318ce49624d8788595c6e6",
    "io.kubernetes.cri-o.SandboxName": "k8s_account-operator-controller-manager-6964f8b87f-4vjbg_account-operator_500bc8c8-0936-4e13-ae64-219f7e07b702_0",
    "io.kubernetes.cri-o.SeccompProfilePath": "Unconfined",
    "io.kubernetes.cri-o.Stdin": "false",
    "io.kubernetes.cri-o.StdinOnce": "false",
    "io.kubernetes.cri-o.TTY": "false",
    "io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/500bc8c8-0936-4e13-ae64-219f7e07b702/etc-hosts\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":false},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/500bc8c8-0936-4e13-ae64-219f7e07b702/containers/manager/ff157364\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":false},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/500bc8c8-0936-4e13-ae64-219f7e07b702/volumes/kubernetes.io~projected/kube-api-access-l7tbv\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":false}]",
    "io.kubernetes.pod.name": "account-operator-controller-manager-6964f8b87f-4vjbg",
    "io.kubernetes.pod.namespace": "account-operator",
    "io.kubernetes.pod.terminationGracePeriod": "10",
    "io.kubernetes.pod.uid": "500bc8c8-0936-4e13-ae64-219f7e07b702",
    "k8s.v1.cni.cncf.io/network-status": "[{\n    \"name\": \"k8s-pod-network\",\n    \"ips\": [\n        \"21.100.19.25\"\n    ],\n    \"default\": true,\n    \"dns\": {}\n}]",
    "k8s.v1.cni.cncf.io/networks-status": "[{\n    \"name\": \"k8s-pod-network\",\n    \"ips\": [\n        \"21.100.19.25\"\n    ],\n    \"default\": true,\n    \"dns\": {}\n}]",
    "kubectl.kubernetes.io/default-container": "account-operator",
    "kubernetes.io/config.seen": "2025-04-08T13:11:51.003935123+08:00",
    "kubernetes.io/config.source": "api",
    "org.systemd.property.After": "['crio.service']",
    "org.systemd.property.DefaultDependencies": "true",
    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
  },
  "linux": {
    "resources": {
      "devices": [
        {
          "allow": false,
          "access": "rwm"
        }
      ],
      "memory": {
        "limit": 1073741824,
        "swap": 1073741824
      },
      "cpu": {
        "shares": 20,
        "quota": 100000,
        "period": 100000,
        "cpus": "0,5-127"
      },
      "pids": {
        "limit": -1
      },
      "hugepageLimits": [
        {
          "pageSize": "16GB",
          "limit": 0
        },
        {
          "pageSize": "2MB",
          "limit": 0
        },
        {
          "pageSize": "512MB",
          "limit": 0
        }
      ]
    },
    "cgroupsPath": "kubepods-burstable-pod500bc8c8_0936_4e13_ae64_219f7e07b702.slice:crio:51b1777092afb9d7d1e86a9777012bdee11bcfb888c7c2ba4ddfc8b22c6f817b",
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "network",
        "path": "/var/run/netns/60b4afcc-43f9-4476-a9a5-be6fc242d7c0"
      },
      {
        "type": "ipc",
        "path": "/var/run/ipcns/60b4afcc-43f9-4476-a9a5-be6fc242d7c0"
      },
      {
        "type": "uts",
        "path": "/var/run/utsns/60b4afcc-43f9-4476-a9a5-be6fc242d7c0"
      },
      {
        "type": "mount"
      }
    ],
    "maskedPaths": [
      "/proc/asound",
      "/proc/acpi",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/proc/scsi",
      "/sys/firmware"
    ],
    "readonlyPaths": [
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ]
  }
}

[Syx5290]

exec:bin/prometheus-config is a hook? you can print config.json. @xuegege5290

No, this is the executable file that the container needs to run as the first process.

[rata]

@xuegege5290 thanks! Great, can you try to trim that config.json and try with the instructions here if you can create a repro using the runc command? You will need to remove probably the cgroup path, all the path component to linux.namespaces (it will just create a new namespace), and maybe remove some mounts (like etc/hosts and friends, that probably don't exist anymore).

[Syx5290]

Another container crio log

```bash

Apr 09 16:59:19 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:19.262739114+08:00" level=info msg="Creating container: test-cnv/hci-compute-controller-65f8cb8b96-mh8fj/kube-rbac-proxy" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:20 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:20.635657575+08:00" level=error msg="Container creation error: time="2025-04-09T16:59:20+08:00" level=error msg="runc create failed: unable to start container process: exec: \"/usr/bin/kube-rbac-proxy\": stat /usr/bin/kube-rbac-proxy: permission denied"\n" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:20 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:20.641673347+08:00" level=info msg="createCtr: deleting container ID e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9 from idIndex" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:20 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:20.641719854+08:00" level=info msg="createCtr: removing container e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:20 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:20.641756420+08:00" level=info msg="createCtr: deleting container e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9 from storage" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:20 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:20.643652421+08:00" level=error msg="Failed to cleanup (probably retrying): failed to cleanup container storage: removing mount point "/var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged": directory not empty" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:21 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:21.143760640+08:00" level=info msg="createCtr: deleting container e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9 from storage" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 16:59:21 sphere17-node-4 crio[7897]: time="2025-04-09 16:59:21.144139759+08:00" level=error msg="Failed to cleanup (probably retrying): failed to cleanup container storage: removing mount point "/var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged": directory not empty" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
......
Apr 09 17:36:16 sphere17-node-4 crio[7897]: time="2025-04-09 17:36:16.501029757+08:00" level=error msg="Retried cleanup function "createCtr: deleting container e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9 from storage" too often, giving up" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer
Apr 09 17:36:16 sphere17-node-4 crio[7897]: time="2025-04-09 17:36:16.501043728+08:00" level=error msg="Unable to cleanup: wait on retry: timed out waiting for the condition" id=75fd4ca0-4b0a-4f5f-b8dd-1e24045c1cae name=/runtime.v1.RuntimeService/CreateContainer

The corresponding container's config.json

```json

{
"ociVersion": "1.1.0+dev",
"process": {
"user": {
"uid": 65534,
"gid": 65534,
"additionalGids": [
65534
]
},
"args": [
"/usr/bin/kube-rbac-proxy",
"--secure-listen-address=0.0.0.0:8443",
"--upstream=http://127.0.0.1:8080/",
"--logtostderr=true",
"--v=0"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=hci-compute-controller-65f8cb8b96-mh8fj",
"NSS_SDB_USE_CACHE=no",
"VIC_CONTROLLER_MANAGER_PORT_30001_TCP=tcp://21.101.219.154:30001",
"KUBEVIRT_OPERATOR_WEBHOOK_SERVICE_PORT_WEBHOOKS=443",
"KUBEVIRT_OPERATOR_WEBHOOK_PORT_443_TCP_PORT=443",
"YUM_REPO_SERVICE_PORT=tcp://21.101.126.246:80",
"VIC_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=21.101.42.180",
"VIC_IMAGE_REGISTRY_SERVICE_PORT_HTTPS=443",
"VIC_IMAGE_REGISTRY_PORT_443_TCP_PROTO=tcp",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_PORT=443",
"NODE_MAINTENANCE_OPERATOR_SERVICE_SERVICE_PORT_8443=8443",
"COMPUTE_SERVICE_PORT_443_TCP_PORT=443",
"YUM_REPO_SERVICE_PORT_80_TCP_ADDR=21.101.126.246",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_PORT_AWEBHOOK=543",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP=tcp://21.101.191.219:443",
"VIC_IMAGE_REGISTRY_SERVICE_PORT=443",
"KUBERNETES_SERVICE_HOST=21.101.0.1",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_PORT_VWEBHOOK=443",
"CCM_PROTECTOR_SERVICE_PORT=8086",
"COMPUTE_SERVICE_PORT_443_TCP=tcp://21.101.45.184:443",
"CDI_API_SERVICE_PORT=443",
"COMPUTE_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP_PORT=8443",
"VIRT_API_SERVICE_PORT=443",
"VIRT_API_PORT=tcp://21.101.195.118:443",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT=tcp://21.101.200.153:8443",
"YUM_REPO_SERVICE_SERVICE_HOST=21.101.126.246",
"HCI_REST_SERVICE_PORT_443_TCP_PROTO=tcp",
"COMPUTE_SERVICE_PORT_443_TCP_ADDR=21.101.45.184",
"KUBEVIRT_PROMETHEUS_METRICS_PORT=tcp://21.101.104.104:443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_PORT=443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP_PROTO=tcp",
"HCI_REST_SERVICE_PORT_443_TCP=tcp://21.101.39.40:443",
"NODE_MAINTENANCE_OPERATOR_SERVICE_SERVICE_PORT=8443",
"PXE_CNV_SERVICE_PORT=443",
"HCO_WEBHOOK_SERVICE_PORT_4343_TCP_ADDR=21.101.220.99",
"DOWNUPLOAD_SERVICE_PORT_443_TCP_PROTO=tcp",
"VIC_WEBHOOK_SERVICE_PORT=tcp://21.101.42.180:443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_HOST=21.101.44.4",
"HCO_WEBHOOK_SERVICE_SERVICE_PORT=4343",
"DOWNUPLOAD_SERVICE_PORT_443_TCP_PORT=443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp",
"VIC_WEBHOOK_SERVICE_PORT_443_TCP=tcp://21.101.42.180:443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP_PORT=543",
"VIC_CONTROLLER_MANAGER_SERVICE_PORT=30001",
"COMPUTE_CONTROLLER_METRICS_SERVICE_SERVICE_PORT_HTTPS=8443",
"DOWNUPLOAD_SERVICE_SERVICE_PORT=443",
"KUBEVIRT_PROMETHEUS_METRICS_SERVICE_PORT=443",
"KUBEVIRT_PROMETHEUS_METRICS_PORT_443_TCP=tcp://21.101.104.104:443",
"VIC_WEBHOOK_SERVICE_SERVICE_PORT=443",
"HCI_REST_SERVICE_PORT_443_TCP_ADDR=21.101.39.40",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443",
"NODE_MAINTENANCE_OPERATOR_SERVICE_PORT_8443_TCP_PROTO=tcp",
"CCM_PROTECTOR_PORT=tcp://21.101.193.23:8086",
"PXE_CNV_PORT=tcp://21.101.100.255:443",
"KUBEVIRT_PROMETHEUS_METRICS_SERVICE_PORT_METRICS=443",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT_HTTPS=8443",
"VIRT_API_PORT_443_TCP=tcp://21.101.195.118:443",
"COMPUTE_SERVICE_SERVICE_PORT_HTTPS=443",
"COMPUTE_SERVICE_SERVICE_PORT_HTTP=80",
"KUBEVIRT_OPERATOR_WEBHOOK_PORT_443_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP_ADDR=21.101.0.1",
"HCI_REST_SERVICE_SERVICE_PORT_REST=443",
"VIC_CONTROLLER_MANAGER_PORT_30001_TCP_PORT=30001",
"COMPUTE_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP=tcp://21.101.54.196:8443",
"DOWNUPLOAD_SERVICE_PORT_443_TCP=tcp://21.101.196.114:443",
"VIC_IMAGE_REGISTRY_PORT_443_TCP_ADDR=21.101.153.133",
"VIC_IMAGE_REGISTRY_PORT_443_TCP=tcp://21.101.153.133:443",
"COMPUTE_SERVICE_PORT=tcp://21.101.45.184:80",
"COMPUTE_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP_PROTO=tcp",
"CDI_API_PORT_443_TCP_PORT=443",
"CCM_PROTECTOR_PORT_8086_TCP_PORT=8086",
"PXE_CNV_PORT_443_TCP_PORT=443",
"KUBEVIRT_OPERATOR_WEBHOOK_PORT_443_TCP=tcp://21.101.151.204:443",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR=21.101.200.153",
"DOWNUPLOAD_SERVICE_PORT=tcp://21.101.196.114:443",
"NODE_MAINTENANCE_OPERATOR_SERVICE_PORT_8443_TCP=tcp://21.101.178.65:8443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_HOST=21.101.191.219",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443",
"NODE_MAINTENANCE_OPERATOR_SERVICE_PORT_8443_TCP_ADDR=21.101.178.65",
"KUBEVIRT_OPERATOR_WEBHOOK_PORT=tcp://21.101.151.204:443",
"VIC_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443",
"PXE_CNV_PORT_443_TCP_PROTO=tcp",
"CDI_UPLOADPROXY_PORT_443_TCP_PORT=443",
"VIC_WEBHOOK_SERVICE_SERVICE_HOST=21.101.42.180",
"PXE_CNV_PORT_443_TCP_ADDR=21.101.100.255",
"COMPUTE_SERVICE_PORT_80_TCP_PROTO=tcp",
"HCO_WEBHOOK_SERVICE_PORT_4343_TCP_PORT=4343",
"KUBEVIRT_OPERATOR_WEBHOOK_PORT_443_TCP_ADDR=21.101.151.204",
"KUBERNETES_PORT_443_TCP_PORT=443",
"CDI_API_PORT_443_TCP_ADDR=21.101.171.221",
"HCI_REST_SERVICE_SERVICE_HOST=21.101.39.40",
"VIC_CONTROLLER_MANAGER_PORT_30001_TCP_PROTO=tcp",
"PXE_CNV_PORT_443_TCP=tcp://21.101.100.255:443",
"HCO_WEBHOOK_SERVICE_SERVICE_HOST=21.101.220.99",
"CDI_UPLOADPROXY_PORT=tcp://21.101.225.9:443",
"CDI_UPLOADPROXY_PORT_443_TCP_ADDR=21.101.225.9",
"CDI_API_SERVICE_HOST=21.101.171.221",
"DOWNUPLOAD_SERVICE_SERVICE_HOST=21.101.196.114",
"VIC_IMAGE_REGISTRY_PORT_443_TCP_PORT=443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_PORT_AWEBHOOK=543",
"CCM_PROTECTOR_SERVICE_PORT_HTTP=8086",
"YUM_REPO_SERVICE_PORT_80_TCP_PROTO=tcp",
"CDI_UPLOADPROXY_PORT_443_TCP=tcp://21.101.225.9:443",
"KUBEVIRT_PROMETHEUS_METRICS_SERVICE_HOST=21.101.104.104",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP=tcp://21.101.191.219:543",
"HCI_REST_SERVICE_PORT_443_TCP_PORT=443",
"VIC_CONTROLLER_MANAGER_PORT=tcp://21.101.219.154:30001",
"NODE_MAINTENANCE_OPERATOR_SERVICE_SERVICE_HOST=21.101.178.65",
"COMPUTE_CONTROLLER_METRICS_SERVICE_SERVICE_PORT=8443",
"VIRT_API_SERVICE_HOST=21.101.195.118",
"VIRT_API_PORT_443_TCP_PROTO=tcp",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_SERVICE_PORT_VWEBHOOK=443",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO=tcp",
"CCM_PROTECTOR_PORT_8086_TCP=tcp://21.101.193.23:8086",
"PXE_CNV_SERVICE_PORT_HTTPS=443",
"YUM_REPO_SERVICE_SERVICE_PORT_HTTP=80",
"KUBERNETES_SERVICE_PORT=443",
"YUM_REPO_SERVICE_PORT_80_TCP_PORT=80",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP_PORT=543",
"COMPUTE_SERVICE_PORT_80_TCP_ADDR=21.101.45.184",
"HCO_WEBHOOK_SERVICE_PORT_4343_TCP=tcp://21.101.220.99:4343",
"KUBEVIRT_OPERATOR_WEBHOOK_SERVICE_HOST=21.101.151.204",
"YUM_REPO_SERVICE_SERVICE_PORT=80",
"YUM_REPO_SERVICE_PORT_80_TCP=tcp://21.101.126.246:80",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP_PROTO=tcp",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP=tcp://21.101.200.153:8443",
"HCI_REST_SERVICE_PORT=tcp://21.101.39.40:443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT=tcp://21.101.44.4:443",
"KUBEVIRT_OPERATOR_WEBHOOK_SERVICE_PORT=443",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PORT=8443",
"DOWNUPLOAD_SERVICE_SERVICE_PORT_HTTPS=443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=21.101.191.219",
"CCM_PROTECTOR_SERVICE_HOST=21.101.193.23",
"CCM_PROTECTOR_PORT_8086_TCP_PROTO=tcp",
"COMPUTE_SERVICE_PORT_80_TCP=tcp://21.101.45.184:80",
"COMPUTE_CONTROLLER_METRICS_SERVICE_SERVICE_HOST=21.101.54.196",
"COMPUTE_CONTROLLER_METRICS_SERVICE_PORT=tcp://21.101.54.196:8443",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP_ADDR=21.101.191.219",
"KUBERNETES_PORT_443_TCP=tcp://21.101.0.1:443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP=tcp://21.101.44.4:543",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_543_TCP_ADDR=21.101.44.4",
"HCO_WEBHOOK_SERVICE_PORT_4343_TCP_PROTO=tcp",
"VIRT_API_PORT_443_TCP_ADDR=21.101.195.118",
"CDI_UPLOADPROXY_SERVICE_PORT=443",
"CDI_UPLOADPROXY_PORT_443_TCP_PROTO=tcp",
"VIC_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp",
"CDI_API_PORT=tcp://21.101.171.221:443",
"KUBERNETES_PORT=tcp://21.101.0.1:443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp",
"CCM_PROTECTOR_PORT_8086_TCP_ADDR=21.101.193.23",
"PXE_CNV_SERVICE_HOST=21.101.100.255",
"HCO_WEBHOOK_SERVICE_PORT=tcp://21.101.220.99:4343",
"KUBEVIRT_PROMETHEUS_METRICS_PORT_443_TCP_ADDR=21.101.104.104",
"NODE_MAINTENANCE_OPERATOR_SERVICE_PORT=tcp://21.101.178.65:8443",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT=8443",
"HCI_REST_SERVICE_SERVICE_PORT=443",
"NODE_MAINTENANCE_OPERATOR_SERVICE_PORT_8443_TCP_PORT=8443",
"VIC_IMAGE_REGISTRY_SERVICE_HOST=21.101.153.133",
"VIC_CONTROLLER_MANAGER_PORT_30001_TCP_ADDR=21.101.219.154",
"HCO_WEBHOOK_SERVICE_SERVICE_PORT_4343=4343",
"VIRT_API_PORT_443_TCP_PORT=443",
"CDI_UPLOADPROXY_SERVICE_HOST=21.101.225.9",
"VIC_IMAGE_REGISTRY_PORT=tcp://21.101.153.133:443",
"CDI_API_PORT_443_TCP=tcp://21.101.171.221:443",
"CDI_API_PORT_443_TCP_PROTO=tcp",
"CEASPHERE_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT=tcp://21.101.191.219:443",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP=tcp://21.101.44.4:443",
"HCI_CONTROLLER_MANAGER_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=21.101.44.4",
"COMPUTE_SERVICE_SERVICE_HOST=21.101.45.184",
"KUBEVIRT_PROMETHEUS_METRICS_PORT_443_TCP_PROTO=tcp",
"COMPUTE_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP_ADDR=21.101.54.196",
"DOWNUPLOAD_SERVICE_PORT_443_TCP_ADDR=21.101.196.114",
"COMPUTE_SERVICE_PORT_443_TCP_PROTO=tcp",
"VIC_CONTROLLER_MANAGER_SERVICE_HOST=21.101.219.154",
"COMPUTE_SERVICE_SERVICE_PORT=80",
"COMPUTE_SERVICE_PORT_80_TCP_PORT=80",
"VIC_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_HOST=21.101.200.153",
"KUBEVIRT_PROMETHEUS_METRICS_PORT_443_TCP_PORT=443",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"cwd": "/",
"capabilities": {},
"noNewPrivileges": true,
"oomScoreAdj": 999
},
"root": {
"path": "/var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged"
},
"hostname": "hci-compute-controller-65f8cb8b96-mh8fj",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"options": [
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
},
{
"destination": "/dev/shm",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/d5ffbedeb8d27a6231257276f16ed401591fba3de722978f1c3a237263fbc93d/userdata/shm",
"options": [
"rw",
"bind"
]
},
{
"destination": "/etc/resolv.conf",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/d5ffbedeb8d27a6231257276f16ed401591fba3de722978f1c3a237263fbc93d/userdata/resolv.conf",
"options": [
"rw",
"bind",
"nodev",
"nosuid",
"noexec"
]
},
{
"destination": "/etc/hostname",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/d5ffbedeb8d27a6231257276f16ed401591fba3de722978f1c3a237263fbc93d/userdata/hostname",
"options": [
"rw",
"bind"
]
},
{
"destination": "/run/.containerenv",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/d5ffbedeb8d27a6231257276f16ed401591fba3de722978f1c3a237263fbc93d/userdata/.containerenv",
"options": [
"rw",
"bind"
]
},
{
"destination": "/etc/hosts",
"type": "bind",
"source": "/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/etc-hosts",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/dev/termination-log",
"type": "bind",
"source": "/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/containers/kube-rbac-proxy/664b7b43",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/run/secrets",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9/userdata/run/secrets",
"options": [
"bind",
"rprivate",
"bind"
]
},
{
"destination": "/tmp/k8s-webhook-server/serving-certs/",
"type": "bind",
"source": "/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/volumes/kubernetes.iosecret/cert",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/var/run/secrets/kubernetes.io/serviceaccount",
"type": "bind",
"source": "/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/volumes/kubernetes.ioprojected/kube-api-access-54x86",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
}
],
"annotations": {
"test.io/scc": "privileged",
"cni.projectcalico.org/containerID": "c1323a25151fcfcf3e2a972e0afb6a48b5545138767d351856e0a814babfea94",
"cni.projectcalico.org/podIP": "21.100.249.49/32",
"cni.projectcalico.org/podIPs": "21.100.249.49/32",
"include.release.test.io/ibm-cloud-managed": "true",
"include.release.test.io/self-managed-high-availability": "true",
"include.release.test.io/single-node-developer": "true",
"io.container.manager": "cri-o",
"io.kubernetes.container.hash": "919bd97e",
"io.kubernetes.container.name": "kube-rbac-proxy",
"io.kubernetes.container.ports": "[{"name":"https","containerPort":8443,"protocol":"TCP"}]",
"io.kubernetes.container.restartCount": "1",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.container.terminationMessagePolicy": "File",
"io.kubernetes.cri-o.Annotations": "{"io.kubernetes.container.hash":"919bd97e","io.kubernetes.container.ports":"[{\"name\":\"https\",\"containerPort\":8443,\"protocol\":\"TCP\"}]","io.kubernetes.container.restartCount":"1","io.kubernetes.container.terminationMessagePath":"/dev/termination-log","io.kubernetes.container.terminationMessagePolicy":"File","io.kubernetes.pod.terminationGracePeriod":"0"}",
"io.kubernetes.cri-o.ContainerID": "e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9",
"io.kubernetes.cri-o.ContainerType": "container",
"io.kubernetes.cri-o.Created": "2025-04-09T16:59:19.305634795+08:00",
"io.kubernetes.cri-o.IP.0": "21.100.248.61",
"io.kubernetes.cri-o.Image": "7150b1b9c5c696a563e6d5825ab7486e1a3f127d7974fb42007a1b5046ff5e77",
"io.kubernetes.cri-o.ImageName": "image.test.io/test-ceasphere/ceasphere@sha256:4c4bb367d942e532e8e367f4ce0638974d3f953d10349de9ae495d940403c58f",
"io.kubernetes.cri-o.ImageRef": "7150b1b9c5c696a563e6d5825ab7486e1a3f127d7974fb42007a1b5046ff5e77",
"io.kubernetes.cri-o.Labels": "{"io.kubernetes.container.name":"kube-rbac-proxy","io.kubernetes.pod.name":"hci-compute-controller-65f8cb8b96-mh8fj","io.kubernetes.pod.namespace":"test-cnv","io.kubernetes.pod.uid":"1262ed27-23b0-42cd-b437-115732368b22"}",
"io.kubernetes.cri-o.LogPath": "/var/log/pods/test-cnv_hci-compute-controller-65f8cb8b96-mh8fj_1262ed27-23b0-42cd-b437-115732368b22/kube-rbac-proxy/1.log",
"io.kubernetes.cri-o.Metadata": "{"name":"kube-rbac-proxy","attempt":1}",
"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged",
"io.kubernetes.cri-o.Name": "k8s_kube-rbac-proxy_hci-compute-controller-65f8cb8b96-mh8fj_test-cnv_1262ed27-23b0-42cd-b437-115732368b22_1",
"io.kubernetes.cri-o.PlatformRuntimePath": "",
"io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/d5ffbedeb8d27a6231257276f16ed401591fba3de722978f1c3a237263fbc93d/userdata/resolv.conf",
"io.kubernetes.cri-o.SandboxID": "d5ffbedeb8d27a6231257276f16ed401591fba3de722978f1c3a237263fbc93d",
"io.kubernetes.cri-o.SandboxName": "k8s_hci-compute-controller-65f8cb8b96-mh8fj_test-cnv_1262ed27-23b0-42cd-b437-115732368b22_0",
"io.kubernetes.cri-o.SeccompProfilePath": "Unconfined",
"io.kubernetes.cri-o.Stdin": "false",
"io.kubernetes.cri-o.StdinOnce": "false",
"io.kubernetes.cri-o.TTY": "false",
"io.kubernetes.cri-o.Volumes": "[{"container_path":"/etc/hosts","host_path":"/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/etc-hosts","readonly":false,"propagation":0,"selinux_relabel":false},{"container_path":"/dev/termination-log","host_path":"/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/containers/kube-rbac-proxy/664b7b43","readonly":false,"propagation":0,"selinux_relabel":false},{"container_path":"/tmp/k8s-webhook-server/serving-certs/","host_path":"/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/volumes/kubernetes.iosecret/cert","readonly":true,"propagation":0,"selinux_relabel":false},{"container_path":"/var/run/secrets/kubernetes.io/serviceaccount","host_path":"/var/lib/kubelet/pods/1262ed27-23b0-42cd-b437-115732368b22/volumes/kubernetes.ioprojected/kube-api-access-54x86","readonly":true,"propagation":0,"selinux_relabel":false}]",
"io.kubernetes.pod.name": "hci-compute-controller-65f8cb8b96-mh8fj",
"io.kubernetes.pod.namespace": "test-cnv",
"io.kubernetes.pod.terminationGracePeriod": "0",
"io.kubernetes.pod.uid": "1262ed27-23b0-42cd-b437-115732368b22",
"k8s.v1.cni.cncf.io/network-status": "[{\n "name": "k8s-pod-network",\n "ips": [\n "21.100.249.49"\n ],\n "default": true,\n "dns": {}\n}]",
"k8s.v1.cni.cncf.io/networks-status": "[{\n "name": "k8s-pod-network",\n "ips": [\n "21.100.249.49"\n ],\n "default": true,\n "dns": {}\n}]",
"kubectl.kubernetes.io/default-container": "manager",
"kubernetes.io/config.seen": "2025-04-09T16:58:40.909075595+08:00",
"kubernetes.io/config.source": "api",
"org.systemd.property.After": "['crio.service']",
"org.systemd.property.DefaultDependencies": "true",
"org.systemd.property.TimeoutStopUSec": "uint64 0000000"
},
"linux": {
"resources": {
"devices": [
{
"allow": false,
"access": "rwm"
}
],
"memory": {
"limit": 134217728,
"swap": 134217728
},
"cpu": {
"shares": 10,
"quota": 10000,
"period": 100000,
"cpus": "0,4-48,52-95"
},
"pids": {
"limit": -1
},
"hugepageLimits": [
{
"pageSize": "2MB",
"limit": 0
},
{
"pageSize": "1GB",
"limit": 0
}
]
},
"cgroupsPath": "kubepods-burstable-pod1262ed27_23b0_42cd_b437_115732368b22.slice:crio:e4c54e11bdc0a385eb0d6d131ca94277094544ab6d4a4f45841b4d3ca0837aa9",
"namespaces": [
{
"type": "pid"
},
{
"type": "network",
"path": "/var/run/netns/1beb06fe-fe75-4281-8923-83854ff43f26"
},
{
"type": "ipc",
"path": "/var/run/ipcns/1beb06fe-fe75-4281-8923-83854ff43f26"
},
{
"type": "uts",
"path": "/var/run/utsns/1beb06fe-fe75-4281-8923-83854ff43f26"
},
{
"type": "mount"
}
],
"maskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"readonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
}
}

I attempted to use the content of this config.json file and created a directory called mybundle.
I'm not sure if my approach is correct. @rata

$ mkdir -p $BUNDLE_DIR
$ cp config.json $BUNDLE_DIR
#        "root": {
#                "path": #"/var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged"
#        },
$ ln -s /var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged $BUNDLE_DIR/rootfs

$ runc --debug create --bundle $BUNDLE_DIR mycontainer
DEBU[0000] nsexec[1080040]: => nsexec container setup   
DEBU[0000] nsexec[1080040]: update /proc/self/oom_score_adj to '999' 
DEBU[0000] nsexec[1080040]: set process as non-dumpable 
DEBU[0000] nsexec-0[1080040]: ~> nsexec stage-0         
DEBU[0000] nsexec-0[1080040]: spawn stage-1             
DEBU[0000] nsexec-0[1080040]: -> stage-1 synchronisation loop 
DEBU[0000] nsexec-1[1080042]: ~> nsexec stage-1         
DEBU[0000] nsexec-1[1080042]: setns(0x8000000) into ipc namespace (with path /var/run/ipcns/1beb06fe-fe75-4281-8923-83854ff43f26) 
DEBU[0000] nsexec-1[1080042]: setns(0x4000000) into uts namespace (with path /var/run/utsns/1beb06fe-fe75-4281-8923-83854ff43f26) 
DEBU[0000] nsexec-1[1080042]: setns(0x40000000) into net namespace (with path /var/run/netns/1beb06fe-fe75-4281-8923-83854ff43f26) 
DEBU[0000] nsexec-1[1080042]: unshare remaining namespaces (except cgroupns) 
DEBU[0000] nsexec-1[1080042]: spawn stage-2             
DEBU[0000] nsexec-1[1080042]: request stage-0 to forward stage-2 pid (1080043) 
DEBU[0000] nsexec-0[1080040]: stage-1 requested pid to be forwarded 
DEBU[0000] nsexec-0[1080040]: forward stage-1 (1080042) and stage-2 (1080043) pids to runc 
DEBU[0000] nsexec-1[1080042]: signal completion to stage-0 
DEBU[0000] nsexec-1[1080042]: <~ nsexec stage-1         
DEBU[0000] nsexec-0[1080040]: stage-1 complete          
DEBU[0000] nsexec-0[1080040]: <- stage-1 synchronisation loop 
DEBU[0000] nsexec-0[1080040]: -> stage-2 synchronisation loop 
DEBU[0000] nsexec-0[1080040]: signalling stage-2 to run 
DEBU[0000] nsexec-2[1]: ~> nsexec stage-2               
DEBU[0000] nsexec-2[1]: signal completion to stage-0    
DEBU[0000] nsexec-2[1]: <= nsexec container setup       
DEBU[0000] nsexec-2[1]: booting up go runtime ...       
DEBU[0000] nsexec-0[1080040]: stage-2 complete          
DEBU[0000] nsexec-0[1080040]: <- stage-2 synchronisation loop 
DEBU[0000] nsexec-0[1080040]: <~ nsexec stage-0         
DEBU[0000] child process in init()                      
ERRO[0000]utils.go:61 main.fatalWithCode() runc create failed: unable to start container process: exec: "/usr/bin/kube-rbac-proxy": stat /usr/bin/kube-rbac-proxy: permission denied


$  ls -alh
total 24K
drwxr-x--- 9 root root 117 Apr  9 17:45 .
drwx------ 3 root root  41 Apr  9 17:42 ..
-rw-r----- 1 root root 21K Apr  9 17:45 config.json
drwxr-x--- 2 root root   6 Apr  9 17:45 dev
drwxr-x--- 2 root root  54 Apr  9 17:45 etc
drwxr-x--- 2 root root   6 Apr  9 17:45 proc
lrwxrwxrwx 1 root root 107 Apr  9 17:44 rootfs -> /var/lib/containers/storage/overlay/0a6c5e8bb6ba4eb00edb087ea5bb6d8c4786e96829814dc7694877c9880a3e83/merged
drwxr-x--- 3 root root  42 Apr  9 17:45 run
drwxr-x--- 2 root root   6 Apr  9 17:45 sys
drwxr-x--- 3 root root  32 Apr  9 17:45 tmp
drwxr-x--- 3 root root  17 Apr  9 17:45 var

[rata]

@xuegege5290 it seems so, great! You can probably copy the rootfs to another directory and trim it to only keep the binary that we fail to execute.

With that, I guess we have a repro!

Does it happen always? If you can try with different runc versions, it would be great.

I can try to have a look next week probably, not sure if I'll manage before that.

[Syx5290]

The second method involves using the config.json file and then creating the rootfs using an image.

# Create the top-level bundle directory
mkdir /mycontainer
cd /mycontainer

# Create the rootfs directory
mkdir rootfs

# Export the specified image via Docker into the rootfs directory
podman export $(podman create mage.test.io/ceake/ceake@sha256:0a93f13da1adab6b708342469585c7380ce364b874a2a2b99294d4a9876db166) | tar -C rootfs -xvf -

# Copy the config.json to /mycontainer
cp /var/lib/container/xxx/config.json /mycontainer

# Change the root configuration in config.json to:
#"root": {
#  "path": "rootfs",
#  "readonly": true
#},

# Delete the namespace.Path and cgroupsPath fields from config.json

cd /mycontainer

runc run mycontainerid
DEBU[0000] nsexec[2238380]: => nsexec container setup   
DEBU[0000] nsexec[2238380]: update /proc/self/oom_score_adj to '999' 
DEBU[0000] nsexec-0[2238380]: ~> nsexec stage-0         
DEBU[0000] nsexec-0[2238380]: spawn stage-1             
DEBU[0000] nsexec-0[2238380]: -> stage-1 synchronisation loop 
DEBU[0000] nsexec-1[2238384]: ~> nsexec stage-1         
DEBU[0000] nsexec-1[2238384]: unshare remaining namespaces (except cgroupns) 
DEBU[0000] nsexec-1[2238384]: spawn stage-2             
DEBU[0000] nsexec-1[2238384]: request stage-0 to forward stage-2 pid (2238388) 
DEBU[0000] nsexec-0[2238380]: stage-1 requested pid to be forwarded 
DEBU[0000] nsexec-0[2238380]: forward stage-1 (2238384) and stage-2 (2238388) pids to runc 
DEBU[0000] nsexec-1[2238384]: signal completion to stage-0 
DEBU[0000] nsexec-1[2238384]: <~ nsexec stage-1         
DEBU[0000] nsexec-0[2238380]: stage-1 complete          
DEBU[0000] nsexec-0[2238380]: <- stage-1 synchronisation loop 
DEBU[0000] nsexec-0[2238380]: -> stage-2 synchronisation loop 
DEBU[0000] nsexec-0[2238380]: signalling stage-2 to run 
DEBU[0000] nsexec-2[1]: ~> nsexec stage-2               
DEBU[0000] nsexec-2[1]: signal completion to stage-0    
DEBU[0000] nsexec-0[2238380]: stage-2 complete          
DEBU[0000] nsexec-2[1]: <= nsexec container setup       
DEBU[0000] nsexec-0[2238380]: <- stage-2 synchronisation loop 
DEBU[0000] nsexec-2[1]: booting up go runtime ...       
DEBU[0000] nsexec-0[2238380]: <~ nsexec stage-0         
DEBU[0000] child process in init()                      
ERRO[0000]utils.go:61 main.fatalWithCode() runc run failed: unable to start container process: exec: "/usr/bin/kube-rbac-proxy": stat /usr/bin/kube-rbac-proxy: permission denied

 ls ./rootfs/
afs  bin  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

ls -alh ./rootfs/usr/bin/kube-rbac-proxy 
-rwxr-xr-x 1 root root 62M Mar  7 12:02 ./rootfs/usr/bin/kube-rbac-proxy

I'll try using a higher version of runc later.

[Syx5290]

I tried using the higher version of runc, release-1.2. @rata

[root@sphere17-node-4 mycontainer]# /root/runc --debug create --bundle $BUNDLE_DIR mycontainer
DEBU[0000]libcontainer/dmz/cloned_binary_linux.go:206 github.com/opencontainers/runc/libcontainer/dmz.IsCloned() F_GET_SEALS on /proc/self/exe failed: invalid argument 
DEBU[0000]libcontainer/dmz/cloned_binary_linux.go:231 github.com/opencontainers/runc/libcontainer/dmz.CloneSelfExe() runc-dmz: using overlayfs for sealed /proc/self/exe 
DEBU[0000]libcontainer/container_linux.go:507 github.com/opencontainers/runc/libcontainer.(*Container).newParentProcess() runc-dmz: using /proc/self/exe clone         
DEBU[0000] nsexec[3111809]: => nsexec container setup   
DEBU[0000] nsexec[3111809]: update /proc/self/oom_score_adj to '999' 
DEBU[0000] nsexec[3111809]: set process as non-dumpable 
DEBU[0000] nsexec-0[3111809]: ~> nsexec stage-0         
DEBU[0000] nsexec-0[3111809]: spawn stage-1             
DEBU[0000] nsexec-0[3111809]: -> stage-1 synchronisation loop 
DEBU[0000] nsexec-1[3111810]: ~> nsexec stage-1         
DEBU[0000] nsexec-1[3111810]: setns(0x8000000) into ipc namespace (with path /var/run/ipcns/1beb06fe-fe75-4281-8923-83854ff43f26) 
DEBU[0000] nsexec-1[3111810]: setns(0x4000000) into uts namespace (with path /var/run/utsns/1beb06fe-fe75-4281-8923-83854ff43f26) 
DEBU[0000] nsexec-1[3111810]: setns(0x40000000) into net namespace (with path /var/run/netns/1beb06fe-fe75-4281-8923-83854ff43f26) 
DEBU[0000] nsexec-1[3111810]: unshare remaining namespaces 
DEBU[0000] nsexec-1[3111810]: spawn stage-2             
DEBU[0000] nsexec-1[3111810]: request stage-0 to forward stage-2 pid (3111811) 
DEBU[0000] nsexec-0[3111809]: stage-1 requested pid to be forwarded 
DEBU[0000] nsexec-0[3111809]: forward stage-1 (3111810) and stage-2 (3111811) pids to runc 
DEBU[0000] nsexec-1[3111810]: signal completion to stage-0 
DEBU[0000] nsexec-1[3111810]: <~ nsexec stage-1         
DEBU[0000] nsexec-0[3111809]: stage-1 complete          
DEBU[0000] nsexec-0[3111809]: <- stage-1 synchronisation loop 
DEBU[0000] nsexec-0[3111809]: -> stage-2 synchronisation loop 
DEBU[0000] nsexec-0[3111809]: signalling stage-2 to run 
DEBU[0000] nsexec-2[1]: ~> nsexec stage-2               
DEBU[0000] nsexec-2[1]: signal completion to stage-0    
DEBU[0000] nsexec-2[1]: <= nsexec container setup       
DEBU[0000] nsexec-0[3111809]: stage-2 complete          
DEBU[0000] nsexec-2[1]: booting up go runtime ...       
DEBU[0000] nsexec-0[3111809]: <- stage-2 synchronisation loop 
DEBU[0000] nsexec-0[3111809]: <~ nsexec stage-0         
DEBU[0000]libcontainer/process_linux.go:464 github.com/opencontainers/runc/libcontainer.(*initProcess).goCreateMountSources.func1() mount source thread: successfully running in container mntns 
DEBU[0000]libcontainer/sync.go:127 github.com/opencontainers/runc/libcontainer.doReadSync() reading sync                                 
DEBU[0000] child process in init()                      
DEBU[0000] writing sync type:procHooks                  
DEBU[0000] reading sync                                 
DEBU[0000]libcontainer/sync.go:139 github.com/opencontainers/runc/libcontainer.doReadSync() read sync type:procHooks                     
DEBU[0000]libcontainer/sync.go:95 github.com/opencontainers/runc/libcontainer.doWriteSync() writing sync type:procHooksDone              
DEBU[0000]libcontainer/sync.go:127 github.com/opencontainers/runc/libcontainer.doReadSync() reading sync                                 
DEBU[0000] read sync type:procHooksDone                 
DEBU[0000] writing sync type:procReady                  
DEBU[0000] reading sync                                 
DEBU[0000] read sync type:procReady                     
DEBU[0000] writing sync type:procRun                    
DEBU[0000] reading sync                                 
DEBU[0000] read sync type:procRun                       
DEBU[0000] writing sync type:procError arg:{"message":"exec: \"/usr/bin/kube-rbac-proxy\": stat /usr/bin/kube-rbac-proxy: permission denied"} 
DEBU[0000]libcontainer/sync.go:139 github.com/opencontainers/runc/libcontainer.doReadSync() read sync type:procError arg:{"message":"exec: \"/usr/bin/kube-rbac-proxy\": stat /usr/bin/kube-rbac-proxy: permission denied"} 
DEBU[0000] mount source thread: closing thread: context canceled 
ERRO[0000]utils.go:62 main.fatalWithCode() runc create failed: unable to start container process: error during container init: exec: "/usr/bin/kube-rbac-proxy": stat /usr/bin/kube-rbac-proxy: permission denied

[root@sphere17-node-4 mycontainer]# /root/runc --version
runc version 1.2.0
spec: 1.2.0
go: go1.21.11
libseccomp: 2.5.2