Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hard time installing kubernetes distributions in defualt image (k3s / k0s) #26

Closed
lonerkim opened this issue Feb 29, 2024 · 4 comments
Closed

hard time installing kubernetes distributions in defualt image (k3s / k0s) #26

lonerkim opened this issue Feb 29, 2024 · 4 comments

Comments

@lonerkim
Copy link

lonerkim commented Feb 29, 2024 •
edited
Loading

When I tried to k0s install and start services, this happened

Error: pre-flight checks failed, check out `k0s sysinfo`: lookup localhost on 8.8.8.8:53: no such host

so I tried to edit my /etc/hosts to include

127.0.0.1       localhost lpi3h-c6dd
::1             localhost ip6-localhost ip6-loopback

and service starts

but then when I tried to join my worker nodes, below happened.

root@lpi3h-(devicename)~# sudo k0s worker "mytoken"
Error: pre-flight checks failed, check out `k0s sysinfo`: failed to load/attach eBPF device filter program: load ext_infos: detect support for Program BTF (func/line_info): detect support for BTF: function not implemented

root@lpi3h-(devicename):~# k0s sysinfo

Machine ID: "(machine ID)" (from machine) (pass)
Total memory: 3.9 GiB (pass)
Disk space available for /var/lib/k0s: 24.6 GiB (pass)
Name resolution: localhost: [127.0.0.1 ::1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.7.0-rc3+ (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": error: failed to load/attach eBPF device filter program: load ext_infos: detect support for Program BTF (func/line_info): detect support for BTF: function not implemented
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": unavailable (warning)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: no kernel config found (warning)
  CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning)
  CONFIG_NET: Networking support: no kernel config found (warning)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning)
  CONFIG_PROC_FS: /proc file system support: no kernel config found (warning)
Error: sysinfo failed

so I tried other distribution (k3s)

and this happened

root@lpi3h-(devicename):~# curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.28.6+k3s2 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.28.6+k3s2/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.28.6+k3s2/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
sh: 581: [: Illegal number:
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
iptables-restore/1.8.10 (nf_tables) Failed to initialize nft: Protocol not supported

so I tried

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
=========================================
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
update-alternatives: using /usr/sbin/ip6tables-legacy to provide /usr/sbin/ip6tables (ip6tables) in manual mode
update-alternatives: error: alternative /usr/sbin/arptables-legacy for arptables not registered; not setting
update-alternatives: error: alternative /usr/sbin/ebtables-legacy for ebtables not registered; not setting

it seems can start a service at first, but when I checked status

root@lpi3h-(devicename):~# systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-02-28 22:36:11 CST; 20s ago
       Docs: https://k3s.io
    Process: 776 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 779 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
    Process: 782 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
   Main PID: 784 (k3s-server)
      Tasks: 26
     Memory: 549.0M (peak: 554.3M)
        CPU: 1min 31.407s
     CGroup: /system.slice/k3s.service
             └─784 "/usr/local/bin/k3s server"

Feb 28 22:36:31 lpi3h--(devicename)k3s[784]: time="2024-02-28T22:36:31+08:00" level=info msg="Waiting for control-plane node lpi3h-(devicename)startup: nodes \"lpi3h-(devicename)\" not found"
Feb 28 22:36:31 lpi3h-(devicename)k3s[784]: I0228 22:36:31.928469     784 shared_informer.go:318] Caches are synced for garbage collector
Feb 28 22:36:31 lpi3h-(devicename)k3s[784]: I0228 22:36:31.936853     784 shared_informer.go:318] Caches are synced for garbage collector
Feb 28 22:36:31 lpi3h-(devicename)k3s[784]: I0228 22:36:31.936943     784 garbagecollector.go:166] "All resource monitors have synced. Proceeding to collect garbage"
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: I0228 22:36:32.142245     784 event.go:307] "Event occurred" object="kube-system/local-path-provisioner-84db5d44d9" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: local-path-p>
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: I0228 22:36:32.172260     784 event.go:307] "Event occurred" object="kube-system/coredns-6799fbcd5" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-6799fbcd5-v7hrv"
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: W0228 22:36:32.188201     784 handler_proxy.go:93] no RequestInfo found in the context
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: E0228 22:36:32.188476     784 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: E0228 22:36:32.188747     784 handler_proxy.go:137] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
Feb 28 22:36:32 lpi3h-(devicename)k3s[784]: I0228 22:36:32.199985     784 replica_set.go:676] "Finished syncing" kind="ReplicaSet" key="kube-system/local-path-provisioner-84db5d44d9" duration="689.235263ms"

also when I checked br_netfilter / overlay myself

root@lpi3h-(devicename):~# sudo modprobe br_netfilter sudo modprobe overlay
modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/6.7.0-rc3+
modprobe: FATAL: Module overlay not found in directory /lib/modules/6.7.0-rc3+

this came out.
br_netfilter and overlay / eBPF, BTF is not supported I guess?
@wu-yue-yu
Copy link
Collaborator

You can check this pr #27 to enable some kernel configs.

@lonerkim
Copy link
Author

lonerkim commented Mar 1, 2024 via email

@lonerkim
Copy link
Author

lonerkim commented Apr 8, 2024 •
edited
Loading

seems some moudle activaton with recent patch for 'main' sucessfully activate k3s.services
still not to test anything due to #30 on my side issue, but there was major progress to share.
(thus issue is still open till more detailed test)
;) thanks for helping me

sipeed@lpi3h-c6dd:~$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-04-08 22:50:08 KST; 4min 1s ago
       Docs: https://k3s.io
   Main PID: 1824 (k3s-server)
      Tasks: 29
     Memory: 586.4M (peak: 613.5M)
        CPU: 7min 12.762s
     CGroup: /system.slice/k3s.service
             └─1824 "/usr/local/bin/k3s server"
sipeed@lpi3h-c6dd:~$ k3s check-config
cat: /sys/kernel/security/apparmor/profiles: No such file or directory

Verifying binaries in /var/lib/rancher/k3s/data/15bd42671378bf2abf3d83fb74cd4854c083c3c857c6bead9802329b4f1fc5d3/bin:
- sha256sum: good
- aux/ip6tables: symlink to xtables-legacy-multi
- aux/ip6tables-restore: symlink to xtables-legacy-multi
- aux/ip6tables-save: symlink to xtables-legacy-multi
- aux/iptables: symlink to xtables-legacy-multi
- aux/iptables-restore: symlink to xtables-legacy-multi
- aux/iptables-save: symlink to xtables-legacy-multi
- links: good

System:
- /var/lib/rancher/k3s/data/15bd42671378bf2abf3d83fb74cd4854c083c3c857c6bead9802329b4f1fc5d3/bin/aux iptables v1.8.8 (legacy): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/6.7.0-rc3+
info: reading kernel config from /boot/config-6.7.0-rc3+ ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
 - "overlay":
   - CONFIG_VXLAN: enabled
     Optional (for encrypted networks):
     - CONFIG_CRYPTO: enabled
     - CONFIG_CRYPTO_AEAD: enabled
     - CONFIG_CRYPTO_GCM: enabled
     - CONFIG_CRYPTO_SEQIV: enabled
     - CONFIG_CRYPTO_GHASH: enabled
     - CONFIG_XFRM: enabled
     - CONFIG_XFRM_USER: enabled
     - CONFIG_XFRM_ALGO: enabled
     - CONFIG_INET_ESP: enabled
     - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
 - "overlay":
   - CONFIG_OVERLAY_FS: enabled

STATUS: pass

@lonerkim
Copy link
Author

lonerkim commented Apr 20, 2024 •
edited
Loading

image

k3s is running on LonganPi 3H Ubuntu-CLI
addtional configs I tried on

seems there isn't signifiacnt issue for now, I'll mark this issues as cloesed.
thanks for help.

@lonerkim lonerkim mentioned this issue Apr 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lonerkim @wu-yue-yu