the muscle | ProgrammerAH

11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.108952     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.209293     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.310543     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.411121     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.511949     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.612822     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.713249     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.781263     974 controller.go:144] failed to ensure lease exists, will retry in 7s, error: leases.coordination.k8s.io "localhost.localdomain" is forbidden: User "system:node:k8s222" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.813355     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"
11月 05 10:58:35 localhost.localdomain kubelet[974]: E1105 10:58:35.913495     974 kubelet.go:2412] "Error getting node" err="node "localhost.localdomain" not found"

1.1 this node is always notready

[root@crust-m01 ~]# kubectl get node
NAME        STATUS     ROLES                  AGE   VERSION
k8s220   NotReady   control-plane,master   44d   v1.21.3
k8s221   NotReady   <none>                 44d   v1.21.3
k8s222   NotReady   <none>                 44d   v1.21.3

1.2 view details of this node

[root@localhost ~]# kubectl describe node k8s221


……
Unschedulable:      false
Lease:
  HolderIdentity:  k8s221
  AcquireTime:     <unset>
  RenewTime:       Tue, 28 Sep 2021 14:37:08 +0800
Conditions:
  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----             ------    -----------------                 ------------------                ------              -------
  MemoryPressure   Unknown   Tue, 28 Sep 2021 14:32:16 +0800   Tue, 28 Sep 2021 14:38:17 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure     Unknown   Tue, 28 Sep 2021 14:32:16 +0800   Tue, 28 Sep 2021 14:38:17 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure      Unknown   Tue, 28 Sep 2021 14:32:16 +0800   Tue, 28 Sep 2021 14:38:17 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready            Unknown   Tue, 28 Sep 2021 14:32:16 +0800   Tue, 28 Sep 2021 14:38:17 +0800   NodeStatusUnknown   Kubelet stopped posting node status.
……

1.3 view kubelet logs on this node

[root@crust-m2 ~]# service kubelet status -l
Redirecting to /bin/systemctl status  -l kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 二 2021-09-28 14:51:57 CST; 4min 6s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 21165 (kubelet)
    Tasks: 19
   Memory: 43.0M
   CGroup: /system.slice/kubelet.service
           └─21165 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.4.1

9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.119645   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.220694   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.321635   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.385100   21165 eviction_manager.go:255] "Eviction manager: failed to get summary stats" err="failed to get node info: node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.422387   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.523341   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.624021   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.724418   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.825475   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"
9月 28 14:56:03 crust-m2 kubelet[21165]: E0928 14:56:03.926199   21165 kubelet.go:2291] "Error getting node" err="node "crust-m2" not found"

2. [troubleshooting]

The startup command of log output in 1.3 is as follows:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.4.1

There is no problem viewing and analyzing all configuration files in the startup command

The error err = "node" localhost. Output in 1.3 was found localdomain "Not found
the information of kubectl get node on the master is k8s220, k8s221, k8s222

conclusion
when kubernetes was installed before, the name of the master was k8s220, and the node was k8s221, k8s222, because /etc/hostname was written as localhost.localdomain by default, kubelet has always been report errors

3. [modification]

Modify the hostname file, execute the hostname command, modify the server name,

Restart kubelete

preface

An error is reported when adding a k8s-node using kubedm join , as follows:

error execution phase kubelet-start: error uploading crisocket: timed out waiting for the condition

Solution:

1 reset the cluster configuration of this machine

kubeadm reset -f

2 stop service

sudo tee /tmp/test.sh <<-'EOF'
systemctl stop kubelet kube-proxy docker containerd
ps -ef | grep containerd | awk '{print $2}' | xargs -I {} kill -9 {}
rm -rf /etc/systemd/system/kube-proxy.service
rm -rf /etc/systemd/system/kubelet.service
systemctl stop zabbix-agent.service
systemctl disable zabbix-agent.service
systemctl stop node_exporter_custom.service
systemctl disable node_exporter_custom.service
systemctl reset-failed
rm -rf /usr/local/bin/kube*
EOF
chmod +x /tmp/test.sh && bash /tmp/test.sh

3 start docker

systemctl start docker

4 join the cluster

Run kubedm token create -- print-join-command on Master to get token information

Then execute the following command on the node node to rejoin the k8s cluster

# k8s-node
kubeadm join 172.18.12.23:6443 --token okt9c2.edidk4m2yfim31su \
--discovery-token-ca-cert-hash sha256:0a6cb9f97106782102840e142eda33dc3dc071aff1af436f90d43699acdc3751

Success information

5 verify on k8s-master

kubectl get node
kubectl get pod -A

ProgrammerAH

Programmer Guide, Tips and Tutorial

Tag Archives: the muscle

Node Kubelet Error: node “xxxxx“ not found [How to Solve]

[Solved] kubeadm join Cluster Error: error execution phase kubelet-start