Kubeadm 添加新 Master 节点到集群出现 ETCD 健康检查失败错误

系统环境

Docker 版本:18.06.3
Kubeadm 版本:1.17.4
Kubernetes 版本:1.17.4
Kubernetes Master 数量:3
Kubernetes 安装方式:Kubeadm

问题描述

Kubernetes 集群中总共有三台 Master,分别是:

k8s-master-2-11、k8s-master-2-12、k8s-master-2-13

对其中 k8s-master-2-11 Master 节点服务器进行了内核和软件升级操作,从而先将其暂时剔出集群,然后进行升级,完成后准备重新加入到 Kubernetes 集群,通过 Kubeadm 执行,输入下面命令:

1
2
3
4
$ kubeadm join mydlq.club:16443 \
--token 6w0nwi.zag57qgfcdhi76vd \
--discovery-token-ca-cert-hash sha256:efa49231e4ffd836ff996921741c98ac4c5655dc729d7c32aa48c608232f0f08 \
--control-plane --certificate-key a64e9da7346153bd64dba1e5126a644a97fdb63c878bb73de07911d1add8e26b

在执行过程中,输出下面日志,提示 etcd 监控检查失败:

1
2
3
4
5
6
7
8
9
10
......
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0329 00:01:51.364121 19209 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0329 00:01:51.373807 19209 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[check-etcd] Checking that the etcd cluster is healthy

error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://10.8.18.105:2379
with maintenance client: context deadline exceeded
To see the stack trace of this error execute with --v=5 or higher

根据关键信息 "error execution phase check-etcd" 可知,可能是在执行加入 etcd 时候出现的错误,导致 master 无法加入原先的 kubernetes 集群。

分析问题

查看集群节点列表

1
2
3
4
5
6
7
8
$ kubectl get node

NAME STATUS ROLES VERSION
k8s-master-2-12 Ready master v1.17.4
k8s-master-2-13 Ready master v1.17.4
k8s-node-2-14 Ready <none> v1.17.4
k8s-node-2-15 Ready <none> v1.17.4
k8s-node-2-16 Ready <none> v1.17.4

可以看到,k8s-master-2-11 节点确实不在节点列表中

查看 Kubeadm 配置信息

在看看 Kubernetes 集群中的 kubeadm 配置信息:

1
$ kubectl describe configmaps kubeadm-config -n kube-system

获取到的内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Name:         kubeadm-config
Namespace: kube-system
Labels: <none>
Annotations: <none>
...
ClusterStatus:
----
apiEndpoints:
k8s-master-2-11:
advertiseAddress: 192.168.2.11
bindPort: 6443
k8s-master-2-12:
advertiseAddress: 192.168.2.12
bindPort: 6443
k8s-master-2-13:
advertiseAddress: 192.168.2.13
bindPort: 6443
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterStatus

可也看到 k8s-master-2-11 节点信息还存在与 kubeadm 配置中,说明 etcd 中还存储着 k8s-master-2-11 相关信息。

分析问题所在及解决方案

因为集群是通过 kubeadm 工具搭建的,且使用了 etcd 镜像方式与 master 节点一起,所以每个 Master 节点上都会存在一个 etcd 容器实例。当剔除一个 master 节点时 etcd 集群未删除剔除的节点的 etcd 成员信息,该信息还存在 etcd 集群列表中。

所以,我们需要 进入 etcd 手动删除 etcd 成员信息

解决

获取 Etcd 镜像列表

1
2
3
4
$ kubectl get pods -n kube-system | grep etcd

etcd-k8s-master-2-12 1/1 Running 0
etcd-k8s-master-2-13 1/1 Running 0

进入 Etcd 容器并删除节点信息

选择上面两个 etcd 中任意一个 pod,通过 kubectl 工具进入 pod 内部:

1
$ kubectl exec -it etcd-k8s-master-2-12 sh -n kube-system

进入容器后,按下面步执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 配置环境
$ export ETCDCTL_API=3
$ alias etcdctl='etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'

# 查看 etcd 集群成员列表
$ etcdctl member list

63bfe05c4646fb08, started, k8s-master-2-11, https://192.168.2.11:2380, https://192.168.2.11:2379, false
8e41efd8164c6e3d, started, k8s-master-2-12, https://192.168.2.12:2380, https://192.168.2.12:2379, false
a61d0bd53c1cbcb6, started, k8s-master-2-13, https://192.168.2.13:2380, https://192.168.2.13:2379, false

# 删除 etcd 集群成员 k8s-master-2-11
$ etcdctl member remove 63bfe05c4646fb08

Member 63bfe05c4646fb08 removed from cluster ed984b9o8w35cap2

# 再次查看 etcd 集群成员列表
$ etcdctl member list

8e41efd8164c6e3d, started, k8s-master-2-12, https://192.168.2.12:2380, https://192.168.2.12:2379, false
a61d0bd53c1cbcb6, started, k8s-master-2-13, https://192.168.2.13:2380, https://192.168.2.13:2379, false

# 退出容器
$ exit

通过 kubeadm 命令再次尝试加入集群

通过 kubeadm 命令再次尝试将 k8s-master-2-11 节点加入集群,在执行前首先进入到 k8s-master-2-11 节点服务器,执行 kubeadm 的清除命令:

1
$ kubeadm reset

然后尝试加入 kubernetes 集群:

1
2
3
4
$ kubeadm join mydlq.club:16443 \
--token 6w0nwi.zag57qgfcdhi76vd \
--discovery-token-ca-cert-hash sha256:efa49231e4ffd836ff996921741c98ac4c5655dc729d7c32aa48c608232f0f08 \
--control-plane --certificate-key a64e9da7346153bd64dba1e5126a644a97fdb63c878bb73de07911d1add8e26b

参考:mydlq.club

-------------本文结束感谢您的阅读-------------