Kubeadm证书过期时间调整

kubeadm 默认证书为一年,一年过期后,会导致api service不可用,使用过程中会出现:x509: certificate has expired or is not yet valid

如何进行调整,下面给了两个方案,供大家选择

方案一 通过修改kubeadm 调整证书过期时间

修改代码,调整过期时间

克隆代码:git clone https://github.com/kubernetes/kubernetes.git, 切换到指定的tag或者版本修改vendor/k8s.io/client-go/util/cert/cert.go文件,git diff 对比如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
diff --git a/staging/src/k8s.io/client-go/util/cert/cert.go b/staging/src/k8s.io/client-go/util/cert/cert.go
index fb7f5fa..e800962 100644
--- a/staging/src/k8s.io/client-go/util/cert/cert.go
+++ b/staging/src/k8s.io/client-go/util/cert/cert.go
@@ -104,7 +104,7 @@ func NewSignedCert(cfg Config, key *rsa.PrivateKey, caCert *x509.Certificate, ca
IPAddresses: cfg.AltNames.IPs,
SerialNumber: serial,
NotBefore: caCert.NotBefore,
- NotAfter: time.Now().Add(duration365d).UTC(),
+ NotAfter: time.Now().Add(duration365d * 10).UTC(),
KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
ExtKeyUsage: cfg.Usages,
}
@@ -149,7 +149,7 @@ func GenerateSelfSignedCertKey(host string, alternateIPs []net.IP, alternateDNS
CommonName: fmt.Sprintf("%s-ca@%d", host, time.Now().Unix()),
},
NotBefore: time.Now(),
- NotAfter: time.Now().Add(time.Hour * 24 * 365),
+ NotAfter: time.Now().Add(time.Hour * 24 * 3650),

KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
BasicConstraintsValid: true,

编译代码

编译环境我已经做了对应的1.11.5、1.12.3、1.13.0、1.13.2、1.13.4、1.14.1、1.15.3,已上传到docker hub 上,大家可下载使用,地址如下:

1
2
3
4
5
6
7
docker pull icyboy/k8s_build:v1.11.5  # 基于 golang:1.10.3
docker pull icyboy/k8s_build:v1.12.3 # 基于 golang:1.10.4
docker pull icyboy/k8s_build:v1.13.0 # 基于 golang:1.11.2
docker pull icyboy/k8s_build:v1.13.2 # 基于 golang:1.11.4
docker pull icyboy/k8s_build:v1.13.4 # 基于 golang:1.11.5
docker pull icyboy/k8s_build:v1.14.1 # 基于 golang:1.12.2
docker pull icyboy/k8s_build:v1.15.3 # 基于 golang:1.12.9

编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
docker run --rm -v 你修改后的代码目录:/go/src/k8s.io/kubernetes -it icyboy/k8s_build:v1.11.5 bash

cd /go/src/k8s.io/kubernetes

# 编译kubeadm, 这里主要编译kubeadm 即可
make all WHAT=cmd/kubeadm GOFLAGS=-v

# 编译kubelet
# make all WHAT=cmd/kubelet GOFLAGS=-v

# 编译kubectl
# make all WHAT=cmd/kubectl GOFLAGS=-v

#编译完产物在 _output/bin/kubeadm 目录下
#将kubeadm 文件拷贝出来,替换系统中的kubeadm

对应的kubeadm 文件我也编译好后放到百度云中,大家可放心下载使用,可通过kubeadm version 查看对应的版本信息和官方的进行比对

1
2
3
4
5
#编译过后的
kubeadm version: &version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-dirty", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"dirty", BuildDate:"2018-12-07T05:58:18Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

#官方的
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:30Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

kubeadm 下载地址:https://pan.baidu.com/s/1PplHyDkYDTusx46j9uHwDA
提取码:dy6f

替换证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#用新的kubeadm 替换官方的kubeadm
chmod +x kubeadm && \cp -f kubeadm /usr/bin

#备份原有的证书
mv /etc/kubernetes/pki /etc/kubernetes/pki.old

#生成新的证书,kubeadm.yaml 指定你自己服务器上的
kubeadm alpha phase certs all --config ~/kubeadm.yaml

#备份原有的conf文件
mv /etc/kubernetes/*conf /etc/kubernetes/*conf-old

#根据新证书重新生成新的配置文件
kubeadm alpha phase kubeconfig all --config ~/kubeadm.yaml

#替换老的config文件
\cp -f /etc/kubernetes/admin.conf ~/.kube/config

验证

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cd /etc/kubernetes/pki
openssl x509 -in apiserver-etcd-client.crt -text -noout
#Certificate:
# Data:
# Version: 3 (0x2)
# Serial Number: 2755977466456048186 (0x263f32e76918023a)
# Signature Algorithm: sha256WithRSAEncryption
# Issuer: CN=kubernetes
# Validity
# Not Before: Dec 7 09:33:32 2018 GMT
Not After : Dec 4 09:33:32 2028 GMT #这里变成10年了
# Subject: O=system:masters, CN=kube-apiserver-etcd-client
# Subject Public Key Info:
# ....

# 批量验证证书
for crt in $(find /etc/kubernetes/pki/ -name "*.crt"); do openssl x509 -in $crt -noout -dates; done

方案二 启用自动轮换kubelet 证书

kubelet证书分为server和client两种, k8s 1.9默认启用了client证书的自动轮换,但server证书自动轮换需要用户开启

增加 kubelet 参数

1
2
# 在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 增加如下参数
Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"

增加 controller-manager 参数

1
2
3
4
5
6
# 在/etc/kubernetes/manifests/kube-controller-manager.yaml 添加如下参数
- command:
- kube-controller-manager
- --experimental-cluster-signing-duration=87600h0m0s
- --feature-gates=RotateKubeletServerCertificate=true
- ....

创建 rbac 对象

创建rbac对象,允许节点轮换kubelet server证书:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeserver
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
EOF

kubectl create –f ca-update.yaml

如果证书已经过期,如何进行重新签发证书

针对kubeadm 1.13.x 及以上处理

准备kubeadm.conf 配置文件一份
1
2
3
4
5
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.1 #-->这里改成你集群对应的版本
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
#这里使用国内的镜像仓库,否则在重新签发的时候会报错:could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"
重新签发命令
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
kubeadm alpha certs renew all --config=/root/kubeadm.conf

运行如上命令会重新生成以下证书
#-- /etc/kubernetes/pki/apiserver.key
#-- /etc/kubernetes/pki/apiserver.crt

#-- /etc/kubernetes/pki/apiserver-etcd-client.key
#-- /etc/kubernetes/pki/apiserver-etcd-client.crt

#-- /etc/kubernetes/pki/apiserver-kubelet-client.key
#-- /etc/kubernetes/pki/apiserver-kubelet-client.crt

#-- /etc/kubernetes/pki/front-proxy-client.key
#-- /etc/kubernetes/pki/front-proxy-client.crt

#-- /etc/kubernetes/pki/etcd/healthcheck-client.key
#-- /etc/kubernetes/pki/etcd/healthcheck-client.crt

#-- /etc/kubernetes/pki/etcd/peer.key
#-- /etc/kubernetes/pki/etcd/peer.crt

#-- /etc/kubernetes/pki/etcd/server.key
#-- /etc/kubernetes/pki/etcd/server.crt
完成后重启kube-apiserver,kube-controller,kube-scheduler,etcd这4个容器

针对kubeadm 1.13.0(不包含1.13.0) 以下处理

移动证书和配置【注意!必须移动,不然会使用现有的证书,不会重新生成】
1
2
3
4
5
6
7
8
9
10
11
12
13
cd /etc/kubernetes
mkdir ./pki_bak
mkdir ./pki_bak/etcd
mkdir ./conf_bak
mv pki/apiserver* ./pki_bak/
mv pki/front-proxy-client.* ./pki_bak/
mv pki/etcd/healthcheck-client.* ./pki_bak/etcd/
mv pki/etcd/peer.* ./pki_bak/etcd/
mv pki/etcd/server.* ./pki_bak/etcd/
mv ./admin.conf ./conf_bak/
mv ./kubelet.conf ./conf_bak/
mv ./controller-manager.conf ./conf_bak/
mv ./scheduler.conf ./conf_bak/
创建证书
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
kubeadm alpha phase certs all --apiserver-advertise-address=${MASTER_API_SERVER_IP} --apiserver-cert-extra-sans=主机内网ip,主机公网ip

运行如上命令会重新生成以下证书
#-- /etc/kubernetes/pki/apiserver.key
#-- /etc/kubernetes/pki/apiserver.crt

#-- /etc/kubernetes/pki/apiserver-etcd-client.key
#-- /etc/kubernetes/pki/apiserver-etcd-client.crt

#-- /etc/kubernetes/pki/apiserver-kubelet-client.key
#-- /etc/kubernetes/pki/apiserver-kubelet-client.crt

#-- /etc/kubernetes/pki/front-proxy-client.key
#-- /etc/kubernetes/pki/front-proxy-client.crt

#-- /etc/kubernetes/pki/etcd/healthcheck-client.key
#-- /etc/kubernetes/pki/etcd/healthcheck-client.crt

#-- /etc/kubernetes/pki/etcd/peer.key
#-- /etc/kubernetes/pki/etcd/peer.crt

#-- /etc/kubernetes/pki/etcd/server.key
#-- /etc/kubernetes/pki/etcd/server.crt

不移动证书会有如下提示
#[certificates] Using the existing apiserver certificate and key.
#[certificates] Using the existing apiserver-kubelet-client certificate and key.
#[certificates] Using the existing front-proxy-client certificate and key.
#[certificates] Using the existing etcd/server certificate and key.
#[certificates] Using the existing etcd/peer certificate and key.
#[certificates] Using the existing etcd/healthcheck-client certificate and key.
#[certificates] Using the existing apiserver-etcd-client certificate and key.
#[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
#[certificates] Using the existing sa key.
生成新配置文件
1
kubeadm alpha phase kubeconfig all --apiserver-advertise-address=${MASTER_API_SERVER_IP}
将新生成的admin配置文件覆盖掉原本的admin文件
1
2
3
4
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
sudo chmod 777 $HOME/.kube/config
完成后重启kube-apiserver,kube-controller,kube-scheduler,etcd这4个容器
如果有多台master,则将第一台生成的相关证书拷贝到其余master即可。

离线一键安装包

k8s 离线一键安装包教程&&地址:一键安装

kubeadm 1.14 证书调整教程

教程地址

-------------本文结束感谢您的阅读-------------

本文标题:Kubeadm证书过期时间调整

文章作者:icyboy

发布时间:2018年12月11日 - 20:00

最后更新:2019年08月21日 - 13:59

原始链接:http://icyxp.github.io/blog/2018/12/k8s-kubeadm-ca-upgdate.html

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。