使用Kubeadm 1.11.x + etcd集群 部署多Master集群

发表于 | 分类于 ， ， | | 热度: ℃
字数统计: 4,993 字 | 阅读时长 ≈ 28 分钟

环境介绍

IP Hostname 备注
172.19.170.183 master-1 私有云安装keepalived + haproxy + etcd
172.19.170.184 master-2 私有云安装keepalived + haproxy + etcd
172.19.170.185 master-3 私有云安装keepalived + haproxy + etcd
172.19.170.186 node-1 无
172.19.170.187 node-2 无
172.19.170.100 vip 私有云使用keepalive+haproxy搭建，公有云使用内部负载均衡器

环境初始化

修改hosts信息，在所有master机器上运行

1
2
3
4
5
cat <<EOF >> /etc/hosts
172.19.170.183 master-1
172.19.170.184 master-2
172.19.170.185 master-3
EOF

在master-1 机器上执行以下命令：

1
2
3
ssh-keygen -t rsa
ssh-copy-id root@master-2
ssh-copy-id root@master-3

优化所有master & node 节点的机器环境

在所有master & node 机器上都要执行以下命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126

#################################################################
# 安装4.18版本内核
# 由于最新稳定版4.19内核将nf_conntrack_ipv4更名为nf_conntrack，目前的kube-proxy不支持在4.19版本内核下开启ipvs
# 详情可以查看：https://github.com/kubernetes/kubernetes/issues/70304
# 对于该问题的修复10月30日刚刚合并到代码主干，所以目前还没有包含此修复的kubernetes版本发出
# 可以选择安装4.18版本内核，或者不开启IPVS
#################################################################
############################# start #############################
# 升级内核
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm ;yum --enablerepo=elrepo-kernel install kernel-ml-devel kernel-ml -y
 
# 检查默认内核版本是否大于4.14，否则请调整默认启动参数
grub2-editenv list
 
#重启以更换内核
reboot

# 开启ip_vs
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules_dir="/usr/lib/modules/\`uname -r\`/kernel/net/netfilter/ipvs"
for i in \`ls \$ipvs_modules_dir | sed  -r 's#(.*).ko.*#\1#'\`; do
    /sbin/modinfo -F filename \$i  &> /dev/null
    if [ \$? -eq 0 ]; then
        /sbin/modprobe \$i
    fi
done
EOF

#检查是否开启了ip_vs
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs

############################# end #############################

# 优化内核
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.netfilter.nf_conntrack_max=2310720
fs.may_detach_mounts = 1
fs.inotify.max_user_watches=1048576
fs.inotify.max_user_instances = 8192
fs.file-max=52706963
fs.nr_open=52706963
vm.swappiness = 0
vm.panic_on_oom=0
vm.overcommit_memory=1
EOF
sysctl --system

# 执行sysctl报错请参考centos7添加bridge-nf-call-ip6tables出现No such file or directory: https://blog.csdn.net/airuozhaoyang/article/details/40534953

# 优化文件打开数
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

# 关闭Selinux/firewalld
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

# 关闭交换分区
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

# 同步时间
yum install -y ntpdate
ntpdate -u ntp.api.bz

# 设置kubernet yum源
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 安装kubernet
yum install kubeadm-1.11.5-0 kubelet-1.11.5-0 kubectl-1.11.5-0 --disableexcludes=kubernetes -y
systemctl enable kubelet

# 设置docker源
cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name=Docker Repository
baseurl=https://mirrors.tuna.tsinghua.edu.cn/docker/yum/repo/centos7
enabled=1
gpgcheck=1
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/docker/yum/gpg
EOF

# 安装docker-engine
yum -y install docker-engine-1.13.1 --disableexcludes=docker
systemctl enable docker
systemctl start docker

# 设置kubelet启动配置
cgroupDriver=$(docker info|grep Cg)
driver=${cgroupDriver##*: }
echo "driver is ${driver}"

cat <<EOF > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=${driver}"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
Environment="KUBELET_EXTRA_ARGS=--system-reserved=cpu=200m,memory=250Mi --kube-reserved=cpu=200m,memory=250Mi --eviction-soft=memory.available<500Mi,nodefs.available<2Gi --eviction-soft-grace-period=memory.available=1m30s,nodefs.available=1m30s --eviction-max-pod-grace-period=120 --eviction-hard=memory.available<500Mi,nodefs.available<1Gi --eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi --node-status-update-frequency=10s --eviction-pressure-transition-period=30s"
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_SYSTEM_PODS_ARGS \$KUBELET_NETWORK_ARGS \$KUBELET_DNS_ARGS \$KUBELET_AUTHZ_ARGS \$KUBELET_CADVISOR_ARGS \$KUBELET_CGROUP_ARGS \$KUBELET_CERTIFICATE_ARGS \$KUBELET_EXTRA_ARGS
EOF

搭建keepalived + haproxy (公有云不需要搭建)

在master-1上配置

keepalived 安装配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
yum install -y keepalived

cat << EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived
    global_defs {
        notification_email {
            root@localhost      #发送邮箱
        }
        notification_email_from keepalived@localhost    #邮箱地址   
        smtp_server 127.0.0.1   #邮件服务器地址
        smtp_connect_timeout 30 
        router_id node1         #主机名，每个节点不同即可
    }       
        
vrrp_instance VI_1 {
    state MASTER        #在另一个节点上为BACKUP
    interface eth0      #IP地址漂移到的网卡
    virtual_router_id 6 #多个节点必须相同
    priority 100        #优先级，备用节点的值必须低于主节点的值
    advert_int 1        #通告间隔1秒
    authentication {
        auth_type PASS      #预共享密钥认证
        auth_pass 571f97b2  #密钥
    }
    virtual_ipaddress {
        172.19.170.100    #VIP地址
    }
}    
EOF

systemctl enable keepalived
systemctl start keepalived

haproxy 安装配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
yum install  -y haproxy

cat << EOF > /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

defaults
    mode                    tcp
    log                     global
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend kubernetes
    bind *:8443
    mode tcp
    default_backend kubernetes-master

backend kubernetes-master
    balance roundrobin
    server master-1 172.19.170.183:6443 check maxconn 2000
    server master-2 172.19.170.183:6443 check maxconn 2000
    server master-3 172.19.170.183:6443 check maxconn 2000
EOF

systemctl enable haproxy
systemctl start haproxy

在master-2上配置

keepalived 安装配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
yum install -y keepalived

cat << EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived
    global_defs {
        notification_email {
            root@localhost      #发送邮箱
        }
        notification_email_from keepalived@localhost    #邮箱地址   
        smtp_server 127.0.0.1   #邮件服务器地址
        smtp_connect_timeout 30 
        router_id node2         #主机名，每个节点不同即可
    }       
        
vrrp_instance VI_1 {
    state BACKUP        #在另一个节点上为BACKUP
    interface eth0      #IP地址漂移到的网卡
    virtual_router_id 6 #多个节点必须相同
    priority 80        #优先级，备用节点的值必须低于主节点的值
    advert_int 1        #通告间隔1秒
    authentication {
        auth_type PASS      #预共享密钥认证
        auth_pass 571f97b2  #密钥
    }
    virtual_ipaddress {
        172.19.170.100    #VIP地址
    }
}    
EOF

systemctl enable keepalived
systemctl start keepalived

haproxy 安装配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
yum install  -y haproxy

cat << EOF > /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

defaults
    mode                    tcp
    log                     global
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend kubernetes
    bind *:8443
    mode tcp
    default_backend kubernetes-master

backend kubernetes-master
    balance roundrobin
    server master-1 172.19.170.183:6443 check maxconn 2000
    server master-2 172.19.170.183:6443 check maxconn 2000
    server master-3 172.19.170.183:6443 check maxconn 2000
EOF

systemctl enable haproxy
systemctl start haproxy

在master-3上配置

keepalived 安装配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
yum install -y keepalived

cat << EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived
    global_defs {
        notification_email {
            root@localhost      #发送邮箱
        }
        notification_email_from keepalived@localhost    #邮箱地址   
        smtp_server 127.0.0.1   #邮件服务器地址
        smtp_connect_timeout 30 
        router_id node3         #主机名，每个节点不同即可
    }       
        
vrrp_instance VI_1 {
    state BACKUP        #在另一个节点上为BACKUP
    interface eth0      #IP地址漂移到的网卡
    virtual_router_id 6 #多个节点必须相同
    priority 80        #优先级，备用节点的值必须低于主节点的值
    advert_int 1        #通告间隔1秒
    authentication {
        auth_type PASS      #预共享密钥认证
        auth_pass 571f97b2  #密钥
    }
    virtual_ipaddress {
        172.19.170.100    #VIP地址
    }
}    
EOF

systemctl enable keepalived
systemctl start keepalived

haproxy 安装配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
yum install  -y haproxy

cat << EOF > /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

defaults
    mode                    tcp
    log                     global
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m

frontend kubernetes
    bind *:8443
    mode tcp
    default_backend kubernetes-master

backend kubernetes-master
    balance roundrobin
    server master-1 172.19.170.183:6443 check maxconn 2000
    server master-2 172.19.170.183:6443 check maxconn 2000
    server master-3 172.19.170.183:6443 check maxconn 2000
EOF

systemctl enable haproxy
systemctl start haproxy

搭建etcd

在master-1上执行

安装cfssl

1
2
3
4
wget -O /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget -O /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget -O /bin/cfssl-certinfo  https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
for cfssl in `ls /bin/cfssl*`;do chmod +x $cfssl;done;

配置生成etcd 证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
# 配置证书
mkdir -pv $HOME/ssl && cd $HOME/ssl

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h" //10年
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF


cat > etcd-ca-csr.json << EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JiangSu",
      "L": "SuZhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

cat > etcd-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "172.19.170.183",
      "172.19.170.184",
      "172.19.170.185"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "JiangSu",
            "L": "SuZhou",
            "O": "etcd",
            "OU": "Etcd Security"
        }
    ]
}
EOF

# 生成证书并复制证书至其他etcd节点
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

# 移动证书
mkdir -pv /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl

# 将证书拷贝到其它节点
scp -r /etc/etcd master-2:/etc/
scp -r /etc/etcd master-3:/etc/

安装配置etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
yum install -y etcd 
cat << EOF > /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.19.170.183:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://172.19.170.183:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.19.170.183:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379,https://172.19.170.183:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.19.170.183:2380,etcd2=https://172.19.170.184:2380,etcd3=https://172.19.170.185:2380"
ETCD_INITIAL_CLUSTER_TOKEN="BigBoss"
ETCD_INITIAL_CLUSTER_STATE="new" #这里要设置为new
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
EOF

chown -R etcd.etcd /etc/etcd
systemctl enable etcd
#启动会卡住，等待其它节点的加入
systemctl start etcd

在master-2 上执行

安装配置etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
yum install -y etcd 

cat << EOF > /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.19.170.184:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://172.19.170.184:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.19.170.184:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379,https://172.19.170.184:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.19.170.183:2380,etcd2=https://172.19.170.184:2380,etcd3=https://172.19.170.185:2380"
ETCD_INITIAL_CLUSTER_TOKEN="BigBoss"
ETCD_INITIAL_CLUSTER_STATE="existing" #如果etcd 启动报错，可先把这行注释掉，然后在启动
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
EOF

chown -R etcd.etcd /etc/etcd
systemctl enable etcd
systemctl start etcd

在master-3 上执行

安装配置etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat << EOF > /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.19.170.185:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://172.19.170.185:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.19.170.185:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379,https://172.19.170.185:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.19.170.183:2380,etcd2=https://172.19.170.184:2380,etcd3=https://172.19.170.185:2380"
ETCD_INITIAL_CLUSTER_TOKEN="BigBoss"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
EOF

chown -R etcd.etcd /etc/etcd
systemctl enable etcd
systemctl start etcd

etcd 集群验证

随意在任何一台etcd (master) 节点上进行验证

1
2
etcdctl --endpoints "https://172.19.170.183:2379,https://172.19.170.184:2379,https://172.19.170.185:2379"   --ca-file=/etc/etcd/ssl/etcd-ca.pem  \
--cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   cluster-health

搭建kubernetes master 集群

初始化master-1

在master-1 上执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
token=$(kubeadm token generate)
echo $token
cat << EOF > $HOME/kubeadm-1.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.5
api:
  advertiseAddress: 172.19.170.183
  bindPort: 6443
  controlPlaneEndpoint: 172.19.170.100:8443    
apiServerCertSANs:
- 172.19.170.183
- 172.19.170.184
- 172.19.170.185
- master-1
- master-2
- master-3
- 172.19.170.100
- 127.0.0.1
kubeProxy:
  config:
    mode: iptables
etcd:
  external:
    endpoints:
    - "https://172.19.170.183:2379"
    - "https://172.19.170.184:2379"
    - "https://172.19.170.185:2379"
    caFile: /etc/etcd/ssl/etcd-ca.pem
    certFile: /etc/etcd/ssl/etcd.pem
    keyFile: /etc/etcd/ssl/etcd-key.pem
token: $token
tokenTTL: "0"
networking:
  podSubnet: 192.168.0.0/16
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
EOF

kubeadm config images pull --config $HOME/kubeadm-1.yaml
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1  k8s.gcr.io/pause:3.1
kubeadm init --config $HOME/kubeadm-1.yaml

mkdir -p $HOME/.kube
cp -f /etc/kubernetes/admin.conf ${HOME}/.kube/config

# 查看etcd是否启动，等待返回Running 说明启动成功
kubectl get pods -n kube-system 2>&1|grep etcd|awk '{print $3}'

# 设置其他master
ssh master-2 "mkdir -p /etc/kubernetes/pki/etcd; mkdir -p ~/.kube"
ssh master-3 "mkdir -p /etc/kubernetes/pki/etcd; mkdir -p ~/.kube"

# 拷贝文件至 master-2
scp /etc/kubernetes/pki/ca.crt master-2:/etc/kubernetes/pki/ca.crt
scp /etc/kubernetes/pki/ca.key master-2:/etc/kubernetes/pki/ca.key
scp /etc/kubernetes/pki/sa.key master-2:/etc/kubernetes/pki/sa.key
scp /etc/kubernetes/pki/sa.pub master-2:/etc/kubernetes/pki/sa.pub
scp /etc/kubernetes/pki/front-proxy-ca.crt master-2:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.key master-2:/etc/kubernetes/pki/front-proxy-ca.key
scp /etc/kubernetes/admin.conf master-2:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf master-2:~/.kube/config

#拷贝文件至 master-3
scp /etc/kubernetes/pki/ca.crt master-3:/etc/kubernetes/pki/ca.crt
scp /etc/kubernetes/pki/ca.key master-3:/etc/kubernetes/pki/ca.key
scp /etc/kubernetes/pki/sa.key master-3:/etc/kubernetes/pki/sa.key
scp /etc/kubernetes/pki/sa.pub master-3:/etc/kubernetes/pki/sa.pub
scp /etc/kubernetes/pki/front-proxy-ca.crt master-3:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.key master-3:/etc/kubernetes/pki/front-proxy-ca.key
scp /etc/kubernetes/admin.conf master-3:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf master-3:~/.kube/config

初始化master-2

在master-2 上执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1  k8s.gcr.io/pause:3.1

cat << EOF > $HOME/kubeadm-2.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.5
api:
  advertiseAddress: 172.19.170.184
  bindPort: 6443
  controlPlaneEndpoint: 172.19.170.100:8443 
apiServerCertSANs:
- 172.19.170.183
- 172.19.170.184
- 172.19.170.185
- master-1
- master-2
- master-3
- 172.19.170.100
- 127.0.0.1
kubeProxy:
  config:
    mode: iptables
etcd:
  external:
    endpoints:
    - "https://172.19.170.183:2379"
    - "https://172.19.170.184:2379"
    - "https://172.19.170.185:2379"
    caFile: /etc/etcd/ssl/etcd-ca.pem
    certFile: /etc/etcd/ssl/etcd.pem
    keyFile: /etc/etcd/ssl/etcd-key.pem
token: $token
tokenTTL: "0"
networking:
  podSubnet: 192.168.0.0/16
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
EOF

kubeadm alpha phase certs all --config $HOME/kubeadm-2.yaml
kubeadm alpha phase kubeconfig controller-manager --config $HOME/kubeadm-2.yaml
kubeadm alpha phase kubeconfig scheduler --config $HOME/kubeadm-2.yaml
kubeadm alpha phase kubelet config write-to-disk --config $HOME/kubeadm-2.yaml
kubeadm alpha phase kubelet write-env-file --config $HOME/kubeadm-2.yaml
kubeadm alpha phase kubeconfig kubelet --config $HOME/kubeadm-2.yaml
systemctl restart kubelet
kubeadm alpha phase kubeconfig all --config $HOME/kubeadm-2.yaml
kubeadm alpha phase controlplane all --config $HOME/kubeadm-2.yaml
kubeadm alpha phase mark-master --config $HOME/kubeadm-2.yaml

初始化master-3

在master-3 上执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1  k8s.gcr.io/pause:3.1

cat << EOF > $HOME/kubeadm-3.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.5
api:
  advertiseAddress: 172.19.170.185
  bindPort: 6443
  controlPlaneEndpoint: 172.19.170.100:8443    
apiServerCertSANs:
- 172.19.170.183
- 172.19.170.184
- 172.19.170.185
- master-1
- master-2
- master-3
- 172.19.170.100
- 127.0.0.1
kubeProxy:
  config:
    mode: iptables
etcd:
  external:
    endpoints:
    - "https://172.19.170.183:2379"
    - "https://172.19.170.184:2379"
    - "https://172.19.170.185:2379"
    caFile: /etc/etcd/ssl/etcd-ca.pem
    certFile: /etc/etcd/ssl/etcd.pem
    keyFile: /etc/etcd/ssl/etcd-key.pem
token: $token
tokenTTL: "0"
networking:
  podSubnet: 192.168.0.0/16
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
EOF

kubeadm alpha phase certs all --config $HOME/kubeadm-3.yaml
kubeadm alpha phase kubeconfig controller-manager --config $HOME/kubeadm-3.yaml
kubeadm alpha phase kubeconfig scheduler --config $HOME/kubeadm-3.yaml
kubeadm alpha phase kubelet config write-to-disk --config $HOME/kubeadm-3.yaml
kubeadm alpha phase kubelet write-env-file --config $HOME/kubeadm-3.yaml
kubeadm alpha phase kubeconfig kubelet --config $HOME/kubeadm-3.yaml
systemctl restart kubelet
kubeadm alpha phase kubeconfig all --config $HOME/kubeadm-3.yaml
kubeadm alpha phase controlplane all --config $HOME/kubeadm-3.yaml
kubeadm alpha phase mark-master --config $HOME/kubeadm-3.yaml
1
2
3
######################################################
################## 至此集群搭建完成 #####################
######################################################

开启kubernetes 插件

在随意一台master执行

安装calico 网络插件

确保 Kubernetes controller manager (/etc/kubernetes/manifests/kube-controller-manager.yaml) 设置了以下标志：--cluster-cidr=192.168.0.0/16和--allocate-node-cidrs=true。

提示：在kubeadm上，您可以传递--pod-network-cidr=192.168.0.0/16 给kubeadm以设置两个Kubernetes控制器标志。

在ConfigMap命名中calico-config，找到typha_service_name，删除none值，并替换为calico-typha

我们建议每200个节点至少复制一个副本，不超过20个副本。在生产中，我们建议至少使用三个副本来减少滚动升级和故障的影响。

警告：如果您设置typha_service_name而不增加其默认值为0 Felix的副本计数将尝试连接到Typha，找不到要连接的Typha实例，并且无法启动。

1
kubectl apply -f www.jiunile.com/k8s/plugin/1.11.5/calico.yaml

安装heapster 监控插件

1
kubectl apply -f www.jiunile.com/k8s/plugin/1.11.5/heapster.yaml

安装dashboard 插件

1
kubectl apply -f www.jiunile.com/k8s/plugin/1.11.5/dashboard.yaml

获取加入master集群命令

1
kubeadm token create --print-join-command

初始化所有kubernetes node

1
2
3
4
5
6
7
8
9
10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1  k8s.gcr.io/pause:3.1

# 拉取calico网络对应需要的镜像，拉取比教缓慢
docker pull quay.io/calico/typha:v3.2.4
docker pull quay.io/calico/node:v3.2.4
docker pull quay.io/calico/cni:v3.2.4

# 加入集群
kubeadm join 172.19.170.100:8443 --token mrvxeb.mfr1wx6upq5bbwqt --discovery-token-ca-cert-hash sha256:8ee893d8bf69f1f622f55a983f1401a9f2a236ffa9248894cb614c972de47f48

可以在master机器上查看对应的node状态

1
2
kubectl get node
kubectl get pod -n kube-system

测试

测试应用和dns是否正常

在随意一台master机器上运行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# 部署一个服务
cd /root && mkdir nginx && cd nginx
cat << EOF > nginx.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
  - port: 80
    nodePort: 31000
    name: nginx-port
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
EOF

kubectl apply -f nginx.yaml

# 创建一个POD来测试DNS解析
kubectl run curl --image=radial/busyboxplus:curl -i --tty
nslookup kubernetes
# Server:    10.96.0.10
# Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

# Name:      kubernetes
# Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
curl nginx
# 有返回结果表明ok
exit
kubectl delete deployment curl

测试master & Haproxy高可用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#随机关掉一台master机器
init 0

# 在别的master机器上执行命令
kubectl get node
#NAME      STATUS     ROLES     AGE       VERSION
#master-1   NotReady   master    1h        v1.11.5
#master-2   Ready      master    59m       v1.11.5
#master-3   Ready      master    59m       v1.11.5
#node-1     Ready      <none>    58m       v1.11.5
#node-2     Ready      <none>    58m       v1.11.5

#重新创建一个pod，看看是否能创建成功
kubectl run curl --image=radial/busyboxplus:curl -i --tty
exit
kubectl delete deployment curl
-------------本文结束感谢您的阅读-------------

本文标题:使用Kubeadm 1.11.x + etcd集群 部署多Master集群

文章作者:icyboy

发布时间:2018年12月29日 - 12:00

最后更新:2019年02月15日 - 14:36

原始链接:http://team.jiunile.com/blog/2018/12/k8s-kubeadm-etcd-ha-1-11-x.html

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。