kuberenetes 云应用实践

如何使用云盘

aws ebs

设置IAM角色并分配到机器上

具有controlplane(控制面板)角色的节点的IAM策略:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyVolume",
"ec2:AttachVolume",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateRoute",
"ec2:DeleteRoute",
"ec2:DeleteSecurityGroup",
"ec2:DeleteVolume",
"ec2:DetachVolume",
"ec2:RevokeSecurityGroupIngress",
"ec2:DescribeVpcs",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:AttachLoadBalancerToSubnets",
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateLoadBalancerPolicy",
"elasticloadbalancing:CreateLoadBalancerListeners",
"elasticloadbalancing:ConfigureHealthCheck",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DeleteLoadBalancerListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DetachLoadBalancerFromSubnets",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancerPolicies",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
"iam:CreateServiceLinkedRole",
"kms:DescribeKey"
],
"Resource": [
"*"
]
}
]
}

worker角色的节点的IAM策略

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRegions",
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage"
],
"Resource": "*"
}
]
}

配置ClusterID

需要在机器上打上标签,如果不配置会产生以下错误

1
failed to run Kubelet: could not init cloud provider "aws": AWS cloud failed to find ClusterID

标 签 Key : kubernetes.io/cluster/CLUSTERID
标签Value: owned

CLUSTERID 可以为任何值

kubernetes配置

需要kube-apiserver, kube-controller-manager以及kubelet 启动加上 --cloud-provider=aws参数 , 注意需要在所有node上的kubelet 开启--cloud-provider=aws

使用kubeadm 初始化kubernetes设置

1
2
3
4
5
6
7
8
9
10
apiServerExtraArgs:
cloud-provider: aws
controllerManagerExtraArgs:
cloud-provider: aws

nodeRegistration:
#建议将Kubelet的名称设置为EC2中节点的私有DNS条目(这可确保它与主机名匹配,如本文前面所述)
name: ip-10-15-30-45.cn-northwest-1.compute.internal
kubeletExtraArgs:
cloud-provider: aws

使用kubeadm 命令安装后的情况下

  • kube-apiserver 修改 /etc/kubernetes/manifests/kube-apiserver.yaml下的启动参数
  • kube-controller-manager 修改 /etc/kubernetes/manifests/kube-controller-manager.yaml下的启动参数
  • kubelet 修改 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 下的启动参数

开始使用EBS

创建StorageClass
1
2
3
4
5
6
7
8
9
10
11
# cat standard-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: ap-northeast-1b
reclaimPolicy: Delete # 在删除 Pod 的同时,也删除 EBS 磁盘
# kubectl apply -f standard-storageclass.yaml
创建PVC
1
2
3
4
5
6
7
8
9
10
11
12
13
# cat nginx-pv-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: standard
# kubectl apply -f nginx-pv-claim.yaml
Pod
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# cat nginx-pv-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-pv
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- name: nginx-pv-claim
mountPath: /data
volumes:
- name: nginx-pv-claim
persistentVolumeClaim:
claimName: nginx-pv-claim
# kubectl apply -f nginx-pv-pod.yaml

参考链接

阿里云云盘

创建阿里云Provisioner 控制器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: alicloud-disk-common
provisioner: alicloud/disk
parameters:
type: cloud

---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: alicloud-disk-efficiency
provisioner: alicloud/disk
parameters:
type: cloud_efficiency

---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: alicloud-disk-ssd
provisioner: alicloud/disk
parameters:
type: cloud_ssd

---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: alicloud-disk-available
provisioner: alicloud/disk
parameters:
type: available

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: alicloud-disk-controller-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: alicloud-disk-controller
namespace: kube-system

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: run-alicloud-disk-controller
subjects:
- kind: ServiceAccount
name: alicloud-disk-controller
namespace: kube-system
roleRef:
kind: ClusterRole
name: alicloud-disk-controller-runner
apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-config
namespace: kube-system
data:
special.keyid: 你阿里云的access key
special.keysecret: 你阿里云的secret key

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: alicloud-disk-controller
namespace: kube-system
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: alicloud-disk-controller
spec:
tolerations:
- effect: NoSchedule
operator: Exists
key: node-role.kubernetes.io/master
- effect: NoSchedule
operator: Exists
key: node.cloudprovider.kubernetes.io/uninitialized
nodeSelector:
node-role.kubernetes.io/master: ""
serviceAccount: alicloud-disk-controller
containers:
- name: alicloud-disk-controller
image: icyboy/alicloud-disk-controller:3652ddf
env:
- name: ACCESS_KEY_ID
valueFrom:
configMapKeyRef:
name: cloud-config
key: special.keyid
- name: ACCESS_KEY_SECRET
valueFrom:
configMapKeyRef:
name: cloud-config
key: special.keysecret
volumeMounts:
- name: cloud-config
mountPath: /etc/kubernetes/
- name: logdir
mountPath: /var/log/alicloud/
volumes:
- name: cloud-config
hostPath:
path: /etc/kubernetes/
- name: logdir
hostPath:
path: /var/log/alicloud/

创建PVC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: disk
spec:
accessModes:
- ReadWriteOnce
#alicloud-disk-common 普通云盘
#alicloud-disk-efficiency 高效云盘
#alicloud-disk-ssd ssd云盘
#alicloud-disk-available
storageClassName: alicloud-disk-common
resources:
requests:
storage: 20Gi

效果

1
2
3
4
5
6
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
disk Bound d-bp1cz8sslda31ld2snbq 20Gi RWO alicloud-disk-common 11s
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
d-bp1cz8sslda31ld2snbq 20Gi RWO Delete Bound default/disk alicloud-disk-common 14s

参考链接

IAM 使用

kube2iam in aws iam

通过AWS IAM创建一个名为kube2iam的策略,策略规则如下

1
2
3
4
5
6
7
8
9
10
11
12
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Resource": "arn:aws-cn:iam::955466075186:role/k8s-*"
}
]
}

通过AWS IAM创建一个名为:node-k8s-role的角色,并将kube2iam策略赋予它,注意!!同时将这个角色赋予到机器上

通过AWS IAM创建一个名为:k8s-kube2iam-demo的角色,将AmazonS3FullAccess策略赋予它,同时编辑信任关系,调整为如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com.cn"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws-cn:iam::955466075186:role/node-k8s-role"
},
"Action": "sts:AssumeRole"
}
]
}

部署kube2iam

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
# Source: kube2iam/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: kube2iam
chart: kube2iam-1.0.0
heritage: Tiller
release: ardent-wildebeest
name: ardent-wildebeest-kube2iam
---
# Source: kube2iam/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
labels:
app: kube2iam
chart: kube2iam-1.0.0
heritage: Tiller
release: ardent-wildebeest
name: ardent-wildebeest-kube2iam
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
verbs:
- list
- watch
- get
---
# Source: kube2iam/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
labels:
app: kube2iam
chart: kube2iam-1.0.0
heritage: Tiller
release: ardent-wildebeest
name: ardent-wildebeest-kube2iam
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ardent-wildebeest-kube2iam
subjects:
- kind: ServiceAccount
name: ardent-wildebeest-kube2iam
namespace: default
---
# Source: kube2iam/templates/daemonset.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
app: kube2iam
chart: kube2iam-1.0.0
heritage: Tiller
release: ardent-wildebeest
name: ardent-wildebeest-kube2iam
spec:
template:
metadata:
labels:
app: kube2iam
release: ardent-wildebeest
spec:
containers:
- name: kube2iam
image: "jtblin/kube2iam:0.10.4"
imagePullPolicy: "IfNotPresent"
args:
- "--host-interface=cali+"
- "--node=$(NODE_NAME)"
- "--host-ip=$(HOST_IP)"
- "--iptables=true"
- "--base-role-arn=arn:aws-cn:iam::955466075186:role/node-k8s-role"
- "--app-port=8181"
- "--use-regional-sts-endpoint"
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: AWS_REGION
value: cn-northwest-1
ports:
- name: http
containerPort: 8181
livenessProbe:
httpGet:
path: /healthz
port: 8181
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
failureThreshold: 3
timeoutSeconds: 1
resources:
{}

securityContext:
privileged: true
hostNetwork: true
serviceAccountName: ardent-wildebeest-kube2iam
tolerations:
[]

updateStrategy:
type: OnDelete

测试pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: apps/v1
kind: Deployment
metadata:
name: awscli-deployment
spec:
selector:
matchLabels:
app: awscli
replicas: 1
template:
metadata:
annotations:
iam.amazonaws.com/role: arn:aws-cn:iam::955466075186:role/k8s/k8s-kube2iam-demo
labels:
app: awscli
spec:
containers:
- name: awscli
image: mesosphere/aws-cli
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 3000; done;" ]

云应用疑难杂症

使用云负载均衡器代理APIServer问题

阿里云/AWS的负载均衡是四层TCP负责,不支持后端ECS实例既作为Real Server又作为客户端向所在的负载均衡实例发送请求。因为返回的数据包只在云服务器内部转发,不经过负载均衡,所以在后端ECS实例上去访问负载均衡的服务地址是不通的。什么意思?就是如果你要使用阿里云的SLB的话,那么你不能在apiserver节点上使用SLB(比如在apiserver 上安装kubectl,然后将apiserver的地址设置为SLB的负载地址使用),因为这样的话就可能造成回环了,所以简单的做法是另外用两个新的节点做HA实例,然后将这两个实例添加到SLB 机器组中。

  • aws 使用nlb,指定ip
  • aliyun 在apiserver上搭建keepalived
-------------本文结束感谢您的阅读-------------

本文标题:kuberenetes 云应用实践

文章作者:icyboy

发布时间:2019年03月15日 - 12:00

最后更新:2019年07月29日 - 10:24

原始链接:http://team.jiunile.com/blog/2019/03/k8s-cloud.html

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。