kuberenetes 云应用实践

发表于 | 分类于 ， | | 热度: ℃
字数统计: 2,207 字 | 阅读时长 ≈ 12 分钟

如何使用云盘

aws ebs

设置IAM角色并分配到机器上

具有controlplane(控制面板)角色的节点的IAM策略：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
{
"Version": "2012-10-17",
"Statement": [
  {
    "Effect": "Allow",
    "Action": [
      "autoscaling:DescribeAutoScalingGroups",
      "autoscaling:DescribeLaunchConfigurations",
      "autoscaling:DescribeTags",
      "ec2:DescribeInstances",
      "ec2:DescribeRegions",
      "ec2:DescribeRouteTables",
      "ec2:DescribeSecurityGroups",
      "ec2:DescribeSubnets",
      "ec2:DescribeVolumes",
      "ec2:CreateSecurityGroup",
      "ec2:CreateTags",
      "ec2:CreateVolume",
      "ec2:ModifyInstanceAttribute",
      "ec2:ModifyVolume",
      "ec2:AttachVolume",
      "ec2:AuthorizeSecurityGroupIngress",
      "ec2:CreateRoute",
      "ec2:DeleteRoute",
      "ec2:DeleteSecurityGroup",
      "ec2:DeleteVolume",
      "ec2:DetachVolume",
      "ec2:RevokeSecurityGroupIngress",
      "ec2:DescribeVpcs",
      "elasticloadbalancing:AddTags",
      "elasticloadbalancing:AttachLoadBalancerToSubnets",
      "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
      "elasticloadbalancing:CreateLoadBalancer",
      "elasticloadbalancing:CreateLoadBalancerPolicy",
      "elasticloadbalancing:CreateLoadBalancerListeners",
      "elasticloadbalancing:ConfigureHealthCheck",
      "elasticloadbalancing:DeleteLoadBalancer",
      "elasticloadbalancing:DeleteLoadBalancerListeners",
      "elasticloadbalancing:DescribeLoadBalancers",
      "elasticloadbalancing:DescribeLoadBalancerAttributes",
      "elasticloadbalancing:DetachLoadBalancerFromSubnets",
      "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
      "elasticloadbalancing:ModifyLoadBalancerAttributes",
      "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
      "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
      "elasticloadbalancing:AddTags",
      "elasticloadbalancing:CreateListener",
      "elasticloadbalancing:CreateTargetGroup",
      "elasticloadbalancing:DeleteListener",
      "elasticloadbalancing:DeleteTargetGroup",
      "elasticloadbalancing:DescribeListeners",
      "elasticloadbalancing:DescribeLoadBalancerPolicies",
      "elasticloadbalancing:DescribeTargetGroups",
      "elasticloadbalancing:DescribeTargetHealth",
      "elasticloadbalancing:ModifyListener",
      "elasticloadbalancing:ModifyTargetGroup",
      "elasticloadbalancing:RegisterTargets",
      "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
      "iam:CreateServiceLinkedRole",
      "kms:DescribeKey"
    ],
    "Resource": [
      "*"
    ]
  }
]
}

worker角色的节点的IAM策略

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "ec2:DescribeInstances",
            "ec2:DescribeRegions",
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:BatchGetImage"
        ],
        "Resource": "*"
    }
]
}

配置ClusterID

需要在机器上打上标签，如果不配置会产生以下错误

1
failed to run Kubelet: could not init cloud provider "aws": AWS cloud failed to find ClusterID

标 签 Key ： kubernetes.io/cluster/CLUSTERID
标签Value： owned

CLUSTERID 可以为任何值

kubernetes配置

需要kube-apiserver, kube-controller-manager以及kubelet 启动加上 --cloud-provider=aws参数 , 注意需要在所有node上的kubelet 开启--cloud-provider=aws

使用kubeadm 初始化kubernetes设置

1
2
3
4
5
6
7
8
9
10
apiServerExtraArgs:
  cloud-provider: aws
controllerManagerExtraArgs:
  cloud-provider: aws

nodeRegistration:
  #建议将Kubelet的名称设置为EC2中节点的私有DNS条目（这可确保它与主机名匹配，如本文前面所述）
  name: ip-10-15-30-45.cn-northwest-1.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws

使用kubeadm 命令安装后的情况下

  • kube-apiserver 修改 /etc/kubernetes/manifests/kube-apiserver.yaml下的启动参数
  • kube-controller-manager 修改 /etc/kubernetes/manifests/kube-controller-manager.yaml下的启动参数
  • kubelet 修改 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 下的启动参数

开始使用EBS

创建StorageClass
1
2
3
4
5
6
7
8
9
10
11
# cat standard-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  zone: ap-northeast-1b
reclaimPolicy: Delete       # 在删除 Pod 的同时，也删除 EBS 磁盘
# kubectl apply -f standard-storageclass.yaml
创建PVC
1
2
3
4
5
6
7
8
9
10
11
12
13
# cat nginx-pv-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: standard
# kubectl apply -f nginx-pv-claim.yaml
Pod
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# cat nginx-pv-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pv
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    volumeMounts:
    - name: nginx-pv-claim
      mountPath: /data
  volumes:
  - name: nginx-pv-claim
    persistentVolumeClaim:
      claimName: nginx-pv-claim
# kubectl apply -f nginx-pv-pod.yaml

参考链接

阿里云云盘

创建阿里云Provisioner 控制器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-common
provisioner: alicloud/disk
parameters:
  type: cloud

---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-efficiency
provisioner: alicloud/disk
parameters:
  type: cloud_efficiency

---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-ssd
provisioner: alicloud/disk
parameters:
  type: cloud_ssd

---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: alicloud-disk-available
provisioner: alicloud/disk
parameters:
  type: available

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: alicloud-disk-controller-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: alicloud-disk-controller
  namespace: kube-system

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: run-alicloud-disk-controller
subjects:
  - kind: ServiceAccount
    name: alicloud-disk-controller
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: alicloud-disk-controller-runner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system
data:
  special.keyid: 你阿里云的access key
  special.keysecret: 你阿里云的secret key

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: alicloud-disk-controller
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: alicloud-disk-controller
    spec:
      tolerations:
      - effect: NoSchedule
        operator: Exists
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        operator: Exists
        key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
         node-role.kubernetes.io/master: ""
      serviceAccount: alicloud-disk-controller
      containers:
        - name: alicloud-disk-controller
          image: icyboy/alicloud-disk-controller:3652ddf
          env:
          - name: ACCESS_KEY_ID
            valueFrom:
              configMapKeyRef:
                name: cloud-config
                key: special.keyid
          - name: ACCESS_KEY_SECRET
            valueFrom:
              configMapKeyRef:
                name: cloud-config
                key: special.keysecret
          volumeMounts:
            - name: cloud-config
              mountPath: /etc/kubernetes/
            - name: logdir
              mountPath: /var/log/alicloud/
      volumes:
        - name: cloud-config
          hostPath:
            path: /etc/kubernetes/
        - name: logdir
          hostPath:
            path: /var/log/alicloud/

创建PVC

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: disk
spec:
  accessModes:
    - ReadWriteOnce
#alicloud-disk-common     普通云盘
#alicloud-disk-efficiency 高效云盘
#alicloud-disk-ssd        ssd云盘
#alicloud-disk-available
  storageClassName: alicloud-disk-common
  resources:
    requests:
      storage: 20Gi

效果

1
2
3
4
5
6
# kubectl get pvc
NAME      STATUS    VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS           AGE
disk      Bound     d-bp1cz8sslda31ld2snbq   20Gi       RWO            alicloud-disk-common   11s
# kubectl get pv
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM          STORAGECLASS           REASON    AGE
d-bp1cz8sslda31ld2snbq   20Gi       RWO            Delete           Bound     default/disk   alicloud-disk-common             14s

参考链接

IAM 使用

kube2iam in aws iam

通过AWS IAM创建一个名为kube2iam的策略，策略规则如下

1
2
3
4
5
6
7
8
9
10
11
12
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws-cn:iam::955466075186:role/k8s-*"
        }
    ]
}

通过AWS IAM创建一个名为：node-k8s-role的角色，并将kube2iam的策略赋予它，注意！！同时将这个角色赋予到机器上

通过AWS IAM创建一个名为：k8s-kube2iam-demo的角色，将AmazonS3FullAccess的策略赋予它，同时编辑信任关系，调整为如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com.cn"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws-cn:iam::955466075186:role/node-k8s-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

部署kube2iam

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
# Source: kube2iam/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: kube2iam
    chart: kube2iam-1.0.0
    heritage: Tiller
    release: ardent-wildebeest
  name: ardent-wildebeest-kube2iam
---
# Source: kube2iam/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: kube2iam
    chart: kube2iam-1.0.0
    heritage: Tiller
    release: ardent-wildebeest
  name: ardent-wildebeest-kube2iam
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
      - pods
    verbs:
      - list
      - watch
      - get
---
# Source: kube2iam/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app: kube2iam
    chart: kube2iam-1.0.0
    heritage: Tiller
    release: ardent-wildebeest
  name: ardent-wildebeest-kube2iam
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ardent-wildebeest-kube2iam
subjects:
  - kind: ServiceAccount
    name: ardent-wildebeest-kube2iam
    namespace: default
---
# Source: kube2iam/templates/daemonset.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: kube2iam
    chart: kube2iam-1.0.0
    heritage: Tiller
    release: ardent-wildebeest
  name: ardent-wildebeest-kube2iam
spec:
  template:
    metadata:
      labels:
        app: kube2iam
        release: ardent-wildebeest
    spec:
      containers:
        - name: kube2iam
          image: "jtblin/kube2iam:0.10.4"
          imagePullPolicy: "IfNotPresent"
          args:
            - "--host-interface=cali+"
            - "--node=$(NODE_NAME)"
            - "--host-ip=$(HOST_IP)"
            - "--iptables=true"
            - "--base-role-arn=arn:aws-cn:iam::955466075186:role/node-k8s-role"
            - "--app-port=8181"
            - "--use-regional-sts-endpoint"
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: AWS_REGION
              value: cn-northwest-1
          ports:
            - name: http
              containerPort: 8181
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8181
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 1
          resources:
            {}

          securityContext:
            privileged: true
      hostNetwork: true
      serviceAccountName: ardent-wildebeest-kube2iam
      tolerations:
        []

  updateStrategy:
    type: OnDelete

测试pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: apps/v1
kind: Deployment
metadata:
  name: awscli-deployment
spec:
  selector:
    matchLabels:
      app: awscli
  replicas: 1
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: arn:aws-cn:iam::955466075186:role/k8s/k8s-kube2iam-demo
      labels:
        app: awscli
    spec:
      containers:
      - name: awscli
        image: mesosphere/aws-cli
        command: [ "/bin/sh", "-c", "--" ]
        args: [ "while true; do sleep 3000; done;" ]

云应用疑难杂症

使用云负载均衡器代理APIServer问题

阿里云/AWS的负载均衡是四层TCP负责，不支持后端ECS实例既作为Real Server又作为客户端向所在的负载均衡实例发送请求。因为返回的数据包只在云服务器内部转发，不经过负载均衡，所以在后端ECS实例上去访问负载均衡的服务地址是不通的。什么意思？就是如果你要使用阿里云的SLB的话，那么你不能在apiserver节点上使用SLB（比如在apiserver 上安装kubectl，然后将apiserver的地址设置为SLB的负载地址使用），因为这样的话就可能造成回环了，所以简单的做法是另外用两个新的节点做HA实例，然后将这两个实例添加到SLB 机器组中。

  • aws 使用nlb，指定ip
  • aliyun 在apiserver上搭建keepalived
-------------本文结束感谢您的阅读-------------

本文标题:kuberenetes 云应用实践

文章作者:icyboy

发布时间:2019年03月15日 - 12:00

最后更新:2019年07月29日 - 10:24

原始链接:http://team.jiunile.com/blog/2019/03/k8s-cloud.html

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。