kubeadm

Kubernetes 一键部署脚本（使用 docker 运行时）
1
# on master
2
export USE_MIRROR=true #国内用户必须使用MIRROR
3
git clone https://github.com/feiskyer/ops
4
cd ops
5
kubernetes/install-kubernetes.sh
6
# 记住控制台输出的 TOEKN 和 MASTER 地址，在其他 Node 安装时会用到
7
​
8
# on all nodes
9
git clone https://github.com/feiskyer/ops
10
cd ops
11
# Setup token and CIDR first.
12
# replace this with yours.
13
export TOKEN="xxxx"
14
export MASTER_IP="x.x.x.x"
15
export CONTAINER_CIDR="10.244.2.0/24"
16
​
17
# Setup and join the new node.
18
./kubernetes/add-node.sh
Copied!
以下是详细的 kubeadm 部署集群步骤。
初始化系统
所有机器都需要初始化 docker 和 kubelet。
ubuntu
1
# for ubuntu 16.04
2
apt-get update
3
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
4
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
5
add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
6
apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
7
​
8
apt-get update && apt-get install -y apt-transport-https curl
9
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
10
    cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
11
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
12
EOF
13
apt-get update
14
apt-get install -y kubelet kubeadm kubectl
Copied!
CentOS
1
yum install -y docker
2
systemctl enable docker && systemctl start docker
3
​
4
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
5
[kubernetes]
6
name=Kubernetes
7
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
8
enabled=1
9
gpgcheck=1
10
repo_gpgcheck=1
11
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
12
EOF
13
setenforce 0
14
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
15
yum install -y kubelet kubeadm kubectl
16
systemctl enable kubelet && systemctl start kubelet
Copied!
安装 master
1
# --api-advertise-addresses <ip-address>
2
# for flannel, setup --pod-network-cidr 10.244.0.0/16
3
kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version latest
4
​
5
# enable schedule pods on the master
6
export KUBECONFIG=/etc/kubernetes/admin.conf
7
kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-
Copied!
如果需要修改 kubernetes 服务的配置选项，则需要创建一个 kubeadm 配置文件，其格式为
1
apiVersion: kubeadm.k8s.io/v1alpha3
2
kind: InitConfiguration
3
bootstrapTokens:
4
- token: "9a08jv.c0izixklcxtmnze7"
5
  description: "kubeadm bootstrap token"
6
  ttl: "24h"
7
- token: "783bde.3f89s0fje9f38fhf"
8
  description: "another bootstrap token"
9
  usages:
10
  - signing
11
  groups:
12
  - system:anonymous
13
nodeRegistration:
14
  name: "ec2-10-100-0-1"
15
  criSocket: "/var/run/dockershim.sock"
16
  taints:
17
  - key: "kubeadmNode"
18
    value: "master"
19
    effect: "NoSchedule"
20
  kubeletExtraArgs:
21
    cgroupDriver: "cgroupfs"
22
apiEndpoint:
23
  advertiseAddress: "10.100.0.1"
24
  bindPort: 6443
25
---
26
apiVersion: kubeadm.k8s.io/v1alpha3
27
kind: ClusterConfiguration
28
etcd:
29
  # one of local or external
30
  local:
31
    image: "k8s.gcr.io/etcd-amd64:3.2.18"
32
    dataDir: "/var/lib/etcd"
33
    extraArgs:
34
      listen-client-urls: "http://10.100.0.1:2379"
35
    serverCertSANs:
36
    -  "ec2-10-100-0-1.compute-1.amazonaws.com"
37
    peerCertSANs:
38
    - "10.100.0.1"
39
  external:
40
    endpoints:
41
    - "10.100.0.1:2379"
42
    - "10.100.0.2:2379"
43
    caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
44
    certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
45
    certKey: "/etcd/kubernetes/pki/etcd/etcd.key"
46
networking:
47
  serviceSubnet: "10.96.0.0/12"
48
  podSubnet: "10.100.0.1/24"
49
  dnsDomain: "cluster.local"
50
kubernetesVersion: "v1.12.0"
51
controlPlaneEndpoint: "10.100.0.1:6443"
52
apiServerExtraArgs:
53
  authorization-mode: "Node,RBAC"
54
controlManagerExtraArgs:
55
  node-cidr-mask-size: 20
56
schedulerExtraArgs:
57
  address: "10.100.0.1"
58
apiServerExtraVolumes:
59
- name: "some-volume"
60
  hostPath: "/etc/some-path"
61
  mountPath: "/etc/some-pod-path"
62
  writable: true
63
  pathType: File
64
controllerManagerExtraVolumes:
65
- name: "some-volume"
66
  hostPath: "/etc/some-path"
67
  mountPath: "/etc/some-pod-path"
68
  writable: true
69
  pathType: File
70
schedulerExtraVolumes:
71
- name: "some-volume"
72
  hostPath: "/etc/some-path"
73
  mountPath: "/etc/some-pod-path"
74
  writable: true
75
  pathType: File
76
apiServerCertSANs:
77
- "10.100.1.1"
78
- "ec2-10-100-0-1.compute-1.amazonaws.com"
79
certificatesDir: "/etc/kubernetes/pki"
80
imageRepository: "k8s.gcr.io"
81
unifiedControlPlaneImage: "k8s.gcr.io/controlplane:v1.12.0"
82
auditPolicy:
83
  # https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy
84
  path: "/var/log/audit/audit.json"
85
  logDir: "/var/log/audit"
86
  logMaxAge: 7 # in days
87
featureGates:
88
  selfhosting: false
89
clusterName: "example-cluster"
Copied!
注意：JoinConfiguration 重命名自 v1alpha2 API 中的 NodeConfiguration，而 InitConfiguration 重命名自 v1alpha2 API 中的 MasterConfiguration。
然后，在初始化 master 的时候指定 kubeadm.yaml 的路径：
1
kubeadm init --config ./kubeadm.yaml
Copied!
配置 Network plugin
CNI bridge
1
mkdir -p /etc/cni/net.d
2
cat >/etc/cni/net.d/10-mynet.conf <<-EOF
3
{
4
    "cniVersion": "0.3.0",
5
    "name": "mynet",
6
    "type": "bridge",
7
    "bridge": "cni0",
8
    "isGateway": true,
9
    "ipMasq": true,
10
    "ipam": {
11
        "type": "host-local",
12
        "subnet": "10.244.0.0/16",
13
        "routes": [
14
            {"dst": "0.0.0.0/0"}
15
        ]
16
    }
17
}
18
EOF
19
cat >/etc/cni/net.d/99-loopback.conf <<-EOF
20
{
21
    "cniVersion": "0.3.0",
22
    "type": "loopback"
23
}
24
EOF
Copied!
flannel
注意：需要 kubeadm init 时设置 --pod-network-cidr=10.244.0.0/16
1
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
Copied!
weave
1
sysctl net.bridge.bridge-nf-call-iptables=1
2
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
Copied!
calico
注意：需要 kubeadm init 时设置 --pod-network-cidr=192.168.0.0/16
1
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
2
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
Copied!
添加 Node
1
kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
Copied!
跟 Master 一样，添加 Node 的时候也可以自定义 Kubernetes 服务的选项，格式为
1
apiVersion: kubeadm.k8s.io/v1alpha2
2
caCertPath: /etc/kubernetes/pki/ca.crt
3
clusterName: kubernetes
4
discoveryFile: ""
5
discoveryTimeout: 5m0s
6
discoveryToken: abcdef.0123456789abcdef
7
discoveryTokenAPIServers:
8
- kube-apiserver:6443
9
discoveryTokenUnsafeSkipCAVerification: true
10
kind: NodeConfiguration
11
nodeRegistration:
12
  criSocket: /var/run/dockershim.sock
13
  name: thegopher
14
tlsBootstrapToken: abcdef.0123456789abcdef
15
token: abcdef.0123456789abcdef
Copied!
在把 Node 加入集群的时候，指定 NodeConfiguration 配置文件的路径
1
kubeadm join --config ./nodeconfig.yml --token $token ${master_ip}
Copied!
Cloud Provider
默认情况下，kubeadm 不包括 Cloud Provider 的配置，在 Azure 或者 AWS 等云平台上运行时，还需要配置 Cloud Provider。如
1
kind: MasterConfiguration
2
apiVersion: kubeadm.k8s.io/v1alpha2
3
apiServerExtraArgs:
4
  cloud-provider: "{cloud}"
5
  cloud-config: "{cloud-config-path}"
6
apiServerExtraVolumes:
7
- name: cloud
8
  hostPath: "{cloud-config-path}"
9
  mountPath: "{cloud-config-path}"
10
controllerManagerExtraArgs:
11
  cloud-provider: "{cloud}"
12
  cloud-config: "{cloud-config-path}"
13
controllerManagerExtraVolumes:
14
- name: cloud
15
  hostPath: "{cloud-config-path}"
16
  mountPath: "{cloud-config-path}"
Copied!
删除安装
1
# drain and delete the node first
2
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
3
kubectl delete node <node name>
4
​
5
# then reset kubeadm
6
kubeadm reset
Copied!
动态升级
kubeadm v1.8 开始支持动态升级，升级步骤为
首先上传 kubeadm 配置，如 kubeadm config upload from-flags [flags]（使用命令行参数）或 kubeadm config upload from-file --config [config]（使用配置文件）
在 master 上检查新版本 kubeadm upgrade plan， 当有新版本（如 v1.8.0）时，执行 kubeadm upgrade apply v1.8.0 升级控制平面
 手动  升级 CNI 插件（如果有新版本的话）
添加自动证书回滚的 RBAC 策略 kubectl create clusterrolebinding kubeadm:node-autoapprove-certificate-rotation --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
最后升级 kubelet
1
$ kubectl drain $HOST --ignore-daemonsets
2
​
3
# 升级软件包
4
$ apt-get update
5
$ apt-get upgrade
6
# CentOS 上面执行 yum 升级
7
# $ yum update
8
​
9
$ kubectl uncordon $HOST
Copied!
手动升级
kubeadm v1.7 以及以前的版本不支持动态升级，但可以手动升级。
升级 Master
假设你已经有一个使用 kubeadm 部署的 Kubernetes v1.6 集群，那么升级到 v1.7 的方法为：
1.升级安装包 apt-get upgrade && apt-get update
2.重启 kubelet systemctl restart kubelet
3.删除 kube-proxy DaemonSet KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete daemonset kube-proxy -n kube-system
4.kubeadm init --skip-preflight-checks --kubernetes-version v1.7.1
5.更新 CNI 插件
升级 Node
1.升级安装包 apt-get upgrade && apt-get update
2.重启 kubelet systemctl restart kubelet
安全选项
默认情况下，kubeadm 会开启 Node 客户端证书的自动批准，如果不需要的话可以选择关闭，关闭方法为
1
$ kubectl delete clusterrole kubeadm:node-autoapprove-bootstrap
Copied!
关闭后，增加新的 Node 时，kubeadm join 会阻塞等待管理员手动批准，匹配方法为
1
$ kubectl get csr
2
NAME                                                   AGE       REQUESTOR                 CONDITION
3
node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   18s       system:bootstrap:878f07   Pending
4
​
5
$ kubectl certificate approve node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ
6
certificatesigningrequest "node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ" approved
7
​
8
$ kubectl get csr
9
NAME                                                   AGE       REQUESTOR                 CONDITION
10
node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   1m        system:bootstrap:878f07   Approved,Issued
Copied!
参考文档
​kubeadm 参考指南​
​Upgrading kubeadm clusters from v1.14 to v1.15​