IT苦工指南 | Kubernetes v1.8.x全手動安裝
最近,有部分用戶飄了……
覺得Rainbond提供的既簡潔、又易用、而且生產就緒的Kubernets體驗不過癮……
想要挑戰一下Kubernetes全手動部署……
並在凌晨一點撥通了客服小哥的電話……
因此本著不重複造輪子並且關愛客服小哥身心健康的主張,我們搬來了Kairen的精彩教程——
開始
Kubernetes官方提供了多種安裝方式Picking the right solution,本文將以全手動安裝方式
來部署Kubernetes v1.8.x版本,學習和了解Kubernetes的構建流程。
版本明細:
- Kubernetes v1.8.6
- CNI v0.6.0
- Etcd v3.2.9
- Calico v2.6.2
- Docker v17.10.0-ce
準備
系統:ubuntu 16.x
或 centos 7.x
- 172.16.35.12 / master1 / 1 CPU / 2G
- 172.16.35.10 / node1 / 1 CPU / 2G
- 172.16.35.11 / node2 / 1 CPU / 2G
master為主要控制節點和部署節點,node為應用運行節點
所有操作均為root
安裝前需確認以下事項:
- 確認所有節點之間網路互通,master1 SSH登入其他節點為passwordless
- 確認防火牆和SELinux已關閉,如centos:
$ systemctl stop firewalld && systemctl disable firewalld$ setenforce 0$ vim /etc/selinux/configSELINUX=disabled
- 所有節點需要設置
/etc/host
解析到所有主機
...172.16.35.10 node1172.16.35.11 node2172.16.35.12 master1
- 所有節點都需要安裝docker$ curl -fsSL "https://get.docker.com/" | sh注意:centos安裝docker完成後需要執行:$ systemctl enable docker && systemctl start docker
編輯
ExecStartPost=/sbin/iptables -A FORWARD -s 0.0.0.0/0 -j ACCEPT完成後重啟docker服務:$ systemctl daemon-reload && systemctl restart docker/lib/systemd/system/docker.service
,在ExecStart=..
加入: - 所有節點都需要設定
/etc/sysctl.d/k8s.conf
系統參數$ cat <<EOF > /etc/sysctl.d/k8s.confnet.ipv4.ip_forward = 1net.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOF$ sysctl -p /etc/sysctl.d/k8s.conf - 在master1安裝
CFSSL
工具,用來建立TLS certificates$ export CFSSL_URL="https://pkg.cfssl.org/R1.2"
$ wget "${CFSSL_URL}/cfssl_linux-amd64" -O /usr/local/bin/cfssl$ wget "${CFSSL_URL}/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson$ chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
Etcd
安裝Kubernetes之前,我們需要完成一些必要的系統配置,高可用共享配置和服務發現存儲Etcd便是其中的重要一環,節點會從Etcd中獲取所需數據。
建立集群CA和certificates
這裡需要生成client和server各組件certificate,代替kubernetes admin user生成client證書。
首先在master1
建立/etc/etcd/ssl
目錄,而後進入目錄進行以下操作:
$ mkdir -p /etc/etcd/ssl && cd /etc/etcd/ssl$ export PKI_URL="https://kairen.github.io/files/manual-v1.8/pki"
下載ca-config.json
和etcd-ca-csr.json
:
$ wget "${PKI_URL}/ca-config.json" "${PKI_URL}/etcd-ca-csr.json"$ cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca$ ls etcd-ca*.pemetcd-ca-key.pem etcd-ca.pem
下載etcd-csr.json
並生成Etcd certificate證書:
$ wget "${PKI_URL}/etcd-csr.json"$ cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd$ ls etcd*.pemetcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem
如果節點IP不同,需要修改
etcd-csr.json
的hosts
完成後刪除不必要的文件:
$ rm -rf *.json
並確認/etc/etcd/ssl
包含:
$ ls /etc/etcd/ssletcd-ca.csr etcd-ca-key.pem etcd-ca.pem etcd.csr etcd-key.pem etcd.pem
Etcd的安裝和設置
首先在master1
節點下載Etcd,解壓到/opt
安裝:
$ export ETCD_URL="https://github.com/coreos/etcd/releases/download"$ cd && wget -qO- --show-progress "${ETCD_URL}/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz" | tar -zx$ mv etcd-v3.2.9-linux-amd64/etcd* /usr/local/bin/ && rm -rf etcd-v3.2.9-linux-amd64
完成後新建Etcd Group和User,並設定Etcd目錄:
$ groupadd etcd && useradd -c "Etcd user" -g etcd -s /sbin/nologin -r etcd
下載etcd相關配置,我們將來管理Etcd:
$ export ETCD_CONF_URL="https://kairen.github.io/files/manual-v1.8/master"$ wget "${ETCD_CONF_URL}/etcd.conf" -O /etc/etcd/etcd.conf$ wget "${ETCD_CONF_URL}/etcd.service" -O /lib/systemd/system/etcd.service
如果沒用本文準備部分的IP,請用自己的IP代替
172.16.35.12
建立var 存放數據,然後啟動Etcd服務:
$ mkdir -p /var/lib/etcd && chown etcd:etcd -R /var/lib/etcd /etc/etcd$ systemctl enable etcd.service && systemctl start etcd.service
通過以下命令驗證:
$ export CA="/etc/etcd/ssl"$ ETCDCTL_API=3 etcdctl --cacert=${CA}/etcd-ca.pem --cert=${CA}/etcd.pem --key=${CA}/etcd-key.pem --endpoints="https://172.16.35.12:2379" endpoint health# outputhttps://172.16.35.12:2379 is healthy: successfully committed proposal: took = 641.36μs
Kubernetes Master
Master是Kubernetes的大總管,通過apiserver
、Controller manager
以及Scheduler
管理所有節點。
本部分將下載Kubernetes並安裝到master1節點上,然後生成相關TLS certificates和CA,供集群組件使用。
下載Kubernetes組件
# Download Kubernetes$ export KUBE_URL="https://storage.googleapis.com/kubernetes-release/release/v1.8.6/bin/linux/amd64"$ wget "${KUBE_URL}/kubelet" -O /usr/local/bin/kubelet$ wget "${KUBE_URL}/kubectl" -O /usr/local/bin/kubectl$ chmod +x /usr/local/bin/kubelet /usr/local/bin/kubectl# Download CNI$ mkdir -p /opt/cni/bin && cd /opt/cni/bin$ export CNI_URL="https://github.com/containernetworking/plugins/releases/download"$ wget -qO- --show-progress "${CNI_URL}/v0.6.0/cni-plugins-amd64-v0.6.0.tgz" | tar -zx
建立集群CA和certificates
與Etcd部分原理一樣,操作也大相徑庭,首先在master1
建立pki
目錄,並進入目錄執行:
$ mkdir -p /etc/kubernetes/pki && cd /etc/kubernetes/pki$ export PKI_URL="https://kairen.github.io/files/manual-v1.8/pki"$ export KUBE_APISERVER="https://172.16.35.12:6443"
下載ca-config.json
和etcd-ca-csr.json
:
$ wget "${PKI_URL}/ca-config.json" "${PKI_URL}/ca-csr.json"$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca$ ls ca*.pemca-key.pem ca.pem
API server certificate
下載apiserver-csr.json
,生成kube-apiserver certificate
證書:
$ wget "${PKI_URL}/apiserver-csr.json"$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -hostname=10.96.0.1,172.16.35.12,127.0.0.1,kubernetes.default -profile=kubernetes apiserver-csr.json | cfssljson -bare apiserver$ ls apiserver*.pemapiserver-key.pem apiserver.pem
如果節點IP不同,需要修改
-hostname
Front proxy certificate
下載front-proxy-ca-csr.json
,生成Front proxy CA,Front proxy主要用在API aggregator上:
$ wget "${PKI_URL}/front-proxy-ca-csr.json"$ cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca$ ls front-proxy-ca*.pemfront-proxy-ca-key.pem front-proxy-ca.pem
下載front-proxy-client-csr.json
,生成front-proxy-client證書:
$ wget "${PKI_URL}/front-proxy-client-csr.json"$ cfssl gencert -ca=front-proxy-ca.pem -ca-key=front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare front-proxy-client$ ls front-proxy-client*.pemfront-proxy-client-key.pem front-proxy-client.pem
Bootstrap Token
手工方式生成CA非常麻煩,只適合少量機器,每次簽證時都需要綁定Node IP,隨著機器增加會帶來很多的不便,因此這裡使用TLS Bootstrapping的方式來進行授權,由apiserver自動為符合條件的Node發送證書授權加入集群。
做法是在kubelet啟動時,向kuber-apiserver傳送TLS Bootstrapping請求,而kube-apiserver驗證kubelet請求的token是否與設定的一樣,如果一樣則自動生成Kuberlet證書和密鑰。具體作法可以參考TLS bootstrapping。
首先生成BOOTSTRAP_TOKEN
,並建立bootstrap.conf
的kubeconfig:
$ export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d )$ cat <<EOF > /etc/kubernetes/token.csv${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"EOF# bootstrap set-cluster$ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=../bootstrap.conf# bootstrap set-credentials$ kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=../bootstrap.conf# bootstrap set-context$ kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=../bootstrap.conf# bootstrap set default context$ kubectl config use-context default --kubeconfig=../bootstrap.conf
如果想用CA的方式來認證,可以參考Kubelet certificate點擊預覽
Admin certificate
下載admin-csr.json
,並生成admin certificate
證書:
$ wget "${PKI_URL}/admin-csr.json"$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin$ ls admin*.pemadmin-key.pem admin.pem
然後執行一下命令生成名為admin.conf
的kubeconfig:
# admin set-cluster$ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=../admin.conf# admin set-credentials$ kubectl config set-credentials kubernetes-admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=../admin.conf# admin set-context$ kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=../admin.conf# admin set default context$ kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=../admin.conf
Controller manager certificate
下載manager-csr.json
,並生成kube-controller-manager certificate
證書:
$ wget "${PKI_URL}/manager-csr.json"$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes manager-csr.json | cfssljson -bare controller-manager$ ls controller-manager*.pem
如果節點IP不同,需要修改
manager-csr.json
的hosts
然後執行命令生成名為controller-manager.conf
的kubeconfig:
# controller-manager set-cluster$ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=../controller-manager.conf# controller-manager set-credentials$ kubectl config set-credentials system:kube-controller-manager --client-certificate=controller-manager.pem --client-key=controller-manager-key.pem --embed-certs=true --kubeconfig=../controller-manager.conf# controller-manager set-context$ kubectl config set-context system:kube-controller-manager@kubernetes --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=../controller-manager.conf# controller-manager set default context$ kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=../controller-manager.conf
Scheduler certificate
下載scheduler-csr.json
,生成kube-scheduler certificate證書:
$ wget "${PKI_URL}/scheduler-csr.json"$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes scheduler-csr.json | cfssljson -bare scheduler$ ls scheduler*.pemscheduler-key.pem scheduler.pem
如果節點IP不同,需要修改
scheduler-csr.json
的hosts
然後執行一下命令生成名為scheduler.conf
的kubeconfig:
# scheduler set-cluster$ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=../scheduler.conf# scheduler set-credentials$ kubectl config set-credentials system:kube-scheduler --client-certificate=scheduler.pem --client-key=scheduler-key.pem --embed-certs=true --kubeconfig=../scheduler.conf# scheduler set-context$ kubectl config set-context system:kube-scheduler@kubernetes --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=../scheduler.conf# scheduler set default context$ kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=../scheduler.conf
Kubelet master certificate
下載kubelet-csr.json
,並生成master node certificate證書:
$ wget "${PKI_URL}/kubelet-csr.json"$ sed -i s/$NODE/master1/g kubelet-csr.json$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -hostname=master1,172.16.35.12 -profile=kubernetes kubelet-csr.json | cfssljson -bare kubelet$ ls kubelet*.pemkubelet-key.pem kubelet.pem
$NODE
需要隨節點名稱不同而改變
然後執行一下命令生成名為kubelet.conf
的kubeconfig:
# kubelet set-cluster$ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=../kubelet.conf# kubelet set-credentials$ kubectl config set-credentials system:node:master1 --client-certificate=kubelet.pem --client-key=kubelet-key.pem --embed-certs=true --kubeconfig=../kubelet.conf# kubelet set-context$ kubectl config set-context system:node:master1@kubernetes --cluster=kubernetes --user=system:node:master1 --kubeconfig=../kubelet.conf# kubelet set default context$ kubectl config use-context system:node:master1@kubernetes --kubeconfig=../kubelet.conf
Service account key
Service account不需要CA認證,也就不需要CA來做Service account key的檢查,這裡我們建立一組private和public的密鑰供Service account key使用:
$ openssl genrsa -out sa.key 2048 $ openssl rsa -in sa.key -pubout -out sa.pub $ ls sa.* sa.key sa.pub
完成後刪除不必要文件:
$ rm -rf *.json *.csr
確認/etc/kubernetes
和/etc/kubernetes/pki
包含以下文件:
$ ls /etc/kubernetes/admin.conf bootstrap.conf controller-manager.conf kubelet.conf pki scheduler.conf token.csv$ ls /etc/kubernetes/pkiadmin-key.pem apiserver-key.pem ca-key.pem controller-manager-key.pem front-proxy-ca-key.pem front-proxy-client-key.pem kubelet-key.pem sa.key scheduler-key.pemadmin.pem apiserver.pem ca.pem controller-manager.pem front-proxy-ca.pem front-proxy-client.pem kubelet.pem sa.pub scheduler.pem
安裝Kubernetes 核心元件
下載Kubernetes核心組件Yaml文件,這裡我們利用Kubernetes Statics Pod來建立Master核心組件,因此下載所有Static Pod文件到etc/kubernetes/manifests
目錄「
$ export CORE_URL="https://kairen.github.io/files/manual-v1.8/master"$ mkdir -p /etc/kubernetes/manifests && cd /etc/kubernetes/manifests$ for FILE in apiserver manager scheduler; do wget "${CORE_URL}/${FILE}.yml.conf" -O ${FILE}.yml done
同樣的,如果IP與本文IP準備不同的話,需要修改
scheduler.yml`apiserver中的apiserver.yml
、manager.yml
、`NodeRestriction
請參考Using Node Authorization
生成一個用來加密Etcd的key
$ head -c 32 /dev/urandom | base64SUpbL4juUYyvxj3/gonV5xVEx8j769/99TSAf8YT/sQ=
在/etc/kubernetes/
目錄建立encryption.yml
的加密YAML文件:
$ cat <<EOF > /etc/kubernetes/encryption.ymlkind: EncryptionConfigapiVersion: v1resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: SUpbL4juUYyvxj3/gonV5xVEx8j769/99TSAf8YT/sQ= - identity: {}EOF
Etcd加密可參考Encrypting data at rest
在/etc/kubernetes/
目錄建立audit-policy.yml
的auditing policay YAML文件:
$ cat <<EOF > /etc/kubernetes/audit-policy.ymlapiVersion: audit.k8s.io/v1beta1kind: Policyrules:- level: MetadataEOF
audit policy請參考Audit
下載kubelet.service
相關文件來管理kubelet:
$ export KUBELET_URL="https://kairen.github.io/files/manual-v1.8/master"$ mkdir -p /etc/systemd/system/kubelet.service.d$ wget "${KUBELET_URL}/kubelet.service" -O /lib/systemd/system/kubelet.service$ wget "${KUBELET_URL}/10-kubelet.conf" -O /etc/systemd/system/kubelet.service.d/10-kubelet.conf
若
cluster-dns
或cluster-domain
有變動,需要修改10-kubelet.conf
最後建立var並啟動kubelet服務:
$ mkdir -p /var/lib/kubelet /var/log/kubernetes$ systemctl enable kubelet.service && systemctl start kubelet.service
完成後需要一段時間來下載鏡像文件並啟動組件:
$ watch netstat -ntlptcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 23012/kubelettcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 22305/kube-scheduletcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 22529/kube-controlltcp6 0 0 :::6443 :::* LISTEN 22956/kube-apiserve
看到上述信息即表明服務啟動正常,如果出現問題可通過docker cli查看
完成後,複製admin kubeconfig並通過以下命令驗證:
$ cp /etc/kubernetes/admin.conf ~/.kube/config$ kubectl get csNAME STATUS MESSAGE ERRORetcd-0 Healthy {"health": "true"}scheduler Healthy okcontroller-manager Healthy ok$ kubectl get nodeNAME STATUS ROLES AGE VERSIONmaster1 NotReady master 1m v1.8.6$ kubectl -n kube-system get poNAME READY STATUS RESTARTS AGEkube-apiserver-master1 1/1 Running 0 4mkube-controller-manager-master1 1/1 Running 0 4mkube-scheduler-master1 1/1 Running 0 4m
確認服務能夠執行logs等命令:
$ kubectl -n kube-system logs -f kube-scheduler-master1Error from server (Forbidden): Forbidden (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy) ( pods/log kube-apiserver-master1)
出現403 Forbidden問題表明
kube-apiserver user
並沒有nodes的許可權
由於上述許可權問題,我們需要建立一個apiserver-to-kubelet-rbac.yml
來定義許可權,以供我們執行logs、exec等命令:
$ cd /etc/kubernetes/$ export URL="https://kairen.github.io/files/manual-v1.8/master"$ wget "${URL}/apiserver-to-kubelet-rbac.yml.conf" -O apiserver-to-kubelet-rbac.yml$ kubectl apply -f apiserver-to-kubelet-rbac.yml# 測試 logs$ kubectl -n kube-system logs -f kube-scheduler-master1...I1031 03:22:42.527697 1 leaderelection.go:184] successfully acquired lease kube-system/kube-scheduler
Kubernetes Node
Node運行容器實例的節點,即工作節點。本部分我們會下載Kubernetes binary並建立node 的certificate來提供給節點註冊認證用。Kubernetes使用Node Authorizer來提供Authorization mode,這種授權模式會替Kubelet生成API request。
開始前,我們先在master1
將需要的ca和cert複製到Node節點上:
$ for NODE in node1 node2; do ssh ${NODE} "mkdir -p /etc/kubernetes/pki/" ssh ${NODE} "mkdir -p /etc/etcd/ssl" # Etcd ca and cert for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do scp /etc/etcd/ssl/${FILE} ${NODE}:/etc/etcd/ssl/${FILE} done # Kubernetes ca and cert for FILE in pki/ca.pem pki/ca-key.pem bootstrap.conf; do scp /etc/kubernetes/${FILE} ${NODE}:/etc/kubernetes/${FILE} done done
下載Kubernetes 元件
首先獲取所有需要執行的文件:
# Download Kubernetes$ export KUBE_URL="https://storage.googleapis.com/kubernetes-release/release/v1.8.6/bin/linux/amd64"$ wget "${KUBE_URL}/kubelet" -O /usr/local/bin/kubelet$ chmod +x /usr/local/bin/kubelet# Download CNI$ mkdir -p /opt/cni/bin && cd /opt/cni/bin$ export CNI_URL="https://github.com/containernetworking/plugins/releases/download"$ wget -qO- --show-progress "${CNI_URL}/v0.6.0/cni-plugins-amd64-v0.6.0.tgz" | tar -zx
設定Kubernetes node
下載Kubernetes相關文件,包括drop-in file、systemd service等:
$ export KUBELET_URL="https://kairen.github.io/files/manual-v1.8/node"$ mkdir -p /etc/systemd/system/kubelet.service.d$ wget "${KUBELET_URL}/kubelet.service" -O /lib/systemd/system/kubelet.service$ wget "${KUBELET_URL}/10-kubelet.conf" -O /etc/systemd/system/kubelet.service.d/10-kubelet.conf
如果
cluster-dns
或cluster-domain
有改變的話,需要修改10-kubelet.conf
然後在所有node建立var,並啟動kubelet服務:
$ mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/kubernetes/manifests$ systemctl enable kubelet.service && systemctl start kubelet.service
授權Kubernetes Node
重複完成所有節點後,在master1節點建立ClusterRoleBinding(因為我們採用的是TLS Bootstrapping):
$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
在master進行驗證,我們可以看到節點處於pending:
$ kubectl get csrNAME AGE REQUESTOR CONDITIONnode-csr-YWf97ZrLCTlr2hmXsNLfjVLwaLfZRsu52FRKOYjpcBE 2s kubelet-bootstrap Pendingnode-csr-eq4q6ffOwT4yqYQNU6sT7mphPOQdFN6yulMVZeu6pkE 2s kubelet-bootstrap Pending
通過kubectl,允許節點加入集群:
$ kubectl get csr | awk /Pending/ {print $1} | xargs kubectl certificate approvecertificatesigningrequest "node-csr-YWf97ZrLCTlr2hmXsNLfjVLwaLfZRsu52FRKOYjpcBE" approvedcertificatesigningrequest "node-csr-eq4q6ffOwT4yqYQNU6sT7mphPOQdFN6yulMVZeu6pkE" approved$ kubectl get csrNAME AGE REQUESTOR CONDITIONnode-csr-YWf97ZrLCTlr2hmXsNLfjVLwaLfZRsu52FRKOYjpcBE 30s kubelet-bootstrap Approved,Issuednode-csr-eq4q6ffOwT4yqYQNU6sT7mphPOQdFN6yulMVZeu6pkE 30s kubelet-bootstrap Approved,Issued$ kubectl get noNAME STATUS ROLES AGE VERSIONmaster1 NotReady master 21m v1.8.6node1 NotReady node 8s v1.8.6node2 NotReady node 8s v1.8.6
Kubernetes Core Addons 部署
完成以上所有步驟,我們還需要安裝一些插件,比如kube-dns、kube-proxy等等。
Kube-proxy addon
Kube-proxy是實現Service的關鍵組件,kube-proxy會在每個節點上執行,然後監聽API Server的Service和Endpoint變化,並根據變化執行iptables實現網路轉發。
這裡我們需要DaemonSet來執行,並需要生成一些certificate。
首先在master1
下載kube-proxy-csr.json
,並生成kube-proxy certificate證書:
$ export PKI_URL="https://kairen.github.io/files/manual-v1.8/pki"$ cd /etc/kubernetes/pki$ wget "${PKI_URL}/kube-proxy-csr.json" "${PKI_URL}/ca-config.json"$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy$ ls kube-proxy*.pemkube-proxy-key.pem kube-proxy.pem
然後通過以下命令生成名為`kube-proxy.conf·的kubeconfig:
# kube-proxy set-cluster$ kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server="https://172.16.35.12:6443" --kubeconfig=../kube-proxy.conf# kube-proxy set-credentials$ kubectl config set-credentials system:kube-proxy --client-key=kube-proxy-key.pem --client-certificate=kube-proxy.pem --embed-certs=true --kubeconfig=../kube-proxy.conf# kube-proxy set-context$ kubectl config set-context system:kube-proxy@kubernetes --cluster=kubernetes --user=system:kube-proxy --kubeconfig=../kube-proxy.conf# kube-proxy set default context$ kubectl config use-context system:kube-proxy@kubernetes --kubeconfig=../kube-proxy.conf
刪除不必要的文件:
$ rm -rf *.json
確認/etc/kubernetes
有以下文件:
$ ls /etc/kubernetes/admin.conf bootstrap.conf encryption.yml kube-proxy.conf pki token.csvaudit-policy.yml controller-manager.conf kubelet.conf manifests scheduler.conf
在master1
上將kube-proxy相關文件複製到Node節點上:
$ for NODE in node1 node2; do echo "--- $NODE ---" for FILE in pki/kube-proxy.pem pki/kube-proxy-key.pem kube-proxy.conf; do scp /etc/kubernetes/${FILE} ${NODE}:/etc/kubernetes/${FILE} done done
完成後,在master1
通過kubectl建立kube-proxy daemon:
$ export ADDON_URL="https://kairen.github.io/files/manual-v1.8/addon"$ mkdir -p /etc/kubernetes/addons && cd /etc/kubernetes/addons$ wget "${ADDON_URL}/kube-proxy.yml.conf" -O kube-proxy.yml$ kubectl apply -f kube-proxy.yml$ kubectl -n kube-system get po -l k8s-app=kube-proxyNAME READY STATUS RESTARTS AGEkube-proxy-bpp7q 1/1 Running 0 47skube-proxy-cztvh 1/1 Running 0 47skube-proxy-q7mm4 1/1 Running 0 47s
Kube-dns addon
Kube DNS是Kubernetes集群內部Pod之間通信的重要插件,允許Pod通過Domain Name鏈接Service,主要由Kube DNS與Sky DNS組合而成,通過Kube DNS監聽Service與Endpoint變化,來提供給Sky DNS信息以更新解析位址。
安裝只需要在master1
通過kubectl建立kube-dns deployment即可:
$ export ADDON_URL="https://kairen.github.io/files/manual-v1.8/addon"$ wget "${ADDON_URL}/kube-dns.yml.conf" -O kube-dns.yml$ kubectl apply -f kube-dns.yml$ kubectl -n kube-system get po -l k8s-app=kube-dnsNAME READY STATUS RESTARTS AGEkube-dns-6cb549f55f-h4zr5 0/3 Pending 0 40s
Calico Network 安裝與設定
Calico是一款純3層協議(不需要Overlay 網路),已與各種雲原生平台有良好的整合,在每個節點節點利用Linux Kernel實現高效的vRouter來負責數據轉發,而當數據中心複雜度增加時,可以用BGP route reflector來達成。
首先在master1
通過kubectl建立Calico policy controller:
$ export CALICO_CONF_URL="https://kairen.github.io/files/manual-v1.8/network"$ wget "${CALICO_CONF_URL}/calico-controller.yml.conf" -O calico-controller.yml$ kubectl apply -f calico-controller.yml$ kubectl -n kube-system get po -l k8s-app=calico-policyNAME READY STATUS RESTARTS AGEcalico-policy-controller-5ff8b4549d-tctmm 0/1 Pending 0 5s
如果節點IP不同,需要修改calico-controller.yml的ETCD_ENDPOINTS
在`master1·下載Calico CLI工具:
$ wget https://github.com/projectcalico/calicoctl/releases/download/v1.6.1/calicoctl$ chmod +x calicoctl && mv calicoctl /usr/local/bin/
然後在所有節點下載Calico,並執行以下命令:
$ export CALICO_URL="https://github.com/projectcalico/cni-plugin/releases/download/v1.11.0"$ wget -N -P /opt/cni/bin ${CALICO_URL}/calico$ wget -N -P /opt/cni/bin ${CALICO_URL}/calico-ipam$ chmod +x /opt/cni/bin/calico /opt/cni/bin/calico-ipam
接著在所有節點下載CNI plugins以及calico-node.service:
$ mkdir -p /etc/cni/net.d$ export CALICO_CONF_URL="https://kairen.github.io/files/manual-v1.8/network"$ wget "${CALICO_CONF_URL}/10-calico.conf" -O /etc/cni/net.d/10-calico.conf$ wget "${CALICO_CONF_URL}/calico-node.service" -O /lib/systemd/system/calico-node.service
如果節點IP不同,需要修改10-calico.conf的etcd_endpoints如果部署機器是虛擬機,需要修改calico-node.service,並在IP_AUTODETECTION_METHOD (包含IP6)部分指定綁定的網卡,以免預設綁定到NAT網路上
之後在所有節點啟動Calico-node:
$ systemctl enable calico-node.service && systemctl start calico-node.service
在master1
查看Calico nodes:
$ cat <<EOF > ~/calico-rcexport ETCD_ENDPOINTS="https://172.16.35.12:2379"export ETCD_CA_CERT_FILE="/etc/etcd/ssl/etcd-ca.pem"export ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"export ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"EOF$ . ~/calico-rc$ calicoctl get node -o wideNAME ASN IPV4 IPV6master1 (64512) 172.16.35.12/24node1 (64512) 172.16.35.10/24node2 (64512) 172.16.35.11/24
查看pending的pod是否已執行:
$ kubectl -n kube-system get poNAME READY STATUS RESTARTS AGEcalico-policy-controller-5ff8b4549d-tctmm 1/1 Running 0 4mkube-apiserver-master1 1/1 Running 0 20mkube-controller-manager-master1 1/1 Running 0 20mkube-dns-6cb549f55f-h4zr5 3/3 Running 0 5mkube-proxy-fnrkb 1/1 Running 0 6mkube-proxy-l72bq 1/1 Running 0 6mkube-proxy-m6rfw 1/1 Running 0 6mkube-scheduler-master1 1/1 Running 0 20m
省事的做法是用Standard Hosted方式安裝。
Kubernetes Extra Addons 部署
本部分說明如何部署官方常用的addons,例如dashboard、heapster等。
Dashboard addon
Dashboard是Kubernetes官方開發的儀錶板,讓我們以可以i 通過web-based方式管理kubernetes集群。
在master1
通過kubectl建立kubernetes dashboard即可:
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml$ kubectl -n kube-system get po,svc -l k8s-app=kubernetes-dashboardNAME READY STATUS RESTARTS AGEpo/kubernetes-dashboard-747c4f7cf-md5m8 1/1 Running 0 56sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEsvc/kubernetes-dashboard ClusterIP 10.98.120.209 <none> 443/TCP 56s
這邊會額外建立一個名稱為open-api
的Cluster Role Binding,放拜年測試使用,一般情況下不開啟(開啟會存取所有API)。
$ cat <<EOF | kubectl create -f -apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata: name: open-api namespace: ""roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects: - apiGroup: rbac.authorization.k8s.io kind: User name: system:anonymousEOF
管理者可以針對特定使用者來開放API存取許可權,這裡我們為了方便直接綁在cluster-admin cluster role。
1.7版本後的Dashboard不再提供所有許可權,需要建立一個service account來綁定cluster-admin role:
$ kubectl -n kube-system create sa dashboard$ kubectl create clusterrolebinding dashboard --clusterrole cluster-admin --serviceaccount=kube-system:dashboard$ SECRET=$(kubectl -n kube-system get sa dashboard -o yaml | awk /dashboard-token/ {print $3})$ kubectl -n kube-system describe secrets ${SECRET} | awk /token:/{print $2}eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtdG9rZW4tdzVocmgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWJmMTFjYzMtZjRlYi0xMWU3LTgzYWUtMDgwMDI3NjdkOWI5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZCJ9.Xuyq34ci7Mk8bI97o4IldDyKySOOqRXRsxVWIJkPNiVUxKT4wpQZtikNJe2mfUBBD-JvoXTzwqyeSSTsAy2CiKQhekW8QgPLYelkBPBibySjBhJpiCD38J1u7yru4P0Pww2ZQJDjIxY4vqT46ywBklReGVqY3ogtUQg-eXueBmz-o7lJYMjw8L14692OJuhBjzTRSaKW8U2MPluBVnD7M2SOekDff7KpSxgOwXHsLVQoMrVNbspUCvtIiEI1EiXkyCNRGwfnd2my3uzUABIHFhm0_RZSmGwExPbxflr8Fc6bxmuz-_jSdOtUidYkFIzvEWw2vRovPgs3MXTv59RwUw
複製token,然後貼到Kubernetes dashboard
Heapster addon
Heapster是Kubernetes社區維護的容器集群監控和分析工具。Heapster會從Kubernetes apiserver取得所有Node數據,然後再通過Node獲取kubelet上的數據,最後再將所有收集到數據送到Heapster後台儲存InfluxDB,最後利用Grafana抓取InfluxDB數據源來進行展示。
在master1
通過kubectl來建立kubernetes monitor即可:
$ export ADDON_URL="https://kairen.github.io/files/manual-v1.8/addon"$ wget ${ADDON_URL}/kube-monitor.yml.conf -O kube-monitor.yml$ kubectl apply -f kube-monitor.yml$ kubectl -n kube-system get po,svcNAME READY STATUS RESTARTS AGE...po/heapster-74fb5c8cdc-62xzc 4/4 Running 0 7mpo/influxdb-grafana-55bd7df44-nw4nc 2/2 Running 0 7mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE...svc/heapster ClusterIP 10.100.242.225 <none> 80/TCP 7msvc/monitoring-grafana ClusterIP 10.101.106.180 <none> 80/TCP 7msvc/monitoring-influxdb ClusterIP 10.109.245.142 <none> 8083/TCP,8086/TCP 7m···
簡單部署Nginx 服務
Kubernetes可以選擇使用指令直接建立應用和服務,或者我們可以寫YAML、JSON文件來配置,如下所示:
$ kubectl run nginx --image=nginx --port=80$ kubectl expose deploy nginx --port=80 --type=LoadBalancer --external-ip=172.16.35.12$ kubectl get svc,poNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEsvc/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1hsvc/nginx LoadBalancer 10.97.121.243 172.16.35.12 80:30344/TCP 22sNAME READY STATUS RESTARTS AGEpo/nginx-7cbc4b4d9c-7796l 1/1 Running 0 28s 192.160.57.181 ,172.16.35.12 80:32054/TCP 21s
這裡type可以選擇NodePort和LoadBalancer在本地裸機部署,兩者差異在於NodePort只映射Host port到Container port,而LoadBalancer則繼承NodePort額外映射Host target port到Container port
擴展服務數量
最後,我們可以通過以下方式來擴展服務數量:
$ kubectl scale deploy nginx --replicas=2$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODEnginx-158599303-0h9lr 1/1 Running 0 25s 10.244.100.5 node2nginx-158599303-k7cbt 1/1 Running 0 1m 10.244.24.3 node1
相關閱讀
技術
Kubernetes Autoscaling是如何工作的?2018/05/07
技術
用戶評測 | Docker管理面板系列——雲幫(Rainbond 出色的k8s管理面板)2018/05/09
技術
如何把應用轉移到Kubernetes2018/05/04
技術
Kubernetes伸縮到2500個節點中遇到的問題和解決方法2018/04/24
技術
kubernetes容器網路介面(CNI) midonet網路插件的設計與實現2017/05/04
技術
在生產環境使用Kuberntes一年後,我們總結了這些經驗和教訓2017/02/23
技術
關於K8s容器集群日誌收集的總結2016/12/15
技術
Kubernetes集群中的高性能網路策略2016/12/08
行業
開了香檳的Kubernetes並不打算放慢成功的腳步2018/05/09
行業
上手kubernetes之前,你應該知道這6件事2018/05/03
行業
為什麼說Kubernetes是雲服務的未來?2016/10/21
開源PaaS Rainbond提供生產就緒的Kubernetes,在線體驗請註冊公有雲(新用戶7天免費)
推薦閱讀:
※一周IT博文精選TOP10(第十期)
※Kubernetes中文指南/雲原生應用架構實踐手冊v1.3發布
※數人云架構師:微服務體系中的K8S&Mesos調度與服務發現
※kubernetes的網路實現
※PPT收藏|KubeCon北美峰會幹貨直播培訓課
TAG:Kubernetes | 科技 | 信息技術IT |