容器編排之etcd集群管理

02-03

隨著CoreOS和Kubernetes的發展，etcd作為支撐它們運行不可或缺的組件也被更多的人認識。

本文主要介紹如何動態搭建etcd集群，並介紹如何動態增加、刪除、修改etcd節點。且如何管理etcd集群。

集群安裝

在官方的文檔中，介紹了三種搭建etcd集群的方式：

靜態
etcd發現
dns發現

本文基於etcd發現來動態搭建集群，搭建之前，首先了解一下etcd的發現服務協議。

發現服務

發現服務協議幫助etcd新節點通過共享的發現URL找到集群中的其他節點。但需注意，發現服務協議只適用於搭建集群的時候，不能用在運行時重新配置或者集群監控中。

該協議使用發現URL引導搭建etcd集群。一個發現URL只能用於一個etcd集群。只要使用該URL的發現服務運行起來了，即使中途down掉了，也不能再用該Token引導另一個集群。

自定義etcd發現服務

發現服務使用已存在的etcd集群引導自身。如果使用私有etcd集群，可以這樣創建發現URL：

$ curl -X PUT http://xxx.xxx.xxx.xxx/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3n

這樣就創建了一個可以搭建3個節點的etcd集群的發現URL。一般可設置為3、5、7。

etcd新節點就可以使用http://xxx.xxx.xxx.xxx/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83作為啟動參數，以便自己加入該集群。

每個節點必須指定不同的name，Hostname或者machine-id都很好。否則發現服務就會因為name重複而崩潰。

接著，我們就可以根據該發現URL引導集群了

$ etcd --name etcd0 n --initial-advertise-peer-urls http://10.1.2.154:2380 n --listen-peer-urls http://10.1.2.154:2380 n --listen-client-urls http://10.1.2.154:2379,http://127.0.0.1:2379 n --advertise-client-urls http://10.1.2.154:2379 n --discovery https://xxx.xxx.xxx.xxx/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83n

$ etcd --name etcd1 n --initial-advertise-peer-urls http://10.1.2.147:2380 n --listen-peer-urls http://10.1.2.147:2380 n --listen-client-urls http://10.1.2.147:2379,http://127.0.0.1:2379 n --advertise-client-urls http://10.1.2.147:2379 n --discovery https://xxx.xxx.xxx.xxx/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83n

$ etcd --name etcd2 n --initial-advertise-peer-urls http://10.1.2.148:2380 n --listen-peer-urls http://10.1.2.148:2380 n --listen-client-urls http://10.1.2.148:2379,http://127.0.0.1:2379 n --advertise-client-urls http://10.1.2.148:2379 n --discovery https://xxx.xxx.xxx.xxx/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83n

這樣，每個節點都使用自定義etcd發現服務註冊自身到集群中，一旦所有的節點（3）都註冊後，集群就運行起來了。

公共etcd發現服務

如果本地沒有etcd集群正在運行著，可以使用http://discovery.etcd.io的公共發現服務。

$ curl https://discovery.etcd.io/new?size=3nhttps://discovery.etcd.io/4c9294a85da70808a0cdd2fde505fc41n

這樣就新建了個有3個節點的etcd集群發現URL。如果不指定size，默認為3。

$ etcd --name etcd0 n --initial-advertise-peer-urls http://10.1.2.154:2380 n --listen-peer-urls http://10.1.2.154:2380 n --listen-client-urls http://10.1.2.154:2379,http://127.0.0.1:2379 n --advertise-client-urls http://10.1.2.154:2379 n --discovery https://discovery.etcd.io/4c9294a85da70808a0cdd2fde505fc41n

$ etcd --name etcd1 n --initial-advertise-peer-urls http://10.1.2.147:2380 n --listen-peer-urls http://10.1.2.147:2380 n --listen-client-urls http://10.1.2.147:2379,http://127.0.0.1:2379 n --advertise-client-urls http://10.1.2.147:2379 n --discovery https://discovery.etcd.io/4c9294a85da70808a0cdd2fde505fc41n

$ etcd --name etcd2 n --initial-advertise-peer-urls http://10.1.2.148:2380 n --listen-peer-urls http://10.1.2.148:2380 n --listen-client-urls http://10.1.2.148:2379,http://127.0.0.1:2379 n --advertise-client-urls http://10.1.2.148:2379 n --discovery https://discovery.etcd.io/4c9294a85da70808a0cdd2fde505fc41n

分別在10.1.2.154、10.1.2.147、10.1.2.148三個節點運行上述命令，即可搭建擁有3個節點的etcd集群。

可以使用etcdctl檢查集群狀態：

$ tcdctl member listn16ad5ae76a270e9c: name=etcd1 peerURLs=http://10.1.2.147:2380 clientURLs=http://10.1.2.147:2379 isLeader=falsen908b97d8d3fa3129: name=etcd2 peerURLs=http://10.1.2.148:2380 clientURLs=http://10.1.2.148:2379 isLeader=falsene79ad49d769934cb: name=etcd0 peerURLs=http://10.1.2.154:2380 clientURLs=http://10.1.2.154:2379 isLeader=truen

集群管理

使用發現服務動態引導集群啟動以後，就不能使用發現服務新增或者修改節點了。必須使用其他方法，這裡介紹使用etcdctl如何管理集群，當然也可以使用SDK編寫代碼操作，這裡不做介紹，有興趣的同學自由查閱。

更新節點advertise-client-urls：可以直接重啟該節點，重啟的同時附上新的advertise-client-urls，節點會自動更新該參數，錯誤的值不會影響集群的健康
更新節點advertise-peer-urls：更新該參數因為會改變集群節點間的通信連接，所以會影響集群的健康，需要小心。首先，先要知道該節點的ID：

$ etcdctl member listn16ad5ae76a270e9c: name=etcd1 peerURLs=http://10.1.2.147:2380 clientURLs=http://10.1.2.147:2379 isLeader=falsen908b97d8d3fa3129: name=etcd2 peerURLs=http://10.1.2.148:2380 clientURLs=http://10.1.2.148:2379 isLeader=falsene79ad49d769934cb: name=etcd0 peerURLs=http://10.1.2.154:2380 clientURLs=http://10.1.2.154:2379 isLeader=truen

$ etcdctl member update 16ad5ae76a270e9c http://10.1.2.147:2380nUpdated member with ID 16ad5ae76a270e9c in clustern

刪除節點：可以很容易的從集群中刪除一個節點

$ etcdctl member remove 16ad5ae76a270e9cnRemoved member 16ad5ae76a270e9c from clustern

新增節點：

$ etcdctl member add etcd1 http://10.1.2.147:2380nAdded member named etcd1 with ID 377f8632d3987921 to clusternnETCD_NAME="etcd1"nETCD_INITIAL_CLUSTER="etcd1=http://10.1.2.147:2380,etcd2=http://10.1.2.148:2380,etcd0=http://10.1.2.154:2380"nETCD_INITIAL_CLUSTER_STATE="existing"n

該命令的意思是會添加一個節點，節點名叫etcd1，advertise-peer-urls為http://10.1.2.147:2380。但是呢，該節點還沒有搭建起來，需要管理員使用返回的三個環境變數，並手動啟動該節點的etcd進程

$ etcd --listen-client-urls http://10.1.2.147:2379 n --advertise-client-urls http://10.1.2.147:2379 n --listen-peer-urls http://10.1.2.147:2380 n --initial-advertise-peer-urls http://10.1.2.147:2380 n --name etcd1 n --initial-cluster etcd1=http://10.1.2.147:2380,etcd2=http://10.1.2.148:2380,etcd0=http://10.1.2.154:2380 n --initial-cluster-state existingn

$ etcdctl member listn908b97d8d3fa3129: name=etcd2 peerURLs=http://10.1.2.148:2380 clientURLs=http://10.1.2.148:2379 isLeader=falsene79ad49d769934cb: name=etcd0 peerURLs=http://10.1.2.154:2380 clientURLs=http://10.1.2.154:2379 isLeader=truene7e2fd9b5c316fed: name=etcd1 peerURLs=http://10.1.2.147:2380 clientURLs=http://10.1.2.147:2379 isLeader=falsen

數據相關

在etcd啟動的時候，使用參數--data-dir指定參數配置等信息的存儲位置。配置存儲在寫日誌中，包括本地memberID，集群ID和初始集群配置。寫日誌和快照文件用於節點操作，或者從重啟中恢復。

使用專有的空間存儲wal文件能提升集群吞吐量。在生產環境中，強烈推薦使用專有的空間存儲wal文件，通過參數--wal-dir設置。

如果某節點的數據文件丟失了或者損壞了，管理員應該使用etcdctl工具把該節點從集群中刪除。

管理員應該避免使用過期的數據備份文件重啟etcd節點，因為etcd使用raft進行共識，容易造成數據不一致。為了集群的安全，如果某個節點遭受了任何形式的數據損失，管理員應該把它從集群中刪除，並使用空數據空間重新加入集群。

wal：提前寫日誌文件
snap：日誌快照文件

如果設置了--wal-dir，etcd會在指定的地方存儲提前寫日誌文件，而不是在默認的data文件夾下。

集群區分

如果管理員同時管理著多個etcd集群，那麼推薦使用參數--initial-cluster-token為不同的集群設置不同的名字。這可以防止因為人為的錯誤造成集群崩潰。

集群監控

etcd單節點暴露了健康檢查介面/health，管理員可以通過該介面檢查etcd節點的健康狀態：

$ curl -L http://127.0.0.1:2379/healthn{"health": "true"}n

也可以使用etcdctl檢查集群的健康信息，也就是分別檢查etcd節點的健康狀態併合並起來

$ etcdctl cluster-healthnmember 908b97d8d3fa3129 is healthy: got healthy result from http://10.1.2.148:2379nmember e79ad49d769934cb is healthy: got healthy result from http://10.1.2.154:2379nmember e7e2fd9b5c316fed is healthy: got healthy result from http://10.1.2.147:2379ncluster is healthyn

集群調試

分散式系統的調試是很困難的，etcd提供了幾種方法使得調試相對簡單些。

運行時調試：

可以在不停止etcd的情況下，修改日誌等級：

$ curl http://127.0.0.1:2379/config/local/log -XPUT -d {"Level":"DEBUG"}n

調試變數：

etcd暴露了調試變數用於實時調試。熟悉etcd的開發者可以分析這些變數確定異常行為。可通過調用/debug/vars來返回json格式的數據。

# curl http://127.0.0.1:2379/debug/vars [18:13:59]n{n"cmdline": ["./etcd","--name","etcd0","--initial-advertise-peer-urls","http://10.1.2.154:2380","--listen-peer-urls","http://10.1.2.154:2380","--listen-client-urls","http://10.1.2.154:2379,http://127.0.0.1:2379","--advertise-client-urls","http://10.1.2.154:2379","--discovery","https://discovery.etcd.io/4c9294a85da70808a0cdd2fde505fc41"],n"file_descriptor_limit": 1024,n"memstats": {"Alloc":30781776,"TotalAlloc":1028671808,"Sys":66865400,"Lookups":34674,"Mallocs":12643977,"Frees":12578043,"HeapAlloc":30781776,"HeapSys":58916864,"HeapIdle":23748608,"HeapInuse":35168256,"HeapReleased":17498112,"HeapObjects":65934,"StackInuse":2555904,"StackSys":2555904,"MSpanInuse":363520,"MSpanSys":622592,"MCacheInuse":2400,"MCacheSys":16384,"BuckHashSys":1547007,"GCSys":2549760,"OtherSys":656889,"NextGC":55934720,"LastGC":1491906059277916627,"PauseTotalNs":24124780,"PauseNs":[137970,274512,89440,115096,146548,860272,4094018,279360,168412,360547,151179,104960,323453,385236,301494,96026,334830,282690,292172,79024,328147,307275,297093,241948,279165,262385,227695,98189,335315,320762,94992,166066,119608,262541,328688,279850,112756,288594,317476,124790,150796,367141,301852,297652,148544,129118,266195,135359,299603,244624,173377,322480,268048,129248,156234,277056,135280,119977,187473,188970,267234,370288,134837,232513,97046,312698,262126,326791,273335,158268,120111,301224,239484,297388,146373,122038,109004,149310,175805,307381,1161925,367421,97289,125290,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[1491896931937881527,1491896947695205166,1491896947711567990,1491896947722595908,1491896947732932030,1491896963098433133,1491897027678599839,1491897147698027075,1491897267707779134,1491897387723715896,1491897507731423150,1491897627744130882,1491897747770780498,1491897867809169948,1491897987829353611,1491898107868276472,1491898227911159554,1491898347923566095,1491898467979927175,1491898587987431529,1491898708007447665,1491898828093686435,1491898948107147576,1491899068118816704,1491899188125778375,1491899308133465767,1491899428167940101,1491899548179070719,1491899668194538860,1491899788212964423,1491899908229511595,1491900015451746301,1491900125736145199,1491900234714735168,1491900338778116476,1491900438833116333,1491900539069087773,1491900615728468300,1491900691437176062,1491900794354832194,1491900898108842736,1491901018119733156,1491901138167082961,1491901258184978986,1491901378208380215,1491901498242417563,1491901618276652772,1491901738307960768,1491901858325470176,1491901978389815529,1491902098407905965,1491902218433972520,1491902338512954497,1491902458529682279,1491902578557471764,1491902698610577372,1491902818644368150,1491902938658408921,1491903058676922200,1491903178707845355,1491903298736201974,1491903418768718805,1491903538808012563,1491903658828638453,1491903778845012030,1491903898860539005,1491904018907099337,1491904138948436931,1491904258963487495,1491904378975741171,1491904498991351129,1491904619007828216,1491904739063503599,1491904859086152961,1491904979106481896,1491905099147427888,1491905219162055647,1491905339198266626,1491905459216237683,1491905579232865004,1491905699244043255,1491905819258224777,1491905939266270866,1491906059277916627,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":84,"GCCPUFraction":4.3618010516851585e-05,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":353596,"Frees":352420},{"Size":16,"Mallocs":3498683,"Frees":3486212},{"Size":32,"Mallocs":2072179,"Frees":2054677},{"Size":48,"Mallocs":1654239,"Frees":1647943},{"Size":64,"Mallocs":756055,"Frees":744679},{"Size":80,"Mallocs":284628,"Frees":282515},{"Size":96,"Mallocs":831888,"Frees":829603},{"Size":112,"Mallocs":350452,"Frees":349275},{"Size":128,"Mallocs":54686,"Frees":54543},{"Size":144,"Mallocs":50740,"Frees":50284},{"Size":160,"Mallocs":44986,"Frees":44470},{"Size":176,"Mallocs":220591,"Frees":219903},{"Size":192,"Mallocs":11825,"Frees":11778},{"Size":208,"Mallocs":195503,"Frees":194741},{"Size":224,"Mallocs":37102,"Frees":36872},{"Size":240,"Mallocs":79228,"Frees":78996},{"Size":256,"Mallocs":32153,"Frees":32125},{"Size":288,"Mallocs":428871,"Frees":427684},{"Size":320,"Mallocs":47373,"Frees":47001},{"Size":352,"Mallocs":84924,"Frees":84604},{"Size":384,"Mallocs":28475,"Frees":28397},{"Size":416,"Mallocs":29543,"Frees":28515},{"Size":448,"Mallocs":7091,"Frees":7034},{"Size":480,"Mallocs":8916,"Frees":8882},{"Size":512,"Mallocs":21786,"Frees":21720},{"Size":576,"Mallocs":72569,"Frees":72465},{"Size":640,"Mallocs":39498,"Frees":35245},{"Size":704,"Mallocs":3834,"Frees":3787},{"Size":768,"Mallocs":1374,"Frees":1369},{"Size":896,"Mallocs":1587,"Frees":1542},{"Size":1024,"Mallocs":18753,"Frees":18651},{"Size":1152,"Mallocs":1927,"Frees":1715},{"Size":1280,"Mallocs":76,"Frees":61},{"Size":1408,"Mallocs":82,"Frees":55},{"Size":1536,"Mallocs":566,"Frees":545},{"Size":1664,"Mallocs":3881,"Frees":3853},{"Size":2048,"Mallocs":322,"Frees":268},{"Size":2304,"Mallocs":122,"Frees":110},{"Size":2560,"Mallocs":37,"Frees":34},{"Size":2816,"Mallocs":33,"Frees":33},{"Size":3072,"Mallocs":96,"Frees":96},{"Size":3328,"Mallocs":39,"Frees":30},{"Size":4096,"Mallocs":10999,"Frees":10909},{"Size":4608,"Mallocs":1121,"Frees":1118},{"Size":5376,"Mallocs":1844,"Frees":1827},{"Size":6144,"Mallocs":1864,"Frees":1844},{"Size":6400,"Mallocs":586,"Frees":586},{"Size":6656,"Mallocs":576,"Frees":575},{"Size":6912,"Mallocs":579,"Frees":579},{"Size":8192,"Mallocs":1033,"Frees":1027},{"Size":8448,"Mallocs":43,"Frees":43},{"Size":8704,"Mallocs":58,"Frees":58},{"Size":9472,"Mallocs":208,"Frees":205},{"Size":10496,"Mallocs":232,"Frees":232},{"Size":12288,"Mallocs":488,"Frees":467},{"Size":13568,"Mallocs":284,"Frees":284},{"Size":14080,"Mallocs":112,"Frees":112},{"Size":16384,"Mallocs":120,"Frees":116},{"Size":16640,"Mallocs":0,"Frees":0},{"Size":17664,"Mallocs":0,"Frees":0}]},n"raft.status": {"id":"e79ad49d769934cb","term":9,"vote":"e7e2fd9b5c316fed","commit":27479,"lead":"e7e2fd9b5c316fed","raftState":"StateFollower","progress":{}}n}n

集群大小

因為容錯的需求，推薦的集群大小為3、5、7，7個節點的集群能夠提供足夠的容錯能力。雖然更大的集群能夠提供更好的容錯，但會降低寫數據的效率，因為需要把數據同步到更多的機子上。

假設允許錯誤的節點數為f，則整個集群需要2f+1個節點才能穩定運行。

節點合併

當需要在不損失數據和改變節點ID的情況下替換etcd宿主機，需要用到節點合併功能。--data-dir指定的數據目錄保存了能使節點恢復到point-in-time狀態的全部數據。合併節點有以下步驟：

停止節點etcd進程
複製當前etcd節點的數據文件夾到新的節點
更新advertise-peer-urls
在新節點啟動etcd，使用相同的配置和數據文件夾的拷貝

災難恢復

etcd集群其實擁有很強的容錯能力，但是也抵不住不可預知的錯誤。etcd提供了備份恢復功能保證集群從災難中恢復。

備份數據：

# etcdctl backup --data-dir /root/etcd/etcd0.etcd --backup-dir /root/etcd/etcd0.etcd.backupn

該命令會在--backup-dir里重建data-dir的數據，如果snap數據和wal數據是分開放置的，還需指定--wal-dir和--backup-wal-dir參數

數據恢復：

備份的數據不一定能夠支撐集群的重新部署，首先我們需要驗證數據的完整性。可以使用-force-new-cluster參數啟動一個新的單節點集群，數據目錄當然指定備份目錄。

etcd n -data-dir=%backup_data_dir% n [-wal-dir=%backup_wal_dir%] n -force-new-cluster n ...n

一旦發現數據是可靠的，就可以把數據拷貝到原來的地方：

pkill etcdn rm -fr %data_dir%n rm -fr %wal_dir%n mv %backup_data_dir% %data_dir%n mv %backup_wal_dir% %wal_dir%n etcd n -data-dir=%data_dir% n [-wal-dir=%wal_dir%] n ...n

接下來就可以使用動態增刪節點功能添加新節點到集群中了。