607的彩的正确网址

现在的位置: 607的彩的正确网址 > 数据库 > 正文

etcd集群运维实践

2020年02月12日 数据库 ⁄ 共 10285字 ⁄ 字号 评论关闭
文章目录

【bianzhedehua】etcd shi kubernetes jiqundeshujuhexin,zuiyanzhongdeqingkuangshi,dang etcd chuwentichediwufahuifudeshihou,jiejuewentidebanfakenengzhiyouzhongxindajianyigehuanjing。yinciweirao etcd xiangguandeyunweizhishijiubijiaozhongyao,etcd keyirongqihuabushu,yekeyizaisuzhujizixingdajian,yixianeirongshitongyongde。

集群的备份和恢复

tianjiabeifen

#!/bin/bashIP=123.123.123.123BACKUP_DIR=/alauda/etcd_bak/mkdir -p $BACKUP_DIRexport ETCDCTL_API=3etcdctl --endpoints=http://$IP:2379 snapshot save $BACKUP/snap-$(date +%Y%m%d%H%M).db# 备份一个节点的数据就可以恢复,实践中,为了防止定时任务配置的节点异常没有生成备份,建议多加几个

 

huifujiqun

#!/bin/bash# 使用 etcdctl snapshot restore 生成各个节点的数据# 比较关键的变量是# --data-dir 需要是实际 etcd 运行时的数据目录# --name  --initial-advertise-peer-urls  需要用各个节点的配置# --initial-cluster  initial-cluster-token 需要和原集群一致ETCD_1=10.1.0.5ETCD_2=10.1.0.6ETCD_3=10.1.0.7for i in ETCD_1 ETCD_2 ETCD_3doexport ETCDCTL_API=3etcdctl snapshot restore snapshot.db --data-dir=/var/lib/etcd --name $i --initial-cluster ${ETCD_1}=http://${ETCD_1}:2380,${ETCD_2}=http://${ETCD_2}:2380,${ETCD_3}=http://${ETCD_3}:2380 --initial-cluster-token k8s_etcd_token --initial-advertise-peer-urls http://$i:2380 && mv /var/lib/etcd/ etcd_$idone# 把 etcd_10.1.0.5 复制到 10.1.0.5节点,覆盖/var/lib/etcd(同--data-dir路径)# 其他节点依次类推

 

yong etcd zidongchuangjiande snapdb huifu

#!/bin/bash export ETCDCTL_API=3etcdctl snapshot restore snapshot.db --skip-hash-check --data-dir=/var/lib/etcd --name 10.1.0.5 --initial-cluster 10.1.0.5=http://10.1.0.5:2380,10.1.0.6=http://10.1.0.6:2380,10.1.0.7=http://10.1.0.7:2380 --initial-cluster-token k8s_etcd_token --initial-advertise-peer-urls http://10.1.0.5:2380# 也是所有节点都需要生成自己的数据目录,参考上一条# 和上一条命令唯一的差别是多了  --skip-hash-check  (跳过完整性校验)# 这种方式不能确保 100% 可恢复,建议还是自己加备份# 通常恢复后需要做一下数据压缩和碎片整理,可参考相应章节

 

caiguodekeng

[ 3.0.14 ban etcd restore gongnengbukeyong ] http://github.com/etcd-io/etcd/issues/7533shiyonggengxinde etcd jike。zongjie:huifujiushiyaona db quba etcd deshujushengchengyifen,yongtongyigejiediande,keyibaozhengchule restore shihouzhidingdecanshuwai,suoyoushujudouyiyang。zheijiushiyongyifen db,caozuosanci(huozhe5ci)deyuanyin。

集群的扩容——从 1 到 3

zhixingtianjia

#!/bin/bashexport ETCDCTL_API=2etcdctl --endpoints=http://10.1.0.6:2379 member add 10.1.0.6 http://10.1.0.6:2380etcdctl --endpoints=http://10.1.0.7:2379 member add 10.1.0.7 http://10.1.0.7:2380# ETCD_NAME="etcd_10.1.0.6" # ETCD_INITIAL_CLUSTER="10.1.0.6=http://10.1.0.6:2380,10.1.0.5=http://10.1.0.5:2380"# ETCD_INITIAL_CLUSTER_STATE="existing"

 

zhunbeitianjiadejiedian etcd canshupeizhi

#!/bin/bash/usr/local/bin/etcd --data-dir=/data.etcd --name 10.1.0.6--initial-advertise-peer-urls http://10.1.0.6:2380 --listen-peer-urls http://10.1.0.6:2380 --advertise-client-urls http://10.1.0.6:2379 --listen-client-urls http://10.1.0.6:2379 --initial-cluster 10.1.0.6=http://10.1.0.6:2380,10.1.0.5=http://10.1.0.5:2380--initial-cluster-state exsiting--initial-cluster-token k8s_etcd_token# --initial-cluster 集群所有节点的 name=ip:peer_url# --initial-cluster-state exsiting 告诉 etcd 自己归属一个已存在的集群,不要自立门户

 

caiguodekeng

cong 1 dao 3 qijian,huijingguojiqunshiliangjiediandezhuangtai,zheishihoukenengjiqundebiaoxianjiuxiangguale,endpoint status zheixieminglingdoubunengyong,suoyiwomenxuyaoyong member add xianbajiqunkuodaosanjiedian,ranhouzaiyiciqidong etcd shili,zheiyangzuojiunengquebao etcd jiushijiankangde。cong 3 daogengduo,qishihaishi member add la,jiufangxingaoba。

集群加证书

shengchengzhengshu

curl -s -L -o /usr/bin/cfssl http://pkg.cfssl.org/R1.2/cfssl_linux-amd64curl -s -L -o /usr/bin/cfssljson http://pkg.cfssl.org/R1.2/cfssljson_linux-amd64chmod +x /usr/bin/{cfssl,cfssljson}cd /etc/kubernetes/pki/etcd

 

#  cat ca-config.json{"signing": {"default": {  "expiry": "100000h"},"profiles": {  "server": {    "usages": ["signing", "key encipherment", "server auth", "client auth"],    "expiry": "100000h"  },  "client": {    "usages": ["signing", "key encipherment", "server auth", "client auth"],    "expiry": "100000h"  }}}} 

 

#  cat ca-csr.json{"CN": "etcd","key": {"algo": "rsa","size": 4096},"names": [{  "C": "CN",  "L": "Beijing",  "O": "Alauda",  "OU": "PaaS",  "ST": "Beijing"}]} 

 

#  cat server-csr.json{"CN": "etcd-server","hosts": ["localhost","0.0.0.0","127.0.0.1","所有master 节点ip ","所有master 节点ip ","所有master 节点ip "],"key": {"algo": "rsa","size": 4096},"names": [{  "C": "CN",  "L": "Beijing",  "O": "Alauda",  "OU": "PaaS",  "ST": "Beijing"}]} 

 

# cat client-csr.json{"CN": "etcd-client","hosts": [""],"key": {"algo": "rsa","size": 4096},"names": [{  "C": "CN",  "L": "Beijing",  "O": "Alauda",  "OU": "PaaS",  "ST": "Beijing"}]} 

 

cd /etc/kubernetes/pki/etcdcfssl gencert -initca ca-csr.json | cfssljson -bare cacfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare servercfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssljson -bare client

cankaolianjie:http://lihaoquan.me/2017/3/29 ... .html

607的彩的正确网址shouxiangengxinjiediandepeer-urls

export ETCDCTL_API=3etcdctl --endpoints=http://x.x.x.x:2379 member list#  1111111111  ..........#  2222222222  ..........#  3333333333  ..........etcdctl --endpoints=http://172.30.0.123:2379 member update 1111111111 --peer-urls=http://x.x.x.x:2380# 执行三次把三个节点的peer-urls都改成https

 

修改配置

#  vim /etc/kubernetes/main*/etcd.yaml#  etcd启动命令部分修改 http 为 https,启动状态改成 existing- --advertise-client-urls=http://x.x.x.x:2379- --initial-advertise-peer-urls=http://x.x.x.x:2380- --initial-cluster=xxx=http://x.x.x.x:2380,xxx=http://x.x.x.x:2380,xxx=http://x.x.x.x:2380- --listen-client-urls=http://x.x.x.x:2379- --listen-peer-urls=http://x.x.x.x:2380- --initial-cluster-state=existing#  etcd 启动命令部分插入- --cert-file=/etc/kubernetes/pki/etcd/server.pem- --key-file=/etc/kubernetes/pki/etcd/server-key.pem- --peer-cert-file=/etc/kubernetes/pki/etcd/server.pem- --peer-key-file=/etc/kubernetes/pki/etcd/server-key.pem- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem- --peer-client-cert-auth=true- --client-cert-auth=true#  检索hostPath在其后插入- hostPath:  path: /etc/kubernetes/pki/etcd  type: DirectoryOrCreatename: etcd-certs#  检索mountPath在其后插入- mountPath: /etc/kubernetes/pki/etcd  name: etcd-certs

 

#  vim /etc/kubernetes/main*/kube-apiserver.yaml#  apiserver 启动部分插入,修改 http 为https- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem- --etcd-certfile=/etc/kubernetes/pki/etcd/client.pem- --etcd-keyfile=/etc/kubernetes/pki/etcd/client-key.pem- --etcd-servers=http://x.x.x.x:2379,http://x.x.x.x:2379,http://x.x.x.x:2379

zongjiexiajiushi,xianzhunbeiyitaozhengshu。ranhouxiugai etcd neibutongxindizhiweihttps,zheishihouetcdrizhihuibaocuo(keyihulve),ranhouyongetcd --daizhengshudecanshuqidong,basuoyoulianjieetcddedifangdouyongshangzhengshu,jike。

yudaodekeng

[ etcd jiazhengshuhou,apiserver dejiankangjianchahaishi http qingqiu,etcd huiyizhishuarizhi ] http://github.com/etcd-io/etcd/issues/9285

2018-02-06 12:41:06.905234 I | embed: rejected connection from "127.0.0.1:35574" (error "EOF", ServerName "")

jiejuebanfa:zhijiequdiao apiserver dejiankangjiancha,huozhebamorendejianchaminglinghuancheng curl(apiserver dejingxiangliyinggaimeiyou curl,ruguoshigangxudehuazijizhongxin build yixiaba)

集群升级

607的彩的正确网址yijingshi v3 dedejiqunbuxuyaotaiduodepeizhi,baoliushujumulu,tihuanjingxiang(huozheerjinzhi)jike;v2 dao v3 deshengjixuyaoyige merge decaozuo,wobingmeiyoushijideshijianguo,yebutaituijianzheiyangzuo。

集群状态检查

607的彩的正确网址qishishangshusuoyoubuzhoudouxuyaozheixieminglingdefuzhu——

#!/bin/bash# 如果证书的话,去掉--cert --key --cacert 即可# --endpoints= 需要写了几个节点的url,endpoint status就输出几条信息export ETCDCTL_API=3etcdctl --endpoints=http://x.x.x.x:2379  --cert=/etc/kubernetes/pki/etcd/client.pem --key=/etc/kubernetes/pki/etcd/client-key.pem --cacert=/etc/kubernetes/pki/etcd/ca.pem endpoint status -w tableetcdctl --endpoints=xxxx endpoint healthetcdctl --endpoints=xxxx member listkubectl get cs

 

数据操作(删除、压缩、碎片整理)

shanchu

ETCDCTL_API=2 etcdctl rm --recursive            # v2 的 api 可以这样删除一个“目录”ETCDCTL_API=3 etcdctl --endpoints=xxx del /xxxxx --prefix # v3 的版本# 带证书的话,参考上一条添加 --cert --key --cacert 即可

yudaodekeng:zaiyigekehuhuanjinglifaxian kubernetes jiqunlide “shijian” chaojiduo,jiushi kubectl describe xxx kandaode events bufenxinxi,shujutaidadaozhi etcd paodehenlei,womenjiuyongzheiyangdefangshishandiaomeiyongdezheixieshuju。

suipianzhengli

ETCDCTL_API=3 etcdctl --endpoints=xx:xx,xx:xx,xx:xx defragETCDCTL_API=3 etcdctl --endpoints=xx:xx,xx:xx,xx:xx endpoint status # 看数据量

 

yasuo

ETCDCTL_API=3 etcdctl --endpoints=xx:xx,xx:xx,xx:xx compact# 这个在只有 K8s 用的 etcd 集群里作用不太大,可能具体场景我没遇到# 可参考这个文档# http://www.cnblogs.com/davygeek/p/8524477.html# 不过跑一下不碍事etcd --auto-compaction-retention=1# 添加这个参数让 etcd 运行时自己去做压缩

 

常见问题

    etcd 对时间很依赖,所以集群里的节点时间一定要同步磁盘空间不足,如果磁盘是被 etcd 自己吃完了,就需要考虑压缩和删数据啦加证书后所有请求就都要带证书了,要不会提示 context deadline exceeded做各个操作时 etcd 启动参数里标明节点状态的要小心,否则需要重新做一遍前面的步骤很麻烦

 

日志收集

etcd derizhizanshizhizhichi syslog he stdout liangzhong——http://github.com/etcd-io/etcd/issues/7936etcd derizhizaipaichaguzhangshihenyouyong,ruguowomenyongsuzhujilaibushu etcd,rizhikeyitongguo systemd jiansuodao,dan kubeadm fangshiqidongde etcd zairongqizhongqihoujiuhuidiushisuoyoulishi。womenkeyiyongyixiadefanganlaizuo——

shell dezhongdingxiang

etcd --xxxx --xxxx   >  /var/log/etcd.log # 配合 logratate 来做日志切割# 将日志通过 volume 挂载到宿主机

 

supervisor

supervisor congrongqigangkaishiliuxingshi,jiushibaochifuwuchixuyunxinghenyouxiaodegongju。

607的彩的正确网址sidecar rongqi(houxuwozai github shangbuchongyigelizi,github.com/jing2uo)

sidecar keyijiandanlijieweiyige pod liyouduogerongqi(biru kubedns)tamenbicikeyikandaoduifangdejincheng,yinciwomenkeyiyongchuantongde strace laibuzhuo etcd jinchengdeshuchu,ranhouzai sidecar zheigerongqilihe shell zhongdingxiangyiyangcaozuo。

strace  -e trace=write -s 200 -f -p 1

 

Kubeadm 1.13 部署的集群

zuijinwomenceshi kubernetes 1.13 jiqunshifaxianleyixieyouqudegaibian,zhayikanwomenshangmiandeminglingjiumeifayongle——http://kubernetes.io/docs/set ... logy/qufenle stacked etcd topology he external etcd topology,guanfangdelianjielezheigetuhenxingxiang——

607的彩的正确网址zheizhongmoshixiade etcd jiqun,zuimingxiandechabieshirongqinei etcd deinitial-cluster qidongcanshuzhiyouzijide ip,huiyoudianmenggualewozheigaizenmequhuifu。qishijibenyuanlimeiyoubian,kubeadm zanglege configmap,qidongcanshubeifangzailezheili——

kubectl get cm  etcdcfg -n kube-system -o yaml

 

etcd:  local:    serverCertSANs:    - "192.168.8.21"    peerCertSANs:    - "192.168.8.21"    extraArgs:      initial-cluster: 192.168.8.21=http://192.168.8.21:2380,192.168.8.22=http://192.168.8.22:2380,192.168.8.20=http://192.168.8.20:2380      initial-cluster-state: new      name: 192.168.8.21      listen-peer-urls: http://192.168.8.21:2380      listen-client-urls: http://192.168.8.21:2379      advertise-client-urls: http://192.168.8.21:2379      initial-advertise-peer-urls: http://192.168.8.21:2380

 

Q&A

Q:请问 etcd 监控和告警如何做的?告警项都有哪些?

607的彩的正确网址a:gaojingyaokanyongdeshenmejiankongba,he kubernetes peitaobijiaochangjiandeshipuluomixiusihe grafana le。gaojingxiangwomeiyoujutipeiguo,keyiguanzhudedianshi:endpoint status -w table likeyikandaoshujuliang,endpoints health kandaojiankangzhuangtai,haiyouneicunshiyongzheixie,jutikeyicankaopuluomixiuside exporter shizenmezuode。

Q:使用 Kubeadm 部署高可用集群是不是相当于先部署三个独立的单点 Master,最后靠 etcd 添加节点操作把数据打通?

607的彩的正确网址a:bushi,kubeadm bushuhuizaizuikaishijiuxianjianyige etcd jiqun,apiserver qidongzhiqianjiuxuyaozhunbeihao etcd,fouze apiserver qibule,jiqunzhijianjiumeifatongxin。keyichangshishoudongdayixiajiqun,buyong kubeadm,yigegebazujiankaiqilai,zhihouduikubernetesdezujianguanxihuilijiegenghaode。

Q:etcd 跨机房高可用如何保证呢?管理 etcd 有好的 UI 工具推荐么?

607的彩的正确网址a:etcd duishijianhewangluoyaoqiuhengao,suoyikuajifangdewangluobuhaodehuaxingnenghencha,guangzaineibianxuanqingshurulianjiemiaoshujuqule。wofenxiangwangletiyige etcd de mirror,keyiqucankaoxiazuofa。kuajifangdehua,wojuedegaosuwangluoshigeqiantiba,buguohaimeizuoguo。ui gongjumeizhaoguo,doushiminglingxingcaozuolaizhe。

Q:Kubeadm 启动的集群内 etcd节 点,kubectl 操作 etcd 的备份恢复有尝试过吗?

607的彩的正确网址a:meiyouyong kubectl quchuliguo etcd debeifenhuifu。etcd dehuifuyilaiyong snapdb shengchengshujumulu,ba etcd jinchengdiujinrongqili,leisidecaozuobimianbule,haiyouqidongdezhuangtaixuyaoxiugai。kubeadm qidongde etcd keyitongguo kubectl chaxunhe exec,danshishujucaozuoyinggaibukeyi,biruhuifu etcd ing shi,wufalianjie etcd,kubectl haizenmegongzuo?

Q:kubeadm-ha 启动 3 个 Master,有 3 个 etcd 节点,怎么跟集群外的 3 个 etcd 做集群,做成 3 Master 6 etcd?

607的彩的正确网址a:keyicankaowendanglidekuorongbufen,zhiyaobaozheng etcd decanshuzhengque,jishiyigejiqunyibufenrongqihua,yibufensuzhuji,doushikeyide(dangranbujianyizheimezuo)。keyixianyong kubeadm dayigejiqun,ranhouyongkuorongdefangshibaqitasangejiedianjiajinlai,huozhezai kubeadm caozuozhiqian,xiandayige etcd jiqun。ranhou kubeadm diaoyongtajiukeyi。

Q:有没有试过 Kubeadm 的滚动升级,etcd 版本变更,各 Master 机分别重启,数据同步是否有异常等等?

a:zuoguo。kubeadm degundongshengjigongsineibuyoucong 1.7 yibubushengjidao 1.11、1.12 dewendang,huoduohuoshaoyouyidianxiaokeng,buguojintianzhutishi etcd suoyimeitizheibufen。gege master fenbiezhongqihoushujudeyizhiwomenceshishimeiwenti,haiyoubijiaojiduandeshizhijiebasan master tingjiyitian,zaiqidonghouyenenghuifu。

以上内容根据2019年1月3日晚微信群分享内容整理。分享人郭靖,灵雀云运维开发工程师,有大规模集群运维经验,对自动化迷之热衷,精通Ansible,HashiCorp工具集,容器和Kubernetes鼓捣了三年,喜欢用Python和Go写小工具,DevOps推崇及践行者,近期关注和期待OpsMop。DockOne每周都会组织定向的技术分享,欢迎感兴趣的同学加微信:liyingjiesd,进群参与,您有想听的话题或者想分享的话题都可以给我们留言。

yishangjiushangyouguanetcdjiqunyunweishijiandexiangguanjieshao,yaolejiegengduokubernetes,devops,alaudaneirongqingdengluxuebuyuan。

baoqian!pinglunyiguanbi.

2元中国福利彩票怎么看 2元买彩票选几个数字 2元中国福利彩票的买法 2元就可以打1毛的斗地主棋牌 2元彩是值得您信赖的 2元斗地主群 2元的彩的网 2元彩票双色球走势图双色球 2元彩票网全国走势图 2元就可以打1毛的炸金花棋牌