🥇 ʻO kā mākou ʻike no ka hana ʻana me ka ʻikepili i loko o ka pūʻulu Kubernetes etcd pololei (me ka ʻole o K8s API)

Ke hoʻonui nei, ke noi mai nei nā mea kūʻai mai iā mākou e hāʻawi i ke komo i ka hui Kubernetes e hiki ke komo i nā lawelawe i loko o ka pūʻulu: i hiki iā lākou ke hoʻopili pololei i kekahi waihona a lawelawe paha, e hoʻopili i kahi noi kūloko me nā noi i loko o ka pūʻulu...

No ka laʻana, pono e hoʻopili mai kāu mīkini kūloko i kahi lawelawe memcached.staging.svc.cluster.local. Hāʻawi mākou i kēia mana me ka hoʻohana ʻana i kahi VPN i loko o ka pūʻulu kahi e hoʻopili ai ka mea kūʻai. No ka hana ʻana i kēia, hoʻolaha mākou i nā subnets o nā pods, nā lawelawe a me ka pahu cluster DNS i ka mea kūʻai aku. No laila, ke ho'āʻo ka mea kūʻai aku e hoʻopili i ka lawelawe memcached.staging.svc.cluster.local, hele ka noi i ka cluster DNS a ma ka pane e loaʻa i ka helu o kēia lawelawe mai ka pūnaewele lawelawe cluster a i ʻole ka helu pod.

Hoʻonohonoho mākou i nā pūʻulu K8 me ka kubeadm, kahi o ka subnet lawelawe paʻamau 192.168.0.0/16, a ʻo ka pūnaewele o nā pods 10.244.0.0/16. Hoʻohana maikaʻi nā mea a pau, akā aia kekahi mau helu:

Uena lalo 192.168.*.* hoʻohana pinepine ʻia i nā pūnaewele keʻena o nā mea kūʻai aku, a ʻoi aku ka nui o nā mea hoʻomohala home. A laila hoʻopau mākou i nā paio: hana nā mea hoʻokele home ma kēia subnet a hoʻokuʻu ka VPN i kēia mau subnets mai ka hui a i ka mea kūʻai.
Loaʻa iā mākou kekahi mau puʻupuʻu (hana, kahua a/a i ʻole kekahi mau puʻupuʻu dev). A laila, ma ka maʻamau, e loaʻa iā lākou nā subnets like no nā pods a me nā lawelawe, kahi e hana ai i nā pilikia nui no ka hana like me nā lawelawe i loko o nā pūʻulu.

Ua hoʻohana lōʻihi mākou i ka hana o ka hoʻohana ʻana i nā subnets ʻokoʻa no nā lawelawe a me nā pods i loko o ka papahana hoʻokahi - ma ke ʻano nui, i loaʻa i nā hui āpau nā pūnaewele ʻokoʻa. Eia nō naʻe, he nui ka nui o nā puʻupuʻu i ka hana ʻaʻole wau makemake e ʻōwili ʻia mai ka wā ʻōpala, no ka mea ke holo nei lākou i nā lawelawe he nui, nā palapala stateful, etc.

A laila nīnau mākou iā mākou iho: pehea e hoʻololi ai i ka subnet i kahi puʻupuʻu e kū nei?

Huli i na olelo hooholo

ʻO ka hana maʻamau ka hana hou nā mea a pau nā lawelawe me ke ʻano ClusterIP. Ma ke ʻano he koho, hiki ke aʻo a ʻo kēia:

He pilikia kēia kaʻina hana: ma hope o ka hoʻonohonoho ʻia ʻana o nā mea a pau, hele mai nā pods me ka IP kahiko ma ke ʻano he DNS nameserver ma /etc/resolv.conf.
No ka mea ʻaʻole i loaʻa iaʻu ka hopena, pono wau e hoʻonohonoho hou i ka hui holoʻokoʻa me ka kubeadm reset a hoʻomaka hou.

Akā ʻaʻole kūpono kēia no kēlā me kēia... Eia nā hoʻolauna kikoʻī hou aku no kā mākou hihia:

Hoʻohana ʻia ka flannel;
Aia nā pūʻulu ma nā ao a ma nā lako lako;
Makemake wau e pale i ka hoʻohana hou ʻana i nā lawelawe āpau i ka pūʻulu;
Pono e hana maʻamau i nā mea āpau me ka liʻiliʻi o nā pilikia;
ʻO ka mana Kubernetes he 1.16.6 (eia naʻe, e like nā pae hou aku no nā mana ʻē aʻe);
ʻO ka hana nui ʻo ia ka hōʻoia ʻana i loko o kahi puʻupuʻu i hoʻohana ʻia me ka kubeadm me kahi subnet lawelawe 192.168.0.0/16, pani me 172.24.0.0/16.

A no laila ua lōʻihi ko mākou hoihoi i ka ʻike ʻana i ka mea a pehea e mālama ʻia ai i loko o Kubernetes i etcd, he aha ka mea hiki ke hana me ia ... No laila ua manaʻo mākou: "No ke aha e hoʻohou ai i ka ʻikepili ma etcd, e hoʻololi i nā IP IP kahiko (subnet) me nā mea hou? "

I ka ʻimi ʻana i nā mea hana i mākaukau no ka hana ʻana me ka ʻikepili ma etcd, ʻaʻole i loaʻa iā mākou kekahi mea e hoʻopau loa i ka pilikia. (Ma ke ala, inā ʻike ʻoe e pili ana i nā pono hana no ka hana ʻana me ka ʻikepili pololei ma etcd, mahalo mākou i nā loulou.) Eia naʻe, ʻo kahi hoʻomaka maikaʻi mea kokua mai OpenShift (mahalo i kāna mea kākau!).

Hiki i kēia pono ke hoʻohui iā etcd me ka hoʻohana ʻana i nā palapala hōʻoia a heluhelu i ka ʻikepili mai laila me ka hoʻohana ʻana i nā kauoha ls, get, dump.

Hoʻohui etcdhelper

ʻO ka manaʻo aʻe he logical: "He aha ka mea e kāohi iā ʻoe mai ka hoʻohui ʻana i kēia pono ma ka hoʻohui ʻana i ka hiki ke kākau i ka ʻikepili i etcd?"

Ua lilo ia i mana hoʻololi o etcdhelper me ʻelua mau hana hou changeServiceCIDR и changePodCIDR. maluna ona hiki iā ʻoe ke ʻike i ke code maanei.

He aha nā hana hou? Algorithm changeServiceCIDR:

hana i kahi deserializer;
hōʻuluʻulu i kahi ʻōlelo maʻamau e pani i ka CIDR;
hele mākou i nā lawelawe āpau me ke ʻano ClusterIP i ka hui:
- decode i ka waiwai mai etcd i kahi mea Go;
- me ka hoʻohana ʻana i kahi ʻōlelo maʻamau e hoʻololi mākou i nā bytes ʻelua o ka helu wahi;
- hāʻawi i ka lawelawe i kahi helu IP mai ka subnet hou;
- e hana i serializer, e hoʻololi i ka Go object i protobuf, e kākau i ka ʻikepili hou i etcd.

kuleana pili i changePodCIDR like loa changeServiceCIDR - ma kahi o ka hoʻoponopono ʻana i ka kikoʻī lawelawe, hana mākou no ka node a hoʻololi .spec.PodCIDR i kahi subnet hou.

E aʻo

Hoʻololi i ka lawelawe CIDR

He maʻalahi loa ka hoʻolālā no ka hoʻokō ʻana i ka hana, akā pili ia i ka downtime ʻoiai e hana hou ʻia nā pods a pau o ka hui. Ma hope o ka wehewehe ʻana i nā ʻanuʻu nui, e kaʻana pū mākou i nā manaʻo pehea, ma ke kumumanaʻo, hiki ke hōʻemi ʻia kēia downtime.

Nā hana hoʻomākaukau:

ka hoʻouka ʻana i ka polokalamu pono a me ka hoʻohui ʻana i ka etcdhelper patched;
backup etcd a /etc/kubernetes.

Hoʻolālā hana pōkole no ka hoʻololi ʻana i ka lawelaweCIDR:

hoʻololi i ka apiserver a me ka mea hoʻokele-manager manifests;
ka hoopuka hou ana i na palapala;
hoʻololi i nā lawelawe ClusterIP ma etcd;
e hoʻomaka hou i nā pods a pau i ka hui.

ʻO ka mea ma lalo nei he kaʻina hana piha i nā kikoʻī.

1. E hoʻouka i ka etcd-client no ka hoʻolei ʻikepili:

apt install etcd-client

2. kūkulu etcdhelper:

E hoʻouka i ka golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

Mālama mākou iā mākou iho etcdhelper.go, hoʻoiho i nā mea hilinaʻi, hōʻiliʻili:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Hana i kahi waihona etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. E hoʻololi i ka subnet lawelawe ma ka hōʻike ʻana o ka mokulele hoʻokele Kubernetes. I nā faila /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml hoʻololi i ka ʻāpana --service-cluster-ip-range i kahi subnet hou: 172.24.0.0/16 ma kahi o 192.168.0.0/16.

5. No ka mea, ke hoʻololi nei mākou i ka subnet lawelawe i hoʻopuka ai ʻo kubeadm i nā palapala hōʻoia no apiserver (me ka pūʻulu), pono e hoʻopuka hou ʻia:

E ʻike kākou i nā kāʻei kapu a me nā helu IP i hoʻopuka ʻia ka palapala hōʻoia o kēia manawa no:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

E hoʻomākaukau i kahi hoʻonohonoho liʻiliʻi no kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла

E holoi mākou i ka crt kahiko a me ke kī, no ka mea ʻaʻole e hoʻopuka ʻia ka palapala hōʻoia hou:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

E hoʻopuka hou i nā palapala hōʻoia no ke kikowaena API:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

E nānā kāua ua hoʻopuka ʻia ka palapala hōʻoia no ka subnet hou:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Ma hope o ka hoʻopuka hou ʻana i ka palapala kikowaena API, e hoʻomaka hou i kāna pahu:
```
docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart
```
E hana hou i ka config no admin.conf:
```
kubeadm alpha certs renew admin.conf
```
E hoʻoponopono i ka ʻikepili ma etcd:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
E hoʻomaikaʻi mai! I kēia manawa, ho'ōki ka hoʻonā ʻāina i ka hana ʻana i ka pūʻulu, no ka mea i loko o nā pods i loaʻa /etc/resolv.conf ua hoʻopaʻa inoa ʻia ka helu CoreDNS kahiko (kube-dns), a hoʻololi ʻo kube-proxy i nā lula iptables mai ka subnet kahiko i ka mea hou. ʻOi aku ma ka ʻatikala ua kākau ʻia e pili ana i nā koho hiki ke hōʻemi i ka downtime.
E hoʻoponopono kākou i ka ConfigMap ma ka inoa inoa kube-system:
```
kubectl -n kube-system edit cm kubelet-config-1.16
```
- pani maanei clusterDNS i ka helu IP hou o ka lawelawe kube-dns: kubectl -n kube-system get svc kube-dns.
```
kubectl -n kube-system edit cm kubeadm-config
```
- e hoʻoponopono mākou data.ClusterConfiguration.networking.serviceSubnet i kahi subnet hou.
Ma muli o ka loli ʻana o ka helu kube-dns, pono e hoʻohou i ka config kubelet ma nā nodes a pau:
```
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```

ʻO nā mea a pau i koe e hoʻomaka hou i nā pods a pau i loko o ka pūʻulu:

kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

E hōʻemi i ka manawa hoʻomaha

Nā manaʻo pehea e hōʻemi ai i ka manawa hoʻomaha:

Ma hope o ka hoʻololi ʻana i ka hōʻike mokulele mana, hana i kahi lawelawe kube-dns hou, no ka laʻana, me ka inoa kube-dns-tmp a me ka helu wahi hou 172.24.0.10.
E hana if i etcdhelper, ʻaʻole ia e hoʻololi i ka lawelawe kube-dns.
E hoʻololi i ka helu wahi ma nā kubelets a pau ClusterDNS i kahi hou, ʻoiai e hoʻomau ka lawelawe kahiko i ka manawa like me ka mea hou.
E kali a hiki i ka ʻōwili ʻia ʻana o nā pods me nā noi no nā kumu kūlohelohe a i ʻole i ka manawa i ʻae ʻia.
Holoi i ka lawelawe kube-dns-tmp a hoololi serviceSubnetCIDR no ka lawelawe kube-dns.

E ʻae kēia hoʻolālā iā ʻoe e hōʻemi i ka manawa haʻahaʻa i ~ hoʻokahi minuke - no ka lōʻihi o ka lawe ʻana i ka lawelawe kube-dns-tmp a hoʻololi i ka subnet no ka lawelawe kube-dns.

Hoʻololi podNetwork

Ma ka manawa like, ua hoʻoholo mākou e nānā pehea e hoʻololi ai i ka podNetwork me ka hoʻohana ʻana i ka etcdhelper i loaʻa. ʻO ke kaʻina o nā hana penei:

hoʻoponopono i nā configs i loko kube-system;
hoʻoponopono i ka hōʻike kube-controller-manager;
hoʻololi pololei i ka podCIDR i etcd;
e hoʻomaka hou i nā pūnana puʻupuʻu a pau.

I kēia manawa hou e pili ana i kēia mau hana:

1. Hoʻololi i ka ConfigMaps ma ka inoa inoa kube-system:

kubectl -n kube-system edit cm kubeadm-config

- hoʻoponopono data.ClusterConfiguration.networking.podSubnet i kahi subnet hou 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- hoʻoponopono data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Hoʻololi i ka hōʻike hoʻomalu-manakia:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- hoʻoponopono --cluster-cidr=10.55.0.0/16.

3. E nana i na waiwai o keia wa .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses no nā node pūʻulu a pau:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Hoʻololi i ka podCIDR ma ka hoʻololi pololei ʻana iā etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. E nānā kākou ua loli maoli ka podCIDR:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. E ho'āla hou kākou i nā pūnana puʻupuʻu a pau.

7. Inā ha'alele 'oe i ho'okahi node podCIDR kahiko, a laila ʻaʻole hiki ke hoʻomaka ka kube-controller-manager, a ʻaʻole e hoʻonohonoho ʻia nā pods i loko o ka pūʻulu.

ʻOiaʻiʻo, hiki ke maʻalahi ka hoʻololi ʻana i ka podCIDR (no ka laʻana, no laila,). Akā makemake mākou e aʻo pehea e hana pololei me etcd, no ka mea aia kekahi mau hihia i ka hoʻoponopono ʻana i nā mea Kubernetes i etcd - ka wale ʻokoʻa hiki. (No ka laʻana, ʻaʻole hiki iā ʻoe ke hoʻololi wale i ke kahua lawelawe me ka ʻole o ka manawa spec.clusterIP.)

ʻO ka hopena

Kūkākūkā ka ʻatikala i ka hiki ke hana me ka ʻikepili ma etcd pololei, i.e. kaʻalo ʻana i ka Kubernetes API. ʻO kekahi manawa kēia ala e hiki ai iā ʻoe ke hana i nā "mea paʻakikī". Ua hoʻāʻo mākou i nā hana i hāʻawi ʻia i ka kikokikona ma nā pūʻulu K8s maoli. Eia naʻe, ko lākou kūlana o ka mākaukau no ka hoʻohana ākea PoC (hōʻoia o ka manaʻo). No laila, inā makemake ʻoe e hoʻohana i kahi mana i hoʻololi ʻia o ka pono etcdhelper ma kāu mau puʻupuʻu, e hana pēlā ma kāu pilikia ponoʻī.

PS

E heluhelu pū ma kā mākou blog:

Source: www.habr.com

ʻO kā mākou ʻike e hana pū me ka ʻikepili i loko o ka pūʻulu Kubernetes etcd pololei (me ka ʻole K8s API)