🥇O matou poto masani i le galulue faʻatasi ma faʻamaumauga i le etcd Kubernetes cluster tuusaʻo (e aunoa ma K8s API)

O lo'o fa'atuputeleina, o lo'o talosaga mai tagata fa'atau ia i matou e tu'uina atu le avanoa i le kulupu Kubernetes ina ia mafai ai ona maua auaunaga i totonu o le fuifui: ia mafai ona fa'afeso'ota'i sa'o i nisi fa'amaumauga po'o auaunaga, fa'afeso'ota'i se talosaga fa'apitonu'u ma talosaga i totonu o le fuifui...

Mo se faʻataʻitaʻiga, e manaʻomia le faʻafesoʻotaʻi mai lau masini faʻapitonuʻu i se auaunaga memcached.staging.svc.cluster.local. Matou te tuʻuina atu lenei gafatia e faʻaaoga ai se VPN i totonu o le fuifui lea e fesoʻotaʻi ai le tagata o tausia. Ina ia faia lenei mea, matou te faʻasalalau atu upega tafaʻilagi o pods, tautua ma tulei le DNS cluster i le kalani. O lea la, pe a taumafai se tagata o tausia e faʻafesoʻotaʻi i le auaunaga memcached.staging.svc.cluster.local, o le talosaga e alu i le DNS cluster ma i le tali e maua ai le tuatusi o lenei 'auʻaunaga mai le 'auʻaunaga faʻapipiʻi poʻo le tuatusi pod.

Matou te faʻapipiʻi K8s fuifui e faʻaaoga ai le kubeadm, lea o loʻo i ai le subnet auaunaga le lelei 192.168.0.0/16, ma le upega o pods o 10.244.0.0/16. E masani lava e lelei mea uma, ae o loʻo i ai ni nai vaega:

Subnet 192.168.*.* e masani ona fa'aoga i feso'ota'iga o ofisa o tagata o tausia, ma e sili atu fo'i i feso'ota'iga i fale atia'e. Ona tatou maua ai lea o feteʻenaʻiga: o loʻo galue le aufaipisinisi ile fale i luga ole laiga ma o le VPN e tuleia nei upega mai le fuifui i le tagata o tausia.
E tele a matou fuifui (gaosiga, tulaga ma/poʻo le tele o fuifui dev). Ma, ona o le le mafai, o latou uma o le ai ai le tutusa subnets mo pods ma auaunaga, lea e mafua ai faigata tele mo galuega faʻatasi ma auaunaga i le tele o fuifui.

Ua leva ona matou faʻaaogaina le faʻaaogaina o subnets eseese mo auaunaga ma pods i totonu o le poloketi tutusa - i se tulaga lautele, ina ia eseese uma fesoʻotaʻiga. Ae ui i lea, o loʻo i ai se numera tele o fuifui o loʻo faʻaogaina ou te le manaʻo e taʻavale mai le sasa, talu ai latou te faʻatautaia le tele o auaunaga, talosaga faʻapitoa, ma isi.

Ona matou fesili lea ia i matou lava: pe faʻafefea ona sui le subnet i se fuifui o loʻo iai?

Su'esu'ega o fa'ai'uga

O le faiga masani o le toe fa'afofoga uma 'au'aunaga ma le ituaiga ClusterIP. O se filifiliga, mafai ona fautua ma lenei:

O le faʻagasologa o loʻo i lalo o loʻo i ai se faʻafitauli: pe a uma ona faʻatulagaina mea uma, e sau le pods ma le IP tuai o se DNS nameserver i /etc/resolv.conf.
Talu ai ou te leʻi mauaina lava le fofo, e tatau ona ou toe setiina le fuifui atoa ma le kubeadm reset ma toe amata.

Ae e le fetaui lenei mea mo tagata uma ... O faʻamatalaga sili atu nei mo la matou mataupu:

E fa'aaogaina le ieila;
E i ai fuifui i ao ma luga o meafaigaluega;
Ou te manaʻo e aloese mai le toe faʻaaogaina o auaunaga uma i le fuifui;
O lo'o iai le mana'oga e fai masani mea uma i se numera la'ititi o fa'afitauli;
Kubernetes version o le 1.16.6 (peita'i, o isi laasaga o le a tutusa mo isi lomiga);
O le galuega autu o le faʻamautinoaina o loʻo faʻapipiʻiina i totonu o se fuifui e faʻaaoga ai le kubeadm ma se subnet auaunaga 192.168.0.0/16, sui i le 172.24.0.0/16.

Ma na tupu lava ua leva ona matou fiafia e vaʻai i le mea ma pe faʻapefea ona teuina i Kubernetes i etcd, o le a le mea e mafai ona fai i ai... O lea na matou mafaufau ai: "Aisea e le naʻo le faʻafouina o faʻamatalaga i etcd, suia tuatusi IP tuai (subnet) i mea fou? »

I le suʻesuʻeina o meafaigaluega ua saunia mo le galue ma faʻamatalaga i etcd, matou te leʻi mauaina se mea e foia atoatoa ai le faʻafitauli. (I le ala, afai e te iloa e uiga i soʻo se mea aoga mo le galue faʻatasi ma faʻamatalaga saʻo i etcd, matou te talisapaia soʻotaga.) Ae ui i lea, o se amataga lelei etcdhelper mai OpenShift (fa'afetai i ona tusitala!).

O lenei aoga e mafai ona faʻafesoʻotaʻi i etcd faʻaaoga tusi faamaonia ma faitau faʻamatalaga mai iina e faʻaaoga ai poloaiga ls, get, dump.

Faaopoopo etcdhelper

O le isi manatu e talafeagai: "O le a le mea e taofia ai oe mai le faʻaopoopoina o lenei aoga e ala i le faʻaopoopoina o le tomai e tusi ai faʻamatalaga i etcd?"

Na avea ma suiga fou o le etcdhelper ma ni galuega fou se lua changeServiceCIDR и changePodCIDR. i luga o ia e mafai ona e vaʻai i le code iinei.

O a mea e fai e foliga fou? Algoritm changeServiceCIDR:

faia se deserializer;
tu'ufa'atasia se fa'aaliga masani e sui ai le CIDR;
matou te uia auaunaga uma ma le ituaiga ClusterIP i le fuifui:
- fa'avasega le tau mai etcd i se mea Go;
- i le fa'aaogaina o se fa'aaliga masani tatou te suia ai le lua paita muamua o le tuatusi;
- tofia le 'au'aunaga se tuatusi IP mai le subnet fou;
- fatu se serializer, liliu le mea Go i protobuf, tusi faʻamatalaga fou i etcd.

galuega tauave changePodCIDR matua tutusa changeServiceCIDR - naʻo nai lo le faʻasaʻoina o faʻamatalaga auʻaunaga, matou te faia mo le node ma suiga .spec.PodCIDR i se subnet fou.

Faataitai

Suia le auaunaga CIDR

O le fuafuaga mo le faʻatinoina o le galuega e matua faigofie lava, ae e aofia ai le downtime i le taimi o le toe faʻafouina o pods uma i le fuifui. A maeʻa ona faʻamatalaina laasaga autu, o le a matou faʻasoa foʻi manatu i le auala, i le teori, e mafai ona faʻaitiitia lenei taimi faʻaletonu.

Laasaga sauniuni:

faʻapipiʻi le polokalama talafeagai ma faʻapipiʻi le patched etcdhelper;
faaleoleo etcd ma /etc/kubernetes.

Fuafuaga fa'atino puupuu mo le suia o le auaunagaCIDR:

suia le apiserver ma le pule-pule fa'aalia;
toe tuuina atu o tusi faamaonia;
suia auaunaga ClusterIP i etcd;
toe amata uma pusa i totonu o le fuifui.

O lo'o mulimuli mai se fa'asologa atoatoa o gaioiga fa'apitoa.

1. Fa'apipi'i etcd-client mo le lafoa'i o fa'amaumauga:

apt install etcd-client

2. Fausia etcdhelper:

Fa'apipi'i golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

Tatou te sefe mo i tatou lava etcdhelper.go, la'uina fa'alagolago, aoina:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Fai se faaleoleo etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Suia le subnet au'aunaga ile fa'aaliga o le va'alele fa'atonutonu Kubernetes. I faila /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml sui le parakalafa --service-cluster-ip-range i se subnet fou: 172.24.0.0/16 nai lo 192.168.0.0/16.

5. Talu ai o lo'o matou suia le 'au'aunaga subnet lea e tu'uina atu ai e le kubeadm tusi pasi mo apiserver (e aofia ai), e mana'omia ona toe tu'uina atu:

Se'i o tatou va'ai po'o fea tuatusi ma tuatusi IP na tu'uina atu i ai nei le tusi pasi mo:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Sei o tatou saunia se laititi config mo kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла

Sei o tatou tape le crt tuai ma le ki, talu ai a aunoa ma lenei mea o le a le tuʻuina atu le tusi faamaonia fou:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

Tatou toe tuuina atu tusi faamaonia mo le API server:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

Sei o tatou siaki pe na tuuina atu le tusi faamaonia mo le subnet fou:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

A maeʻa ona toe tuʻuina atu le API server certificate, toe amata lana pusa:

docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

Tatou toe fa'afouina le config mo admin.conf:
```
kubeadm alpha certs renew admin.conf
```
Sei o tatou faʻasaʻo faʻamatalaga i etcd:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
Faʻamolemole faʻamolemole! I le taimi nei, o le fa'ai'uga o le domain e taofi le galue i le fuifui, talu ai i totonu o pods o lo'o iai /etc/resolv.conf o le tuatusi tuai CoreDNS (kube-dns) ua resitalaina, ma suia e le kube-proxy tulafono iptables mai le subnet tuai i le fou. E le gata i lea i totonu o le tusiga o loʻo tusia ai e uiga i filifiliga e mafai ona faʻaitiitia ai le taimi malolo.
Se'i toe fa'aleleia ConfigMap i le igoa avanoa kube-system:
```
kubectl -n kube-system edit cm kubelet-config-1.16
```
- sui iinei clusterDNS i le tuatusi IP fou o le kube-dns auaunaga: kubectl -n kube-system get svc kube-dns.
```
kubectl -n kube-system edit cm kubeadm-config
```
- o le a tatou faaleleia data.ClusterConfiguration.networking.serviceSubnet i se subnet fou.
Talu ai ua suia le tuatusi kube-dns, e tatau ona faʻafouina le kubelet config i luga o nodes uma:
```
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```

Pau lava le mea o loʻo totoe o le toe amata uma o pods i le fuifui:

kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Fa'aiti'itia taimi malolo

Mafaufauga i le auala e faaitiitia ai le taimi malolo:

A maeʻa ona suia le faʻaaliga o le vaʻalele faʻatonutonu, faia se auaunaga kube-dns fou, mo se faʻataʻitaʻiga, ma le igoa kube-dns-tmp ma le tuatusi fou 172.24.0.10.
Le faia if i etcdhelper, lea o le a le suia ai le auaunaga kube-dns.
Suia le tuatusi i kubelets uma ClusterDNS i se mea fou, ae o le auaunaga tuai o le a faaauau pea ona galulue faatasi ma le auaunaga fou.
Fa'atali se'ia ta'avale solo na'o pusa o lo'o i ai talosaga mo mafua'aga masani po'o se taimi na malilie i ai.
Aveese le tautua kube-dns-tmp ma suiga serviceSubnetCIDR mo le auaunaga kube-dns.

Ole fuafuaga lea ole a fa'ataga ai oe e fa'aiti'itia le taimi fa'aletonu i le ~se minute - mo le umi o le 'au'aunaga kube-dns-tmp ma suia le subnet mo le auaunaga kube-dns.

Suiga podNetwork

I le taimi lava e tasi, na matou filifili e vaʻavaʻai pe faʻafefea ona sui le podNetwork e faʻaaoga ai le mea e maua ai etcdhelper. O le fa'asologa o gaioiga e fa'apea:

faaleleia configs i totonu kube-system;
fa'apipi'i le fa'aaliga kube-controller-manager;
sui tonu podCIDR i etcd;
toe fa'afou uma nodes fuifui.

Le taimi nei e uiga i gaioiga nei:

1. Suia ConfigMap i le igoa avanoa kube-system:

kubectl -n kube-system edit cm kubeadm-config

- faasa'oga data.ClusterConfiguration.networking.podSubnet i se subnet fou 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- faasa'oga data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Suia le fa'aaliga pule-pule:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- faasa'oga --cluster-cidr=10.55.0.0/16.

3. Va'ai i tulaga taua o lo'o iai nei .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses mo nodes fuifui uma:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Sui podCIDR i le faia sa'o o suiga ile etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Sei o tatou siaki ua suia moni le podCIDR:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Tatou toe fa'afou uma nodes fuifui ta'itasi.

7. Afai e te tuua ia le itiiti ifo ma le tasi le pona podCIDR tuai, ona le mafai lea ona amata le kube-controller-manager, ma o pods i le fuifui o le a le faʻatulagaina.

O le mea moni, o le suia o le podCIDR e mafai ona sili atu ona faigofie (mo se faʻataʻitaʻiga, ina). Ae matou te fia aʻoaʻo pe faʻafefea ona galulue faʻatasi ma etcd saʻo, aua e iai mataupu pe a faʻasaʻo mea Kubernetes i etcd - tasi eseese e mafai. (Mo se faʻataʻitaʻiga, e le mafai ona e suia le fanua Auaunaga e aunoa ma se taimi malolo spec.clusterIP.)

Le iʻuga

O loʻo faʻatalanoaina e le tusiga le avanoa e galulue faʻatasi ai ma faʻamatalaga i etcd tuusaʻo, i.e. sopoia le Kubernetes API. O nisi taimi o lenei faiga e mafai ai ona e faia "mea taufaasese." Na matou faʻataʻitaʻiina gaioiga o loʻo tuʻuina atu i le tusitusiga i luga ole K8s fuifui moni. Ae ui i lea, o latou tulaga o le saunia mo le faʻaaogaina lautele o PoC (faamaoniga o manatu). O le mea lea, afai e te manaʻo e faʻaoga se suiga faʻaogaina o le etcdhelper aoga i luga o au fuifui, fai i lou lava tulaga lamatia.

SALA

Faitau foi i la matou blog:

puna: www.habr.com

O matou poto masani i faʻamatalaga i le etcd Kubernetes cluster tuusaʻo (e aunoa ma K8s API)