🥇Zomwe takumana nazo pogwira ntchito ndi data mu etcd ya gulu la Kubernetes mwachindunji (popanda K8s API)

Kuchulukirachulukira, makasitomala akutipempha kuti tipereke mwayi ku gulu la Kubernetes kuti titha kupeza ntchito mkati mwa gululi: kuti titha kulumikizana mwachindunji ndi database kapena ntchito ina, kulumikiza pulogalamu yakomweko ndi mapulogalamu omwe ali mkati mwa gulu ...

Mwachitsanzo, pakufunika kulumikiza makina anu am'deralo kupita ku ntchito memcached.staging.svc.cluster.local. Timapereka izi pogwiritsa ntchito VPN mkati mwa gulu lomwe kasitomala amalumikizana. Kuti tichite izi, timalengeza ma subnets a ma pod, ntchito ndikukankhira gulu la DNS kwa kasitomala. Choncho, pamene kasitomala ayesa kulumikiza utumiki memcached.staging.svc.cluster.local, pempho likupita ku cluster DNS ndipo poyankha amalandira adilesi ya msonkhanowu kuchokera ku network network network kapena pod adilesi.

Timakonza magulu a K8s pogwiritsa ntchito kubeadm, pomwe subnet yokhazikika ili 192.168.0.0/16, ndi netiweki ya ma pod ndi 10.244.0.0/16. Nthawi zambiri zonse zimayenda bwino, koma pali mfundo zingapo:

Subnet 192.168.*.* Nthawi zambiri amagwiritsidwa ntchito m'maofesi amakasitomala, komanso nthawi zambiri pamanetiweki apanyumba. Ndiyeno timapeza mikangano: ma routers akunyumba amagwira ntchito pa subnet iyi ndipo VPN imakankhira ma subnets awa kuchokera pagulu kupita kwa kasitomala.
Tili ndi magulu angapo (kupanga, siteji ndi/kapena magulu angapo a dev). Ndiye, mwachisawawa, onse adzakhala ndi ma subnets ofanana a ma pod ndi mautumiki, zomwe zimapangitsa kuti zikhale zovuta kwambiri pakugwira ntchito nthawi imodzi ndi mautumiki m'magulu angapo.

Tidatengera kalekale mchitidwe wogwiritsa ntchito ma subnets osiyanasiyana pazithandizo ndi ma pod mkati mwa projekiti imodzi - pafupipafupi, kuti magulu onse akhale ndi maukonde osiyanasiyana. Komabe, pali magulu ambiri omwe akugwira ntchito omwe sindikufuna kugubuduza kuyambira pachiyambi, chifukwa amayendetsa ntchito zambiri, mapulogalamu apamwamba, ndi zina zotero.

Ndiyeno tinadzifunsa tokha: momwe tingasinthire subnet mumagulu omwe alipo?

Kufufuza zisudzo

Mchitidwe wofala kwambiri ndi kulenganso onse ntchito ndi mtundu ClusterIP. Monga njira, akhoza kulangiza ndi izi:

Njira yotsatirayi ili ndi vuto: zonse zitakonzedwa, ma pod amabwera ndi IP yakale monga DNS nameserver mu /etc/resolv.conf.
Popeza sindinapeze yankho, ndidayenera kukonzanso gulu lonselo ndikukhazikitsanso kubeadm ndikuyambitsanso.

Koma izi sizoyenera aliyense... Nawa mau oyamba atsatanetsatane ankhani yathu:

Flannel imagwiritsidwa ntchito;
Pali masango onse m'mitambo ndi pa hardware;
Ndikufuna kupewa kuyikanso ntchito zonse mugulu;
Pakufunika kuchita zonse ndi zovuta zochepa;
Mtundu wa Kubernetes ndi 1.16.6 (komabe, masitepe ena adzakhala ofanana ndi matembenuzidwe ena);
Ntchito yayikulu ndikuwonetsetsa kuti mgulu lomwe likugwiritsidwa ntchito pogwiritsa ntchito kubeadm yokhala ndi subnet yantchito 192.168.0.0/16, m'malo mwake 172.24.0.0/16.

Ndipo zidangochitika kuti takhala ndi chidwi chofuna kuwona zomwe komanso momwe Kubernetes zimasungidwira mu etcd, zomwe tingachite nazo ... Kotero tinaganiza kuti: "Bwanji osangosintha zambiri mu etcd, m'malo mwa ma adilesi akale a IP (subnet) ndi atsopano? "

Titafufuza zida zopangidwa kale zogwirira ntchito ndi data mu etcd, sitinapeze chilichonse chomwe chinathetsa vutoli. (Mwa njira, ngati mukudziwa za zida zilizonse zogwiritsira ntchito deta mwachindunji etcd, tingayamikire maulalo.) Komabe, chiyambi chabwino ndi etcdhelper kuchokera ku OpenShift (zikomo kwa olemba ake!).

Izi zitha kulumikizana ndi etcd pogwiritsa ntchito satifiketi ndikuwerenga zomwe zili pamenepo pogwiritsa ntchito malamulo ls, get, dump.

Onjezani etcdhelper

Lingaliro lotsatira ndilomveka: "Nchiyani chikukulepheretsani kuwonjezera izi powonjezera luso lolemba deta ku etcd?"

Inakhala mtundu wosinthidwa wa etcdhelper wokhala ndi ntchito ziwiri zatsopano changeServiceCIDR и changePodCIDR. pa iye mukhoza kuwona kodi apa.

Kodi zatsopanozi zikuchita chiyani? Algorithm changeServiceCIDR:

kupanga deserializer;
phatikiza mawu okhazikika kuti alowe m'malo mwa CIDR;
timadutsa mautumiki onse ndi mtundu wa ClusterIP mgululi:
- zindikirani mtengo kuchokera etcd kukhala chinthu cha Go;
- pogwiritsa ntchito mawu okhazikika timasintha ma byte awiri oyamba a adilesi;
- perekani ma adilesi a IP kuchokera pa subnet yatsopano;
- pangani serializer, sinthani Go chinthu kukhala protobuf, lembani zatsopano ku etcd.

ntchito changePodCIDR zofanana kwenikweni changeServiceCIDR - kokha m'malo mokonza ndondomeko ya utumiki, timachitira node ndikusintha .spec.PodCIDR ku subnet yatsopano.

Yesetsani

Sinthani utumiki CIDR

Dongosolo lokhazikitsa ntchitoyi ndi losavuta, koma limakhudza nthawi yopumira pomwe ma pod onse omwe ali mgululi amapangidwanso. Pambuyo pofotokoza masitepe akuluakulu, tidzagawananso malingaliro a momwe, mwachidziwitso, nthawi yopumayi ingachepetsedwe.

Zokonzekera:

kukhazikitsa mapulogalamu ofunikira ndikusonkhanitsa zigamba etcdhelper;
backup etcd ndi /etc/kubernetes.

Ndondomeko yachidule yosinthira ntchitoCIDR:

kusintha mawonekedwe apiserver ndi woyang'anira-woyang'anira;
kutulutsanso ziphaso;
kusintha mautumiki a ClusterIP mu etcd;
kuyambitsanso mapoto onse mu cluster.

Zotsatirazi ndi ndondomeko yathunthu ya zochita mwatsatanetsatane.

1. Ikani etcd-client kuti muchotse deta:

apt install etcd-client

2. Mangani etcdhelper:

Ikani golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

Timadzisungira tokha etcdhelper.go, tsitsani zodalira, sonkhanitsani:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Pangani zosunga zobwezeretsera etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Sinthani subnet ya utumiki mu Kubernetes control ndege ikuwonetsera. Mu mafayilo /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml kusintha parameter --service-cluster-ip-range ku subnet yatsopano: 172.24.0.0/16 mmalo mwa 192.168.0.0/16.

5. Popeza tikusintha kagawo kakang'ono ka ntchito komwe kubeadm amapereka ziphaso za apiserver (kuphatikiza), ziyenera kutulutsidwanso:

Tiyeni tiwone madera ndi ma adilesi a IP omwe satifiketi yapano idaperekedwa:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Tiyeni tikonzekere kakhazikitsidwe kakang'ono ka kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла

Tiyeni tichotse crt yakale ndi kiyi, popeza popanda izi satifiketi yatsopano sidzaperekedwa:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

Tiyeni titulutsenso ziphaso za seva ya API:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

Tiyeni tiwone ngati satifiketi idaperekedwa ya subnet yatsopano:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Mutaperekanso satifiketi ya seva ya API, yambitsaninso chidebe chake:

docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

Tiyeni tikonzenso config for admin.conf:
```
kubeadm alpha certs renew admin.conf
```
Tiyeni tisinthe data mu etcd:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
Chonde chonde! Pakadali pano, kusamvana kwa madambwe kumasiya kugwira ntchito mgulu, chifukwa m'matumba omwe alipo kale /etc/resolv.conf adilesi yakale ya CoreDNS (kube-dns) idalembetsedwa, ndipo kube-proxy amasintha malamulo a iptables kuchokera ku subnet yakale kupita ku yatsopano. Komanso m'nkhaniyo zalembedwa za njira zomwe zingatheke kuti muchepetse nthawi yopuma.
Tiyeni tikonze ma ConfigMap mu malo a mayina kube-system:
```
kubectl -n kube-system edit cm kubelet-config-1.16
```
- sinthani apa clusterDNS ku adilesi yatsopano ya IP ya ntchito ya kube-dns: kubectl -n kube-system get svc kube-dns.
```
kubectl -n kube-system edit cm kubeadm-config
```
- tidzakonza data.ClusterConfiguration.networking.serviceSubnet ku subnet yatsopano.
Popeza adilesi ya kube-dns yasintha, pakufunika kusinthira kakhazikitsidwe ka kubelet pama node onse:
```
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```

Zomwe zatsala ndikuyambitsanso ma pod onse mgululi:

kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Chepetsani nthawi yopuma

Malingaliro amomwe mungachepetse nthawi yopuma:

Pambuyo posintha mawonekedwe a ndege, pangani ntchito yatsopano ya kube-dns, mwachitsanzo, ndi dzina kube-dns-tmp ndi adilesi yatsopano 172.24.0.10.
Kupanga if mu etcdhelper, zomwe sizingasinthe ntchito ya kube-dns.
Bwezerani adilesi mu ma kubelets onse ClusterDNS kwa watsopano, pamene utumiki wakale udzapitirizabe kugwira ntchito limodzi ndi watsopano.
Dikirani mpaka ma pod omwe ali ndi mapulogalamu agubudulidwe okha pazifukwa zachilengedwe kapena panthawi yomwe mwagwirizana.
Chotsani ntchito kube-dns-tmp ndi kusintha serviceSubnetCIDR za utumiki wa kube-dns.

Dongosololi likuthandizani kuti muchepetse nthawi yotsika kukhala ~ miniti - munthawi yonse yochotsa ntchito kube-dns-tmp ndi kusintha subnet ya utumiki kube-dns.

Kusintha kwa podNetwork

Nthawi yomweyo, tinaganiza zoyang'ana momwe tingasinthire podNetwork pogwiritsa ntchito etcdhelper. Kutsatira kwa zochita ndi motere:

kukonza configs mu kube-system;
kukonza chiwonetsero cha kube-controller-manager;
kusintha podCIDR mwachindunji etcd;
yambitsaninso ma cluster node onse.

Tsopano zambiri za izi:

1. Sinthani ConfigMaps mu malo a mayina kube-system:

kubectl -n kube-system edit cm kubeadm-config

- kukonza data.ClusterConfiguration.networking.podSubnet ku subnet yatsopano 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- kukonza data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Sinthani chiwonetsero cha controller-manager:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- kukonza --cluster-cidr=10.55.0.0/16.

3. Yang'anani pa zikhalidwe zamakono .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses kwa ma cluster node onse:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Sinthani podCIDR posintha mwachindunji etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Tiyeni tiwone ngati podCIDR yasinthadi:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Tiyeni tiyambitsenso mfundo zonse za cluster imodzi ndi imodzi.

7. Ngati mutasiya mfundo imodzi PodCIDR yakale, ndiye kube-controller-manager sangathe kuyambitsa, ndipo ma pods mumagulu sangakonzedwe.

M'malo mwake, kusintha podCIDR kumatha kuchitika mosavuta (mwachitsanzo, kotero). Koma tinkafuna kuphunzira momwe tingagwirire ntchito ndi etcd mwachindunji, chifukwa pali nthawi zina mukamakonza zinthu za Kubernetes mu etcd - chokhacho zotheka zosiyanasiyana. (Mwachitsanzo, simungangosintha gawo la Utumiki popanda nthawi yopuma spec.clusterIP.)

Zotsatira

Nkhaniyi ikufotokoza kuthekera kogwira ntchito ndi deta mu etcd mwachindunji, i.e. kudutsa Kubernetes API. Nthawi zina njira iyi imakulolani kuchita "zinthu zachinyengo". Tinayesa ntchito zomwe zaperekedwa m'mawu pamagulu enieni a K8s. Komabe, mawonekedwe awo okonzeka kugwiritsidwa ntchito kwambiri PoC (umboni wa lingaliro). Chifukwa chake, ngati mukufuna kugwiritsa ntchito mtundu wosinthidwa wa etcdhelper pamagulu anu, chitani izi mwakufuna kwanu.