🥇Khibradayada ku saabsan la shaqaynta xogta ee kutlada Kubernetes si toos ah (la'aanteed K8s API)

Waxaa isa soo taraya, macaamiishu waxa ay na waydiisanayaan in aanu siino gelitaanka kooxda Kubernetes si ay awood ugu yeeshaan helitaanka adeegyada kutlada dhexdeeda: si ay u awoodaan in ay si toos ah ugu xidhmaan xogta ama adeega qaar ka mid ah, si loogu xidho arjiga maxalli ah codsiyada ku jira kutlada...

Tusaale ahaan, waxaa jirta baahi loo qabo in laga xidho mishiinka deegaankaaga iyo adeega memcached.staging.svc.cluster.local. Waxaan bixinaa awoodan anagoo adeegsanayna VPN-ka gudaha kooxda uu macmiilku ku xidho. Si tan loo sameeyo, waxaanu ku dhawaaqnaa subnets of pods, adeegyada iyo riix kooxda DNS macmiilka. Markaa, marka macmiilku isku dayo inuu ku xidho adeegga memcached.staging.svc.cluster.local, codsigu waxa uu aadayaa kutlada DNS jawaab ahaanna waxa uu ciwaanka adeegan ka helayaa shabakada adeega kooxda ama ciwaanka pods

Waxa aanu ku habaynay kooxaha K8s anagoo adeegsanayna kubeadm, halka subnet-ka adeega caadiga ahi yaalo 192.168.0.0/16, iyo shabkada pods waa 10.244.0.0/16. Caadi ahaan wax walba si fiican ayey u shaqeeyaan, laakiin waxaa jira dhowr qodob:

Subnet 192.168.*.* inta badan loo isticmaalo shabakadaha xafiiska macmiilka, iyo xataa marar badan shabakadaha guryaha horumariya. Kadibna waxaan helnaa isku dhacyo: router gurigu waxay ku shaqeeyaan subnet-kan VPN-kuna wuxuu ka riixaa shabakadaha hoose ee kooxda macaamiisha.
Waxaan haynaa dhowr rucubood (wax soo saar, marxalad iyo/ama dhowr kooxood). Dabadeed, sida caadiga ah, dhammaantood waxay yeelan doonaan shabakad-hoosaadyo isku mid ah oo loogu talagalay pods iyo adeegyada, taas oo ku abuurta dhibaato weyn shaqada isku mar ah ee adeegyada dhowr kooxood.

Waxaan mar hore qaadanay dhaqanka ah isticmaalka shabakadaha hoose ee kala duwan ee adeegyada iyo pods hal mashruuc gudahood - guud ahaan, si ay dhammaan kooxuhu waxay leeyihiin shabakado kala duwan. Si kastaba ha ahaatee, waxaa jira kooxo tiro badan oo shaqaynaya oo aanan jeclayn inaan ka rogo xoqan, maadaama ay qabtaan adeegyo badan, codsiyo dawladeed, iwm.

Ka dib waxaan is weydiinnay: sida loo beddelo subnet-ka ee kutlada jira?

Raadinta go'aamada

Dhaqanka ugu caansan waa in dib loo abuuro oo dhan adeegyo leh nooca ClusterIP. Ikhtiyaar ahaan, talo bixin kara iyo kan:

Habka soo socda ayaa dhibaato leh: ka dib wax kasta oo la habeeyey, boodhadhku waxay la imaanayaan IP-gii hore ee magaca DNS ee /etc/resolv.conf.
Maadaama aanan wali helin xalka, waa inaan dib u dajiyaa kutlada oo dhan anigoo wata kubeadm reset oo markale dib u dajiyo.

Laakiin tani kuma habboona qof walba... Halkan waxaa ah hordhacyo faahfaahsan oo ku saabsan kiiskeena:

Flannel waxaa loo isticmaalaa;
Waxaa jira rucubyo labadaba ku jira daruuraha iyo qalabka;
Waxaan jeclaan lahaa in aan iska ilaaliyo in aan dib u hawlgeliyo dhammaan adeegyada kooxda;
Waxaa jirta baahi loo qabo in wax walba lagu sameeyo tirada ugu yar ee dhibaatooyinka;
Nooca Kubernetes waa 1.16.6 (si kastaba ha ahaatee, tillaabooyin dheeraad ah ayaa la mid noqon doona noocyada kale);
Hawsha ugu muhiimsan waa in la hubiyo in kooxdu ay ku jirto kubeadm oo leh shabakad hoosaad adeeg 192.168.0.0/16, ku beddel 172.24.0.0/16.

Oo waxa dhacday in aanu muddo dheer xiisaynnay inaanu aragno waxa iyo sida Kubernetes loogu kaydiyo iwm, maxaa lagu samayn karaa... Markaa waxaanu ku fikirnay: "Waa maxay sababta aan u cusboonaysiin oo keliya xogta ku jirta iwm, oo lagu beddelayo cinwaannadii hore ee IP-ga (subnet) kuwo cusub? »

Ka dib markii aan raadinay qalab diyaarsan oo loogu shaqeynayo xogta iwm, ma aanan helin wax si buuxda u xalliyey dhibaatada. (Jid ahaan, haddii aad ogtahay wax ku saabsan wixii utility ah ee si toos ah xogta loogula shaqaynayo iwm, waan u mahadcelinaynaa isku xirka.) Si kastaba ha ahaatee, bar bilow fiican waa iwm caawiye ka OpenShift (Mahad waxaa leh qorayaashiisa!).

Utility Tani waxay ku xidhi kartaa etcd isticmaalaya shahaadooyinka oo akhri xogta halkaas oo isticmaalaya amarada ls, get, dump.

Ku dar iwm caawiye

Fikirka soo socdaa waa macquul: "Maxaa kaa joojinaya inaad ku darto utility-gan adigoo ku daraya awoodda aad u qori karto xogta iwm?"

Waxa ay noqotay nooc wax laga beddelay oo etcdhhelper ah oo leh laba hawlood oo cusub changeServiceCIDR и changePodCIDR. iyada waxaad arki kartaa koodhka halkan.

Maxay qabtaan sifooyinka cusub? Algorithm changeServiceCIDR:

samee deserializer;
ururin odhaah joogto ah si aad u bedesho CIDR;
Waxaan ku mareynaa dhammaan adeegyada leh nooca ClusterIP ee kutlada:
- go'aamin qiimaha laga bilaabo etcd una beddelo shayga Go;
- annaga oo adeegsanayna tibaax joogto ah waxaan ku beddeleynaa labada bytes ee ugu horreeya cinwaanka;
- u dhiibo adeegga ciwaanka IP-ga ee subnet-ka cusub;
- Waxaan abuurnaa serializer, u beddelo shayga Go protobuf, u qor xog cusub iwm.

function changePodCIDR asal ahaan la mid ah changeServiceCIDR - kaliya halkii aad ka tafatirin lahayd qeexida adeegga, waxaan u sameynaa noodhka iyo isbeddelka .spec.PodCIDR subnet cusub.

Tababarka

Beddel adeegga CIA

Qorshaha fulinta hawshu waa mid aad u fudud, laakiin waxa ay ku lug leedahay waqti dhimis iyadoo dhammaan boodhadhka kooxda dib loo abuurayo. Ka dib markaan sharaxno tillaabooyinka ugu muhiimsan, waxaan sidoo kale wadaagi doonaa fikradaha ku saabsan sida, aragti ahaan, wakhtigan hoos u dhaca loo yareyn karo.

Talaabooyinka diyaarinta:

rakibidda software-ka lagama maarmaanka ah iyo isu-ururinta iwm.
gurmad iwm iyo /etc/kubernetes.

Qorshe hawleed kooban oo lagu beddelayo serviceCIDR:

beddelka apiserver iyo kantaroolaha-maareeyaha muuqaalada;
dib u soo saarida shahaadooyinka;
beddelka adeegyada ClusterIP ee iwm;
dib u bilaabo dhamaan gadafka kutlada

Kuwa soo socdaa waa taxane dhammaystiran oo ficil ah oo faahfaahsan.

1. Ku rakib etcd-client si aad u daadiso xogta:

apt install etcd-client

2. Dhiso caawiye:

Ku rakib golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

Nafteena ayaan wax u badbaadinaa etcdhelper.go, soo deji ku tiirsanaanta, ururi:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Samee kayd iwm.

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Beddel subnet-ka adeegga ee diyaaradda xakamaynta Kubernetes. Faylasha ku jira /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml beddel cabbirka --service-cluster-ip-range ku soo biir subnet cusub: 172.24.0.0/16 halkii 192.168.0.0/16.

5. Maadaama aanu bedelayno subnet-ka adeega kaas oo kubeadm uu bixiyo shahaadooyinka apiserver (ay ku jirto), waxay u baahan yihiin in dib loo soo saaro:

Aynu aragno xayndaabyada iyo ciwaanka IP-ga ee shahaadada hadda la siiyay:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Aan u diyaarino qaabaynta ugu yar ee kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла

Aynu tirno crt-kii hore iyo furihii hore, maadaama tan la'aanteed aan shahaadada cusub la bixin doonin:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

Aan dib u soo saarno shahaadooyinka server-ka API:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

Aynu hubino in shahaadada la siiyay subnet-ka cusub:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Ka dib markii dib loo soo saaro shahaadada server-ka API, dib u bilow weelkeeda:
```
docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart
```
Aynu dib u soo kicinno isku xidhka loogu talagalay admin.conf:
```
kubeadm alpha certs renew admin.conf
```
Aynu ku tafatirno xogta gudaha iwm:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
Digniin Waqtigan xaadirka ah, xallinta bogga waxa uu joogsadaa ka shaqaynta kutlada, tan iyo gundhigyada jira /etc/resolv.conf Ciwaanka hore ee CoreDNS (kube-dns) wuu diiwaangashan yahay, kube-proxy-kuna wuxuu beddelaa xeerarka iptables-ka hoose ee hore una beddelo kan cusub. Faahfaahin dheeraad ah oo ku saabsan maqaalka waxaa lagu qoray wax ku saabsan fursadaha suurtagalka ah ee lagu yareynayo wakhtiga dhimista.
Aan ku hagaajino ConfigMap's meesha magaca kube-system:
```
kubectl -n kube-system edit cm kubelet-config-1.16
```
- halkan ku beddel clusterDNS Ciwaanka IP-ga cusub ee adeega kube-dns: kubectl -n kube-system get svc kube-dns.
```
kubectl -n kube-system edit cm kubeadm-config
```
- waan hagaajin doonaa data.ClusterConfiguration.networking.serviceSubnet subnet cusub.
Maaddaama ciwaanka kube-dns uu isbedelay, waxaa lagama maarmaan ah in la cusboonaysiiyo isku xidhka kubelet dhammaan noodyada:
```
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```

Waxa kaliya ee hadhay waa in dib loo bilaabo dhammaan galbihii kooxda:

kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Yaree wakhtiga dhimista

Fikradaha ku saabsan sida loo yareeyo wakhtiga dhimista:

Ka dib markii la beddelo diyaaradda xakamaynta muujinta, samee adeeg kube-dns cusub, tusaale ahaan, oo leh magaca kube-dns-tmp iyo ciwaan cusub 172.24.0.10.
Samee if iwmdhhelper, taas oo aan wax ka beddeli doonin adeegga kube-dns.
Ku beddel ciwaanka dhammaan kubelets ClusterDNS mid cusub, halka adeeggii hore uu sii wadi doono inuu isku mar la shaqeeyo kan cusub.
Sug ilaa gaboobyada codsiyada wata ay ka soo wareegayaan midkood sababo dabiici ah dartood ama waqti lagu heshiiyey.
Tirtir adeegga kube-dns-tmp iyo isbedel serviceSubnetCIDR adeega kube-dns.

Qorshahani wuxuu kuu ogolaanayaa inaad hoos u dhigto ~ hal daqiiqo - inta lagu jiro ka saarista adeegga kube-dns-tmp iyo beddelka subnet-ka adeegga kube-dns.

Shabakadda wax ka beddelka

Isla mar ahaantaana, waxaan go'aansanay inaan eegno sida loo beddelo podNetwork iyada oo la adeegsanayo etcdhhelper natiijada. Taxanaha falku waa sida soo socota:

hagaajinta habaynta gudaha kube-system;
hagaajinta kube-controller-maareeyaha qoraalka;
u beddel podCIDR si toos ah iwm;
dib u bilow dhammaan qanjidhada kooxda

Hadda wax badan oo ku saabsan falalkan:

1. Wax ka beddel ConfigMaps meesha magaca kube-system:

kubectl -n kube-system edit cm kubeadm-config

- sixid data.ClusterConfiguration.networking.podSubnet subnet cusub 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- sixid data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Wax ka beddel qoraalka maamulaha-maamulaha:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- sixid --cluster-cidr=10.55.0.0/16.

3. Fiiri qiyamka hadda jira .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses dhammaan qanjidhada kooxda:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Beddel podCIDR adiga oo si toos ah u beddelaya etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Aynu eegno in podCIDR ay runtii is bedeshay:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Aynu dib u bilowno dhammaan qanjidhada kooxda mid mid.

7. Haddii aad ka tagto ugu yaraan hal nood podCIDR hore, ka dibna kube-controller-maareeyaha ma awoodi doono inuu bilaabo, iyo boodhadhka kutlada lama ballansan doono.

Dhab ahaantii, beddelka podCIDR waa la samayn karaa xitaa ka fudud (tusaale ahaan, sidaas darteed). Laakiin waxaan rabnay inaan barano sida loola shaqeeyo iwm si toos ah, sababtoo ah waxaa jira kiisas marka la hagaajinayo walxaha Kubernetes ee iwm. kaliya kala duwanaansho suurtagal ah. (Tusaale ahaan, kaliya ma bedeli kartid goobta adeega wakhti la'aan spec.clusterIP.)

Natiijada

Maqaalku wuxuu ka hadlayaa suurtagalnimada in si toos ah loogu shaqeeyo xogta iwm, i.e. ka gudubka Kubernetes API. Mararka qaarkood habkani wuxuu kuu ogolaanayaa inaad sameyso "waxyaabo khiyaano leh." Waxaan ku tijaabinay hawlgallada qoraalka ku qoran ee kutlada dhabta ah ee K8s. Si kastaba ha ahaatee, xaaladdooda diyaargarowga isticmaalka baahsan waa PoC (caddayn fikradda). Sidaa darteed, haddii aad rabto in aad isticmaasho nooca wax laga beddelay ee utility etcdhelper ee kutlooyinkaaga, sidaas u yeelo adiga oo halis u ah.

PS

Sidoo kale ka akhri boggayaga:

Source: www.habr.com

Khibradayada xogta ku jirta kooxda Kubernetes si toos ah (la'aanteed K8s API)