Chiitiko chedu nedata mu etcd Kubernetes cluster zvakananga (isina K8s API)

Kuwedzera, vatengi vari kutikumbira kuti tipe mukana weKubernetes cluster kuti tikwanise kuwana masevhisi mukati meiyo cluster: kuitira kuti vakwanise kubatana zvakananga kune imwe dhatabhesi kana sevhisi, kubatanidza application yemuno nemaapplication mukati meboka...

Semuenzaniso, pane kudikanwa kwekubatanidza kubva muchina wako wepanzvimbo kuenda kune sevhisi memcached.staging.svc.cluster.local. Isu tinopa kugona uku tichishandisa VPN mukati meboka rinosangana nemutengi. Kuti tiite izvi, isu tinozivisa ma subnets epods, masevhisi uye kusunda cluster DNS kumutengi. Saka, kana mutengi achiedza kubatanidza kune sevhisi memcached.staging.svc.cluster.local, chikumbiro chinoenda kune cluster DNS uye mukupindura inogamuchira kero yebasa iri kubva kune cluster service network kana kero yepod.

Isu tinogadzirisa maK8s masumbu tichishandisa kubeadm, uko kune default sevhisi subnet 192.168.0.0/16, uye network yepods ndiyo 10.244.0.0/16. Kazhinji zvese zvinoshanda nemazvo, asi pane akati wandei mapoinzi:

• Subnet 192.168.*.* inowanzo shandiswa mumatengi ehofisi network, uye kunyanya kazhinji mumashandisi emusha ekuvandudza. Uye tobva tawana kupokana: ma routers epamba anoshanda pane iyi subnet uye VPN inosundidzira aya ma subnets kubva musumbu kuenda kumutengi.
• Tine masumbu akati wandei (kugadzira, nhanho uye/kana akati wandei dev masumbu). Zvino, nekusarudzika, ese achange aine ma subnets akafanana epods uye masevhisi, izvo zvinogadzira matambudziko makuru ebasa rimwechete nemasevhisi mumasumbu akati wandei.

Isu takatora kare tsika yekushandisa ma subnets akasiyana-siyana emasevhisi uye pods mukati meprojekiti imwechete - kazhinji, kuitira kuti masumbu ese ave nema network akasiyana. Nekudaro, kune huwandu hukuru hwemasumbu ari kushanda andisingade kuumburudza kubva pakutanga, sezvo achimhanyisa masevhisi mazhinji, maapplication ane hunyanzvi, nezvimwe.

Uye isu takazvibvunza isu: nzira yekushandura iyo subnet muboka riripo?

Kutsvaga zvisarudzo

Chiitiko chinowanzoitwa ndechekugadzira patsva all the masevhisi ane mhando ClusterIP. Sechisarudzo, anogona kupa zano uye ichi:

Maitiro anotevera ane dambudziko: mushure mekunge zvese zvagadziriswa, mapodhi anouya neiyo IP yekare seDNS nameserver mu /etc/resolv.conf.
Sezvo ini ndisati ndawana mhinduro, ndaifanira kuseta sumbu rese nekubeadm reset uye kuipinza zvakare.

Asi izvi hazvina kukodzera kumunhu wese... Heano mamwe akadzama masumo enyaya yedu:

• Flannel inoshandiswa;
• Kune masumbu ese ari mumakore uye pane Hardware;
• Ndinoda kudzivirira kutumira zvakare masevhisi ese muchikwata;
• Pane kudikanwa kwekuwanzoita zvese nenhamba shoma yezvinetso;
• Kubernetes version ndeye 1.16.6 (zvisinei, mamwe matanho achave akafanana kune dzimwe shanduro);
• Basa guru nderekuona kuti muchikwata chakaiswa uchishandisa kubeadm ine sevhisi subnet 192.168.0.0/16, itsive nayo 172.24.0.0/16.

Uye zvakangoitika kuti takanga tichifarira kuona kuti chii uye sei muKubernetes inochengetwa mu etcd, chii chingaitwa nazvo ... Saka takafunga: "Wadii kungovandudza iyo data mu etcd, kutsiva yekare IP kero (subnet) neitsva? »

Tatsvaga maturusi akagadzirira ekushanda nedata mu etcd, hatina kuwana chero chinhu chakanyatsogadzirisa dambudziko. (Nenzira, kana iwe uchiziva nezve chero zvishandiso zvekushanda nedata zvakananga mu etcd, isu tingatenda zvinongedzo.) Zvisinei, nzvimbo yakanaka yekutanga ndiyo etcdhelper kubva kuOpenShift (ndatenda kune vanyori varo!).

Ichi chishandiso chinogona kubatana kune etcd uchishandisa zvitupa uye kuverenga data kubva ipapo uchishandisa mirairo ls, get, dump.

Wedzera etcdhelper

Mufungo unotevera une musoro: "Chii chiri kukutadzisa kuwedzera chishandiso ichi nekuwedzera kugona kunyora data kune etcd?"

Yakava shanduro yakagadziridzwa ye etcdhelper ine mabasa maviri matsva changeServiceCIDR и changePodCIDR. paari unogona kuona kodhi pano.

Zvinhu zvitsva zvinoitei? Algorithm changeServiceCIDR:

• kugadzira deserializer;
• gadzira kutaura kwenguva dzose kutsiva CIDR;
• isu tinofamba nemasevhisi ese ane ClusterIP mhando musumbu:
  • decode kukosha kubva etcd kuita Go chinhu;
  • tichishandisa chirevo chenguva dzose tinotsiva mabheti maviri ekutanga ekero;
  • govera iyo sevhisi kero yeIP kubva kune itsva subnet;
  • gadzira serializer, shandura iyo Go chinhu kuita protobuf, nyora data nyowani kune etcd.

shanda changePodCIDR zvakangofanana changeServiceCIDR - chete pachinzvimbo chekugadzirisa iyo sevhisi yakatarwa, tinoiitira iyo node uye shanduko .spec.PodCIDR kune subnet itsva.

Dzidzira

Shandura sevhisi CIDR

Urongwa hwekuita basa iri nyore kwazvo, asi hunosanganisira kuderera panguva yekusikwa patsva kwemapodhi ese ari musumbu. Mushure mekutsanangura matanho makuru, isu tichagoverawo pfungwa pamusoro pekuti, muchirevo, iyi nguva yekudzikisa inogona kuderedzwa.

Matanho ekugadzirira:

• kuisa software inodiwa uye kuunganidza zvigamba etcdhelper;
• backup etcd uye /etc/kubernetes.

Muchidimbu chirongwa chekuchinja sevhisiCIDR:

• kushandura apiserver uye controller-maneja anoratidza;
• kuburitswazve kwezvitupa;
• kuchinja ClusterIP masevhisi mune etcd;
• kutangazve kwemapodhi ese ari musumbu.

Izvi zvinotevera kutevedzana kwakakwana kwezviito zvakadzama.

1. Isa etcd-client yekurasa data:

apt install etcd-client

2. Vaka etcdhelper:

• Isa golang:

  GOPATH=/root/golang
  mkdir -p $GOPATH/local
  curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
  echo "export GOPATH="$GOPATH"" >> ~/.bashrc
  echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
  echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

• Tinozvichengetera etcdhelper.go, download dependencies, unganidza:

  wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
  go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
  go build -o etcdhelper etcdhelper.go

3. Gadzira backup etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Shandura sevhisi subnet muKubernetes control ndege inoratidza. Mumafaira /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml shandura parameter --service-cluster-ip-range kune subnet itsva: 172.24.0.0/16 panzvimbo ye 192.168.0.0/16.

5. Sezvo isu tiri kushandura sevhisi subnet iyo kubeadm inoburitsa zvitupa zve apiserver (kusanganisira), ivo vanofanirwa kuburitswazve:

1. Ngationei kuti ndeapi madomasi uye IP kero chitupa chazvino chakapihwa:

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

2. Ngatigadzirire idiki config ye kubeadm:

   cat kubeadm-config.yaml
   apiVersion: kubeadm.k8s.io/v1beta1
   kind: ClusterConfiguration
   networking:
     podSubnet: "10.244.0.0/16"
     serviceSubnet: "172.24.0.0/16"
   apiServer:
     certSANs:
     - "192.168.199.100" # IP-адрес мастер узла

3. Ngatibvise crt yekare uye kiyi, sezvo pasina izvi chitupa chitsva hachizopihwa:

   rm /etc/kubernetes/pki/apiserver.{key,crt}

4. Ngatiburitsezve zvitupa zveiyo API server:

   kubeadm init phase certs apiserver --config=kubeadm-config.yaml

5. Ngatitarisei kuti chitupa chakapihwa subnet itsva:

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

6. Mushure mekuburitsazve API server chitupa, tangazve mudziyo wayo:

   docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

7. Ngatitangezve config ye admin.conf:

   kubeadm alpha certs renew admin.conf

8. Ngatigadzirise data mu etcd:

   ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 

   Cherechedza chinyorwa! Panguva ino, domain resolution inomira kushanda musumbu, kubva mumapodhi aripo /etc/resolv.conf iyo yekare CoreDNS kero (kube-dns) yakanyoreswa, uye kube-proxy inoshandura iptables mitemo kubva kune yekare subnet kuenda kune itsva. Kuwedzera mune iyo chinyorwa zvakanyorwa pamusoro pezvinogoneka sarudzo dzekudzikisa kuderera.

9. Ngatigadzirisei ConfigMap's munzvimbo yemazita kube-system:

   kubectl -n kube-system edit cm kubelet-config-1.16

   - tsiva pano clusterDNS kune iyo itsva IP kero yekube-dns sevhisi: kubectl -n kube-system get svc kube-dns.

   kubectl -n kube-system edit cm kubeadm-config

   - tichazvigadzirisa data.ClusterConfiguration.networking.serviceSubnet kune subnet itsva.

10. Sezvo iyo kube-dns kero yachinja, zvinodikanwa kugadzirisa kubelet config pane ese node:

    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet

11. Chasara kutangazve ese mapodhi musumbu:

    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Deredza nguva yekudzikira

Pfungwa dzekuti ungaderedza sei nguva yekudzikira:

1. Mushure mekushandura ndege yekudzora inoratidzira, gadzira itsva kube-dns sevhisi, semuenzaniso, ine zita kube-dns-tmp uye kero itsva 172.24.0.10.
2. Kuita if mu etcdhelper, iyo isinga gadzirise iyo kube-dns sevhisi.
3. Tsiva kero mune zvese kubelets ClusterDNS kune itsva, nepo sevhisi yekare icharamba ichishanda panguva imwe chete neitsva.
4. Mirira kusvikira mapodhi ane zvikumbiro atenderedzwa ega nekuda kwezvikonzero zvechisikigo kana panguva yakabvumiranwa.
5. Delete sevhisi kube-dns-tmp uye shanduko serviceSubnetCIDR yebasa rekube-dns.

Urongwa uhwu huchakubvumidza kuti uderedze nguva yekudzikira kusvika ~ miniti - yenguva yekubviswa kwesevhisi kube-dns-tmp uye kushandura subnet yebasa racho kube-dns.

Kushandura podNetwork

Panguva imwecheteyo, takasarudza kutarisa maitiro ekugadzirisa podNetwork tichishandisa mhedzisiro etcdhelper. Kutevedzana kwezviito kunotevera:

• kugadzirisa configs mukati kube-system;
• kugadzirisa kube-controller-maneja manifest;
• shandura podCIDR zvakananga mu etcd;
• reboot ese ma cluster node.

Iye zvino zvimwe nezvezviito izvi:

1. Shandura ConfigMap's munzvimbo yemazita kube-system:

kubectl -n kube-system edit cm kubeadm-config

- kugadzirisa data.ClusterConfiguration.networking.podSubnet kune subnet itsva 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- kugadzirisa data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Shandura mutungamiriri-maneja manifest:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- kugadzirisa --cluster-cidr=10.55.0.0/16.

3. Tarisa maitiro azvino .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses kune ese macluster node:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Tsiva podCIDR nekuchinja zvakananga ku etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Ngatitarisei kuti podCIDR yanyatsochinja here:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Ngatitangeizve masumbu ese masumbu rimwe nerimwe.

7. Kana iwe ukasiya imwe node yekare podCIDR, ipapo kube-controller-maneja haazokwanisi kutanga, uye mapodhi ari musumbu haazorongerwi.

Muchokwadi, kuchinja podCIDR kunogona kuitwa kunyange nyore (semuenzaniso, saka) Asi isu taida kudzidza kushanda ne etcd zvakananga, nekuti pane zviitiko pakugadzirisa Kubernetes zvinhu mu etcd - ivo chete zvinogoneka musiyano. (Semuenzaniso, haugone kungochinja iyo Sevhisi ndima pasina nguva yekudzikira spec.clusterIP.)

Mugumisiro

Chinyorwa chinokurukura mukana wekushanda nedata mu etcd zvakananga, i.e. nekupfuura iyo Kubernetes API. Dzimwe nguva nzira iyi inokubvumira kuita "zvinhu zvinonyengera". Takaedza mashandiro akapihwa muzvinyorwa pane chaiwo maK8s masumbu. Zvisinei, chimiro chavo chekugadzirira kushandiswa kwakapararira chiri PoC (uchapupu hwepfungwa). Naizvozvo, kana iwe uchida kushandisa yakagadziridzwa vhezheni ye etcdhelper utility pamasumbu ako, ita izvi nenjodzi yako.

PS

Verenga zvakare pablog yedu:

• «etcd 3.4.3: kuvimbika kwekuchengetedza uye chidzidzo chekuchengetedza";
• «Calico ye networking muKubernetes: sumo uye chiitiko chidiki";
• «6 inovaraidza system bugs mukushanda kweKubernetes [uye mhinduro yavo]";
• «A Visual Guide kune Troubleshooting Kubernetes".

Source: www.habr.com