Amava ethu ngedatha etcd Kubernetes cluster ngokuthe ngqo (ngaphandle kweK8s API)

Ukwandisa, abathengi basicela ukuba sinikeze ukufikelela kwi-cluster ye-Kubernetes ukuze sikwazi ukufikelela kwiinkonzo ngaphakathi kweqela: ukwenzela ukuba bakwazi ukudibanisa ngokuthe ngqo kwi-database ethile okanye inkonzo, ukudibanisa isicelo sendawo kunye nezicelo ngaphakathi kweqela...

Umzekelo, kukho imfuneko yokudibanisa kumatshini wakho wendawo ukuya kwinkonzo memcached.staging.svc.cluster.local. Sinikezela ngobu buchule sisebenzisa i-VPN ngaphakathi kweqela apho umxhasi axhuma khona. Ukwenza oku, sibhengeza i-subnets yeepod, iinkonzo kunye ne-push cluster DNS kumxhasi. Ngaloo ndlela, xa umxhasi ezama ukuqhagamshela kwinkonzo memcached.staging.svc.cluster.local, isicelo siya kwi-cluster DNS kwaye kwimpendulo ifumana idilesi yale nkonzo kwinethiwekhi yenkonzo ye-cluster okanye idilesi ye-pod.

Siqwalasela amaqela e-K8s sisebenzisa i-kubeadm, apho i-subnet yenkonzo engagqibekanga ikhoyo 192.168.0.0/16, kunye nenethiwekhi yeepods 10.244.0.0/16. Ngokuqhelekileyo yonke into isebenza kakuhle, kodwa kukho amanqaku ambalwa:

• Umnatha ongaphantsi 192.168.*.* isetyenziswa rhoqo kuthungelwano lweofisi yomxhasi, kwaye nangakumbi kuthungelwano lwasekhaya lomphuhlisi. Kwaye ke sifumana iingxabano: iirutha zasekhaya zisebenza kule subnet kwaye iVPN ityhala ezi subnets ukusuka kwiqela ukuya kumxhasi.
• Sinamaqela amaninzi (imveliso, iqonga kunye/okanye amaqela amaninzi e-dev). Emva koko, ngokuzenzekelayo, zonke ziya kuba ne-subnets efanayo yeepod kunye neenkonzo, nto leyo eyenza ubunzima obukhulu bokusebenza ngaxeshanye kunye neenkonzo kumaqela amaninzi.

Kudala samkela umkhuba wokusebenzisa ii-subnets ezahlukeneyo kwiinkonzo kunye neepods ngaphakathi kweprojekthi efanayo - ngokubanzi, ukuze onke amaqela abe nothungelwano oluhlukeneyo. Nangona kunjalo, kukho inani elikhulu lamaqela asebenzayo endingathandi ukuwaqengqa ukusuka ekuqaleni, kuba aqhuba iinkonzo ezininzi, izicelo ezifanelekileyo, njl.

Kwaye ke sazibuza: indlela yokutshintsha i-subnet kwiqela elikhoyo?

Ukukhangela izigqibo

Olona qheliselo luqhelekileyo kukudala ngokutsha zonke iinkonzo ezinodidi lwe-ClusterIP. Njengenketho, unokucebisa kwaye oku:

Le nkqubo ilandelayo inengxaki: emva kokuba yonke into ilungisiwe, ii-pods ziza kunye ne-IP endala njenge-DNS nameserver kwi /etc/resolv.conf.
Kuba ndingekasifumani isisombululo, kuye kwafuneka ndisete kwakhona iqela lonke ngokusetha kwakhona kwe-kubeadm kwaye ndiyiqalise kwakhona.

Kodwa oku akufanelanga wonke umntu... Nazi iintshayelelo ezithe vetshe malunga nemeko yethu:

• I-Flannel isetyenziswa;
• Kukho amaqela amabini kumafu nakwi-hardware;
• Ndingathanda ukunqanda ukusasaza kwakhona zonke iinkonzo kwiqela;
• Kukho isidingo sokwenza yonke into ngokubanzi ngenani elincinci leengxaki;
• I-Kubernetes version yi-1.16.6 (nangona kunjalo, amanyathelo angaphezulu aya kufana nezinye iinguqulelo);
• Umsebenzi ophambili kukuqinisekisa ukuba kwiqela elisetyenzisiweyo kusetyenziswa i-beadm nge-subnet yenkonzo 192.168.0.0/16, yibuyisele nge 172.24.0.0/16.

Kwaye kwenzeka nje ukuba kudala sinomdla wokubona ukuba yintoni kwaye njani kwi-Kubernetes igcinwe kwi-etcd, yintoni enokwenziwa ngayo ... Ngoko sacinga: "Kutheni ungahlaziyi nje idatha kwi-etcd, ukubuyisela iidilesi ze-IP endala (i-subnet) kunye nezitsha? »

Emva kokukhangela izixhobo esele zilungile zokusebenza kunye nedatha kwi-etcd, asifumananga nto eyisombulule ngokupheleleyo ingxaki. (Ngendlela, ukuba uyazi malunga naziphi na izinto eziluncedo zokusebenza ngedatha ngokuthe ngqo kwi-etcd, siyakuwaxabisa amakhonkco.) Nangona kunjalo, isiqalo esihle etcdhelper ukusuka kwi-OpenShift (enkosi kubabhali bayo!).

Esi sixhobo sinokuqhagamshela kwi- etcd usebenzisa izatifikethi kwaye ufunde idata ukusuka apho usebenzisa imiyalelo ls, get, dump.

Yongeza etcdhelper

Ingcinga elandelayo isengqiqweni: "Yintoni ekuthintelayo ekongezeni olu ncedo ngokongeza ukukwazi ukubhala idatha ku-etcd?"

Yaba luguqulelo olulungisiweyo etcdhelper imisebenzi emibini emitsha changeServiceCIDR и changePodCIDR. kuye ungabona ikhowudi apha.

Zenza ntoni iimpawu ezintsha? Umgaqo-nkqubo changeServiceCIDR:

• yenza i-deserializer;
• qulunqa intetho eqhelekileyo yokubeka endaweni ye-CIDR;
• sihamba kuzo zonke iinkonzo ngohlobo lwe-ClusterIP kwiqela:
  • gqiba ixabiso ukusuka etcd ukuya kwinto ethi Go;
  • sisebenzisa intetho eqhelekileyo sibuyisela iibhayithi ezimbini zokuqala zedilesi;
  • abele inkonzo idilesi ye IP ukusuka kwi subnet entsha;
  • yenza i-serializer, guqula into ye-Go ibe yiprotobuf, bhala idatha entsha kwi-etcd.

Umsebenzi changePodCIDR ngokufanayo changeServiceCIDR - kuphela endaweni yokuhlela ukucaciswa kwenkonzo, siyenzela i-node kunye nokutshintsha .spec.PodCIDR kwi subnet entsha.

Zenza

Tshintsha inkonzo CIDR

Isicwangciso sokuphumeza umsebenzi silula kakhulu, kodwa sibandakanya ixesha lokuphumla ngexesha lokwenziwa kwakhona kwazo zonke iipods kwiqela. Emva kokuchaza amanyathelo aphambili, siya kuphinda sabelane ngeengcinga malunga nendlela, kwithiyori, eli xesha lokuphumla lingancitshiswa.

Amanyathelo okulungiselela:

• ukufakela isoftware eyimfuneko kunye nokudibanisa i-patched etcdhelper;
• backup etcd kunye /etc/kubernetes.

Isicwangciso esifutshane sokutshintsha inkonzoCIDR:

• ukutshintsha i-apiserver kunye nomlawuli-umphathi ubonakalisa;
• ukukhutshwa kwakhona kwezatifikethi;
• ukutshintsha iinkonzo ze-ClusterIP kwi- etcd;
• qala kwakhona zonke iipod kwiqela.

Oku kulandelayo lulandelelwano olupheleleyo lwezenzo ngokweenkcukacha.

1. Faka etcd-client ukwenzela ukulahla idatha:

apt install etcd-client

2. Yakha etcdhelper:

• Faka igolang:

  GOPATH=/root/golang
  mkdir -p $GOPATH/local
  curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
  echo "export GOPATH="$GOPATH"" >> ~/.bashrc
  echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
  echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

• Siyazigcinela thina etcdhelper.go, Khuphela ukuxhomekeka, qokelela:

  wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
  go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
  go build -o etcdhelper etcdhelper.go

3. Yenza ugcino etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Guqula i-subnet yenkonzo kwi-Kubernetes control plane ibonakalisa. Kwiifayile /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml tshintsha iparameter --service-cluster-ip-range kwi-subnet entsha: 172.24.0.0/16 вместо 192.168.0.0/16.

5. Kuba sitshintsha i-subnet yenkonzo apho kubeadm ikhupha izatifikethi ze-apiserver (kubandakanywa), kufuneka ziphinde zikhutshwe:

1. Makhe sibone ukuba yeyiphi imimandla kunye needilesi ze-IP isatifikethi sangoku esikhutshelwe:

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

2. Masilungiselele uqwalaselo oluncinci lwe-kubeadm:

   cat kubeadm-config.yaml
   apiVersion: kubeadm.k8s.io/v1beta1
   kind: ClusterConfiguration
   networking:
     podSubnet: "10.244.0.0/16"
     serviceSubnet: "172.24.0.0/16"
   apiServer:
     certSANs:
     - "192.168.199.100" # IP-адрес мастер узла

3. Masicime i-crt endala kunye nesitshixo, kuba ngaphandle koku isatifikethi esitsha asiyi kukhutshwa:

   rm /etc/kubernetes/pki/apiserver.{key,crt}

4. Masikhuphe kwakhona izatifikethi zeseva ye-API:

   kubeadm init phase certs apiserver --config=kubeadm-config.yaml

5. Masijonge ukuba isatifikethi sikhutshelwe isubnet entsha:

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

6. Emva kokuphinda ukhuphe isatifikethi seseva ye-API, qala kwakhona isikhongozeli saso:

   docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

7. Masihlaziye uqwalaselo lwe admin.conf:

   kubeadm alpha certs renew admin.conf

8. Masihlele idatha kwi etcd:

   ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 

   Nceda nceda! Ngeli xesha, isisombululo sesizinda siyayeka ukusebenza kwiqela, ekubeni kwiipods ezikhoyo /etc/resolv.conf idilesi endala yeCoreDNS (kube-dns) ibhalisiwe, kwaye kube-proxy itshintsha imithetho ye-iptables ukusuka kwi-subnet endala ukuya kwentsha. Ukuqhubela phambili kwinqaku kubhaliwe malunga neenketho ezinokwenzeka zokunciphisa ixesha lokunciphisa.

9. Masilungise iConfigMap kwindawo yamagama kube-system:

   kubectl -n kube-system edit cm kubelet-config-1.16

   - buyisela apha clusterDNS kwidilesi entsha ye-IP yenkonzo ye-kube-dns: kubectl -n kube-system get svc kube-dns.

   kubectl -n kube-system edit cm kubeadm-config

   - siya kuyilungisa data.ClusterConfiguration.networking.serviceSubnet kwi subnet entsha.

10. Kuba idilesi ye-kube-dns itshintshile, kuyafuneka ukuhlaziya uqwalaselo lwe-kubelet kuzo zonke iindawo:

    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet

11. Ekuphela kwento eseleyo kukuqalisa kwakhona zonke iipods kwiqela:

    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Nciphisa ixesha lokuphumla

Iingcamango malunga nendlela yokunciphisa ixesha lokuphumla:

1. Emva kokutshintsha ukubonakaliswa kwendiza yolawulo, yenza inkonzo entsha ye-kube-dns, umzekelo, ngegama kube-dns-tmp kunye nedilesi entsha 172.24.0.10.
2. Ukwenza if kwi etcdhelper, engayi kuguqula inkonzo ye-kube-dns.
3. Buyisela idilesi kuzo zonke ii-kubelets ClusterDNS kwentsha, ngelixa inkonzo endala iya kuqhubeka isebenza ngaxeshanye nentsha.
4. Linda de iipods ezinezicelo ziqengqeleke ngokwazo ngenxa yezizathu zendalo okanye ngexesha ekuvunyelwene ngalo.
5. Cima inkonzo kube-dns-tmp kwaye utshintshe serviceSubnetCIDR yenkonzo ye-kube-dns.

Esi sicwangciso siya kukuvumela ukuba unciphise ixesha lokuphumla libe ~umzuzu - ngexesha lokususwa kwenkonzo kube-dns-tmp kunye nokutshintsha i-subnet yenkonzo kube-dns.

Ukuguqulwa kwe-podNetwork

Kwangaxeshanye, sagqiba ekubeni sijonge indlela yokuguqula i-podNetwork usebenzisa isiphumo etcdhelper. Ulandelelwano lwezenzo ngolu hlobo lulandelayo:

• ukulungisa uqwalaselo ngaphakathi kube-system;
• ukulungisa kube-controller-manager manifest;
• tshintsha i-podCIDR ngokuthe ngqo kwi- etcd;
• qalisa kwakhona zonke iindawo zeqela.

Ngoku ngakumbi malunga nezi zenzo:

1. Guqula iConfigMap kwindawo yamagama kube-system:

kubectl -n kube-system edit cm kubeadm-config

- ukulungisa data.ClusterConfiguration.networking.podSubnet kwi subnet entsha 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- ukulungisa data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Guqula isilawuli-umphathi we-manifest:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- ukulungisa --cluster-cidr=10.55.0.0/16.

3. Jonga kumaxabiso angoku .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses kuzo zonke iindawo zeqela:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Faka indawo ye-podCIDR ngokwenza utshintsho ngokuthe ngqo kwi-etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Masijonge ukuba i-podCIDR itshintshile ngokwenene na:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Masiqale ngokutsha zonke iinodi ze-cluster nganye nganye.

7. Ukuba ushiya indawo enye ubuncinane i-podCIDR endala, ngoko kube-controller-manager akayi kuba nako ukuqalisa, kwaye iipods kwiqela aziyi kucwangciswa.

Ngapha koko, ukutshintsha i-podCIDR kunokwenziwa lula ngakumbi (umzekelo, kunjalo). Kodwa besifuna ukufunda indlela yokusebenza kunye etcd ngokuthe ngqo, kuba kukho iimeko xa uhlela izinto zeKubernetes kwi etcd - kuphela ukwahluka kunokwenzeka. (Umzekelo, awukwazi ukutshintsha intsimi yeNkonzo ngaphandle kwexesha lokuphumla spec.clusterIP.)

Isiphumo

Inqaku lixubusha ithuba lokusebenza kunye nedatha kwi-etcd ngokuthe ngqo, okt. ugqitha iKubernetes API. Ngamanye amaxesha le ndlela ikuvumela ukuba wenze "izinto ezikhohlisayo". Siye savavanya imisebenzi enikwe kwisicatshulwa kumaqela okwenene e-K8s. Nangona kunjalo, imeko yabo yokulungela ukusetyenziswa ngokubanzi I-PoC (ubungqina bengcamango). Ke ngoko, ukuba ufuna ukusebenzisa uguqulelo olulungisiweyo lwento eluncedo etcdhelper kumaqela akho, yenza oko ngomngcipheko wakho.

PS

Funda nakwibhlog yethu:

• «etcd 3.4.3: ukuthembeka kogcino kunye nesifundo sokhuseleko";
• «I-Calico yokunxibelelana kwi-Kubernetes: isingeniso kunye namava amancinci";
• «6 inkqubo yokonwabisa bugs ekusebenzeni kweKubernetes [kunye nesisombululo sabo]";
• «Isikhokelo esiBonakalayo sokuJonga ingxaki kwi-Kubernetes».

umthombo: www.habr.com