🥇Isipiliyoni sethu sokusebenza ngedatha ku- etcd Kubernetes cluster ngokuqondile (ngaphandle kwe-K8s API)

Ngokukhulayo, amaklayenti asicela ukuthi sinikeze ukufinyelela kuqoqo le-Kubernetes ukuze sikwazi ukufinyelela izinsiza ngaphakathi kweqoqo: ukwazi ukuxhuma ngokuqondile kusizindalwazi esithile noma isevisi, ukuxhuma uhlelo lokusebenza lwasendaweni nezinhlelo zokusebenza ezingaphakathi kweqoqo...

Isibonelo, kunesidingo sokuxhuma emshinini wangakini uye kusevisi memcached.staging.svc.cluster.local. Sinikeza leli khono sisebenzisa i-VPN ngaphakathi kweqoqo lapho iklayenti lixhumeka khona. Ukwenza lokhu, simemezela ama-subnet ama-pods, izinsizakalo kanye ne-push cluster DNS kuklayenti. Ngakho, uma iklayenti lizama ukuxhuma kusevisi memcached.staging.svc.cluster.local, isicelo siya ku-DNS yeqoqo futhi ngokuphendula sithola ikheli lale sevisi kunethiwekhi yesevisi yeqoqo noma ekhelini le-pod.

Simisa amaqoqo e-K8s sisebenzisa i-kubeadm, lapho i-subnet yesevisi ezenzakalelayo ikhona 192.168.0.0/16, kanye nenethiwekhi yama-pods 10.244.0.0/16. Ngokuvamile konke kusebenza kahle, kodwa kukhona amaphuzu ambalwa:

I-Subnet 192.168.*.* evame ukusetshenziswa kumanethiwekhi ehhovisi leklayenti, futhi kaningi kumanethiwekhi asekhaya onjiniyela. Bese sithola ukungqubuzana: amarutha asekhaya asebenza kule subnet futhi i-VPN iphusha lawa ma-subnets ukusuka kuqoqo kuya kuklayenti.
Sinamaqoqo amaningana (ukukhiqiza, isiteji kanye/noma amaqoqo e-dev ambalwa). Khona-ke, ngokuzenzakalelayo, zonke zizoba nama-subnets afanayo ama-pods namasevisi, okudala ubunzima obukhulu bokusebenza ngesikhathi esisodwa nezinsizakalo kumaqoqo amaningana.

Kudala samukela umkhuba wokusebenzisa ama-subnet ahlukene ezinsizeni nama-pods ngaphakathi kwephrojekthi efanayo - ngokuvamile, ukuze wonke amaqoqo abe namanethiwekhi ahlukene. Kodwa-ke, kunenombolo enkulu yamaqoqo asebenzayo engingeke ngithande ukuwahambisa kusukela ekuqaleni, ngoba asebenzisa izinsizakalo eziningi, izinhlelo zokusebenza ezisezingeni eliphezulu, njll.

Futhi-ke sazibuza: indlela yokushintsha i-subnet kuqoqo elikhona?

Ukusesha izinqumo

Umkhuba ojwayelekile uwukudala kabusha konke amasevisi ngohlobo lwe-ClusterIP. Njengenketho, angaluleka futhi lokhu:

Inqubo elandelayo inenkinga: ngemva kokuba konke kulungisiwe, ama-pods avela ne-IP endala njenge-DNS nameserver ku/etc/resolv.conf.
Njengoba bengingakasitholi isisombululo, bekufanele ngisethe kabusha lonke iqoqo ngokusetha kabusha kwe-kubeadm bese ngiyiqalisa futhi.

Kodwa lokhu akulungele wonke umuntu... Nazi izingeniso ezinemininingwane yecala lethu:

I-flannel isetshenziswa;
Kukhona amaqoqo kokubili emafwini naku-hardware;
Ngingathanda ukugwema ukuphinda ngiphakele zonke izinkonzo kuqoqo;
Kunesidingo sokwenza konke ngokujwayelekile ngenani elincane lezinkinga;
Inguqulo ye-Kubernetes ithi 1.16.6 (noma kunjalo, izinyathelo ezengeziwe zizofana nezinye izinguqulo);
Umsebenzi omkhulu uwukuqinisekisa ukuthi kuqoqo elisetshenziswe kusetshenziswa i-beadm ene-subnet yesevisi 192.168.0.0/16, esikhundleni salo 172.24.0.0/16.

Futhi kwenzeka nje ukuthi kade sinesithakazelo sokubona ukuthi yini futhi kanjani ku-Kubernetes egcinwe ku-etcd, yini engenziwa ngakho... Ngakho sacabanga: “Kungani ungavele ubuyekeze idatha ku- etcd, esikhundleni samakheli e-IP amadala (i-subnet) ufake amasha? »

Ngemva kokusesha amathuluzi asevele enziwe okusebenza ngedatha ku- etcd, asitholanga lutho oluyixazulule ngokuphelele inkinga. (Kodwa-ke, uma wazi nganoma yiziphi izinsiza zokusebenza ngedatha ngokuqondile ku- etcd, singazijabulela izixhumanisi.) Nokho, isiqalo esihle njll kusuka ku-OpenShift (sibonga ababhali bayo!).

Lolu hlelo lokusebenza lungaxhuma ku- etcd lusebenzisa izitifiketi futhi lufunde idatha kusuka lapho kusetshenziswa imiyalo ls, get, dump.

Engeza njllshelper

Umcabango olandelayo unengqondo: "Yini ekuvimbela ukuthi ungeze lolu hlelo lokusebenza ngokungeza ikhono lokubhala idatha ku-etcd?"

Yaba inguqulo eguquliwe ye-etcdhelper enemisebenzi emibili emisha changeServiceCIDR и changePodCIDR. kuye ungabona ikhodi lapha.

Zenzani izici ezintsha? I-algorithm changeServiceCIDR:

dala i-deserializer;
hlanganisa isisho esivamile ukuze ungene esikhundleni se-CIDR;
sidlula kuzo zonke izinkonzo ngohlobo lwe-ClusterIP kuqoqo:
- khipha inani elisuka ku- etcd libe into ethi Go;
- sisebenzisa isisho esivamile sishintsha amabhayithi amabili okuqala ekheli;
- nikeza isevisi ikheli le-IP kusuka ku-subnet entsha;
- dala i-serializer, guqula into ethi Hamba ibe yi-protobuf, bhala idatha entsha ku- etcd.

Umsebenzi changePodCIDR ngokufanayo changeServiceCIDR - kuphela esikhundleni sokuhlela ukucaciswa kwesevisi, sikwenzela i-node kanye noshintsho .spec.PodCIDR ku-subnet entsha.

Hlanganisa

Shintsha isevisi CIDR

Uhlelo lokuqalisa umsebenzi lulula kakhulu, kodwa lubandakanya isikhathi sokuphumula ngesikhathi sokwakhiwa kabusha kwawo wonke ama-pods kuqoqo. Ngemva kokuchaza izinyathelo eziyinhloko, sizophinde sabelane ngemicabango yokuthi, ngokombono, lesi sikhathi sokuphumula singancishiswa kanjani.

Izinyathelo zokulungiselela:

ukufaka isofthiwe edingekayo kanye nokuhlanganisa i-etcdhelper enamachibi;
isipele njlld futhi /etc/kubernetes.

Uhlelo lokusebenza olufushane lokushintsha isevisiCIDR:

ukushintsha i-apiserver kanye ne-controller-manager manifests;
ukukhishwa kabusha kwezitifiketi;
ukushintsha izinsiza ze-ClusterIP ku- etcd;
qala kabusha wonke ama-pods ku-cluster.

Okulandelayo ukulandelana okuphelele kwezenzo ngokuningiliziwe.

1. Faka njlld-iklayenti ukuze kulahlwe idatha:

apt install etcd-client

2. Yakha njlldhelper:

Faka i-golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

Sizigcinela thina etcdhelper.go, ukulanda okuncikile, qoqa:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Yenza isipele njlld:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Shintsha i-subnet yesevisi endizeni yokulawula ye-Kubernetes. Kumafayela /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml shintsha ipharamitha --service-cluster-ip-range ku-subnet entsha: 172.24.0.0/16 esikhundleni salokho 192.168.0.0/16.

5. Njengoba sishintsha i-subnet yesevisi lapho i-kubeadm ikhipha khona izitifiketi ze-apiserver (okuhlanganisa), zidinga ukukhishwa kabusha:

Ake sibone ukuthi yiziphi izizinda namakheli e-IP isitifiketi samanje esikhishelwe:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Ake silungiselele ukucushwa okuncane kwe-kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла

Masisuse i-crt endala nokhiye, ngoba ngaphandle kwalokhu isitifiketi esisha ngeke sikhishwe:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

Masikhiphe kabusha izitifiketi zeseva ye-API:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

Ake sihlole ukuthi isitifiketi sikhishelwe i-subnet entsha:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Ngemva kokuphinda ukhiphe isitifiketi seseva ye-API, qala kabusha isiqukathi saso:
```
docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart
```
Masikhiqize kabusha ukulungiselelwa kwe admin.conf:
```
kubeadm alpha certs renew admin.conf
```
Masihlele idatha ku- etcd:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
Ukuqapha Ngalesi sikhathi, ukulungiswa kwesizinda kuyeka ukusebenza kuqoqo, kusukela kuma-pods akhona /etc/resolv.conf ikheli elidala le-CoreDNS (kube-dns) libhalisiwe, futhi i-kube-proxy ishintsha imithetho ye-iptables isuke ku-subnet endala iye kwentsha. Ngokuqhubekayo esihlokweni kubhaliwe mayelana nezinketho ezingenzeka zokunciphisa isikhathi sokuphumula.
Masilungise i-ConfigMap endaweni yamagama kube-system:
```
kubectl -n kube-system edit cm kubelet-config-1.16
```
- buyisela lapha clusterDNS ekhelini le-IP elisha lesevisi ye-kube-dns: kubectl -n kube-system get svc kube-dns.
```
kubectl -n kube-system edit cm kubeadm-config
```
- sizoyilungisa data.ClusterConfiguration.networking.serviceSubnet ku-subnet entsha.
Njengoba ikheli le-kube-dns selishintshile, kuyadingeka ukuthi kubuyekezwe ukucushwa kwe-kubelet kuwo wonke ama-node:
```
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```

Okusele ukuqala kabusha wonke ama-pods ku-cluster:

kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Nciphisa isikhathi sokuphumula

Imicabango yokuthi unganciphisa kanjani isikhathi sokuphumula:

Ngemva kokushintsha isibonakaliso sendiza yokulawula, dala isevisi entsha ye-kube-dns, isibonelo, ngegama kube-dns-tmp kanye nekheli elisha 172.24.0.10.
Yenza if ku- etcdhelper, engeke iguqule isevisi ye-kube-dns.
Faka esikhundleni ikheli kuwo wonke ama-kubelets ClusterDNS entsha, kuyilapho isevisi endala izoqhubeka nokusebenza kanyekanye nentsha.
Linda kuze kube yilapho ama-pods anezicelo aziqengqa ngokwawo ngenxa yezizathu zemvelo noma ngesikhathi okuvunyelwene ngaso.
Susa isevisi kube-dns-tmp futhi ushintshe serviceSubnetCIDR ngesevisi ye-kube-dns.

Lolu hlelo luzokuvumela ukuthi unciphise isikhathi sokuphumula sibe ~umzuzu - ngesikhathi sokususwa kwesevisi kube-dns-tmp kanye nokushintsha i-subnet yesevisi kube-dns.

Ukuguqulwa kwe-podNetwork

Ngesikhathi esifanayo, sinqume ukubheka ukuthi singayishintsha kanjani i-podNetwork usebenzisa i- etcdhelper ewumphumela. Ukulandelana kwezenzo kanje:

ukulungisa ama-configs kube-system;
ukulungisa i-kube-controller-manager manifest;
shintsha i-podCIDR ngokuqondile ku- etcd;
qala kabusha wonke ama-cluster node.

Manje okwengeziwe ngalezi zenzo:

1. Lungisa i-ConfigMap endaweni yamagama kube-system:

kubectl -n kube-system edit cm kubeadm-config

- ukulungisa data.ClusterConfiguration.networking.podSubnet ku-subnet entsha 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- ukulungisa data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Lungisa i-manifest yomphathi wesilawuli:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- ukulungisa --cluster-cidr=10.55.0.0/16.

3. Bheka amanani amanje .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses kuwo wonke ama-cluster node:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Faka esikhundleni i-podCIDR ngokwenza izinguquko ngqo ku- etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Ake sihlole ukuthi i-podCIDR ishintshile ngempela:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Masiqalise kabusha wonke ama-cluster nodes ngayinye ngayinye.

7. Uma ushiya okungenani i-node eyodwa i-podCIDR endala, i-kube-controller-manager ngeke ikwazi ukuqala, futhi ama-pods kuqoqo ngeke ahlelwe.

Eqinisweni, ukushintsha i-podCIDR kungenziwa kalula (ngokwesibonelo, kanjalo). Kepha besifuna ukufunda ukuthi singasebenza kanjani ne-etcd ngokuqondile, ngoba kunezimo lapho uhlela izinto ze-Kubernetes ku- etcd - kuphela okuhlukile okungenzeka. (Isibonelo, awukwazi nje ukushintsha inkambu Yesevisi ngaphandle kwesikhathi sokuphumula spec.clusterIP.)

Umphumela

I-athikili ixoxa ngamathuba okusebenza ngedatha ku-ejjd ngokuqondile, i.e. ngokudlula i-Kubernetes API. Ngezinye izikhathi le ndlela ikuvumela ukuba wenze "izinto ezikhohlisayo." Sihlole ukusebenza okunikezwe embhalweni kumaqoqo wangempela we-K8s. Nokho, isimo sabo sokulungela ukusetshenziswa kabanzi sinjalo I-PoC (ubufakazi bomqondo). Ngakho-ke, uma ufuna ukusebenzisa inguqulo eguquliwe yensiza njll kumaqoqo akho, kwenze lokho ngokuzifaka wena engozini.

PS

Funda futhi kubhulogi yethu:

Source: www.habr.com

Umuzwa wethu ngedatha ku- etcd Kubernetes cluster ngokuqondile (ngaphandle kwe-K8s API)