🥇 Ahụmahụ anyị nke iji data na-arụ ọrụ na ụyọkọ Kubernetes wdg (na-enweghị K8s API)

Na-arịwanye elu, ndị ahịa na-arịọ anyị ka anyị nye ohere na ụyọkọ Kubernetes ka ị nwee ike ịnweta ọrụ n'ime ụyọkọ ahụ: ka ị nwee ike jikọọ ozugbo na ụfọdụ nchekwa data ma ọ bụ ọrụ, jikọọ ngwa mpaghara na ngwa n'ime ụyọkọ ...

Dịka ọmụmaatụ, enwere mkpa ijikọ site na igwe mpaghara gị gaa na ọrụ memcached.staging.svc.cluster.local. Anyị na-enye ike a site na iji VPN n'ime ụyọkọ nke onye ahịa na-ejikọta ya. Iji mee nke a, anyị na-akpọsa subnets nke pọd, ọrụ na ịkwanye ụyọkọ DNS nye onye ahịa. Ya mere, mgbe onye ahịa nwara ijikọ na ọrụ ahụ memcached.staging.svc.cluster.local, arịrịọ ahụ na-aga na ụyọkọ DNS na nzaghachi na-enweta adreesị nke ọrụ a site na netwọk ọrụ ụyọkọ ma ọ bụ adreesị pod.

Anyị na-ahazi ụyọkọ K8s site na iji kubeadm, ebe subnet ọrụ ndabara dị 192.168.0.0/16, na netwọk nke pọd bụ 10.244.0.0/16. Ọtụtụ mgbe, ihe niile na-arụ ọrụ nke ọma, mana enwere isi ihe abụọ:

Netwọk 192.168.*.* A na-ejikarị ya na netwọk ụlọ ọrụ ndị ahịa, na ọbụna karịa na netwọk ụlọ ndị mmepụta. Ma mgbe ahụ, anyị na-enwe esemokwu: ndị na-anya ụgbọ ala na-arụ ọrụ na subnet a na VPN na-ebugharị subnets ndị a site na ụyọkọ na onye ahịa.
Anyị nwere ọtụtụ ụyọkọ (mmepụta, ogbo na/ma ọ bụ ọtụtụ ụyọkọ dev). Mgbe ahụ, na ndabara, ha niile ga-enwe otu subnets maka pods na ọrụ, nke na-emepụta nnukwu ihe isi ike maka ịrụ ọrụ n'otu oge na ọrụ na ọtụtụ ụyọkọ.

Ogologo oge gara aga, anyị nakweere omume nke iji subnets dị iche iche maka ọrụ na pọd n'ime otu ọrụ ahụ - n'ozuzu, ka ụyọkọ niile nwee netwọk dị iche iche. Otú ọ dị, enwere ọnụ ọgụgụ buru ibu nke ụyọkọ na-arụ ọrụ nke na-agaghị amasị m ịtụgharị site na ọkọ, ebe ọ bụ na ha na-arụ ọtụtụ ọrụ, ngwa ngwa, wdg.

Mgbe ahụ, anyị jụrụ onwe anyị: otú ịgbanwe subnet na dị ụyọkọ?

Ịchọ mkpebi

Ihe na-emekarị bụ ịmegharị ihe niile ọrụ nwere ụdị ClusterIP. Dị ka nhọrọ, nwere ike inye ndụmọdụ na nke a:

Usoro na-esonụ nwere nsogbu: mgbe a haziri ihe niile, pọd na-abịa na ochie IP dị ka DNS nameserver na /etc/resolv.conf.
Ebe ọ bụ na ahụghị m ngwọta ya, ekwesịrị m iji kubeadm reset tọgharịa ụyọkọ ahụ ma tinye ya ọzọ.

Mana nke a adabaghị maka onye ọ bụla… Lee okwu mmeghe zuru oke maka ikpe anyị:

A na-eji flannel eme ihe;
Enwere ụyọkọ ma na igwe ojii ma na ngwaike;
Ọ ga-amasị m izere ibugharị ọrụ niile na ụyọkọ ahụ;
Ọ dị mkpa ka a na-eme ihe niile na obere nsogbu;
Ụdị Kubernetes bụ 1.16.6 (agbanyeghị, usoro ndị ọzọ ga-adị ka nsụgharị ndị ọzọ);
Isi ọrụ bụ ịhụ na n'ụyọkọ etinyere na-eji kubeadm nwere subnet ọrụ 192.168.0.0/16, jiri dochie ya 172.24.0.0/16.

Ma ọ dị nnọọ mere na anyị nwere mmasị ogologo oge ịhụ ihe na otú e si echekwa Kubernetes na wdg, ihe a ga-eme ya ... Ya mere, anyị chere: "Gịnị kpatara na ọ bụghị naanị imelite data na wdgd, jiri ndị ọhụrụ dochie adreesị IP ochie (subnet) ochie? »

N'ịchọta ngwaọrụ ndị a kwadebere maka ịrụ ọrụ na data na wdgd, anyị ahụghị ihe ọ bụla na-edozi nsogbu ahụ kpamkpam. (Site n'ụzọ, ọ bụrụ na ịmara maka akụrụngwa ọ bụla maka iji data rụọ ọrụ ozugbo na wdgd, anyị ga-enwe ekele maka njikọ ndị ahụ.) Agbanyeghị, ebe mmalite dị mma bụ wdgdhelper site na OpenShift (ekele ndị dere ya!).

Ngwa a nwere ike jikọọ na wdgd site na iji asambodo wee gụọ data si ebe ahụ site na iji iwu ls, get, dump.

Tinye wdgdhhelper

Echiche na-esote bụ ihe ezi uche dị na ya: "Gịnị na-egbochi gị ịgbakwunye ike a site n'ịgbakwunye ikike ide data na wdgd?"

Ọ ghọrọ ụdị etcdhhelper gbanwetụrụ nwere ọrụ ọhụrụ abụọ changeServiceCIDR и changePodCIDR. na ya ị nwere ike ịhụ koodu ebe a.

Kedu ihe njirimara ọhụrụ na-eme? Algorithm changeServiceCIDR:

mepụta deserializer;
chịkọta okwu mgbe niile iji dochie CIDR;
anyị na-eji ụdị ClusterIP na-aga na ọrụ niile:
- decode uru si etcd n'ime ihe Go;
- na-eji okwu mgbe nile anyị na-edochi mbụ bytes abụọ nke adreesị;
- kenye ọrụ ahụ adreesị IP site na subnet ọhụrụ;
- mepụta serializer, gbanwee ihe Go ka ọ bụrụ protobuf, dee data ọhụrụ na wdgd.

ọrụ changePodCIDR n'ezie yiri changeServiceCIDR - naanị kama dezie nkọwapụta ọrụ, anyị na-eme ya maka ọnụ na mgbanwe .spec.PodCIDR na subnet ọhụrụ.

Mee ihe

Gbanwee ọrụ CIDR

Atụmatụ maka mmejuputa atumatu a dị nnọọ mfe, ma ọ na-agụnye oge nkwụsịtụ n'oge a na-emegharị pods niile na ụyọkọ ahụ. Mgbe anyị kọwachara usoro ndị bụ isi, anyị ga-ekekọrịtakwa echiche maka otu, na tiori, enwere ike ibelata oge ọdịda a.

Usoro nkwado:

ịwụnye ngwanrọ dị mkpa ma na-achịkọta patched etcdhelper;
ndabere etcd na /etc/kubernetes.

Atụmatụ ime ihe dị nkenke maka ịgbanwe serviceCIDR:

na-agbanwe apiserver na njikwa-onye njikwa na-egosipụta;
reissue nke asambodo;
na-agbanwe ọrụ ClusterIP na wdgd;
malitegharịa pọd niile dị na ụyọkọ ahụ.

Ihe a bụ usoro omume zuru oke.

1. Wụnye etcd-client maka mkpofu data:

apt install etcd-client

2. Wulite wdgdhelper:

Wụnye golang:

GOPATH=/root/golang
mkdir -p $GOPATH/local
curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
echo "export GOPATH="$GOPATH"" >> ~/.bashrc
echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

Anyị na-echekwa maka onwe anyị etcdhelper.go, budata ndabere, nakọta:

wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
go build -o etcdhelper etcdhelper.go

3. Mee ndabere etcd:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Gbanwee subnet ọrụ na ụgbọ elu njikwa Kubernetes gosipụtara. Na faịlụ /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml gbanwee oke --service-cluster-ip-range gaa na subnet ọhụrụ: 172.24.0.0/16 kama nke 192.168.0.0/16.

5. Ebe ọ bụ na anyị na-agbanwe subnet ọrụ nke kubeadm na-enye asambodo maka apiserver (gụnyere), ha kwesịrị ịnyeghachi:

Ka anyị hụ ngalaba na adreesị IP nke enyerela asambodo ugbu a maka:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Ka anyị kwadoo obere nhazi maka kubeadm:

cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "172.24.0.0/16"
apiServer:
  certSANs:
  - "192.168.199.100" # IP-адрес мастер узла

Ka anyị hichapụ crt na igodo ochie, ebe ọ bụ na enweghị nke a agaghị enye asambodo ọhụrụ:
```
rm /etc/kubernetes/pki/apiserver.{key,crt}
```

Ka anyị nyegharịa asambodo maka sava API:

kubeadm init phase certs apiserver --config=kubeadm-config.yaml

Ka anyị lelee na e nyere asambodo maka subnet ọhụrụ:

openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
X509v3 Subject Alternative Name:
    DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

Ka iwepụtaghachi akwụkwọ nkesa API, malitegharịa akpa ya:

docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

Ka anyị megharịa nhazi maka admin.conf:
```
kubeadm alpha certs renew admin.conf
```
Ka anyị dezie data na wdgd:
```
./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 
```
Ịkpachara anya N'oge a, ngalaba mkpebi akwụsị ịrụ ọrụ na ụyọkọ, ebe ọ bụ na pọd ndị dị adị /etc/resolv.conf adreesị CoreDNS ochie (kube-dns) debanyere aha, na kube-proxy na-agbanwe iwu iptables site na subnet ochie gaa na nke ọhụrụ. Ọzọkwa n'isiokwu a, e dere banyere nhọrọ enwere ike ibelata oge nkwụsị.
Ka anyị dozie ConfigMap na oghere aha kube-system:
```
kubectl -n kube-system edit cm kubelet-config-1.16
```
- dochie ebe a clusterDNS gaa na adreesị IP ọhụrụ nke ọrụ kube-dns: kubectl -n kube-system get svc kube-dns.
```
kubectl -n kube-system edit cm kubeadm-config
```
- anyị ga-edozi ya data.ClusterConfiguration.networking.serviceSubnet na subnet ọhụrụ.
Ebe ọ bụ na adreesị kube-dns agbanweela, ọ dị mkpa imelite kubelet config na ọnụ niile:
```
kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
```

Naanị ihe fọdụrụ bụ ịmalitegharịa pọd niile dị na ụyọkọ ahụ:

kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Wedata oge nkwụsịtụ

Echiche maka otu esi ewelata oge ezumike:

Mgbe ịgbanwere ụgbọ elu njikwa gosipụtara, mepụta ọrụ kube-dns ọhụrụ, dịka ọmụmaatụ, jiri aha ahụ kube-dns-tmp na adreesị ọhụrụ 172.24.0.10.
Mee if na wdgdhhelper, nke na-agaghị agbanwe ọrụ kube-dns.
Dochie adreesị na kubelets niile ClusterDNS na nke ọhụrụ, mgbe ọrụ ochie ga-anọgide na-arụ ọrụ n'otu oge na nke ọhụrụ.
Chere ruo mgbe pọd ndị nwere ngwa ga-atụgharị ma ọ bụ n'onwe ha maka ebumnuche ebumpụta ụwa ma ọ bụ n'oge ekwekọrịtara.
Hichapụ ọrụ kube-dns-tmp ma gbanwee serviceSubnetCIDR maka ọrụ kube-dns.

Atụmatụ a ga-enye gị ohere iwelata oge nkwụsịtụ ruo ~ otu nkeji - maka oge mwepụ ọrụ ahụ kube-dns-tmp na ịgbanwe subnet maka ọrụ ahụ kube-dns.

Mgbanwe podNetwork

N'otu oge ahụ, anyị kpebiri ileba anya ka esi gbanwee podNetwork site na iji etcdhelper pụta. Usoro omume bụ nke a:

idozi configs na kube-system;
na-edozi ihe ngosi kube-nchịkwa-nchịkwa;
gbanwee podCIDR ozugbo na wdgd;
malitegharịa ọnụ ụyọkọ niile.

Ugbu a karịa gbasara omume ndị a:

1. Gbanwee ConfigMap na oghere aha kube-system:

kubectl -n kube-system edit cm kubeadm-config

- mmezi data.ClusterConfiguration.networking.podSubnet na subnet ọhụrụ 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- mmezi data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Gbanwee ngosipụta njikwa-onye njikwa:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- mmezi --cluster-cidr=10.55.0.0/16.

3. Lelee ụkpụrụ dị ugbu a .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses maka ọnụ ụyọkọ niile:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Dochie podCIDR site na ime mgbanwe ozugbo na etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Ka anyị lelee na podCIDR agbanweela n'ezie:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Ka anyị malitegharịa ọnụ ụyọkọ niile n'otu n'otu.

7. Ọ bụrụ na ị hapụ opekata mpe otu ọnụ ochie podCIDR, mgbe ahụ, kube-controller-manager agaghị enwe ike ịmalite, na pods na ụyọkọ agaghị ahazi.

N'ezie, mgbanwe podCIDR nwere ike ime ọbụna mfe (dịka ọmụmaatụ, ka). Mana anyị chọrọ ịmụta ka esi arụ ọrụ na etcd ozugbo, n'ihi na enwere ikpe mgbe ị na-edezi ihe Kubernetes na wdg - otu enwere ike variant. (Dịka ọmụmaatụ, ị nweghị ike ịgbanwe mpaghara Ọrụ na-enweghị oge nkwụsịtụ spec.clusterIP.)

Nsonaazụ

Isiokwu ahụ na-atụle ohere nke ịrụ ọrụ na data na wdgd ozugbo, i.e. ịgafe Kubernetes API. Mgbe ụfọdụ, usoro a na-enye gị ohere ịme “ihe aghụghọ”. Anyị nwalere arụmọrụ enyere na ederede na ezigbo ụyọkọ K8. Agbanyeghị, ọnọdụ ha dị njikere maka ojiji zuru ebe niile bụ PoC (ihe akaebe nke echiche). Ya mere, ọ bụrụ na ịchọrọ iji ụdị ọrụ wdgdhelper gbanwetụrụ na ụyọkọ gị, mee ya n'ihe egwu nke gị.

PS

Gụọkwa na blọọgụ anyị:

isi: www.habr.com

Ahụmahụ anyị na-arụ ọrụ na data na ụyọkọ Kubernetes wdg (na-enweghị K8s API)