Peb qhov kev paub nrog cov ntaub ntawv hauv etcd Kubernetes pawg ncaj qha (tsis muaj K8s API)

Ntau ntxiv, cov neeg siv tau thov kom peb muab kev nkag mus rau Kubernetes pawg kom nkag tau rau hauv cov kev pabcuam hauv pawg: kom lawv tuaj yeem txuas ncaj qha rau qee qhov chaw khaws ntaub ntawv lossis kev pabcuam, txuas rau daim ntawv thov hauv zos nrog cov ntawv thov hauv pawg ...

Piv txwv li, yuav tsum tau txuas los ntawm koj lub tshuab hauv zos mus rau qhov kev pabcuam memcached.staging.svc.cluster.local. Peb muab lub peev xwm no siv VPN nyob rau hauv pawg uas tus neeg siv khoom txuas. Txhawm rau ua qhov no, peb tshaj tawm subnets ntawm pods, kev pabcuam thiab thawb pawg DNS rau tus neeg siv khoom. Yog li, thaum tus neeg siv khoom sim txuas rau qhov kev pabcuam memcached.staging.svc.cluster.local, qhov kev thov mus rau pawg DNS thiab hauv cov lus teb tau txais qhov chaw nyob ntawm qhov kev pabcuam no los ntawm pawg pabcuam pabcuam lossis chaw nyob ntawm pod.

Peb teeb tsa K8s pawg siv kubeadm, qhov twg qhov kev pabcuam subnet yog 192.168.0.0/16, thiab lub network ntawm pods yog 10.244.0.0/16. Feem ntau txhua yam ua haujlwm zoo, tab sis muaj ob peb lub ntsiab lus:

• Subnet 192.168.*.* feem ntau siv nyob rau hauv cov neeg siv khoom siv tes hauj lwm, thiab ntau zaus hauv cov neeg tsim khoom hauv tsev. Thiab tom qab ntawd peb tau txais kev tsis sib haum xeeb: cov routers hauv tsev ua haujlwm ntawm lub subnet no thiab VPN thawb cov subnets ntawm pawg mus rau tus neeg siv khoom.
• Peb muaj ntau pawg (kev tsim khoom, theem thiab / lossis ntau pawg dev). Tom qab ntawd, los ntawm lub neej ntawd, txhua tus ntawm lawv yuav muaj tib lub subnets rau cov pods thiab cov kev pabcuam, uas tsim teeb meem loj rau kev ua haujlwm ib txhij nrog cov kev pabcuam hauv ntau pawg.

Peb tau ntev dhau los tau txais kev coj ua ntawm kev siv cov subnets sib txawv rau cov kev pabcuam thiab cov pods nyob rau hauv tib txoj haujlwm - feem ntau, kom txhua pawg muaj kev sib txawv. Txawm li cas los xij, muaj ntau pawg hauv kev ua haujlwm uas kuv tsis xav dov los ntawm kos, vim lawv khiav ntau cov kev pabcuam, cov ntawv thov hauv xeev, thiab lwm yam.

Thiab ces peb nug peb tus kheej: yuav ua li cas hloov lub subnet hauv ib pawg uas twb muaj lawm?

Nrhiav kev txiav txim siab

Qhov kev xyaum tshaj plaws yog rov tsim dua tag nrho cov kev pabcuam nrog hom ClusterIP. Raws li kev xaiv, tuaj yeem qhia thiab qhov no:

Cov txheej txheem hauv qab no muaj teeb meem: tom qab txhua yam teeb tsa, cov pods tuaj nrog tus IP qub li DNS nameserver hauv /etc/resolv.conf.
Txij li thaum kuv tseem tsis tau pom qhov kev daws teeb meem, kuv yuav tsum rov pib dua tag nrho pawg nrog kubeadm pib dua thiab rov pib dua.

Tab sis qhov no tsis haum rau txhua tus ... Ntawm no yog cov lus qhia ntxaws ntxiv rau peb rooj plaub:

• Flannel yog siv;
• Muaj pawg ob qho tib si hauv huab thiab ntawm kev kho vajtse;
• Kuv xav kom tsis txhob rov muab tag nrho cov kev pabcuam hauv pawg;
• Feem ntau yuav tsum tau ua txhua yam nrog tsawg kawg ntawm cov teeb meem;
• Kubernetes version yog 1.16.6 (txawm li cas los xij, cov kauj ruam ntxiv yuav zoo sib xws rau lwm cov versions);
• Lub luag haujlwm tseem ceeb yog los xyuas kom meej tias nyob rau hauv ib pawg uas siv kubeadm nrog cov kev pabcuam subnet 192.168.0.0/16, hloov nrog 172.24.0.0/16.

Thiab nws nyuam qhuav tshwm sim uas peb tau xav ntev los pom dab tsi thiab yuav ua li cas hauv Kubernetes khaws cia hauv lwm yam, yuav ua li cas nrog nws ... Yog li peb xav tias: "Vim li cas tsis yog hloov kho cov ntaub ntawv hauv etcd, hloov cov qub IP chaw nyob (subnet) nrog cov tshiab? »

Tau tshawb nrhiav cov cuab yeej npaj ua haujlwm rau kev ua haujlwm nrog cov ntaub ntawv hauv etcd, peb tsis pom ib yam dab tsi uas daws tau qhov teeb meem. (Los ntawm txoj kev, yog tias koj paub txog txhua yam khoom siv rau kev ua haujlwm nrog cov ntaub ntawv ncaj qha hauv lwm yam, peb yuav txaus siab rau cov kev sib txuas.) Txawm li cas los xij, qhov pib zoo yog lwm tus pab los ntawm OpenShift (ua tsaug rau nws cov neeg sau ntawv!).

Qhov kev siv hluav taws xob no tuaj yeem txuas rau lwm yam siv daim ntawv pov thawj thiab nyeem cov ntaub ntawv los ntawm qhov ntawd siv cov lus txib ls, get, dump.

Ntxiv etcdhelper

Qhov kev xav tom ntej no yog qhov laj thawj: "Dab tsi txwv koj los ntawm kev ntxiv cov khoom siv no los ntawm kev ntxiv peev xwm sau cov ntaub ntawv rau lwm yam?"

Nws tau los ua ib qho kev hloov kho ntawm etcdhelper nrog ob txoj haujlwm tshiab changeServiceCIDR и changePodCIDR. ntawm nws koj tuaj yeem pom cov cai no.

Cov yam ntxwv tshiab ua li cas? Algorithm changeServiceCIDR:

• tsim ib tug deserializer;
• sau cov lus qhia tsis tu ncua los hloov CIDR;
• peb mus txog txhua qhov kev pabcuam nrog ClusterIP hom hauv pawg:
  • txiav txim siab tus nqi ntawm etcd rau hauv ib qho khoom Go;
  • siv cov lus qhia tsis tu ncua peb hloov thawj ob bytes ntawm qhov chaw nyob;
  • muab qhov kev pab cuam tus IP chaw nyob los ntawm lub subnet tshiab;
  • tsim cov serializer, hloov cov khoom mus rau hauv protobuf, sau cov ntaub ntawv tshiab rau lwm yam.

muaj nuj nqi changePodCIDR qhov tseem ceeb zoo sib xws changeServiceCIDR - tsuas yog hloov kho cov kev pabcuam tshwjxeeb, peb ua rau ntawm node thiab hloov pauv .spec.PodCIDR mus rau lub subnet tshiab.

Xyaum ua haujlwm

Hloov kev pabcuam CIDR

Txoj kev npaj rau kev ua tiav txoj haujlwm yog qhov yooj yim heev, tab sis nws cuam tshuam nrog lub sijhawm poob thaum lub sijhawm rov tsim dua ntawm txhua lub pods hauv pawg. Tom qab piav qhia txog cov kauj ruam tseem ceeb, peb tseem yuav qhia cov kev xav txog yuav ua li cas, hauv txoj kev xav, lub sijhawm poob qis tuaj yeem txo qis.

Cov kauj ruam npaj:

• txhim kho qhov tsim nyog software thiab assembling lub patched etcdhelper;
• backup thiab lwm yam /etc/kubernetes.

Cov phiaj xwm luv luv rau kev hloov kev pabcuamCIDR:

• hloov lub apiserver thiab controller-manager manifests;
• reissue ntawm daim ntawv pov thawj;
• hloov ClusterIP cov kev pabcuam hauv lwm yam;
• restart ntawm tag nrho cov pods hauv pawg.

Cov hauv qab no yog ib qho ua tiav ntawm kev ua kom ntxaws.

1. Nruab lwm yam-tus neeg siv khoom rau cov ntaub ntawv pov tseg:

apt install etcd-client

2. Tsim thiab lwm yam kev pab:

• Installation ntawm:

  GOPATH=/root/golang
  mkdir -p $GOPATH/local
  curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
  echo "export GOPATH="$GOPATH"" >> ~/.bashrc
  echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
  echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc

• Peb txuag rau peb tus kheej etcdhelper.go, download dependencies, sau:

  wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
  go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
  go build -o etcdhelper etcdhelper.go

3. Ua ib qho thaub qab thiab lwm yam:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Hloov cov kev pabcuam subnet hauv Kubernetes tswj lub dav hlau manifests. Hauv cov ntaub ntawv /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml hloov qhov parameter --service-cluster-ip-range mus rau lub subnet tshiab: 172.24.0.0/16 es tsis txhob 192.168.0.0/16.

5. Txij li thaum peb tab tom hloov cov kev pabcuam subnet uas kubeadm teeb meem daim ntawv pov thawj rau apiserver (xws li), lawv yuav tsum tau muab rov qab:

1. Cia peb pom cov npe thiab IP chaw nyob twg daim ntawv pov thawj tam sim no tau muab rau:

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

2. Cia peb npaj ib qho me me config rau kubeadm:

   cat kubeadm-config.yaml
   apiVersion: kubeadm.k8s.io/v1beta1
   kind: ClusterConfiguration
   networking:
     podSubnet: "10.244.0.0/16"
     serviceSubnet: "172.24.0.0/16"
   apiServer:
     certSANs:
     - "192.168.199.100" # IP-адрес мастер узла

3. Cia peb rho tawm cov qub crt thiab tus yuam sij, vim tias tsis muaj qhov no daim ntawv pov thawj tshiab yuav tsis muab:

   rm /etc/kubernetes/pki/apiserver.{key,crt}

4. Cia peb rov muab daim ntawv pov thawj rau API server:

   kubeadm init phase certs apiserver --config=kubeadm-config.yaml

5. Cia peb xyuas tias daim ntawv pov thawj tau muab rau lub subnet tshiab:

   openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
   X509v3 Subject Alternative Name:
       DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100

6. Tom qab rov muab daim ntawv pov thawj API server, rov pib nws lub thawv:

   docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart

7. Wb regenerate lub config rau admin.conf:

   kubeadm alpha certs renew admin.conf

8. Cia peb hloov cov ntaub ntawv hauv etcd:

   ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 

   Ceev faj Lub sijhawm no, qhov kev daws teeb meem sau tseg tsis ua haujlwm hauv pawg, txij li hauv cov pods uas twb muaj lawm /etc/resolv.conf qhov qub CoreDNS chaw nyob (kube-dns) tau sau npe, thiab kube-proxy hloov cov cai iptables los ntawm cov qub subnet mus rau qhov tshiab. Ntxiv rau hauv tsab xov xwm nws tau sau txog cov kev xaiv tau los txo qis qis qis.

9. Cia peb kho ConfigMap's hauv lub npe kube-system:

   kubectl -n kube-system edit cm kubelet-config-1.16

   - hloov ntawm no clusterDNS mus rau qhov chaw nyob IP tshiab ntawm kube-dns kev pabcuam: kubectl -n kube-system get svc kube-dns.

   kubectl -n kube-system edit cm kubeadm-config

   - peb yuav kho nws data.ClusterConfiguration.networking.serviceSubnet mus rau lub subnet tshiab.

10. Txij li qhov chaw nyob kube-dns tau hloov pauv, nws yog qhov yuav tsum tau hloov kho kubelet config ntawm tag nrho cov nodes:

    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet

11. Txhua yam uas tseem tshuav yog rov pib dua tag nrho cov pods hauv pawg:

    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Minimize downtime

Xav txog yuav ua li cas txo qis downtime:

1. Tom qab hloov lub dav hlau tswj manifests, tsim ib tug tshiab kube-dns kev pab cuam, piv txwv li, nrog lub npe kube-dns-tmp thiab chaw nyob tshiab 172.24.0.10.
2. Ua if hauv etcdhelper, uas yuav tsis hloov cov kev pabcuam kube-dns.
3. Hloov qhov chaw nyob hauv tag nrho cov kubelets ClusterDNS mus rau ib qho tshiab, thaum cov kev pabcuam qub tseem yuav ua haujlwm ib txhij nrog tus tshiab.
4. Tos kom txog thaum cov pods nrog daim ntawv thov dov los ntawm lawv tus kheej rau tej yam ntuj tso los yog raws li lub sij hawm pom zoo.
5. Rho tawm kev pabcuam kube-dns-tmp thiab hloov serviceSubnetCIDR rau qhov kev pabcuam kube-dns.

Txoj kev npaj no yuav tso cai rau koj txo qis qis rau ~ ib feeb - rau lub sijhawm tshem tawm cov kev pabcuam kube-dns-tmp thiab hloov subnet rau kev pabcuam kube-dns.

Hloov kho podNetwork

Nyob rau tib lub sijhawm, peb txiav txim siab saib yuav ua li cas hloov kho podNetwork siv cov txiaj ntsig etcdhelper. Cov txheej txheem ntawm kev ua yog raws li nram no:

• kho configs hauv kube-system;
• kho cov kube-controller-manifest;
• hloov podCIDR ncaj qha hauv lwm yam;
• reboot tag nrho cov pawg nodes.

Tam sim no ntxiv txog cov kev ua no:

1. Hloov kho ConfigMap's hauv lub npe kube-system:

kubectl -n kube-system edit cm kubeadm-config

- kho data.ClusterConfiguration.networking.podSubnet mus rau lub subnet tshiab 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- kho data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Hloov kho tus tswj-tswj manifest:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- kho --cluster-cidr=10.55.0.0/16.

3. Saib cov nqi tam sim no .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses rau tag nrho cov pawg nodes:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Hloov podCIDR los ntawm kev hloov pauv ncaj qha rau lwm yam:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Cia peb xyuas tias podCIDR tau hloov pauv tiag tiag:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Cia peb reboot tag nrho cov pawg ntawm ib qho los ntawm ib qho.

7. Yog tias koj tawm tsawg kawg ib lub ntawm qub podCIDR, ces kube-controller-manager yuav tsis tuaj yeem pib, thiab cov pods hauv pawg yuav tsis teem sijhawm.

Qhov tseeb, hloov podCIDR tuaj yeem ua tau yooj yim dua (piv txwv li, li ntawd,). Tab sis peb xav kawm yuav ua li cas ua hauj lwm nrog etcd ncaj qha, vim hais tias muaj cov ntaub ntawv thaum kho Kubernetes cov khoom nyob rau hauv lwm yam - qhov tsuas yog ua tau variant. (Piv txwv li, koj tsis tuaj yeem hloov qhov Kev Pabcuam tsuas yog tsis muaj sijhawm poob spec.clusterIP.)

Qhov no

Kab lus tham txog qhov ua tau ntawm kev ua haujlwm nrog cov ntaub ntawv hauv etcd ncaj qha, i.e. bypassing Kubernetes API. Qee lub sij hawm txoj hauv kev no tso cai rau koj ua "yam tsis yooj yim". Peb tau sim cov haujlwm uas tau muab rau hauv cov ntawv nyeem ntawm cov K8s tiag tiag. Txawm li cas los xij, lawv cov xwm txheej ntawm kev npaj rau kev siv dav dav yog PoC (pov thawj ntawm lub tswvyim). Yog li ntawd, yog tias koj xav siv ib qho kev hloov kho ntawm cov khoom siv hluav taws xob etcdhelper ntawm koj pawg, ua li ntawd ntawm koj tus kheej qhov kev pheej hmoo.

PS

Nyeem kuj ntawm peb blog:

• «etcd 3.4.3: kawm txog kev cia siab thiab kev ruaj ntseg»;
• «Calico rau kev sib tham hauv Kubernetes: kev taw qhia thiab kev paub me ntsis»;
• «6 lom zem kab kab hauv kev ua haujlwm ntawm Kubernetes [thiab lawv cov kev daws teeb meem]»;
• «Daim Ntawv Qhia Pom Pom Kev daws teeb meem Kubernetes".

Tau qhov twg los: www.hab.com