Kwarewar mu game da bayanai a cikin gungu Kubernetes kai tsaye (ba tare da K8s API ba)

Ƙaruwa, abokan ciniki suna tambayar mu don samar da dama ga gungu na Kubernetes don samun damar yin amfani da ayyuka a cikin gungu: don samun damar haɗa kai tsaye zuwa wasu bayanai ko sabis, don haɗa aikace-aikacen gida tare da aikace-aikace a cikin gungu ...

Kwarewar mu game da bayanai a cikin gungu Kubernetes kai tsaye (ba tare da K8s API ba)

Misali, akwai buƙatar haɗi daga injin ɗin ku zuwa sabis memcached.staging.svc.cluster.local. Muna ba da wannan damar ta amfani da VPN a cikin gungu wanda abokin ciniki ke haɗawa. Don yin wannan, muna sanar da ƙananan ramuka na kwasfan fayiloli, ayyuka da tura tari na DNS ga abokin ciniki. Don haka, lokacin da abokin ciniki yayi ƙoƙarin haɗi zuwa sabis ɗin memcached.staging.svc.cluster.local, Buƙatun yana zuwa ga cluster DNS kuma a cikin martani yana karɓar adireshin wannan sabis ɗin daga cibiyar sadarwar cluster sabis ko adireshi na pod.

Muna saita gungu na K8s ta amfani da kubeadm, inda tsohuwar cibiyar sadarwar sabis take 192.168.0.0/16, kuma cibiyar sadarwa na pods ne 10.244.0.0/16. Yawancin lokaci komai yana aiki da kyau, amma akwai maki biyu:

  • Subnet 192.168.*.* yawanci ana amfani da su a cikin cibiyoyin sadarwar abokin ciniki, har ma fiye da sau da yawa a cikin cibiyoyin sadarwar gida masu haɓaka. Sannan muna samun rikice-rikice: masu amfani da hanyar gida suna aiki akan wannan rukunin yanar gizon kuma VPN yana tura waɗannan rukunin yanar gizo daga gungu zuwa abokin ciniki.
  • Muna da gungu da yawa (samuwa, mataki da/ko ƙungiyoyin dev da yawa). Sa'an nan, ta tsohuwa, dukansu za su sami rahusa iri ɗaya don kwasfan fayiloli da ayyuka, wanda ke haifar da babbar matsala don aiki tare da ayyuka a cikin gungu da yawa.

Mun daɗe da ɗaukar al'adar yin amfani da hanyoyin sadarwa daban-daban don ayyuka da kwasfan fayiloli a cikin aiki ɗaya - gabaɗaya, ta yadda duk gungu suna da hanyoyin sadarwa daban-daban. Koyaya, akwai adadi mai yawa na gungu da ke aiki waɗanda ba zan so in jujjuya su daga karce ba, tunda suna gudanar da ayyuka da yawa, aikace-aikacen hukuma, da sauransu.

Kuma a sa'an nan muka tambayi kanmu: yadda za a canza subnet a cikin wani data kasance gungu?

Neman yanke shawara

Mafi yawan al'adar ita ce sake ƙirƙira duk ayyuka masu nau'in ClusterIP. A matsayin zaɓi, iya shawara kuma wannan:

Tsarin da ke biyo baya yana da matsala: bayan duk abin da aka saita, kwasfan fayiloli suna zuwa tare da tsohon IP azaman uwar garken DNS a /etc/resolv.conf.
Tun da har yanzu ban sami mafita ba, dole ne in sake saita dukkan cluster tare da sake saitin kubeadm sannan in sake shigar da shi.

Amma wannan bai dace da kowa ba... Anan akwai ƙarin cikakkun bayanai game da shari'ar mu:

  • Ana amfani da flannel;
  • Akwai gungu duka biyu a cikin gajimare da kan kayan aiki;
  • Ina so in guji sake tura duk ayyuka a cikin gungu;
  • Akwai buƙatar gabaɗaya yin komai tare da ƙaramin adadin matsaloli;
  • Sigar Kubernetes shine 1.16.6 (duk da haka, ƙarin matakai zasu kasance iri ɗaya ga sauran sigogin);
  • Babban aikin shine tabbatar da cewa a cikin gungu da aka tura ta amfani da kubeadm tare da subnet ɗin sabis 192.168.0.0/16, musanya shi da 172.24.0.0/16.

Kuma haka ya faru da cewa mun dade da sha'awar ganin abin da kuma yadda a cikin Kubernetes aka adana a etcd, abin da za a iya yi da shi ... Don haka mun yi tunani: "Me yasa ba kawai sabunta bayanai a cikin etcd ba, maye gurbin tsoffin adiresoshin IP (subnet) tare da sababbi? "

Bayan neman kayan aikin da aka shirya don aiki tare da bayanai a cikin etcd, ba mu sami wani abu da ya warware matsalar gaba ɗaya ba. (Af, idan kun san game da kowane kayan aiki don aiki tare da bayanai kai tsaye a cikin da dai sauransu, za mu yaba da hanyoyin haɗin gwiwa.) Duk da haka, kyakkyawar farawa shine da dai sauransu daga OpenShift (godiya ga marubutansa!).

Wannan mai amfani zai iya haɗawa zuwa etcd ta amfani da takaddun shaida kuma karanta bayanai daga can ta amfani da umarni ls, get, dump.

Ƙara etcdhelper

Tunani na gaba yana da ma'ana: "Me ke hana ku ƙara wannan kayan aiki ta ƙara ikon rubuta bayanai zuwa da sauransu?"

Ya zama ingantaccen sigar etcdhelper tare da sabbin ayyuka guda biyu changeServiceCIDR и changePodCIDR. a kanta za ka iya ganin code a nan.

Menene sabbin abubuwan ke yi? Algorithm changeServiceCIDR:

  • ƙirƙirar deserializer;
  • tattara magana ta yau da kullun don maye gurbin CIDR;
  • muna tafiya cikin duk ayyuka tare da nau'in ClusterIP a cikin gungu:
    • yanke ƙima daga etcd zuwa abu Go;
    • ta amfani da magana ta yau da kullun muna maye gurbin bytes biyu na farko na adireshin;
    • sanya sabis ɗin adireshin IP daga sabon gidan yanar gizo;
    • ƙirƙirar serializer, canza abin Go zuwa protobuf, rubuta sabbin bayanai zuwa da sauransu.

aiki changePodCIDR kama da gaske changeServiceCIDR - kawai maimakon gyara ƙayyadaddun sabis, muna yin shi don kumburi da canji .spec.PodCIDR zuwa sabon subnet.

Yi aiki

Canza sabis na CDR

Shirin aiwatar da aikin yana da sauƙi, amma ya haɗa da raguwa a lokacin sake ƙirƙirar dukkan kwasfa a cikin tari. Bayan bayyana manyan matakai, za mu kuma raba tunani kan yadda, a ka'idar, za a iya rage girman wannan lokacin.

Matakan shiri:

  • shigar da software da ake buƙata da kuma haɗa facin da sauransu;
  • madadin etcd da /etc/kubernetes.

Takaitaccen tsarin aiki don canza serviceCIDR:

  • canza apiserver da mai sarrafawa-mai sarrafa ya bayyana;
  • sake fitar da takaddun shaida;
  • canza ayyukan ClusterIP a da dai sauransu;
  • sake kunna duk kwas ɗin da ke cikin tari.

Mai zuwa shine cikakken jerin ayyuka daki-daki.

1. Shigar da etcd-client don juji bayanai:

apt install etcd-client

2. Gina da sauransu.

  • Shigar da golan:
    GOPATH=/root/golang
    mkdir -p $GOPATH/local
    curl -sSL https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz | tar -xzvC $GOPATH/local
    echo "export GOPATH="$GOPATH"" >> ~/.bashrc
    echo 'export GOROOT="$GOPATH/local/go"' >> ~/.bashrc
    echo 'export PATH="$PATH:$GOPATH/local/go/bin"' >> ~/.bashrc
  • Muna ajiye wa kanmu etcdhelper.go, zazzage abubuwan dogaro, tattara:
    wget https://raw.githubusercontent.com/flant/examples/master/2020/04-etcdhelper/etcdhelper.go
    go get go.etcd.io/etcd/clientv3 k8s.io/kubectl/pkg/scheme k8s.io/apimachinery/pkg/runtime
    go build -o etcdhelper etcdhelper.go

3. Yi madadin da dai sauransu:

backup_dir=/root/backup
mkdir ${backup_dir}
cp -rL /etc/kubernetes ${backup_dir}
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt --endpoints https://192.168.199.100:2379 snapshot save ${backup_dir}/etcd.snapshot

4. Canja subnet ɗin sabis a cikin jirgin sarrafa Kubernetes yana bayyana. A cikin fayiloli /etc/kubernetes/manifests/kube-apiserver.yaml и /etc/kubernetes/manifests/kube-controller-manager.yaml canza siga --service-cluster-ip-range zuwa sabon subnet: 172.24.0.0/16 maimakon 192.168.0.0/16.

5. Tunda muna canza subnet ɗin sabis wanda kubeadm ke ba da takaddun shaida don apiserver (ciki har da), suna buƙatar sake fitar da su:

  1. Bari mu ga waɗanne yankuna da adiresoshin IP aka ba da takaddun shaida na yanzu don:
    openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
    X509v3 Subject Alternative Name:
        DNS:dev-1-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver, IP Address:192.168.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100
  2. Bari mu shirya ƙaramin tsari don kubeadm:
    cat kubeadm-config.yaml
    apiVersion: kubeadm.k8s.io/v1beta1
    kind: ClusterConfiguration
    networking:
      podSubnet: "10.244.0.0/16"
      serviceSubnet: "172.24.0.0/16"
    apiServer:
      certSANs:
      - "192.168.199.100" # IP-адрес мастер узла
  3. Bari mu share tsohon crt da maɓalli, tunda ba tare da wannan ba za a ba da sabuwar takardar shaidar:
    rm /etc/kubernetes/pki/apiserver.{key,crt}
  4. Bari mu sake fitar da takaddun shaida don uwar garken API:
    kubeadm init phase certs apiserver --config=kubeadm-config.yaml
  5. Bari mu duba cewa an ba da takaddun shaida don sabon gidan yanar gizo:
    openssl x509 -noout -ext subjectAltName </etc/kubernetes/pki/apiserver.crt
    X509v3 Subject Alternative Name:
        DNS:kube-2-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:172.24.0.1, IP Address:10.0.0.163, IP Address:192.168.199.100
  6. Bayan sake fitar da takardar shaidar uwar garken API, sake kunna akwati:
    docker ps | grep k8s_kube-apiserver | awk '{print $1}' | xargs docker restart
  7. Bari mu sake farfado da saitin don admin.conf:
    kubeadm alpha certs renew admin.conf
  8. Bari mu gyara bayanan a cikin etcd:
    ./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-service-cidr 172.24.0.0/16 

    Tsanaki A wannan lokacin, ƙudurin yanki ya daina aiki a cikin gungu, tunda a cikin kwas ɗin da ke akwai /etc/resolv.conf tsohon adireshin CoreDNS (kube-dns) yayi rijista, kuma kube-proxy yana canza ka'idojin iptables daga tsohon gidan yanar gizo zuwa sabon. Bugu da ari a cikin labarin an rubuta game da yiwuwar zažužžukan don rage raguwa.

  9. Bari mu gyara ConfigMap's a cikin filin suna kube-system:
    kubectl -n kube-system edit cm kubelet-config-1.16

    - maye gurbin nan clusterDNS zuwa sabon adireshin IP na sabis na kube-dns: kubectl -n kube-system get svc kube-dns.

    kubectl -n kube-system edit cm kubeadm-config

    - za mu gyara shi data.ClusterConfiguration.networking.serviceSubnet zuwa sabon subnet.

  10. Tunda adireshin kube-dns ya canza, ya zama dole a sabunta kubelet config akan duk nodes:
    kubeadm upgrade node phase kubelet-config && systemctl restart kubelet
  11. Abin da ya rage shi ne sake kunna dukkan kwas ɗin da ke cikin tarin:
    kubectl get pods --no-headers=true --all-namespaces |sed -r 's/(S+)s+(S+).*/kubectl --namespace 1 delete pod 2/e'

Rage raguwar lokaci

Tunani kan yadda ake rage raguwar lokaci:

  1. Bayan canza yanayin jirgin sama mai sarrafawa, ƙirƙirar sabon sabis na kube-dns, misali, tare da sunan kube-dns-tmp da sabon adireshin 172.24.0.10.
  2. Don yin if a etcdhelper, wanda ba zai canza sabis ɗin kube-dns ba.
  3. Sauya adireshin a duk kubelets ClusterDNS zuwa sabon, yayin da tsohon sabis zai ci gaba da aiki lokaci guda tare da sabon.
  4. Jira har sai kwas ɗin tare da aikace-aikace sun mirgine ko dai da kansu don dalilai na halitta ko a lokacin da aka yarda.
  5. Share sabis kube-dns-tmp kuma canza serviceSubnetCIDR don sabis na kube-dns.

Wannan shirin zai ba ku damar rage lokacin raguwa zuwa ~ minti daya - na tsawon lokacin cire sabis ɗin kube-dns-tmp da canza subnet don sabis ɗin kube-dns.

Gyara podNetwork

A lokaci guda, mun yanke shawarar duba yadda ake canza podNetwork ta amfani da sakamakon etcdhelper. Jerin ayyuka kamar haka:

  • gyara saiti a ciki kube-system;
  • gyara kube-mai kula-mai sarrafa bayyana;
  • canza podCIDR kai tsaye a etcd;
  • sake kunna duk nodes na tari.

Yanzu ƙarin game da waɗannan ayyukan:

1. Gyara ConfigMap's a cikin filin suna kube-system:

kubectl -n kube-system edit cm kubeadm-config

- gyarawa data.ClusterConfiguration.networking.podSubnet zuwa sabon subnet 10.55.0.0/16.

kubectl -n kube-system edit cm kube-proxy

- gyarawa data.config.conf.clusterCIDR: 10.55.0.0/16.

2. Gyara bayanan mai sarrafawa-mai gudanarwa:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- gyarawa --cluster-cidr=10.55.0.0/16.

3. Dubi ƙimar halin yanzu .spec.podCIDR, .spec.podCIDRs, .InternalIP, .status.addresses ga dukkan nodes na gungu:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.244.0.0/24",
    "podCIDRs": [
      "10.244.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.244.1.0/24",
    "podCIDRs": [
      "10.244.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

4. Sauya podCIDR ta yin canje-canje kai tsaye zuwa etcd:

./etcdhelper -cacert /etc/kubernetes/pki/etcd/ca.crt -cert /etc/kubernetes/pki/etcd/server.crt -key /etc/kubernetes/pki/etcd/server.key -endpoint https://127.0.0.1:2379 change-pod-cidr 10.55.0.0/16

5. Bari mu duba cewa podCIDR ya canza da gaske:

kubectl get no -o json | jq '[.items[] | {"name": .metadata.name, "podCIDR": .spec.podCIDR, "podCIDRs": .spec.podCIDRs, "InternalIP": (.status.addresses[] | select(.type == "InternalIP") | .address)}]'

[
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "192.168.199.2"
  },
  {
    "name": "kube-2-master",
    "podCIDR": "10.55.0.0/24",
    "podCIDRs": [
      "10.55.0.0/24"
    ],
    "InternalIP": "10.0.1.239"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "192.168.199.222"
  },
  {
    "name": "kube-2-worker-01f438cf-579f9fd987-5l657",
    "podCIDR": "10.55.1.0/24",
    "podCIDRs": [
      "10.55.1.0/24"
    ],
    "InternalIP": "10.0.4.73"
  }
]

6. Bari mu sake yin duk nodes na gungu ɗaya bayan ɗaya.

7. Idan ka bar aƙalla kumburi ɗaya tsohon podCIDR, to kube-controller-manager ba zai iya farawa ba, kuma ba za a tsara kwas ɗin da ke cikin cluster ba.

A zahiri, canza podCIDR za a iya yi har ma mafi sauƙi (misali, haka). Amma muna so mu koyi yadda ake aiki tare da etcd kai tsaye, saboda akwai lokuta lokacin gyara abubuwan Kubernetes a cikin da dai sauransu - kadai mai yiwuwa bambancin. (Misali, ba za ku iya canza filin Sabis kawai ba tare da raguwar lokaci ba spec.clusterIP.)

Sakamakon

Labarin ya tattauna yiwuwar yin aiki tare da bayanai a cikin etcd kai tsaye, watau. Ketare Kubernetes API. Wani lokaci wannan hanyar tana ba ku damar yin "abubuwa masu ban tsoro." Mun gwada ayyukan da aka bayar a cikin rubutu akan gungu na K8s na gaske. Koyaya, matsayinsu na shirye don amfani da yawa shine PoC (tabbacin ra'ayi). Don haka, idan kuna son yin amfani da gyare-gyaren sigar kayan amfani da etcdhelper akan gungu naku, kuyi haka cikin haɗarin ku.

PS

Karanta kuma a kan shafinmu:

source: www.habr.com

Add a comment