Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes
Nkhaniyi ikuthandizani kumvetsetsa momwe kusanja kwa katundu kumagwirira ntchito ku Kubernetes, zomwe zimachitika mukakulitsa kulumikizana kwanthawi yayitali, komanso chifukwa chake muyenera kuganizira za kusanja kwa kasitomala ngati mugwiritsa ntchito HTTP/2, gRPC, RSockets, AMQP, kapena ma protocol ena anthawi yayitali. . 

Zambiri za momwe magalimoto amagawidwiranso ku Kubernetes 

Kubernetes imapereka njira ziwiri zosavuta zopangira ntchito: Ntchito ndi Kutumiza.

Kutumiza kumalongosola momwe ndi kuchuluka kwa makope a pulogalamu yanu yomwe iyenera kuyendetsedwa nthawi iliyonse. Ntchito iliyonse imayikidwa ngati Pod ndipo imapatsidwa adilesi ya IP.

Ntchito zimafanana ndi ntchito yolinganiza katundu. Amapangidwa kuti azigawira kuchuluka kwa magalimoto pamagulu angapo.

Tiyeni tiwone momwe zimawonekera.

  1. Pachithunzi chomwe chili pansipa mutha kuwona zochitika zitatu zakugwiritsa ntchito komweko ndi chowerengera cholemetsa:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  2. The load balancer amatchedwa Service ndipo amapatsidwa adilesi ya IP. Pempho lililonse lomwe likubwera limatumizidwa ku imodzi mwama pod:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  3. Zomwe zimatumizidwa zimatsimikizira kuchuluka kwa zochitika zomwe zimagwiritsidwa ntchito. Simudzafunikanso kukulitsa molunjika pansi:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  4. Pod iliyonse imapatsidwa adilesi yake ya IP:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Ndizothandiza kuganiza za ntchito ngati gulu la ma adilesi a IP. Nthawi iliyonse mukapeza ma adilesi, imodzi mwa ma adilesi a IP imasankhidwa pamndandanda ndikugwiritsidwa ntchito ngati adilesi yolowera.

Zikuwoneka chonchi.

  1. Pempho la curl 10.96.45.152 lalandiridwa ku ntchito:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  2. Ntchitoyi imasankha imodzi mwa ma adilesi atatu ngati kopita:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  3. Magalimoto amatumizidwa kumalo enaake:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Ngati ntchito yanu ili ndi kutsogolo ndi kumbuyo, ndiye kuti mudzakhala ndi ntchito komanso kutumizidwa kwa aliyense.

Pamene kutsogolo kumapanga pempho lakumbuyo, sikuyenera kudziwa ndendende ndi ma pod angati omwe amatumikira: pakhoza kukhala imodzi, khumi, kapena zana.

Komanso, kutsogolo sadziwa kalikonse za maadiresi a pods omwe akutumikira kumbuyo.

Pamene kutsogolo kumapanga pempho ku backend, amagwiritsa ntchito adilesi ya IP ya utumiki wakumbuyo, womwe susintha.

Umu ndi momwe zimawonekera.

  1. Pansi pa 1 imapempha gawo lamkati lakumbuyo. M'malo mosankha ina ya backend, imapanga pempho ku utumiki:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  2. Ntchitoyi imasankha imodzi mwamagawo akumbuyo ngati adilesi yopitira:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  3. Magalimoto amachoka ku Pod 1 kupita ku Pod 5, osankhidwa ndi ntchito:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  4. Pansi pa 1 sindikudziwa ndendende kuti ndi ma pod angati omwe ali pansi pa 5 omwe amabisika kuseri kwa ntchitoyi:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Koma kodi ntchitoyi imagawa bwanji zopempha? Zikuwoneka ngati kusanja kwa robin kozungulira kumagwiritsidwa ntchito? Tiyeni tiganizire. 

Kusinthanitsa mu ntchito za Kubernetes

Ntchito za Kubernetes palibe. Palibe ndondomeko ya ntchito yomwe imapatsidwa adilesi ya IP ndi doko.

Mutha kutsimikizira izi polowa mu node iliyonse pagulu ndikuyendetsa netstat -ntlp command.

Simungathe ngakhale kupeza adilesi ya IP yoperekedwa ku ntchitoyo.

Adilesi ya IP ya ntchitoyo ili mugawo lowongolera, mu owongolera, ndikujambulidwa mu database - etcd. Adilesi yomweyi imagwiritsidwa ntchito ndi gawo lina - kube-proxy.
Kube-proxy amalandira mndandanda wa ma adilesi a IP a mautumiki onse ndikupanga malamulo a iptables pa node iliyonse mumagulu.

Malamulowa amati: "Ngati tiwona adilesi ya IP ya ntchitoyo, tifunika kusintha adilesi yofunsira ndikuitumiza ku imodzi mwamagawo."

Adilesi ya IP yautumiki imagwiritsidwa ntchito ngati polowera ndipo sichimaperekedwa ndi njira iliyonse yomvera adilesi ya IP ndi doko.

Tiyeni tione izi

  1. Taganizirani gulu la mfundo zitatu. Node iliyonse ili ndi ma pod:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  2. Madontho omangidwa opaka utoto wa beige ndi gawo lautumiki. Chifukwa ntchitoyo kulibe ngati njira, ikuwonetsedwa mu imvi:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  3. Poda yoyamba imapempha ntchito ndipo ikuyenera kupita ku imodzi mwamapodi ogwirizana nawo:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  4. Koma utumiki kulibe, ndondomeko kulibe. Zimagwira ntchito bwanji?

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  5. Pempho lisanachoke pa node, limadutsa malamulo a iptables:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  6. Malamulo a iptables amadziwa kuti ntchitoyi kulibe ndikulowetsa adilesi yake ya IP ndi imodzi mwa ma adilesi a IP a ma pods okhudzana ndi ntchitoyi:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  7. Pempho limalandira adilesi yovomerezeka ya IP monga adilesi yopitira ndipo imakonzedwa bwino:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  8. Kutengera ndi topology ya netiweki, pempholi pamapeto pake limafika poda:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Kodi ma iptables amatha kutsitsa bwino?

Ayi, ma iptables amagwiritsidwa ntchito posefa ndipo sanapangidwe kuti azilinganiza.

Komabe, ndizotheka kulemba malamulo omwe amagwira ntchito ngati pseudo-balancer.

Ndipo izi ndizomwe zimakhazikitsidwa ku Kubernetes.

Ngati muli ndi ma pod atatu, kube-proxy adzalemba malamulo awa:

  1. Sankhani gawo loyamba lokhala ndi mwayi wa 33%, apo ayi pitani ku lamulo lotsatira.
  2. Sankhani yachiwiri ndi mwayi wa 50%, mwinamwake pitani ku lamulo lotsatira.
  3. Sankhani lachitatu pansi.

Dongosololi limapangitsa kuti pod iliyonse isankhidwe ndikuthekera kwa 33%.

Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Ndipo palibe chitsimikizo kuti Pod 2 idzasankhidwa pambuyo pa Pod 1.

ndemanga: iptables amagwiritsa ntchito gawo lowerengera ndi kugawa mwachisawawa. Chifukwa chake, algorithm yofananira imatengera kusankha mwachisawawa.

Tsopano popeza mwamvetsetsa momwe mautumiki amagwirira ntchito, tiyeni tiwone zochitika zosangalatsa zautumiki.

Kulumikizana kwanthawi yayitali ku Kubernetes sikumakula mwachisawawa

Pempho lililonse la HTTP kuchokera kutsogolo kupita ku backend limatumizidwa ndi kugwirizana kosiyana kwa TCP, komwe kumatsegulidwa ndi kutsekedwa.

Ngati kutsogolo kumatumiza zopempha za 100 pa sekondi iliyonse ku backend, ndiye kuti 100 yolumikizana ndi TCP yosiyana imatsegulidwa ndikutsekedwa.

Mutha kuchepetsa nthawi yofunsira ndikutsegula potsegula kulumikizana kumodzi kwa TCP ndikuigwiritsa ntchito pazofunsira zonse za HTTP.

Protocol ya HTTP ili ndi gawo lotchedwa HTTP keep-live, kapena kugwiritsanso ntchito kulumikizana. Pankhaniyi, kulumikizana kumodzi kwa TCP kumagwiritsidwa ntchito kutumiza ndi kulandira zopempha ndi mayankho angapo a HTTP:

Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Izi sizimathandizidwa mwachisawawa: seva ndi kasitomala ziyenera kukonzedwa moyenera.

Kukhazikitsa komweko ndikosavuta komanso kupezeka m'zilankhulo zambiri zamapulogalamu ndi malo.

Nawa maulalo a zitsanzo m'zilankhulo zosiyanasiyana:

Kodi chimachitika ndi chiyani ngati tigwiritsa ntchito kusunga moyo mu ntchito ya Kubernetes?
Tiyerekeze kuti kutsogolo ndi kumbuyo kumathandizira kukhalabe ndi moyo.

Tili ndi kopi imodzi ya frontend ndi makope atatu a backend. Kutsogolo kumapanga pempho loyamba ndikutsegula kulumikiza kwa TCP ku backend. Pempho likufika pautumiki, imodzi mwa mapepala akumbuyo amasankhidwa ngati adilesi yopita. The backend imatumiza yankho, ndipo frontend amalandira izo.

Mosiyana ndi momwe zimakhalira nthawi zonse pomwe kulumikizana kwa TCP kumatsekedwa atalandira yankho, tsopano kumatsegulidwa kuti apemphe zina za HTTP.

Kodi chimachitika ndi chiyani ngati kutsogolo kumatumiza zopempha zambiri ku backend?

Kuti mutumize zopemphazi, kugwirizana kotseguka kwa TCP kudzagwiritsidwa ntchito, zopempha zonse zidzapita kumbuyo komweko komwe pempho loyamba linapita.

Kodi iptables sayenera kugawanso magalimoto?

Osati mu nkhani iyi.

Pamene kugwirizana kwa TCP kupangidwa, kumadutsa mu malamulo a iptables, omwe amasankha kumbuyo kwapadera komwe magalimoto adzapita.

Popeza zopempha zonse zotsatila zili pa mgwirizano wotseguka wa TCP, malamulo a iptables sakutchedwanso.

Tiyeni tiwone momwe zimawonekera.

  1. Gulu loyamba limatumiza pempho ku utumiki:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  2. Mukudziwa kale zomwe zidzachitike. Ntchitoyi kulibe, koma pali malamulo a iptables omwe angagwire ntchito:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  3. Imodzi mwamapod akumbuyo idzasankhidwa ngati adilesi yopita:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  4. Pempho lifika podi. Pakadali pano, kulumikizana kosalekeza kwa TCP pakati pa ma pod awiriwa kudzakhazikitsidwa:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  5. Pempho lililonse lotsatira kuchokera ku pod yoyamba lidzadutsa mulumikizidwe womwe wakhazikitsidwa kale:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Zotsatira zake ndi nthawi yoyankha mwachangu komanso kutulutsa kwapamwamba, koma mumataya kuthekera kokweza kumbuyo.

Ngakhale mutakhala ndi ma pod awiri kumbuyo, ndi kulumikizana kosalekeza, magalimoto amapita kwa amodzi mwa iwo nthawi zonse.

Kodi izi zingakonzeke?

Popeza Kubernetes sadziwa momwe angagwiritsire ntchito kulumikizana kosalekeza, ntchitoyi igwera kwa inu.

Ntchito ndi gulu la ma adilesi a IP ndi madoko otchedwa endpoints.

Pulogalamu yanu imatha kupeza mndandanda wazomwe zimathera pautumiki ndikusankha momwe mungagawire zopempha pakati pawo. Mutha kutsegulira kulumikizana kosalekeza kumtundu uliwonse ndikufunsira zoyeserera pakati pa maulumikizidwewa pogwiritsa ntchito robin yozungulira.

Kapena funsani zambiri zovuta kusanja ma aligorivimu.

Khodi ya mbali ya kasitomala yomwe imayang'anira kusanja iyenera kutsatira mfundo iyi:

  1. Pezani mndandanda wazomaliza kuchokera muutumiki.
  2. Tsegulani kugwirizana kosalekeza kwa mapeto aliwonse.
  3. Pamene pempho likufunika kupangidwa, gwiritsani ntchito imodzi mwamalumikizidwe otseguka.
  4. Sinthani pafupipafupi mndandanda wazomaliza, pangani zatsopano kapena kutseka zolumikizira zakale ngati mndandanda wasintha.

Izi ndi momwe zidzawonekere.

  1. M'malo mwa pod yoyamba kutumiza pempho ku ntchito, mutha kulinganiza zopempha kumbali ya kasitomala:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  2. Muyenera kulemba khodi yomwe imafunsa kuti ndi ma pod omwe ali gawo la ntchitoyo:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  3. Mukakhala ndi mndandanda, sungani kumbali ya kasitomala ndikugwiritsa ntchito kuti mulumikizane ndi ma pod:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

  4. Muli ndi udindo pa algorithm yoyezera katundu:

    Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Tsopano funso likubuka: kodi vutoli limagwira ntchito pa HTTP kukhalabe wamoyo?

Kusanja katundu kumbali ya kasitomala

HTTP si protocol yokhayo yomwe ingagwiritse ntchito maulumikizidwe osalekeza a TCP.

Ngati ntchito yanu ikugwiritsa ntchito nkhokwe, ndiye kuti kulumikizidwa kwa TCP sikutsegulidwa nthawi iliyonse yomwe mukufuna pempho kapena kupeza chikalata kuchokera ku database. 

M'malo mwake, kulumikizana kosalekeza kwa TCP ku database kumatsegulidwa ndikugwiritsidwa ntchito.

Ngati nkhokwe yanu yasungidwa pa Kubernetes ndipo mwayi umaperekedwa ngati ntchito, ndiye kuti mudzakumana ndi mavuto omwe afotokozedwa m'gawo lapitalo.

Chifaniziro chimodzi cha database chidzadzaza kwambiri kuposa enawo. Kube-proxy ndi Kubernetes sizingathandize kulumikizana bwino. Muyenera kusamala kuti musamalire mafunso ku database yanu.

Kutengera laibulale yomwe mumagwiritsa ntchito kuti mulumikizane ndi database, mutha kukhala ndi zosankha zosiyanasiyana zothetsera vutoli.

Pansipa pali chitsanzo chofikira gulu la database la MySQL kuchokera ku Node.js:

var mysql = require('mysql');
var poolCluster = mysql.createPoolCluster();

var endpoints = /* retrieve endpoints from the Service */

for (var [index, endpoint] of endpoints) {
  poolCluster.add(`mysql-replica-${index}`, endpoint);
}

// Make queries to the clustered MySQL database

Pali ma protocol ena ambiri omwe amagwiritsa ntchito kulumikizana kwa TCP kosalekeza:

  • WebSockets ndi ma WebSockets otetezedwa
  • HTTP / 2
  • gRPC
  • RSockets
  • Zamgululi

Muyenera kudziwa zambiri za ma protocol awa.

Koma ngati ma protocol awa ndi otchuka kwambiri, chifukwa chiyani palibe njira yofananira yofananira? Chifukwa chiyani malingaliro a kasitomala akufunika kusintha? Kodi pali yankho lakwawo la Kubernetes?

Kube-proxy ndi iptables adapangidwa kuti azigwira ntchito zomwe zimagwiritsidwa ntchito nthawi zambiri potumiza ku Kubernetes. Izi ndizosavuta.

Ngati mukugwiritsa ntchito intaneti yomwe imawulula REST API, muli ndi mwayi - pakadali pano, kulumikizana kosalekeza kwa TCP sikugwiritsidwa ntchito, mutha kugwiritsa ntchito ntchito iliyonse ya Kubernetes.

Koma mukangoyamba kugwiritsa ntchito maulumikizidwe osalekeza a TCP, muyenera kudziwa momwe mungagawire katunduyo molingana ndi ma backends. Kubernetes ilibe mayankho okonzeka pankhaniyi.

Komabe, pali njira zingapo zomwe zingathandize.

Kulinganiza maulalo omwe akhalapo nthawi yayitali ku Kubernetes

Pali mitundu inayi ya mautumiki ku Kubernetes:

  1. ClusterIP
  2. NodePort
  3. LoadBalancer
  4. Popanda mutu

Ntchito zitatu zoyambirira zimagwira ntchito motengera adilesi ya IP, yomwe imagwiritsidwa ntchito ndi kube-proxy kupanga malamulo a iptables. Koma maziko ofunikira a mautumiki onse ndi ntchito yopanda mutu.

Ntchito yopanda mutu ilibe adilesi ya IP yolumikizidwa nayo ndipo imangopereka njira yopezeranso mndandanda wa ma adilesi a IP ndi madoko a ma pod (mapeto) okhudzana nawo.

Ntchito zonse zimatengera ntchito yopanda mutu.

Ntchito ya ClusterIP ndi ntchito yopanda mutu yokhala ndi zowonjezera zina: 

  1. Kasamalidwe kagawo kamapereka adilesi ya IP.
  2. Kube-proxy imapanga malamulo oyenerera a iptables.

Mwanjira iyi mutha kunyalanyaza kube-proxy ndikugwiritsa ntchito mwachindunji mndandanda wazomaliza zomwe mwapeza kuchokera ku ntchito yopanda mutu kuti muyike bwino ntchito yanu.

Koma tingawonjezere bwanji malingaliro ofanana ndi mapulogalamu onse omwe ayikidwa mumagulu?

Ngati ntchito yanu yatumizidwa kale, ntchitoyi ingawoneke ngati yosatheka. Komabe, pali njira ina.

Service Mesh ikuthandizani

Inu mwina kale anazindikira kuti kasitomala-mbali katundu kusanja njira ndithu muyezo.

Ntchito ikayamba, izi:

  1. Imapeza mndandanda wama adilesi a IP kuchokera muutumiki.
  2. Amatsegula ndi kusunga malo olumikizirana.
  3. Nthawi ndi nthawi imasintha dziwe powonjezera kapena kuchotsa zomaliza.

Pomwe pulogalamuyo ikufuna kupanga pempho, izi:

  1. Imasankha kulumikizana komwe kulipo pogwiritsa ntchito malingaliro (monga round-robin).
  2. Ikuchita pempho.

Masitepewa amagwira ntchito pamalumikizidwe onse a WebSockets, gRPC, ndi AMQP.

Mutha kulekanitsa mfundo iyi kukhala laibulale yosiyana ndikuigwiritsa ntchito mu mapulogalamu anu.

Komabe, mutha kugwiritsa ntchito ma meshes ngati Istio kapena Linkerd m'malo mwake.

Service Mesh imawonjezera ntchito yanu ndi njira yomwe:

  1. Amasaka ma adilesi a IP a ntchito.
  2. Imayesa kulumikizana monga WebSockets ndi gRPC.
  3. Kulinganiza zopempha pogwiritsa ntchito protocol yolondola.

Service Mesh imathandizira kuyang'anira kuchuluka kwa magalimoto mkati mwa tsango, koma ndiyofunika kwambiri. Zosankha zina zikugwiritsa ntchito malaibulale a chipani chachitatu ngati Netflix Ribbon kapena ma proxies osinthika ngati Envoy.

Kodi chimachitika ndi chiyani ngati munyalanyaza zovuta za kusanja?

Mutha kusankha kusagwiritsa ntchito kusanja katundu ndipo osazindikira kusintha kulikonse. Tiyeni tiwone zochitika zingapo zantchito.

Ngati muli ndi makasitomala ambiri kuposa ma seva, ili si vuto lalikulu.

Tiyerekeze kuti pali makasitomala asanu omwe amalumikizana ndi ma seva awiri. Ngakhale ngati palibe kulinganiza, ma seva onsewa adzagwiritsidwa ntchito:

Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Malumikizidwe sangagawidwe mofanana: mwina makasitomala anayi olumikizidwa ku seva yomweyo, koma pali mwayi woti ma seva onsewa agwiritsidwe ntchito.

Chomwe chavuta kwambiri ndi chosiyana.

Ngati muli ndi makasitomala ochepera komanso ma seva ochulukirapo, zinthu zanu zitha kugwiritsidwa ntchito mochepera ndipo vuto lomwe lingakhalepo lidzawonekera.

Tinene kuti pali makasitomala awiri ndi ma seva asanu. Pabwino kwambiri, padzakhala zolumikizira ziwiri zokhazikika kwa ma seva awiri mwa asanu.

Ma seva otsalawo adzakhala opanda ntchito:

Kwezani kusanja ndikukulitsa kulumikizana kwanthawi yayitali ku Kubernetes

Ngati ma seva awiriwa sangathe kuthana ndi zopempha zamakasitomala, kukweza kopingasa sikungathandize.

Pomaliza

Ntchito za Kubernetes zidapangidwa kuti zizigwira ntchito m'mawonekedwe ambiri ogwiritsira ntchito intaneti.

Komabe, mukangoyamba kugwira ntchito ndi ma protocol omwe amagwiritsa ntchito kulumikizana kwa TCP kosalekeza, monga nkhokwe, gRPC kapena WebSockets, mautumiki sakhalanso oyenera. Kubernetes sapereka njira zamkati zolumikizira kulumikizana kosalekeza kwa TCP.

Izi zikutanthauza kuti muyenera kulemba mapulogalamu poganizira za kasitomala.

Kumasulira kokonzedwa ndi gulu Kubernetes aaS kuchokera ku Mail.ru.

Chinanso choti muwerenge pamutuwu:

  1. Miyezo itatu ya autoscaling ku Kubernetes ndi momwe mungagwiritsire ntchito bwino
  2. Kubernetes mu mzimu wa piracy wokhala ndi template yokhazikitsidwa.
  3. Njira yathu ya Telegraph yokhudza kusintha kwa digito.

Source: www.habr.com

Kuwonjezera ndemanga