Phawula. transl.: Ababhali beli nqaku ziinjineli ezivela kwinkampani encinci yaseCzech, i-pipetail. Bakwazile ukuhlanganisa uluhlu olumangalisayo [ngamanye amaxesha banal, kodwa kunjalo] iingxaki ezicinezela kakhulu kunye neengcamango eziphosakeleyo ezinxulumene nokusebenza kwamaqela e-Kubernetes.

Ukutyhubela iminyaka yokusebenzisa i-Kubernetes, siye sasebenza kunye nenani elikhulu lamaqela (zombini zilawulwa kwaye zingalawulwa - kwi-GCP, i-AWS kunye ne-Azure). Ekuhambeni kwexesha, saqalisa ukuphawula ukuba ezinye iimpazamo zazisoloko ziphindaphindwa. Nangona kunjalo, akukho ntloni kule nto: uninzi lwazo sizenzele ngokwethu!
Inqaku liqulathe ezona mpazamo zixhaphakileyo kwaye likwakhankanya nendlela yokuzilungisa.
1. Izibonelelo: izicelo kunye nemida
Le nto ngokuqinisekileyo ifanelwe ingqalelo esondeleyo kunye nendawo yokuqala kuluhlu.
Isicelo seCPU ngokuqhelekileyo mhlawumbi ayichazwanga kwaphela okanye inexabiso eliphantsi kakhulu (ukubeka iipod ezininzi kwindawo nganye kangangoko kunokwenzeka). Ngaloo ndlela, ii-nodes ziba nzima kakhulu. Ngexesha lomthwalo ophezulu, amandla okusebenza e-node asetyenziswa ngokupheleleyo kwaye umthwalo othile womsebenzi ufumana kuphela oko "ubucelile" CPU throttling. Oku kukhokelela ekwandeni kokubambezeleka kwesicelo, ukuphuma kwexesha, kunye nezinye iziphumo ezingathandekiyo. (Funda ngakumbi malunga noku kwenye inguqulelo yethu yamva nje: “"- malunga. guqulela.)
BestEffort (kakhulu hayi kucetyiswa):
resources: {}Isicelo esisezantsi kakhulu seCPU (kakhulu hayi kucetyiswa):
resources:
Requests:
cpu: "1m"Kwelinye icala, ubukho bomda we-CPU bunokukhokelela ekutsibeni okungenangqiqo kwimijikelo yewotshi ngeepod, nokuba iprosesa ye-node ayilayishwanga ngokupheleleyo. Kwakhona, oku kunokukhokelela ekulibazisekeni okwandisiweyo. Ingxabano iyaqhubeka malunga nepharamitha CPU CFS quota embindini Linux Ukuxinana kweCPU ngokuxhomekeke kwimida ebekiweyo, kunye nokukhubaza i-CFS quota... Ngelishwa, imida yeCPU inokubangela iingxaki ezingaphezulu kunezo zisombululwayo. Ungafunda okungakumbi ngale nto kwikhonkco elingezantsi.
Ukukhetha ngokugqithisileyo (ukuzinikela kakhulu) iingxaki zememori zingakhokelela kwiingxaki ezinkulu. Ukufikelela kumda we-CPU kubandakanya ukutsiba imijikelo yewotshi, ngelixa ukufikelela kumda wememori kubandakanya ukubulala i-pod. Ngaba ukhe waphawula OOMkill? Ewe, yiloo nto kanye sithetha ngayo.
Ngaba uyafuna ukunciphisa amathuba okuba kwenzeke? Musa ukwaba ngokugqithisileyo imemori kwaye usebenzise i-Guaranteed QoS (Umgangatho weNkonzo) ngokucwangcisa isicelo sememori kumda (njengoko kumzekelo ongezantsi). Funda ngakumbi ngale nto kwi (Injineli ekhokelayo eZalando).
Ukuqhawuka (amathuba aphezulu okufumana i-OOMkilled):
resources:
requests:
memory: "128Mi"
cpu: "500m"
limits:
memory: "256Mi"
cpu: 2Qi nyiweyo:
resources:
requests:
memory: "128Mi"
cpu: 2
limits:
memory: "128Mi"
cpu: 2Yintoni enokuba luncedo xa kumiselwa izixhobo?
Ngo kunceda i-metrics-server ungabona ukusetyenziswa kwe-CPU yangoku kunye nokusetyenziswa kwememori ngee-pods (kunye nezikhongozeli ezingaphakathi kuzo). Ngokunokwenzeka, sele uyisebenzisa. Qhuba nje le miyalelo ilandelayo:
kubectl top pods
kubectl top pods --containers
kubectl top nodesNangona kunjalo, zibonisa kuphela ukusetyenziswa kwangoku. Inokukunika umbono orhabaxa womyalelo wobungakanani, kodwa ekugqibeleni uya kuyidinga imbali yotshintsho kwiimethrikhi ngokuhamba kwexesha (ukuphendula imibuzo efana nale: "Ibiyintoni incopho yomthwalo we-CPU?", "Ibiyintoni umthwalo izolo ekuseni?", njl.). Kule nto ungasebenzisa Prometheus, DataDog kunye nezinye izixhobo. Bafumana ngokulula iimethrikhi kwi-metrics-server kwaye bazigcine, kwaye umsebenzisi unokuzibuza kwaye azicwangcise ngokufanelekileyo.
it ivumela izenzekela le nkqubo. Ilandelela i-CPU kunye nembali yokusetyenziswa kwememori kwaye ibeka izicelo ezintsha kunye nemida esekelwe kule ngcaciso.
Ukusebenzisa amandla ekhompyutha ngokufanelekileyo akungomsebenzi olula. Kufana nokudlala iTetris ngalo lonke ixesha. Ukuba uhlawula kakhulu umbane wokubala ngokusetyenziswa komndilili ophantsi (yithi ~ 10%), sincoma ukujonga iimveliso ezisekelwe kwi-AWS Fargate okanye kwi-Virtual Kubelet. Zakhelwe kwimodeli yentlawulo yokusetyenziswa kwe-serverless/intlawulo-nganye, enokuthi ibonakale inexabiso eliphantsi kwiimeko ezinjalo.
2. Ubomi kunye nokulungela ukuphonononga
Ngokungagqibekanga, ukusebenza kunye nokulungela ukujonga akwenziwa ukuba kusebenze kwi-Kubernetes. Kwaye ngamanye amaxesha bayalibala ukuwavula...
Kodwa ungayiqalisa njani enye inkonzo kwakhona xa kukho impazamo ebulalayo? Kwaye i-balancer yomthwalo yazi njani ukuba i-pod ikulungele ukwamkela i-traffic? Okanye ukuba inokusingatha i-traffic eninzi?
Olu vavanyo luhlala lubhidaniswa enye kwenye:
- Ukuphila — ukukhangela "ukusinda", eqala kwakhona i-pod ukuba iyasilela;
- Ukulungela — ukujonga ukulungela, ukuba kuyasilela, kuqhawula umdibaniso wepod kwinkonzo ye Kubernetes (oku kunokujongwa kusetyenziswa
kubectl get endpoints) kwaye i-traffic ayifiki kuyo de itsheki elandelayo igqitywe ngempumelelo.
Zombini ezi itshekhi OWENZIWA NGEXESHA LONKE LOKUJIKELEKA KOBOMI BEPOD. Ibaluleke kakhulu.
Ingcamango ephosakeleyo eqhelekileyo kukuba i-probes yokulungela iqhutywe kuphela ekuqaleni ukuze i-balancer ikwazi ukuba i-pod ilungile (Ready) kwaye inokuqalisa ukusetyenzwa kwetrafikhi. Nangona kunjalo, oku kuphela kolunye ukhetho lokusetyenziswa kwabo.
Enye inokwenzeka yokufumanisa ukuba i-traffic kwi-pod igqithise kwaye ilayisha kakhulu (okanye i-pod yenza izibalo ezinzulu kwimithombo). Kule meko, ukujonga ukulungela kunceda ukunciphisa umthwalo kwi-pod kwaye "uyipholise".. Ukugqitywa ngempumelelo kokuhlolwa kokulungela kwixa elizayo kuyakuvumela ukwandisa umthwalo kwi-pod kwakhona. Kule meko (ukuba uvavanyo lokulungela aluphumeleli), ukungaphumeleli kuvavanyo lwe-liveness kuya kuba nemveliso kakhulu. Kutheni uphinda uqalise i-pod esempilweni kwaye isebenza nzima?
Ke ngoko, kwezinye iimeko, akukho zitsheki konke kungcono kunokubavumela ngeeparamitha eziqwalaselwe ngokungachanekanga. Njengoko kuchaziwe ngasentla, ukuba liveness khangela iikopi ukulungela ukujonga, ngoko usengxakini enkulu. Ukhetho olunokwenzeka kukuqwalasela , kwaye shiya ecaleni.
Zombini iindidi zokutshekisha akufuneki zisilele xa ukuxhomekeka okuqhelekileyo kungaphumeleli, kungenjalo oku kuya kukhokelela ekungaphumelelini kwe-cascading (i-avalanche-like) yazo zonke iipods. Ngamanye amazwi, .
3. LoadBalancer kwinkonzo nganye yeHTTP
Ngokunokwenzeka, uneenkonzo ze-HTTP kwiqela lakho ongathanda ukuzithumela kwihlabathi langaphandle.
Ukuba uvula inkonzo njenge type: LoadBalancer, umlawuli wayo (kuxhomekeke kumnikezeli wenkonzo) uya kubonelela kwaye axoxisane ne-LoadBalancer yangaphandle (engaqhubekiyo kwi-L7, kodwa kunoko nakwi-L4), kwaye oku kunokuchaphazela iindleko (idilesi ye-static IPv4 yangaphandle, amandla e-computing, i-bill-second billing. ) ngenxa yesidingo sokudala inani elikhulu lezixhobo ezinjalo.
Kule meko, kunengqiqo ngakumbi ukusebenzisa i-balancer yomthwalo wangaphandle, ukuvula iinkonzo njenge type: NodePort. Okanye ngcono noko, yandisa into enje nginx-ingress-controller (okanye Umzila), oya kuba yedwa I-NodePort isiphelo esinxulunyaniswa nesilinganisi somthwalo wangaphandle kwaye siya kuthungatha izithuthi kwiqela usebenzisa ingress-Kubernetes izixhobo.
Ezinye iinkonzo ze-intra-cluster (micro) ezinxibelelana enye kwenye zino “kunxibelelana” zisebenzisa iinkonzo ezifana ClusterIP kunye nendlela yokufumanisa inkonzo eyakhelwe-ngaphakathi nge-DNS. Musa nje ukusebenzisa i-DNS / IP yabo kawonkewonke, njengoko oku kunokuchaphazela ukubambezeleka kunye nokunyusa iindleko zeenkonzo zefu.
4. I-autoscaling iqoqo ngaphandle kokuqwalasela iimpawu zayo
Xa usongeza ii-nodes kwaye uzisusa kwiqela, akufuneki uxhomekeke kwezinye iimetriki ezisisiseko ezifana nokusetyenziswa kwe-CPU kwezo ndawo. Ukucwangcisa iPod kufuneka kuthathele ingqalelo uninzi izithintelo, ezifana ne-pod / node affinity, ukungcola kunye nokunyamezela, izicelo zezibonelelo, i-QoS, njl. Ukusebenzisa i-autoscaler yangaphandle engathathi le miba ingakhokelela kwiingxaki.
Khawucinge ukuba i-pod ethile kufuneka icwangciswe, kodwa onke amandla akhoyo e-CPU aceliwe / achithwa kwaye i-pod uxinga ebumeni Pending. I-autoscaler yangaphandle ibona umndilili womthwalo we-CPU wangoku (hayi lowo uceliweyo) kwaye ayiqali ukwanda. (isikali) - ayongezi enye i-node. Ngenxa yoko, le pod ayiyi kucwangciswa.
Kule meko, ukubuyisela umva ukulinganisa (isikali) - ukususa i-node kwi-cluster kuhlala kunzima ukuphumeza. Khawufane ucinge ukuba unepod esemgangathweni (enokugcinwa okuzingileyo okuqhagamshelweyo). Imiqulu eqhubekayo ngokuqhelekileyo yeya indawo yokufumaneka ethile kwaye aziphindaphindwa kwingingqi. Ngaloo ndlela, ukuba i-autoscaler yangaphandle icima i-node kunye nale pod, umcwangcisi akayi kukwazi ukucwangcisa le pod kwenye i-node, ekubeni oku kunokwenziwa kuphela kwindawo yokufumaneka apho ukugcinwa okuzingileyo kukho. I-pod iya kubambelela kurhulumente Pending.
Idume kakhulu kuluntu lwaseKubernetes . Isebenza kwiqela, isekela i-APIs evela kubaboneleli befu abakhulu, ithathela ingqalelo yonke imiqobo kwaye inokulinganisa kwiimeko ezingentla. Iyakwazi kwakhona ukukala ngelixa igcina yonke imida ebekiweyo, ngaloo ndlela igcina imali (enokuthi ngenye indlela isetyenziswe kumthamo ongasetyenziswanga).
5. Ukungahoywa kwezakhono ze-IAM/RBAC
Kulumkele ukusebenzisa abasebenzisi be-IAM abaneemfihlo eziqhubekayo ze oomatshini kunye nezicelo. Lungiselela ukufikelela okwethutyana usebenzisa iindima kunye neeakhawunti zenkonzo (iiakhawunti zenkonzo).
Sihlala sidibana nenyaniso yokuba izitshixo zokufikelela (kunye neemfihlo) zifakwe kwi-hardcode kuqwalaselo lwesicelo, kunye nokutyeshela ukujikeleza kweemfihlo nangona ukufikelela kwi-Cloud IAM. Sebenzisa iindima ze-IAM kunye neeakhawunti zenkonzo endaweni yabasebenzisi apho kufanelekileyo.

Ulibale nge-kube2iam kwaye uye ngqo kwiindima ze-IAM kwiiakhawunti zenkonzo (njengoko kuchaziwe kwi Štěpán Vraný):
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role
name: my-serviceaccount
namespace: defaultInkcazo enye. Akunzima kangako, akunjalo?
Kwakhona, musa ukunika ii-akhawunti zenkonzo kunye namalungelo eprofayili umzekelo admin и cluster-adminukuba abafuni. Oku kunzima ngakumbi ukuphumeza, ngakumbi kwi-RBAC K8s, kodwa ngokuqinisekileyo kuwufanele umzamo.
6. Musa ukuxhomekeka kwi-anti-affinity ezenzekelayo kwiipods
Khawufane ucinge ukuba uneekopi ezithathu zokusasazwa kwindawo ethile. I-node iyawa, kwaye kunye nayo yonke i-replicas. Imeko engathandekiyo, akunjalo? Kodwa kwakutheni ukuze zonke iikopi zikwindawo enye? Ngaba iKubernetes ayifanelanga ukuba ibonelele ngokufumaneka okuphezulu (HA)?!
Ngelishwa, umcwangcisi we-Kubernetes, ngokwawo, akahambelani nemithetho yobukho obahlukeneyo (anti-affinity) ukwenzela imidumba. Kufuneka zichazwe ngokucacileyo:
// опущено для краткости
labels:
app: zk
// опущено для краткости
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname" Kuko konke. Ngoku iipods ziya kucwangciswa kwiindawo ezahlukeneyo (le meko ijongwa kuphela ngexesha lokucwangciswa, kodwa hayi ngexesha lokusebenza kwabo - ngenxa yoko requiredDuringSchedulingIgnoredDuringExecution).
Apha sithetha ngayo podAntiAffinity kwiindawo ezahlukeneyo: topologyKey: "kubernetes.io/hostname", - kwaye hayi malunga neendawo ezahlukeneyo zokufumaneka. Ukuphumeza i-HA epheleleyo, kuya kufuneka umbe ngokunzulu kwesi sihloko.
7. Ukungahoyi iPodDisruptionBudgets
Khawucinge ukuba unomthwalo wemveliso kwiqela le-Kubernetes. Ngamaxesha athile, iindawo zokuhlala kunye neqela ngokwalo kufuneka zihlaziywe (okanye zipheliswe). I-PodDisruptionBudget (PDB) yinto efana nesivumelwano sesiqinisekiso senkonzo phakathi kwabalawuli beqela kunye nabasebenzisi.
I-PDB ikuvumela ukuba uphephe ukuphazamiseka kwenkonzo okubangelwa kukunqongophala kweendawo:
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: zookeeperKulo mzekelo, wena, njengomsebenzisi weqela, uthi kubaphathi: "Heyi, ndinenkonzo yomgcini-zilwanyana, kwaye nokuba wenza ntoni na, ndingathanda ukuba neekopi ezi-2 zale nkonzo ezihlala zikhona."
Unokufunda ngakumbi malunga noku .
8. Abasebenzisi abaninzi okanye iindawo ezisingqongileyo kwiqela elinye
Kubernetes izithuba zamagama (izithuba zamagama) musa ukubonelela nge-insulation enamandla.
Umbono ongalunganga oqhelekileyo kukuba, ukuba ubeka umthwalo ongeyomveliso kwindawo yegama elinye kunye nomthwalo weprod kwenye, ke baya aziyi kuphembelelana nangayiphi na indlela... Nangona kunjalo, umgangatho othile wokuzibekela bucala unokufikelelwa kusetyenziswa izicelo zezibonelelo/imida, ukusetwa kwezabelo, kunye nokuseta iiDidi eziphambili. Ukwahlukaniswa "okwenyama" kwinqwelomoya yedatha kubonelelwa ngezinto ezihambelanayo, ukunyamezela, ukungcola (okanye ii-nodeselectors), kodwa ukwahlukana okunjalo kulungile. kunzima phumeza.
Abo bafuna ukudibanisa zombini iintlobo zemithwalo yomsebenzi kwiqela elifanayo baya kufuneka bajongane nobunzima. Ukuba akukho mfuneko enjalo, kwaye unokukwazi ukufumana enye elinye iqela ngaphezulu (yithi, kwilifu likawonkewonke), ke kungcono ukwenza njalo. Oku kuya kufikelela kwinqanaba eliphezulu kakhulu lokugquma.
9. Umgaqo-nkqubo weTrafikhi wangaphandle: Iqela
Ngokuqhelekileyo sibona ukuba zonke izithuthi ngaphakathi kweqela ziza ngenkonzo efana neNodePort, apho umgaqo-nkqubo ongagqibekanga usetiwe. externalTrafficPolicy: Cluster... Ithetha ukuba I-NodePort ivuliwe kuyo yonke i-node kwiqela, kwaye ungasebenzisa nayiphi na kuzo ukusebenzisana nenkonzo efunekayo (iseti yeepods).

Kwangaxeshanye, iipods zokwenyani ezinxulumene nenkonzo yeNodePort ekhankanywe ngasentla zihlala zifumaneka kuphela kwindawo ethile iseti esezantsi yezi nodi. Ngamanye amazwi, ukuba ndidibanisa kwi-node engenayo i-pod efunekayo, iya kuthumela itrafikhi kwenye indawo, Ukongeza i-hop kunye nokunyuka kwe-latency (ukuba ii-nodes zifumaneka kwiindawo ezahlukeneyo zokufumaneka / kumaziko edatha, i-latency ingaba phezulu kakhulu; ukongeza, iindleko ze-traffic egress ziya kwanda).
Ngakolunye uhlangothi, ukuba inkonzo ethile ye-Kubernetes inomgaqo-nkqubo omiselweyo externalTrafficPolicy: Local, emva koko i-NodePort ivula kuphela kwezo ndawo ii-pods ezifunekayo zisebenza ngokwenene. Xa usebenzisa i-balancer yomthwalo wangaphandle ohlola urhulumente (ujongo lwempilo) isiphelo (yenza njani AWS ELB), Yena iya kuthumela itrafikhi kuphela kwiindawo eziyimfuneko, eya kuba nefuthe elihle ekulibazisekeni, iimfuno zekhompyutha, iibhilidi ze-egress (kunye nengqiqo eqhelekileyo ichaza okufanayo).
Kukho ithuba eliphezulu lokuba sele usebenzisa into enje Umzila okanye nginx-ingress-controller njengesiphelo seNodePort (okanye iLoadBalancer, ekwasebenzisa iNodePort) kwindlela ye-HTTP yokungena kwitrafikhi, kwaye ukucwangcisa olu khetho kunokunciphisa kakhulu ukubambezeleka kwezicelo ezinjalo.
В Unokufunda ngakumbi malunga nePolicy yeTraffic yangaphandle, iingenelo zayo kunye nokungalunganga.
10. Musa ukubophelela kumaqela kwaye ungayisebenzisi kakubi inqwelomoya yokulawula
Ngaphambili, yayilisiko ukubiza iiseva ngamagama afanelekileyo: , HAL9000 kunye ne-Colossus... Namhlanje zithathelwe indawo zizazisi ezenziwe ngokungenamkhethe. Nangona kunjalo, umkhwa wahlala, kwaye ngoku amagama afanelekileyo aya kumaqela.
Ibali eliqhelekileyo (elisekwe kwiziganeko zokwenyani): konke kwaqala ngobungqina bengcamango, ngoko ke iqela linegama elinebhongo. uvavanyo… Iminyaka idlulile kwaye isasetyenziswa kwimveliso, kwaye wonke umntu uyoyika ukuyiphatha.
Akukho nto imnandi malunga neqela eliguqukayo libe zizilwanyana zasekhaya, ke sicebisa ukuba zisuswe ngamaxesha athile ngelixa siziqhelanisa ukuchacha kwintlekele (oku kuya kunceda - malunga. guqulela.). Ukongeza, bekungayi kuba buhlungu ukusebenza kwinqanaba lolawulo (inqwelomoya). Ukoyika ukumphatha asilophawu lulungileyo. njl.njl ufile? Madoda nisengxakini nyani!
Kwelinye icala, akufuneki uthabatheke ngokuyilawula. Ngexesha umaleko wolawulo unokucotha. Okunokwenzeka, oku kungenxa yenani elikhulu lezinto ezenziweyo ngaphandle kokujikeleza kwazo (imeko eqhelekileyo xa usebenzisa iHelm enezicwangciso ezingagqibekanga, yiyo loo nto imeko yayo kwi-configmaps/emfihlekweni ingahlaziywayo - ngenxa yoko, amawaka ezinto aqokelelana kwi umaleko wolawulo) okanye ngokuhlelwa rhoqo kwezinto ze-kube-api (ukulinganisa okuzenzekelayo, kwi-CI / CD, ukubeka iliso, iilogi zesiganeko, abalawuli, njl.).
Ukongezelela, sincoma ukujonga izivumelwano ze-SLA / SLO kunye nomboneleli olawulayo we-Kubernetes kwaye ubeke ingqalelo kwiziqinisekiso. Umthengisi unokuqinisekisa ukufumaneka komgangatho wolawulo (okanye amacandelwana ayo), kodwa hayi ukulibaziseka kwe-p99 kwezicelo ozithumela kuyo. Ngamanye amazwi, ungangena kubectl get nodes, kwaye ufumane impendulo kuphela emva kwemizuzu eyi-10, kwaye oku akuyi kuba kukuphulwa kwemimiselo yesivumelwano senkonzo.
11. Ibhonasi: usebenzisa ithegi yamva nje
Kodwa oku sele kukudala. Kutshanje siye safumana obu buchule kancinci, kuba abaninzi, befunde kumava abuhlungu, bayekile ukusebenzisa ithegi. :latest kwaye waqala ukupina iinguqulelo. Uxolo!
ECR ; Sicebisa ukuba uziqhelanise nolu phawu lumangalisayo.
Isishwankathelo
Ungalindeli ukuba yonke into isebenze ngobusuku: iKubernetes ayisiyompatho. Usetyenziso olubi (kwaye mhlawumbi kuya kuba kubi ngakumbi). Ukungakhathali kuya kukhokelela kubunzima obugqithisileyo, umsebenzi ocothayo kunye noxinzelelo lwenqanaba lolawulo. Ukongeza, usengozini yokushiywa ngaphandle kwesicwangciso sokubuyisela intlekele. Ungalindeli ukuba uKubernetes abonelele yedwa kunye nokufumaneka okuphezulu ngaphandle kwebhokisi. Chitha ixesha usenza isicelo sakho ukuba sifike kwilifu ngokwenene.
Ungaqhelana namava angaphumelelanga amaqela ahlukeneyo kwi nguHenning Jacobs.
Abo banqwenela ukongeza kuluhlu lweempazamo ezinikwe kweli nqaku banokuqhagamshelana nathi kwi-Twitter (, ).
PS evela kumguquleli
Funda nakwibhlog yethu:
- «";
- «";
- «"(uphononongo kunye nengxelo yevidiyo);
- «».
umthombo: www.habr.com
