Ingabe ukuqapha kufile? - Ukuqapha bukhoma

Kusukela ngo-2008, inkampani yethu ibilokhu ibambe iqhaza ekuphathweni kwengqalasizinda kanye nokwesekwa kwezobuchwepheshe ubusuku nemini kumaphrojekthi wewebhu: sinamakhasimende angaphezu kwama-400, okungaba yi-15% ye-e-commerce yaseRussia. Ngokuvumelana nalokho, kusekelwa i-architecture ehluke kakhulu. Uma okuthile kuwa, siphoqelekile ukuthi sikulungise phakathi nemizuzu engu-15. Kodwa ukuze uqonde ukuthi ingozi yenzeke, udinga ukuqapha iphrojekthi futhi uphendule izigameko. Ukwenza kanjani lokhu?

Ngikholelwa ukuthi kunenkinga ekuhleleni uhlelo olufanele lokuqapha. Ukube bekungekho nkinga, inkulumo yami ibizoba nethesis eyodwa: “Sicela ufake i-Prometheus + Grafana nama-plugin 1, 2, 3.” Ngeshwa, akusasebenzi ngaleyo ndlela. Futhi inkinga enkulu ukuthi wonke umuntu uyaqhubeka nokukholelwa kokuthile okwakukhona ngo-2008, ngokwezingxenye zesofthiwe.

Mayelana nokuhlelwa kohlelo lokuqapha, ngingazama ukusho ukuthi... amaphrojekthi anokuqapha okunekhono awekho. Futhi isimo sibi kangangokuthi uma into iwa, kunengozi yokuthi ngeke ibonakale - phela, wonke umuntu unesiqiniseko sokuthi "konke kugadiwe."
Mhlawumbe yonke into igadiwe. Kodwa kanjani?

Sonke sihlangabezane nendaba enjengale elandelayo: ama-devops athile, umlawuli othile uyasebenza, ithimba labathuthukisi liza kubo lithi - “sikhululiwe, manje siqaphe.” Gada ini? Isebenza kanjani?

KULUNGILE. Siqapha indlela yakudala. Futhi isivele ishintsha, futhi kuvela ukuthi uqaphe isevisi A, eyaba yisevisi B, esebenzisana nesevisi C. Kodwa ithimba lokuthuthukiswa liyakutshela: "Faka isofthiwe, kufanele iqaphe yonke into!"

Manje yini eshintshile? - Konke kushintshile!

2008 Konke kuhamba kahle

Kukhona abathuthukisi abambalwa, iseva eyodwa, iseva eyodwa yesizindalwazi. Konke kusuka lapha. Sinolwazi oluthile, sifaka i-zabbix, i-Nagios, i-cacti. Bese sibeka izexwayiso ezicacile ku-CPU, ekusebenzeni kwediski, kanye nesikhala sediski. Futhi senza ukuhlola okumbalwa mathupha ukuze siqinisekise ukuthi isayithi liyaphendula nokuthi ama-oda afika kusizindalwazi. Futhi yilokho - sivikelekile kakhulu noma kancane.

Uma siqhathanisa inani lomsebenzi owenziwa umlawuli ukuze ahlinzeke ngokuqapha, khona-ke i-98% yawo yayizenzakalela: umuntu owenza ukuqapha kufanele aqonde indlela yokufaka i-Zabbix, ukuthi ungayilungisa kanjani futhi ulungiselele izexwayiso. Futhi i-2% - yokuhlola kwangaphandle: ukuthi isayithi liyaphendula futhi lenze isicelo ku-database, ukuthi ama-oda amasha afikile.

2010 Umthwalo uyakhula

Siqala ukukala iwebhu, sengeza injini yokusesha. Sifuna ukwenza isiqiniseko sokuthi ikhathalogi yomkhiqizo iqukethe yonke imikhiqizo. Futhi lokho kusesha umkhiqizo kuyasebenza. Ukuthi i-database iyasebenza, ukuthi ama-oda ayenziwa, ukuthi isayithi liphendula ngaphandle futhi liphendule kusuka kumaseva amabili futhi umsebenzisi akakhishelwa ngaphandle kwesayithi ngenkathi elinganiswa kwenye iseva, njll. Kukhona amabhizinisi amaningi.

Ngaphezu kwalokho, inkampane ehlotshaniswa nengqalasizinda iseyiyona enkulu ekhanda lomphathi. Kusenombono ekhanda lami wokuthi umuntu owenza i-monitor ngumuntu ozofaka i-zabbix futhi akwazi ukuyilungisa.

Kodwa ngesikhathi esifanayo, umsebenzi uvela ekuhloleni kwangaphandle, ekwakheni isethi yemibhalo yombuzo we-indexer yokusesha, isethi yemibhalo yokuhlola ukuthi ukusesha kuyashintsha yini ngesikhathi senqubo yokukhomba, isethi yemibhalo ehlola ukuthi izimpahla zidluliselwa ku isevisi yokulethwa, njll. njalo njalo.

Qaphela: Ngibhale “iqoqo lemibhalo” izikhathi ezi-3. Okusho ukuthi, umuntu obhekene nokuqapha akuseyena ovele afake i-zabbix. Lona ngumuntu oqala ukubhala amakhodi. Kodwa akukho okushintshile emiqondweni yeqembu okwamanje.

Kodwa izwe liyashintsha, liba yinkimbinkimbi ngokwengeziwe. Isendlalelo se-virtualization kanye nezinhlelo ezintsha ezimbalwa ziyengezwa. Baqala ukuxhumana bodwa. Ubani owathi "kunuka njengama-microservices?" Kodwa isevisi ngayinye isabukeka njengewebhusayithi ngayinye. Singaphendukela kuyo futhi siqonde ukuthi inikeza ulwazi oludingekayo futhi isebenza ngokwayo. Futhi uma ungumqondisi njalo uhileleke kuphrojekthi oye waqala ukuthuthukisa iminyaka engu-5-7-10, lolu lwazi luyanqwabelana: kuvela izinga elisha - ukubonile, kuvela elinye izinga - ukubonile ...

Kodwa kuyaqabukela umuntu ephelezela iphrojekthi iminyaka eyi-10.

Qalisa kabusha i-Monitoringman

Ake sithi ufike esiqalisweni esisha esiqashe ngokushesha onjiniyela abangama-20, sabhala ama-microservices ayi-15, futhi ungumqondisi otshelwa ukuthi: “Yakha i-CI/CD. Ngiyacela." Wakhe i-CI / CD futhi ngokuzumayo uzwa: "Kunzima ngathi ukusebenza ngokukhiqiza "ku-cube", ngaphandle kokuqonda ukuthi isicelo sizosebenza kanjani kuyo. Senze ibhokisi lesihlabathi “ku-cube” efanayo.
Wenza ibhokisi lesihlabathi kule cube. Ngokushesha bakutshela ukuthi: "Sifuna isizindalwazi sesiteji esibuyekezwa nsuku zonke kusukela ekukhiqizweni, ukuze siqonde ukuthi sisebenza kusizindalwazi, kodwa ngesikhathi esifanayo singonakalisi imininingwane yokukhiqiza."

Uphila kukho konke lokhu. Kukhona amasonto angu-2 asele ngaphambi kokukhululwa, bayakutshela: "Manje ake siqaphe konke lokhu ...". ukuqapha ingqalasizinda yeqoqo, ukuqapha ukwakheka kwe-microservice, ukuqapha umsebenzi ngezinsizakalo zangaphandle...

Futhi ozakwethu bakhipha isikimu esivamile emakhanda abo bathi: “Awu, yonke into icacile lapha! Faka uhlelo oluzoqapha konke lokhu.” Yebo, yebo: ama-plugin we-Prometheus + Grafana +.
Futhi bayanezela: “Unamasonto amabili, qiniseka ukuthi yonke into iphephile.”

Kumaphrojekthi amaningi esiwabonayo, umuntu oyedwa wabelwe ukuqapha. Cabanga ukuthi sifuna ukuqasha umuntu ukuthi enze ukuqapha amaviki angu-2, futhi simbhalela i-resume. Yimaphi amakhono lo muntu okufanele abe nawo, uma sibheka konke esesikushilo kuze kube manje?

• Kumele aqonde ukugadwa kanye nemininingwane yokusebenza kwengqalasizinda yensimbi.
• Kumelwe aqonde imininingwane yokuqapha u-Kubernetes (futhi wonke umuntu ufuna ukuya "ku-cube", ngoba ungakwazi ukukhipha yonke into, ufihle, ngoba umlawuli uzobhekana nokunye) - uqobo, ingqalasizinda yayo, futhi aqonde indlela yokuqapha izinhlelo zokusebenza. ngaphakathi.
• Kufanele aqonde ukuthi izinsiza zixhumana zodwa ngezindlela ezikhethekile, futhi azi imininingwane ecacile yokuthi izinsiza zisebenzisana kanjani. Kungenzeka impela ukubona iphrojekthi lapho ezinye izinsizakalo zixhumana ngokuhambisanayo, ngoba ayikho enye indlela. Isibonelo, i-backend ihamba nge-REST, nge-gRPC kuya kusevisi yekhathalogi, ithola uhlu lwemikhiqizo bese iyibuyisela emuva. Awukwazi ukulinda lapha. Futhi nezinye izinsiza kusebenza ngokulinganayo. Dlulisela i-oda kusevisi yokulethwa, thumela incwadi, njll.
  Cishe usuvele wabhukuda kukho konke lokhu? No admin odinga ukugada lento wavele wadideka.
• Kumele akwazi ukuhlela nokuhlela ngendlela efanele - njengoba umsebenzi uya ukhula.
• Ngakho-ke kufanele enze isu elivela kusevisi edaliwe ukuze aqonde indlela yokuqapha ngokuqondile. Udinga ukuqonda ngesakhiwo sephrojekthi kanye nokuthuthukiswa kwayo + ukuqonda kobuchwepheshe obusetshenziswa ekuthuthukisweni.

Masikhumbule icala elijwayelekile: ezinye izinsiza ziku-PHP, ezinye izinsiza ziku-Go, ezinye izinkonzo ziku-JS. Ngandlela thize bayasebenzisana. Kulapho igama elithi “microservice” livela khona: maningi kakhulu amasistimu angawodwana abathuthukisi abangakwazi ukuqonda iphrojekthi iyonke. Enye ingxenye yethimba ibhala izinsiza ku-JS ezisebenza ngokwazo futhi engazi ukuthi lonke uhlelo lusebenza kanjani. Enye ingxenye ibhala izinkonzo ngePython futhi ayiphazamisi ukuthi ezinye izinsizakalo zisebenza kanjani; zihlukaniswe zindawo zazo. Okwesithathu ukubhala izinsizakalo ku-PHP noma enye into.
Bonke laba bantu abangu-20 bahlukene ngezinkonzo ezingu-15, futhi munye kuphela umlawuli okufanele akuqonde konke lokhu. Ima! sivele sihlukanise uhlelo lube yizinkonzo ezincane eziyi-15 ngoba abantu abangama-20 abakwazi ukuqonda lonke uhlelo.

Kodwa idinga ukugadwa ngandlela thize...

Uyini umphumela? Ngenxa yalokho, kukhona umuntu oyedwa oza nakho konke ithimba lonke labathuthukisi elingakwazi ukukuqonda, futhi ngesikhathi esifanayo kufanele azi futhi akwazi ukwenza lokho esikubonise ngenhla - ingqalasizinda ye-hardware, ingqalasizinda ye-Kubernetes, njll.

Ngingathini... Houston, sinezinkinga.

Ukuqapha iphrojekthi yesofthiwe yesimanje kuyiphrojekthi yesofthiwe ngokwayo

Kusukela enkolelweni engamanga yokuthi ukuqapha kuyisofthiwe, sihlakulela ukukholelwa ezimangalisweni. Kodwa izimangaliso, maye, azenzeki. Awukwazi ukufaka i-zabbix futhi ulindele ukuthi konke kusebenze. Asikho isidingo sokufaka iGrafana futhi sinethemba lokuthi konke kuzolunga. Isikhathi esiningi sizosetshenziswa ekuhleleni amasheke okusebenza kwezinsizakalo nokusebenzisana kwazo, kubhekwa ukuthi izinhlelo zangaphandle zisebenza kanjani. Eqinisweni, i-90% yesikhathi izosetshenziswa hhayi ekubhaleni imibhalo, kodwa ekuthuthukiseni isofthiwe. Futhi kufanele isingathwe ithimba eliwuqondayo umsebenzi wephrojekthi.
Uma kulesi simo umuntu oyedwa ephonswa ekuqashweni, kuyokwenzeka inhlekelele. Okwenzeka yonke indawo.

Isibonelo, kunezinsizakalo ezimbalwa ezixhumana nomunye nge-Kafka. I-oda lafika, sathumela umlayezo mayelana ne-oda e-Kafka. Kukhona isevisi elalela ulwazi mayelana ne-oda bese ithumela izimpahla. Kukhona isevisi elalela ulwazi mayelana ne-oda bese ithumela incwadi kumsebenzisi. Bese kuvela inqwaba yezinkonzo, bese siqala ukudideka.

Futhi uma futhi unikeza lokhu umlawuli nabathuthukisi esiteji lapho kunesikhathi esifushane esisele ngaphambi kokukhishwa, umuntu uzodinga ukuqonda yonke le phrothokholi. Labo. Iphrojekthi yalesi sikali ithatha isikhathi esiningi, futhi lokhu kufanele kufakwe ekuthuthukisweni kwesistimu.
Kodwa kaningi, ikakhulukazi ekuqaleni, sibona ukuthi ukuqapha kuhlehliswa kanjani kuze kube kamuva. “Manje sizokwenza iProof of Concept, sizoyethula ngayo, ayiwe - sesikulungele ukuzidela. Bese sizokuqapha konke.” Lapho (noma uma) iphrojekthi iqala ukwenza imali, ibhizinisi lifuna ukwengeza izici ezengeziwe - ngoba isiqalile ukusebenza, ngakho-ke idinga ukukhishwa ngokuqhubekayo! Futhi usendaweni lapho udinga kuqala ukuqapha konke okwedlule, okungathathi u-1% wesikhathi, kodwa okuningi kakhulu. Futhi ngendlela, abathuthukisi bazodingeka ukuze kuqashwe, futhi kulula ukubavumela ukuthi basebenze ezicini ezintsha. Ngenxa yalokho, izici ezintsha ziyabhalwa, yonke into iyaqina, futhi usesimweni esingapheli.

Ngakho-ke indlela yokuqapha iphrojekthi kusukela ekuqaleni, nokuthi yini okufanele uyenze uma uthola iphrojekthi edinga ukugadwa, kodwa ungazi ukuthi ungaqala kuphi?

Okokuqala, udinga ukuhlela.

Ukwehla kwezwi: imvamisa baqala ngokuqapha ingqalasizinda. Isibonelo, sine-Kubernetes. Ake siqale ngokufaka i-Prometheus nge-Grafana, sifake ama-plugin okuqapha “i-cube”. Akubona abathuthukisi kuphela, kodwa nabaphathi banomkhuba ongemuhle wokuthi: "Sizofaka le plugin, kodwa i-plugin cishe iyazi ukuthi kwenziwa kanjani." Abantu bathanda ukuqala ngezinto ezilula neziqondile, hhayi ngezenzo ezibalulekile. Futhi ukuqapha ingqalasizinda kulula.

Okokuqala, nquma ukuthi yini ofuna ukuyiqapha futhi kanjani, bese ukhetha ithuluzi, ngoba abanye abantu abakwazi ukukucabangela. Futhi kufanele? Abanye abantu bazicabangela bona, mayelana nesistimu yendawo yonke - noma abazange bacabange nhlobo ngenkathi kubhalwa le plugin. Futhi ngenxa yokuthi le plugin inabasebenzisi abayizinkulungwane ezingu-5 akusho ukuthi iyasebenza. Mhlawumbe uzoba ngo-5001st ngoba bekuvele kukhona abantu abangama-5000 ngaphambili.

Uma uqala ukuqapha ingqalasizinda bese ingemuva lohlelo lwakho lokusebenza liyeka ukuphendula, bonke abasebenzisi bazolahlekelwa ukuxhumana nohlelo lokusebenza lweselula. Kuzovela iphutha. Bazoza kuwe bathi "Isicelo asisebenzi, ufunani lapha?" - "Siqapha." — "Uqapha kanjani uma ungaboni ukuthi uhlelo lokusebenza alusebenzi?!"

1. Ngikholwa ukuthi udinga ukuqala ukuqapha ngqo kusuka endaweni yokungena yomsebenzisi. Uma umsebenzisi engaboni ukuthi uhlelo lokusebenza luyasebenza, yilokho, ukwehluleka. Futhi uhlelo lokuqapha kufanele luxwayise ngalokhu kuqala.
2. Kungaleso sikhathi kuphela lapho singakwazi ukuqapha ingqalasizinda. Noma ukwenze ngokuhambisana. Kulula ngengqalasizinda - lapha ekugcineni singakwazi ukufaka i-zabbix.
3. Futhi manje udinga ukuya ezimpandeni zohlelo lokusebenza ukuze uqonde lapho izinto zingasebenzi.

Umbono wami omkhulu ukuthi ukuqapha kufanele kuhambisane nenqubo yentuthuko. Uma uphazamisa ithimba eliqapha eminye imisebenzi (ukudala i-CI/CD, i-sandboxing, ukuhlelwa kabusha kwengqalasizinda), ukuqapha kuzoqala ukunensa futhi ungase ungalokothi ubambe iqhaza ekuthuthukisweni (noma ngokuhamba kwesikhathi kuzodingeka ukumise).

Konke ngamaleveli

Lena yindlela engibona ngayo ukuhlelwa kohlelo lokuqapha.

1) Izinga lohlelo lokusebenza:

• ukuqapha uhlelo lokusebenza logic yebhizinisi;
• ukuqapha amamethrikhi ezinsizakalo zezempilo;
• ukuqapha ukuhlanganiswa.

2) Izinga lengqalasizinda:

• ukuqapha izinga le-orchestration;
• ukuqapha isofthiwe yesistimu;
• ukuqapha izinga lensimbi.

3) Futhi izinga lesicelo - kodwa njengomkhiqizo wobunjiniyela:

• ukuqoqa nokuqapha izingodo zezicelo;
• APM;
• ukulandelela.

4) Isexwayiso:

• ukuhlelwa kohlelo lokuxwayisa;
• ukuhlelwa kohlelo lwemisebenzi;
• inhlangano "yesisekelo solwazi" kanye nokuhamba komsebenzi kokucubungula isigameko.

ezibalulekile: sithola uqaphile hhayi ngemva, kodwa ngokushesha! Asikho isidingo sokuqalisa ukuqapha futhi “ngandlela thize kamuva” uthole ukuthi ubani ozothola izexwayiso. Phela, uyini umsebenzi wokuqapha: ukuqonda ukuthi kukhona okungahambi kahle ohlelweni, nokwazisa abantu abalungile ngakho. Uma ushiya lokhu kuze kube sekupheleni, khona-ke abantu abalungile bazokwazi ukuthi kukhona okungahambi kahle ngokubiza kuphela "akukho okusisebenzelayo."

Isendlalelo Sohlelo - Business Logic Monitoring

Lapha sikhuluma ngokubheka iqiniso lokuthi uhlelo lokusebenza lusebenzela umsebenzisi.

Leli zinga kufanele lenziwe phakathi nesigaba sokuthuthukiswa. Isibonelo, sine-Prometheus enemibandela: iya kuseva eyenza amasheke, idonse indawo yokugcina, futhi iphoyinti lokugcina liyahamba lihlole i-API.

Lapho kuvame ukucelwa ukuthi ugade ikhasi lasekhaya ukuze uqiniseke ukuthi isayithi liyasebenza, abahleli banikeza isibambo esingadonswa ngaso sonke isikhathi uma bedinga ukuqinisekisa ukuthi i-API iyasebenza. Futhi abahleli bohlelo okwamanje basathatha futhi babhale /api/test/helloworld
Indlela kuphela yokuqinisekisa ukuthi yonke into iyasebenza? - Cha!

• Ukudala ukuhlola okunjalo kuwumsebenzi wonjiniyela. Ukuhlolwa kwamayunithi kufanele kubhalwe abahleli ababhala ikhodi. Ngoba uma uyivuza kumphathi, “Mfo, nalu uhlu lwezivumelwano ze-API zayo yonke imisebenzi engama-25, sicela uqaphe yonke into!” - akukho okuzophumelela.
• Uma uphrinta "sawubona mhlaba", akekho ozokwazi ukuthi i-API kufanele futhi isebenze. Lonke ushintsho lwe-API kufanele luholele ekushintsheni kokuhlola.
• Uma usuvele unenkinga enjalo, yeka izici futhi unikeze abathuthukisi abazobhala lokhu kuhlola, noma bamukele ukulahlekelwa, yamukele ukuthi akukho okuhloliwe futhi okuzohluleka.

Amathiphu Ezobuchwepheshe:

• Qiniseka ukuthi uhlela iseva yangaphandle ukuze uhlele ukuhlola - kufanele uqiniseke ukuthi iphrojekthi yakho ifinyeleleka emhlabeni wangaphandle.
• Hlela amasheke kuyo yonke iphrothokholi ye-API, hhayi amaphoyinti angawodwana kuphela.
• Dala i-prometheus-endpoint ngemiphumela yokuhlolwa.

Isendlalelo sohlelo lokusebenza - ukuqapha amamethrikhi ezempilo

Manje sikhuluma ngamamethrikhi ezempilo angaphandle ezinsizakalo.

Sinqume ukuthi siqaphe zonke "izibambo" zohlelo lokusebenza sisebenzisa ukuhlola kwangaphandle, esikubiza ngohlelo lokuqapha lwangaphandle. Kodwa lezi “yizibambo” umsebenzisi “azibonayo”. Sifuna ukwenza isiqiniseko sokuthi izinsiza zethu ziyasebenza. Kunendaba engcono lapha: Ama-K8 ayahlolwa impilo, ukuze okungenani “ikhiyubhu” ngokwayo iqiniseke ukuthi isevisi iyasebenza. Kodwa ingxenye yamasheke engiwabonile ayimibhalo efanayo “sawubona mhlaba”. Labo. Ngakho udonsa kanye ngemva kokuthunyelwa, waphendula ngokuthi konke kuhamba kahle - yilokho kuphela. Futhi insizakalo, uma inikeza i-API yayo, inenani elikhulu lamaphoyinti okungena kuleyo API efanayo, nayo edinga ukugadwa, ngoba sifuna ukwazi ukuthi iyasebenza. Futhi sesivele siyiqaphile ngaphakathi.

Ukusebenzisa kanjani lokhu ngendlela efanele ngobuchwepheshe: isevisi ngayinye iveza isiphetho mayelana nokusebenza kwayo kwamanje, futhi kumagrafu e-Grafana (noma yiluphi olunye uhlelo lokusebenza) sibona isimo sazo zonke izinkonzo.

• Lonke ushintsho lwe-API kufanele luholele ekushintsheni kokuhlola.
• Dala isevisi entsha ngokushesha ngamamethrikhi ezempilo.
• Umlawuli angeza konjiniyela abuze ukuthi “ngingeze izici ezimbalwa ukuze ngiqonde yonke into futhi ngengeze ulwazi mayelana nalokhu ohlelweni lwami lokuqapha.” Kodwa onjiniyela bavamise ukuphendula, "Ngeke sengeze lutho emasontweni amabili ngaphambi kokukhishwa."
  Kwazise abaphathi bentuthuko ukuthi kuzoba nokulahlekelwa okunjalo, kwazise abaphathi babaphathi bentuthuko. Ngoba uma yonke into iwa, othile usazofona futhi afune ukuqapha “isevisi ewa njalo” (c)
• By the way, yabela abathuthukisi ukuthi babhale ama-plugin e-Grafana - lokhu kuzoba usizo oluhle kubaphathi.

Uhlelo Lokusebenza - Ukuqapha Ukuhlanganisa

Ukuqapha ukuhlanganiswa kugxile ekuqapheni ukuxhumana phakathi kwezinhlelo ezibucayi zebhizinisi.

Isibonelo, kunezinsizakalo eziyi-15 ezixhumana zodwa. Lawa awasewona amasayithi ahlukene. Labo. asikwazi ukudonsa insiza ngokwayo, sithole / helloworld futhi siqonde ukuthi isevisi iyasebenza. Ngoba isevisi yewebhu yoku-oda kufanele ithumele ulwazi mayelana ne-oda ebhasini - isuka ebhasini, isevisi ye-warehouse kufanele ithole lo mlayezo futhi isebenze nawo ngokuqhubekayo. Futhi insizakalo yokusabalalisa i-imeyili kufanele icubungule lokhu ngandlela thile, njll.

Ngakho-ke, asikwazi ukuqonda, ngokuhlola insizakalo ngayinye, ukuthi konke kuyasebenza. Ngoba sinebhasi elithile lapho yonke into ixhumana futhi ixhumana.
Ngakho-ke, lesi sigaba kufanele siphawule isigaba sezinsizakalo zokuhlola ukuxhumana nezinye izinsizakalo. Akunakwenzeka ukuhlela ukuqapha ukuxhumana ngokuqapha umthengisi womlayezo. Uma kukhona isevisi ekhipha idatha kanye nesevisi eyitholayo, lapho siqapha umdayisi sizobona kuphela idatha ehamba ngapha nangapha. Noma ngabe ngandlela thize sikwazile ukuqapha ukusebenzisana kwale datha ngaphakathi - ukuthi umkhiqizi othile uthumela idatha, othile uyayifunda, lokhu kugeleza kuyaqhubeka ukuya e-Kafka - lokhu namanje ngeke kusinike ulwazi uma isevisi eyodwa ithumele umlayezo ngenguqulo eyodwa. , kodwa enye isevisi ayizange ilindele le nguqulo futhi iyeqa. Ngeke sazi ngalokhu, ngoba izinsiza zizositshela ukuthi konke kuyasebenza.

Engincoma ukukwenza:

• Ngokuxhumana okuvumelanayo: indawo yokugcina yenza izicelo kumasevisi ahlobene. Labo. sithatha lesi siphetho, sidonse iskripthi ngaphakathi kwenkonzo, esiya kuwo wonke amaphuzu bese sithi "Ngingadonsa lapho, futhi ngidonsele lapho, ngingakwazi ukudonsa lapho..."
• Ukuxhumana okuvumelanayo: imilayezo engenayo - indawo yokugcina ihlola ibhasi ngemilayezo yokuhlola futhi ibonise isimo sokucubungula.
• Ukuxhumana okuvumelanayo: imilayezo ephumayo - indawo yokugcina ithumela imiyalezo yokuhlola ebhasini.

Njengoba kuvamise ukwenzeka: sinesevisi ephonsa idatha ebhasini. Siza kule nkonzo futhi sikucela ukuthi usitshele mayelana nempilo yayo yokuhlanganiswa. Futhi uma isevisi idinga ukukhiqiza umlayezo kwenye indawo (WebApp), izokhiqiza lo mlayezo wokuhlola. Futhi uma sisebenzisa isevisi ngasohlangothini lwe-OrderProcessing, iqala ngokuthumela lokho engakwazi ukukuthumela ngokuzimela, futhi uma kukhona izinto ezincikile, bese ifunda isethi yemilayezo yokuhlola evela ebhasini, iqonde ukuthi ingakwazi ukuyicubungula, ibike futhi. , uma kunesidingo, zithumele ngokuqhubekayo, futhi mayelana nalokhu uthi - konke kulungile, ngiyaphila.

Kaningi sizwa umbuzo othi "singakuhlola kanjani lokhu kudatha yokulwa?" Isibonelo, sikhuluma ngesevisi efanayo yoku-oda. I-oda lithumela imilayezo endaweni yokugcina izimpahla lapho kukhishwa khona izimpahla: asikwazi ukukuhlola lokhu kudatha yokulwa, ngoba "impahla yami izosulwa!" Isixazululo: Hlela konke lokhu kuhlolwa ekuqaleni. Uphinde ube nokuhlolwa kwamayunithi okwenza ukuhlekisa. Ngakho-ke, kwenze ezingeni elijulile lapho unesiteshi sokuxhumana esingalimazi ukusebenza kwebhizinisi.

Izinga lengqalasizinda

Ukuqapha ingqalasizinda yinto osekunesikhathi eside ibhekwa njengokuyiqapha.

• Ukuqapha ingqalasizinda kungenziwa futhi kufanele kwethulwe njengenqubo ehlukile.
• Akufanele uqale ngokuqapha ingqalasizinda kuphrojekthi esebenzayo, noma ngabe ufuna ngempela. Lokhu kubuhlungu kuwo wonke ama-devops. "Okokuqala ngizoqapha iqoqo, ngizoqapha ingqalasizinda" - i.e. Okokuqala, izoqapha ukuthi yini engezansi, kodwa ngeke ingene kuhlelo lokusebenza. Ngoba uhlelo lokusebenza luyinto engaqondakali yama-devops. Kwaputshuzelwa kuye, futhi akaqondi ukuthi kusebenza kanjani. Futhi uyayiqonda ingqalasizinda futhi aqale ngayo. Kodwa cha - udinga njalo ukuqapha uhlelo lokusebenza kuqala.
• Ungadluli ngenani lezexwayiso. Uma kucatshangelwa ubunkimbinkimbi bezinhlelo zesimanje, izexwayiso zihlale zindiza, futhi kufanele ngandlela thize uphile nalesi sixuku sezaziso. Futhi umuntu osocingweni, esebheke izexwayiso eziyikhulu ezilandelayo, uzonquma ukuthi "Angifuni ukucabanga ngakho." Izexwayiso kufanele zazise ngezinto ezibucayi kuphela.

Izinga lesicelo njengeyunithi yebhizinisi

Amaphuzu abalulekile:

• ELK. Leli izinga lemboni. Uma ngesizathu esithile ungahlanganisi izingodo, qala ukwenza kanjalo ngokushesha.
• APM. Ama-APM angaphandle njengendlela yokuvala ngokushesha ukuqapha kohlelo lokusebenza (i-NewRelic, i-BlackFire, i-Datadog). Ungakwazi ukufaka le nto okwesikhashana ukuze okungenani uqonde ukuthi kwenzakalani ngawe.
• Ukulandelela. Kuma-microservices amaningi, kufanele ulandele yonke into, ngoba isicelo asisahlali sodwa. Kunzima kakhulu ukungeza kamuva, ngakho-ke kungcono ukuhlela ngokushesha ukulandelela ekuthuthukisweni - lokhu kuwumsebenzi kanye nokusetshenziswa kwabathuthukisi. Uma ungakayisebenzisi, isebenzise! Bheka uJaeger/Zipkin

Iyaxwayisa

• Ukuhlelwa kwesistimu yezaziso: ezimeni zokuqapha inqwaba yezinto, kufanele kube nesistimu ehlanganisiwe yokuthumela izaziso. Ungakwazi e Grafana. ENtshonalanga, wonke umuntu usebenzisa i-PagerDuty. Izaziso kufanele zicace (isb. lapho zivela khona...). Futhi kuhle ukulawula ukuthi izaziso zitholwa nhlobo
• Inhlangano yesistimu yomsebenzi: izexwayiso akufanele zithunyelwe kuwo wonke umuntu (noma wonke umuntu uzosabela esixukwini, noma akekho ozosabela). Onjiniyela nabo badinga ukushayelwa ucingo: qiniseka ukuthi uchaza izindawo ezinomthwalo wemfanelo, wenze imiyalelo ecacile futhi ubhale kuyo ukuthi ubani okufanele ashayele ngoMsombuluko nangoLwesithathu, nokuthi ubani okufanele ashayele ngoLwesibili nangoLwesihlanu (ngaphandle kwalokho ngeke bashayele muntu umcimbi wenkinga enkulu - bazosaba ukukuvusa noma ukukuphazamisa : abantu ngokuvamile abathandi ukushayela nokuvusa abanye abantu, ikakhulukazi ebusuku). Futhi chaza ukuthi ukucela usizo akuyona inkomba yokuntula ikhono (“Ngicela usizo, lokho kusho ukuthi ngiyisisebenzi esibi”), khuthaza izicelo zosizo.
• Ukuhlelwa “kwesisekelo solwazi” nokugeleza komsebenzi wokucubungula isigameko: esigamekweni ngasinye esibucayi, ukuhlolwa kwesidumbu kufanele kuhlelwe, futhi njengesinyathelo sesikhashana, izenzo ezizoxazulula isigameko kufanele zirekhodwe. Futhi kwenze umkhuba ukuthi izixwayiso eziphindaphindiwe ziyisono; zidinga ukulungiswa kukhodi noma umsebenzi wengqalasizinda.

Isitaki sobuchwepheshe

Ake sicabange ukuthi isitaki sethu simi kanje:

• ukuqoqwa kwedatha - Prometheus + Grafana;
• ukuhlaziywa kwelogi - ELK;
• okwe-APM noma Ukulandela - Jaeger (Zipkin).

Ukukhethwa kwezinketho akubalulekile. Ngoba uma ekuqaleni uqonde indlela yokuqapha uhlelo futhi ubhale phansi uhlelo, bese uqala ukukhetha amathuluzi ahambisana nezidingo zakho. Umbuzo uwukuthi yini okhethe ukuyigada kwasekuqaleni. Ngoba mhlawumbe ithuluzi olikhethile ekuqaleni alihambisani nezidingo zakho nhlobo.

Amaphoyinti ezobuchwepheshe ambalwa engiwabona yonke indawo kamuva nje:

U-Prometheus uphushwa ngaphakathi ku-Kubernetes - ubani oze nalokhu?! Uma iqoqo lakho liphahlazeka, uzokwenzani? Uma uneqoqo eliyinkimbinkimbi ngaphakathi, kufanele kube nohlobo oluthile lwesistimu yokuqapha ngaphakathi kweqoqo, futhi okunye ngaphandle, okuzoqoqa idatha ngaphakathi kweqoqo.

Ngaphakathi kweqoqo siqoqa izingodo nakho konke okunye. Kodwa uhlelo lokuqapha kumele lube ngaphandle. Ngokuvamile, kuqoqo lapho kukhona i-Promtheus efakwe ngaphakathi, kukhona nezinhlelo ezenza ukuhlola kwangaphandle kokusebenza kwesayithi. Kuthiwani uma ukuxhumana kwakho nomhlaba wangaphandle kwehlile futhi uhlelo lokusebenza lungasebenzi? Kuvele ukuthi yonke into ihamba kahle ngaphakathi, kodwa akwenzi izinto zibe lula kubasebenzisi.

okutholakele

• Ukuthuthukiswa kokuqapha akukhona ukufakwa kwezinsiza, kodwa ukuthuthukiswa komkhiqizo wesofthiwe. I-98% yokuqapha kwanamuhla ifaka amakhodi. Ukufaka amakhodi kumasevisi, ukubhala amakhodi amasheke angaphandle, ukuhlola izinsizakalo zangaphandle, futhi yilokho kuphela.
• Ungamoshi isikhathi sonjiniyela bakho ekuqapheni: kungathatha kufika ku-30% womsebenzi wabo, kodwa kuyafaneleka.
• Devops, ungakhathazeki ngokuthi awukwazi ukuqapha okuthile, ngoba ezinye izinto ziyindlela ehluke ngokuphelele yokucabanga. Ubungeyena umhleli, futhi umsebenzi wokuqapha kuwumsebenzi wabo.
• Uma ngabe iphrojekthi isivele iyasebenza futhi ingagadiwe (futhi ungumphathi), nikeza izinsiza zokuqapha.
• Uma umkhiqizo usuvele ukhiqizwa, futhi ungama-devops owatshelwa ukuthi "umise ukuqapha" - zama ukuchaza kubaphathi ukuthi ngibhale ini ngakho konke lokhu.

Lena inguqulo enwetshiwe yombiko engqungqutheleni ye-Saint Highload++.

Uma unentshisekelo emibonweni yami nemicabango ngakho kanye nezihloko ezihlobene, khona-ke lapha ungakwazi funda isiteshi 🙂

Source: www.habr.com