I-SRE: Ukuhlaziywa Kokusebenza. Indlela yokumisa usebenzisa iseva yewebhu elula ku-Go

Ukuhlaziywa kokusebenza nokushuna kuyithuluzi elinamandla lokuqinisekisa ukuthobela ukusebenza kwamaklayenti.

Ukuhlaziywa kokusebenza kungasetshenziselwa ukuhlola izingqinamba ohlelweni ngokusebenzisa indlela yesayensi ekuhloleni ukuhlola ukushuna. Lesi sihloko sichaza indlela evamile yokuhlaziya ukusebenza nokushuna, kusetshenziswa i-Go webserver njengesibonelo.

I-Go inhle ikakhulukazi lapha ngoba inamathuluzi okwenza iphrofayela pprof kumtapo wolwazi ojwayelekile.

Isu

Masidale uhlu olufingqiwe lokuhlaziywa kwethu kwesakhiwo. Sizozama ukusebenzisa idatha ethile ukuze senze izinqumo esikhundleni sokwenza izinguquko ezisuselwe ekuziqondeni kwengqondo noma ekuqageleni. Ukuze senze lokhu sizokwenza lokhu:

• Sinquma imingcele yokwenza kahle (izidingo);
• Sibala umthwalo wokwenziwe wesistimu;
• Senza ukuhlola (dala idatha);
• Siyabuka;
• Siyahlaziya - ingabe zonke izidingo ziyahlangatshezwa?
• Sikumisa ngokwesayense, senze inkoleloze;
• Senza isilingo ukuze sihlole le nkolelo-mbono.

Isakhiwo Esilula Seseva ye-HTTP

Kulesi sihloko sizosebenzisa iseva encane ye-HTTP e-Golang. Yonke ikhodi evela kulesi sihloko ingatholakala lapha.

Uhlelo lokusebenza oluhlaziywayo luyiseva ye-HTTP evotela i-Postgresql ngesicelo ngasinye. Ukwengeza, kukhona i-Prometheus, i-node_exporter ne-Grafana yokuqoqa nokubonisa amamethrikhi ohlelo lokusebenza kanye nesistimu.

Ukwenza lula, sicabanga ukuthi ekukaleni okuvundlile (kanye nokwenza lula izibalo) isevisi ngayinye kanye nesizindalwazi kusetshenziswa ndawonye:

Ukuchaza imigomo

Kulesi sinyathelo, sinquma ngomgomo. Yini esizama ukuyihlaziya? Sazi kanjani ukuthi sekuyisikhathi sokuphela? Kulesi sihloko, sizocabanga ukuthi sinamakhasimende nokuthi insizakalo yethu izocubungula izicelo eziyi-10 ngomzuzwana.

В I-Google SRE Book Izindlela zokukhetha nokumodela zixoxwa ngokuningiliziwe. Masenze okufanayo futhi sakhe amamodeli:

• Ukubambezeleka: U-99% wezicelo kufanele ugcwaliswe ngaphansi kuka-60ms;
• Izindleko: Isevisi kufanele idle inani elincane lemali esicabanga ukuthi lingenzeka ngokuphusile. Ukuze senze lokhu, sikhulisa ukuphuma;
• Ukuhlela amandla: Kudinga ukuqonda kanye nokubhala ukuthi zingaki izimo zesicelo ezizodinga ukwenziwa, okuhlanganisa ukusebenza kokukalwa okuphelele, nokuthi zingaki izimo ezizodingeka ukuze kuhlangatshezwane nezidingo zokuqala zomthwalo kanye nokuhlinzekwa. ukungabi namsebenzi n+1.

Ukubambezeleka kungase kudinge ukuthuthukiswa ngokungeziwe ekuhlaziyeni, kodwa ukusebenza kahle kudinga ukuhlaziya ngokucacile. Uma usebenzisa inqubo ye-SRE SLO, isicelo sokulibaziseka sivela kukhasimende noma ibhizinisi, elimelwe umnikazi womkhiqizo. Futhi inkonzo yethu izofeza lesi sibopho kusukela ekuqaleni ngaphandle kwezilungiselelo!

Ukusetha indawo yokuhlola

Ngosizo lwendawo yokuhlola, sizokwazi ukubeka umthwalo olinganisiwe kusistimu yethu. Ukuze kuhlaziywe, idatha yokusebenza kwesevisi yewebhu izokhiqizwa.

Umthwalo wokwenziwe

Le ndawo isetshenziswa Vegeta ukuze udale isilinganiso sesicelo se-HTTP sangokwezifiso kuze kumiswe:

$ make load-test LOAD_TEST_RATE=50
echo "POST http://localhost:8080" | vegeta attack -body tests/fixtures/age_no_match.json -rate=50 -duration=0 | tee results.bin | vegeta report

Ukubuka

Umthwalo wokwenziwe uzosetshenziswa ngesikhathi sokusebenza. Ngokungeziwe kumamethrikhi wohlelo lokusebenza (inani lezicelo, ukubambezeleka kwempendulo) kanye namamethrikhi esistimu yokusebenza (inkumbulo, i-CPU, i-IOPS), ukuphrofayela yohlelo lokusebenza kuzoqhutshwa ukuze kuqondwe ukuthi kunezinkinga kuphi nokuthi isikhathi se-CPU sisetshenziswa kanjani.

Ukwenza iphrofayela

Ukwenza iphrofayela kuwuhlobo lokulinganisa olukuvumela ukuthi ubone ukuthi isikhathi se-CPU siyaphi uma uhlelo lokusebenza lusebenza. Ikuvumela ukuthi unqume ukuthi kusetshenziswe isikhathi esingakanani futhi singakanani iphrosesa:

Le datha ingasetshenziswa ngesikhathi sokuhlaziya ukuze uthole ukuqonda mayelana nesikhathi esichithiwe se-CPU kanye nomsebenzi ongadingekile owenziwayo. I-Go (pprof) ingakhiqiza amaphrofayili futhi iwabone ngeso lengqondo njengamagrafu omlilo isebenzisa isethi evamile yamathuluzi. Ngizokhuluma ngokusetshenziswa kwazo kanye nomhlahlandlela wokusetha kamuva esihlokweni.

Ukwenza, ukubheka, ukuhlaziya.

Asenze isilingo. Sizokwenza, sibheke futhi sihlaziye size saneliseke ngokusebenza. Masikhethe inani lomthwalo eliphansi ngokunganaki ukuze silisebenzise ukuze sithole imiphumela yokuqaphela kokuqala. Esinyathelweni ngasinye esilandelayo sizokwandisa umthwalo ngesici esithile sokukala, esikhethwe ngokuhlukahluka okuthile. Ukuhlolwa kokulayisha ngakunye kwenziwa ngenani lezicelo ezilungisiwe: make load-test LOAD_TEST_RATE=X.

Izicelo ezingama-50 ngomzuzwana

Naka amagrafu amabili aphezulu. Phezulu kwesokunxele kukhombisa ukuthi uhlelo lwethu lokusebenza lucubungula izicelo ezingama-50 ngomzuzwana (liyacabanga) futhi kwesokudla phezulu kukhombisa ubude besikhathi sesicelo ngasinye. Womabili amapharamitha asisiza ukuthi sibheke futhi sihlaziye ukuthi singaphakathi kwemingcele yethu yokusebenza noma cha. Umugqa obomvu kugrafu Ukubambezeleka kwesicelo se-HTTP ibonisa i-SLO ngo-60ms. Ulayini ubonisa ukuthi singaphansi kakhulu kwesikhathi sethu sokuphendula esiphezulu.

Ake sibheke uhlangothi lwezindleko:

Izicelo eziyi-10000 ngomzuzwana / izicelo ezingama-50 ngeseva ngayinye = amaseva angama-200 + 1

Sisengakwazi ukuthuthukisa lesi sibalo.

Izicelo ezingama-500 ngomzuzwana

Izinto ezithokozisayo ezengeziwe ziqala ukwenzeka lapho umthwalo ufika ezicelweni ezingama-500 ngomzuzwana:

Futhi, kugrafu engenhla kwesokunxele ungabona ukuthi uhlelo lokusebenza luqopha umthwalo ojwayelekile. Uma kungenjalo, kunenkinga kuseva lapho uhlelo lokusebenza lusebenza khona. Igrafu yokubambezeleka kwempendulo itholakala phezulu kwesokudla, okubonisa ukuthi izicelo ezingama-500 ngomzuzwana zibangele ukubambezeleka kwempendulo okungu-25-40ms. I-99th percentile isangena kahle ku-60ms SLO ekhethwe ngenhla.

Ngokwezindleko:

Izicelo eziyi-10000 ngomzuzwana / izicelo ezingama-500 ngeseva ngayinye = amaseva angama-20 + 1

Konke kusengathuthukiswa.

Izicelo ezingama-1000 ngomzuzwana

Ukwethulwa okuhle! Isicelo sibonisa ukuthi sicubungule izicelo eziyi-1000 ngomzuzwana, kodwa umkhawulo wokubambezeleka wephulwa yi-SLO. Lokhu kungabonakala kulayini p99 kugrafu ephezulu kwesokudla. Naphezu kweqiniso lokuthi umugqa we-p100 uphakeme kakhulu, ukubambezeleka kwangempela kuphakeme kunomkhawulo ongu-60ms. Asingene ekuphrofayilini ukuze sithole ukuthi uhlelo lokusebenza lwenzani ngempela.

Ukwenza iphrofayela

Ukuze wenze iphrofayili, sibeka umthwalo kuzicelo eziyi-1000 ngomzuzwana, bese usebenzisa pprof ukuze uthwebule idatha ukuthola lapho uhlelo lokusebenza luchitha isikhathi se-CPU. Lokhu kungenziwa ngokuvula iphoyinti lokugcina le-HTTP pprof, bese, ngaphansi komthwalo, gcina imiphumela usebenzisa i-curl:

$ curl http://localhost:8080/debug/pprof/profile?seconds=29 > cpu.1000_reqs_sec_no_optimizations.prof

Imiphumela ingaboniswa kanje:

$ go tool pprof -http=:12345 cpu.1000_reqs_sec_no_optimizations.prof

Igrafu ibonisa ukuthi uhlelo lokusebenza luchitha isikhathi esingakanani se-CPU. Kusukela encazelweni evela UBrendan Gregg:

I-eksisi engu-X iyinani lephrofayela yesitaki, ehlelwa ngokulandelana kwezinhlamvu (lesi akusona isikhathi), i-eksisi engu-Y ibonisa ukujula kwesitaki, ibala ukusuka kuziro kuye [phezulu]. Unxande ngamunye uwuhlaka lwesitaki. Uma uhlaka luba banzi, luvame ukuba khona ezitaki. Okuphezulu kusebenza ku-CPU, futhi okungezansi kuyizinto zengane. Imibala ngokuvamile ayisho lutho, kodwa ivele ikhethwe ngokungahleliwe ukuze ihlukanise ozimele.

Ukuhlaziya - i-hypothesis

Ukuze sishune, sizogxila ekuzameni ukuthola isikhathi esimoshekile se-CPU. Sizobheka imithombo emikhulu yokusebenzisa imali engenamsebenzi futhi siyisuse. Nokho, uma kubhekwa ukuthi ukwenza iphrofayela kuveza ngokunembe kakhulu lapho uhlelo lokusebenza lisebenzisa isikhathi salo sokucubungula, kungase kudingeke ukwenze izikhathi ezimbalwa, futhi uzodinga futhi ukushintsha ikhodi yomthombo wohlelo lokusebenza, uqalise kabusha izivivinyo futhi ubone ukuthi ukusebenza kusondela kulokho okuhlosiwe.

Ngokulandela izincomo zikaBrendan Gregg, sizofunda ishadi ukusuka phezulu kuye phansi. Umugqa ngamunye ubonisa uhlaka lwesitaki (ucingo lomsebenzi). Umugqa wokuqala indawo yokungena ohlelweni, umzali wazo zonke ezinye izingcingo (ngamanye amazwi, zonke ezinye izingcingo zizoba nazo kusitaki sazo). Umugqa olandelayo usuvele uhlukile:

Uma uhambisa ikhesa phezu kwegama lomsebenzi kugrafu, isikhathi esiphelele ebikuso isitaki phakathi nokulungisa iphutha sizoboniswa. Umsebenzi we-HTTPServe wawukhona u-65% wesikhathi, eminye imisebenzi yesikhathi sokusebenza runtime.mcall, mstart и gc, kwathatha isikhathi esisele. Iqiniso elijabulisayo: 5% wesikhathi esiphelele sichithwa emibuzweni ye-DNS:

Amakheli afunwa wuhlelo awe-Postgresql. Chofoza ku- FindByAge:

Kuyathakazelisa ukuthi lolu hlelo lubonisa ukuthi, empeleni, kunemithombo emithathu eyinhloko engeza ukubambezeleka: ukuvula nokuvala ukuxhumana, ukucela idatha, nokuxhuma ku-database. Igrafu ikhombisa ukuthi izicelo ze-DNS, ukuvula nokuvala ukuxhumana kuthatha cishe u-13% wesikhathi esiphelele sokwenza.

I-hypothesis: Ukusebenzisa kabusha ukuxhumana usebenzisa i-pooling kufanele kunciphise isikhathi sesicelo esisodwa se-HTTP, okuvumela ukuphuma okuphezulu nokubambezeleka okuphansi..

Ukusetha uhlelo lokusebenza - ukuhlolwa

Sibuyekeza ikhodi yomthombo, zama ukususa ukuxhumana ku-Postgresql ngesicelo ngasinye. Inketho yokuqala ukusebenzisa indawo yokuxhumana ezingeni lesicelo. Kulesi sivivinyo thina asiyimise ukuhlanganisa uxhumano usebenzisa umshayeli we-sql ukuze uhambe:

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)

if err != nil {
   return nil, err
}

Ukwenza, ukubheka, ukuhlaziya

Ngemva kokuqalisa kabusha ukuhlolwa ngezicelo eziyi-1000 ngomzuzwana, kuyacaca ukuthi amazinga okubambezeleka kwe-p99 abuyele kokujwayelekile nge-SLO ye-60ms!

Ziyini izindleko?

Izicelo eziyi-10000 ngomzuzwana / izicelo ezingama-1000 ngeseva ngayinye = amaseva angama-10 + 1

Ake sikwenze kangcono nakakhulu!

Izicelo ezingama-2000 ngomzuzwana

Ukuphinda kabili umthwalo kubonisa into efanayo, igrafu engenhla kwesokunxele ibonisa ukuthi uhlelo lokusebenza lukwazi ukucubungula izicelo ezingu-2000 ngomzuzwana, i-p100 ingaphansi kuka-60ms, i-p99 yanelisa i-SLO.

Ngokwezindleko:

Izicelo eziyi-10000 ngomzuzwana / izicelo ezingama-2000 ngeseva ngayinye = amaseva angama-5 + 1

Izicelo ezingama-3000 ngomzuzwana

Lapha isicelo singacubungula izicelo eziyi-3000 nge-p99 latency engaphansi kuka-60ms. I-SLO ayiphuliwe, futhi izindleko zamukelwa kanje:

Izicelo eziyi-10000 ngomzuzwana / izicelo eziyi-3000 ngeseva ngayinye = amaseva ama-4 + 1 (umbhali uqoqe, cishe. umhumushi)

Ake sizame omunye umjikelezo wokuhlaziya.

Ukuhlaziya - i-hypothesis

Siqoqa futhi sibonise imiphumela yokususa iphutha kuhlelo lokusebenza ngezicelo eziyi-3000 ngomzuzwana:

Noma kunjalo u-6% wesikhathi esichithwa ekusunguleni ukuxhumana. Ukusetha i-pool kuthuthukise ukusebenza, kodwa usengabona ukuthi uhlelo lokusebenza luyaqhubeka nokusebenza ekudaleni ukuxhumana okusha kusizindalwazi.

I-hypothesis: Izixhumanisi, ngaphandle kokuba khona kwechibi, zisawiswa futhi zihlanzwe, ngakho-ke uhlelo lokusebenza ludinga ukusethwa kabusha. Ukusetha inombolo yoxhumo olusalindile kusayizi wechibi kufanele kusize ngokubambezeleka ngokunciphisa isikhathi esichithwa uhlelo lokusebenza udala uxhumano..

Ukusetha uhlelo lokusebenza - ukuhlolwa

Izama ukufaka I-MaxIdleConns ilingana nosayizi wechibi (futhi kuchaziwe lapha):

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)
db.SetMaxIdleConns(8)
if err != nil {
   return nil, err
}

Ukwenza, ukubheka, ukuhlaziya

Izicelo ezingama-3000 ngomzuzwana

i-p99 ingaphansi kuka-60ms ene-p100 engaphansi kakhulu!

Ukuhlola igrafu yomlilo kubonisa ukuthi uxhumano alusabonakali! Ake sihlole kabanzi pg(*conn).query - futhi asiboni ukuxhumana okwenziwa lapha.

isiphetho

Ukuhlaziywa kokusebenza kubalulekile ekuqondeni ukuthi okulindelwe ngamakhasimende kanye nezimfuneko ezingasebenzi kuyahlangatshezwana nazo. Ukuhlaziya ngokuqhathanisa ukuqaphela nokulindelwe yikhasimende kungasiza ekunqumeni ukuthi yini eyamukelekayo nengamukeleki. I-Go ihlinzeka ngamathuluzi anamandla akhelwe kulabhulali ejwayelekile enza ukuhlaziya kube lula futhi kufinyeleleke.

Source: www.habr.com