SRE: Kusanthula Ntchito. Njira yosinthira pogwiritsa ntchito seva yosavuta yapaintaneti mu Go

Kusanthula Magwiridwe ndi Kukonza ndi chida champhamvu chotsimikizira kuti makasitomala akutsatiridwa.

Kusanthula kagwiridwe ka ntchito kungagwiritsidwe ntchito kuzindikira zolepheretsa pulogalamu, kugwiritsa ntchito njira yasayansi yotsimikizira zoyeserera zoyeserera. Nkhaniyi ikufotokoza njira yanthawi zonse yowunikira magwiridwe antchito ndikusintha, kugwiritsa ntchito seva yapaintaneti ya Go monga chitsanzo.

Go ndiyoyenera kwambiri pano chifukwa ili ndi zida zama mbiri. pprof mu laibulale yokhazikika.

Njira

Tiyeni tipange mndandanda wachidule wa kusanthula kwathu masanjidwe. Tidzayesa kugwiritsa ntchito data ina kupanga zisankho m'malo mosintha zinthu mongoganizira chabe. Kuti tichite izi, tidzachita zotsatirazi:

• Timatanthauzira malire okhathamiritsa (zofunika);
• Timawerengera katundu wogulitsira dongosolo;
• Timayesa mayeso (pangani deta);
• Tikuwona;
• Timasanthula - kodi zonse zofunikira zimakwaniritsidwa?
• Timaziyika mwasayansi ndikupanga lingaliro;
• Timachita zoyeserera kuti tiyese lingaliro ili.

Zomangamanga Zosavuta za HTTP Server

Kwa nkhaniyi tidzagwiritsa ntchito seva yaing'ono ya HTTP ku Golang. Ma code onse a m'nkhaniyi angapezeke apa.

Ntchito yowunikidwa ndi seva ya HTTP yomwe imafunsa PostgreSQL pa pempho lililonse. Kuphatikiza apo, Prometheus, node_exporter, ndi Grafana amagwiritsidwa ntchito kusonkhanitsa ndikuwonetsa ma metrics ogwiritsira ntchito ndi dongosolo.

Kuti zikhale zosavuta, tikuganiza kuti pakukweza kopingasa (ndi kuwerengera mosavuta), ntchito iliyonse ndi nkhokwe zimayikidwa pamodzi:

Kufotokozera zolinga

Mu sitepe iyi, timafotokozera cholinga chathu. Kodi tikuyesera kusanthula chiyani? Kodi timadziwa bwanji nthawi yoti tisiye? M'nkhaniyi, tiganiza kuti tili ndi makasitomala ndipo ntchito yathu ikonza zopempha 10,000 pa sekondi iliyonse.

В Buku la Google SRE Njira zosankhidwa ndi zitsanzo zinakambidwa mwatsatanetsatane. Tiyeni tichite zomwezo ndikumanga zitsanzo:

• Latency: 99% ya zopempha ziyenera kumaliza zosakwana 60ms;
• Mtengo: Ntchitoyi iyenera kuwononga ndalama zochepa zomwe tikuwona kuti zingatheke. Kuti tikwaniritse izi, timakulitsa zotuluka;
• Kukonzekera luso: Kumvetsetsa ndikulemba kuchuluka kwa ntchito zomwe zidzafunikire kuyendetsedwa, kuphatikiza magwiridwe antchito, komanso kuchuluka kwanthawi zomwe zidzafunikire kukwaniritsa zofunikira zoyambira ndi zoperekera. ntchito n+1.

Latency ingafunike kukhathamiritsa kuwonjezera pa kusanthula, koma zotulukapo ziyenera kuwunikiridwa. Mukamagwiritsa ntchito njira ya SRE SLO, chosowa cha latency chimachokera kwa kasitomala ndi / kapena bizinesi, yoimiridwa ndi mwiniwake wazinthu. Ndipo ntchito yathu ikwaniritsa kudziperekaku kuyambira pachiyambi, popanda makonda!

Kukhazikitsa malo oyesera

Pogwiritsa ntchito malo oyesera, tikhoza kuyika katundu woyezedwa ku dongosolo lathu. Deta ya magwiridwe antchito a pa intaneti idzapangidwa kuti iwunikenso.

Katundu wamalonda

Chilengedwe ichi chimagwiritsa ntchito Vegeta kuti mupange makonda a pempho la HTTP mpaka itayimitsidwa:

$ make load-test LOAD_TEST_RATE=50
echo "POST http://localhost:8080" | vegeta attack -body tests/fixtures/age_no_match.json -rate=50 -duration=0 | tee results.bin | vegeta report

Kuwona

Munthawi yantchito, ntchito yobwereketsa idzagwiritsidwa ntchito. Kuphatikiza pa ma metrics ogwiritsira ntchito (chiwerengero cha zopempha, latency ya kuyankha) ndi ma metrics ogwiritsira ntchito (memory, CPU, IOPS), mbiri ya pulogalamu idzayendetsedwa kuti mumvetsetse komwe mavuto akuchitika komanso momwe CPU ikugwiritsidwira ntchito.

Mbiri

Kulemba mbiri ndi mtundu wa muyeso womwe umakulolani kuti muwone komwe nthawi ya CPU imathera pomwe pulogalamu ikugwira ntchito. Zimakupatsani mwayi wodziwa komwe ndi nthawi yayitali bwanji ya CPU:

Izi zitha kugwiritsidwa ntchito pakuwunika kuti mumvetsetse nthawi ya CPU yotayika komanso ntchito yosafunikira. Go (pprof) amatha kupanga mbiri ndikuwonera ngati chithunzi chamoto pogwiritsa ntchito zida zokhazikika. Ndikambirana zakugwiritsa ntchito kwawo komanso kalozera wokhazikitsa pambuyo pake m'nkhaniyi.

Kukhazikitsa, kuyang'ana, kusanthula.

Tiyeni tiyambe kuyesa. Tidzathamanga, kuyang'ana, ndi kusanthula mpaka titakhutira ndi momwe ntchitoyi ikuyendera. Tidzasankha katundu wocheperako kuti tigwiritse ntchito kuti tipeze zowonera zoyambirira. Pa sitepe iliyonse yotsatira, tidzawonjezera katunduyo ndi chinthu chokulitsa chosankhidwa ndi kusiyana kwina. Kuyesa kulikonse kudzachitidwa ndi kuchuluka kwa zopempha zosinthidwa: make load-test LOAD_TEST_RATE=X.

Zopempha 50 pa sekondi iliyonse

Samalani ma graph awiri apamwamba. Chithunzi chakumanzere chakumanzere chikuwonetsa kuti ntchito yathu imapanga zopempha 50 pa sekondi imodzi (malinga ndi mawerengedwe ake), ndipo chithunzi chakumanja chakumanja chikuwonetsa kutalika kwa pempho lililonse. Magawo onse awiriwa amatithandiza kuyang'anira ndikuwunika ngati tili m'malire omwe timachita. Mzere wofiira pa graph HTTP Pempho Latency ikuwonetsa SLO ya 60ms. Mzerewu ukuwonetsa kuti tatsala pang'ono kuyankha.

Tiyeni tiwone mbali ya mtengo:

Zopempha 10000 pamphindikati / zopempha 50 pa seva = ma seva 200 + 1

Tikhozabe kuwongolera chiwerengerochi.

Zopempha 500 pa sekondi iliyonse

Zinthu zosangalatsa kwambiri zimayamba kuchitika katundu akafika zopempha 500 pamphindikati:

Apanso, graph yakumanzere ikuwonetsa kuti pulogalamuyo ikulembetsa katundu wamba. Ngati sizili choncho, pali vuto ndi seva yomwe ikuyendetsa pulogalamuyi. Kuyankha kwa latency graph, yomwe ili kumanja kumanja, ikuwonetsa kuti zopempha za 500 pa sekondi iliyonse zidapangitsa kuyankha kwa 25-40ms. 99th percentile ikadali yabwino mkati mwa 60ms SLO yosankhidwa pamwambapa.

Pankhani ya mtengo:

Zopempha 10000 pamphindikati / zopempha 500 pa seva = ma seva 20 + 1

Pali malo oti tiwongolere.

Zopempha 1000 pa sekondi iliyonse

Kukhazikitsa kwakukulu! Pulogalamuyi ikuwonetsa kuti idakonza zopempha 1000 pamphindikati, koma malire a latency adaphwanyidwa ndi SLO. Izi zikuwonekera pamzere wa p99 pa graph yapamwamba kumanja. Ngakhale mzere wa p100 ndi wapamwamba kwambiri, latency yeniyeni ili pamwamba pa 60ms maximum. Tiyeni tilowe mu mbiri kuti tiwone zomwe pulogalamuyi ikuchita.

Mbiri

Polemba mbiri timayika zopempha 1000 pamphindikati, kenako gwiritsani ntchito pprof kusonkhanitsa deta kuti mudziwe komwe pulogalamuyo ikuwonongera nthawi ya CPU. Izi zitha kuchitika poyambitsa HTTP endpoint. pprof, kenako sungani zotsatirazo ndikuziyika pogwiritsa ntchito ma curl:

$ curl http://localhost:8080/debug/pprof/profile?seconds=29 > cpu.1000_reqs_sec_no_optimizations.prof

Zotsatira zitha kuwonetsedwa motere:

$ go tool pprof -http=:12345 cpu.1000_reqs_sec_no_optimizations.prof

Grafu ikuwonetsa komwe ndi nthawi yochuluka ya CPU yomwe pulogalamuyo ikuwononga. Kuchokera kukufotokozera Brendan Gregg:

X-axis imawonetsa kudzaza kwa mbiri ya stack, yosankhidwa motsatira zilembo (osati nthawi). Y-axis ikuwonetsa kuya kwa stack, kuwerengera kuyambira ziro mpaka [pamwamba]. Rectangle iliyonse imayimira chimango cha stack. Kukula kwa chimango, m'pamenenso kumawonekera pafupipafupi. Chimango chapamwamba chikuyenda pa CPU, pomwe mafelemu apansi ndi zinthu zamwana. Mitundu nthawi zambiri siyiyimira chilichonse, koma imasankhidwa mwachisawawa kuti isiyanitse mafelemu.

Analysis - zongopeka

Pakusintha, timayang'ana kwambiri kuyesa kupeza zowononga nthawi ya CPU. Tidzayang'ana magwero akuluakulu owononga ndalama ndikuchotsa. Chabwino, popeza kufotokozera kumawulula molondola komwe pulogalamu imathera nthawi yake ya CPU, mungafunike kuchita izi kangapo, ndipo mudzafunikanso kusintha kachidindo ka pulogalamuyo, kuyesanso kuyesa, ndikuwona kuti ntchitoyo ikuyandikira chandamale.

Kutsatira malingaliro a Brendan Gregg, tiwerenga graph kuchokera pamwamba mpaka pansi. Mzere uliwonse umayimira stack frame (kuyitana kwa ntchito). Mzere woyamba ndi malo olowera pulogalamu, kholo la mafoni ena onse (mwanjira ina, mafoni ena onse adzakhala nawo mumndandanda wawo). Mzere wotsatira ndi wosiyana:

Ngati muyang'ana pa dzina lachidziwitso pa graph, mudzawona nthawi yonse yomwe idathera pa stack panthawi yokonza zolakwika. Ntchito ya HTTPServe inalipo kwa 65% ya nthawiyo, pomwe ntchito zina zothamanga runtime.mcall, mstart и gc, zinatenga nthawi yotsalayo. Chochititsa chidwi: 5% ya nthawi yonse yomwe idagwiritsidwa ntchito pa mafunso a DNS:

Maadiresi omwe pulogalamuyi ikuyang'ana ndi a Postgresql. Dinani pa FindByAge:

Chosangalatsa ndichakuti, pulogalamuyi ikuwonetsa kuti pali magwero atatu akulu a latency: kutsegulira ndi kutseka kulumikizana, zopempha za data, ndi kulumikizana kwa database. Chithunzichi chikuwonetsa kuti zopempha za DNS ndikutsegula ndi kutseka zolumikizira zimakhala pafupifupi 13% ya nthawi yonse yophedwa.

Zongoyerekeza: Kugwiritsanso ntchito kulumikizana ndi dziwe kuyenera kuchepetsa nthawi ya pempho limodzi la HTTP, kulola kutulutsa kwapamwamba komanso kutsika kwachedwa..

Kukhazikitsa pulogalamu - kuyesa

Tikusintha kachidindo koyambira ndikuyesera kuchotsa kulumikizana kwa PostgreSQL pafunso lililonse. Njira yoyamba ndiyo kugwiritsa ntchito dziwe lolumikizana pamlingo wofunsira. Mukuyesera uku, ife tiyeni tiyime kulumikizana kolumikizana pogwiritsa ntchito sql driver kuti mupite:

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)

if err != nil {
   return nil, err
}

Kukhazikitsa, kuyang'ana, kusanthula

Pambuyo poyambitsanso mayeso ndi zopempha za 1000 pamphindi, zikuwonekeratu kuti p99 latency yabwerera mwakale ndi SLO ya 60ms!

Mtengo wake ndi wotani?

Zopempha 10000 pamphindikati / zopempha 1000 pa seva = ma seva 10 + 1

Tiyeni tipange bwino!

Zopempha 2000 pa sekondi iliyonse

Kuwirikiza katunduyo kukuwonetsa zomwezo, chithunzi chakumanzere chakumanzere chikuwonetsa kuti pulogalamuyo imatha kukonza zopempha 2000 pamphindikati, p100 ndiyotsika kuposa 60ms, p99 imakwaniritsa SLO.

Pankhani ya mtengo:

Zopempha 10000 pamphindikati / zopempha 2000 pa seva = ma seva 5 + 1

Zopempha 3000 pa sekondi iliyonse

Apa, pulogalamuyo imatha kukonza zopempha 3000 ndi p99 latency yochepera 60 ms. SLO sinaphwanyidwe, ndipo mtengo wake umayikidwa motere:

Zopempha 10000 pamphindikati / zopempha 3000 pa seva = ma seva 4 + 1 (wolemba anamaliza, pafupifupi. womasulira)

Tiyeni tiyese kuzungulira kwina.

Analysis - zongopeka

Timasonkhanitsa ndikuwonetsa zotsatira zakusintha pulogalamuyo pazopempha 3000 pamphindikati:

6% ya nthawi imagwiritsidwabe ntchito kukhazikitsa maulumikizidwe. Kukhazikitsa dziwe kwayenda bwino, koma zikuwonekerabe kuti pulogalamuyi ikupitiliza kupanga malumikizano atsopano ku database.

Zongoyerekeza: Malumikizidwe, ngakhale pali dziwe, akugwetsedwabe ndikuchotsedwa, kotero pulogalamuyo ikufunika kuwakhazikitsanso. Kuyika chiwerengero cha maulumikizidwe omwe akudikirira kukula kwa dziwe kuyenera kuthandizira kuchedwa pochepetsa nthawi yomwe pulogalamuyo imagwiritsa ntchito kukhazikitsa kulumikizana..

Kukhazikitsa pulogalamu - kuyesa

Tiyeni tiyese kukhazikitsa MaxIdleConns zofanana ndi kukula kwa dziwe (lofotokozedwanso apa):

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)
db.SetMaxIdleConns(8)
if err != nil {
   return nil, err
}

Kukhazikitsa, kuyang'ana, kusanthula

Zopempha 3000 pa sekondi iliyonse

p99 zosakwana 60ms zotsika kwambiri p100!

Kuwunika kwa graph yamoto kukuwonetsa kuti kulumikizana sikukuwonekanso! Tiyeni tione bwinobwino. pg(*conn).query - Sitikuzindikiranso kulumikizana komwe kukukhazikitsidwa pano.

Pomaliza

Kusanthula kagwiridwe ka ntchito ndikofunikira kuti timvetsetse ngati zomwe kasitomala amayembekeza ndi zomwe sizikugwira ntchito zikukwaniritsidwa. Kuyerekeza kuwunika ndi zomwe kasitomala amayembekeza kungathandize kudziwa zomwe zili zovomerezeka ndi zomwe siziri. Go imapereka zida zamphamvu zomangidwa mulaibulale yokhazikika zomwe zimapangitsa kusanthula uku kukhala kosavuta komanso kupezeka.

Source: www.habr.com