SRE: Kusanthula Ntchito. Njira yosinthira pogwiritsa ntchito seva yosavuta yapaintaneti mu Go

Kusanthula kagwiridwe ndi kukonza ndi chida champhamvu chotsimikizira kuti makasitomala akutsatiridwa.

Kusanthula kagwiridwe ka ntchito kungagwiritsidwe ntchito poyang'ana zovuta mu pulogalamu pogwiritsa ntchito njira yasayansi pakuyesa kuyesa kosintha. Nkhaniyi ikufotokoza njira wamba pakuwunika magwiridwe antchito ndikusintha, pogwiritsa ntchito Go webserver mwachitsanzo.

Go ndikwabwino kwambiri pano chifukwa ili ndi zida zama mbiri pprof mu laibulale yokhazikika.

Njira

Tiyeni tipange mndandanda wachidule wa kusanthula kwathu masanjidwe. Tidzayesa kugwiritsa ntchito zidziwitso zina kupanga zisankho m'malo mosintha motengera mwadzidzidzi kapena kungoyerekeza. Kuti tichite izi:

• Timazindikira malire okhathamiritsa (zofunika);
• Timawerengera katundu wogulitsira dongosolo;
• Timayesa (kupanga deta);
• Timaona;
• Timasanthula - kodi zonse zofunikira zimakwaniritsidwa?
• Ife timaziyika izo mwasayansi, kupanga lingaliro;
• Timayesa kuyesa lingaliro ili.

Zomangamanga Zosavuta za HTTP Server

Kwa nkhaniyi tidzagwiritsa ntchito seva yaing'ono ya HTTP ku Golang. Ma code onse a m'nkhaniyi angapezeke apa.

Ntchito yomwe ikuwunikidwa ndi seva ya HTTP yomwe imasankha Postgresql pa pempho lililonse. Kuphatikiza apo, pali Prometheus, node_exporter ndi Grafana kuti atolere ndikuwonetsa ma metrics ogwiritsira ntchito ndi dongosolo.

Kuti muchepetse, timawona kuti pakukulitsa kopingasa (ndi kuwerengera kosavuta) ntchito iliyonse ndi nkhokwe zimayikidwa palimodzi:

Kufotokozera zolinga

Pa sitepe iyi, timasankha pa cholinga. Kodi tikuyesera kusanthula chiyani? Kodi timadziwa bwanji nthawi yoti titha? M'nkhaniyi, tilingalira kuti tili ndi makasitomala komanso kuti ntchito yathu ikonza zopempha 10 pamphindikati.

В Buku la Google SRE Njira zosankha ndi kutsanzira zikukambidwa mwatsatanetsatane. Tiyeni tichite zomwezo ndikumanga zitsanzo:

• Latency: 99% ya zopempha ziyenera kumalizidwa pasanathe 60ms;
• Mtengo: Ntchitoyi iyenera kuwononga ndalama zochepa zomwe tikuganiza kuti zingatheke. Kuti tichite izi, timawonjezera kutulutsa;
• Kukonzekera luso: Kumafunikira kumvetsetsa ndikulemba kuchuluka kwa ntchito zomwe zidzafunikire kuyendetsedwa, kuphatikiza magwiridwe antchito onse, ndi nthawi zingati zomwe zidzafunike kukwaniritsa zofunikira zoyambira ndikupereka. ntchito n+1.

Latency ingafunike kukhathamiritsa kuwonjezera pa kusanthula, koma kutulutsa kumafunika kuunikanso. Mukamagwiritsa ntchito njira ya SRE SLO, pempho lochedwa limachokera kwa kasitomala kapena bizinesi, yoimiridwa ndi eni ake. Ndipo utumiki wathu udzakwaniritsa udindo umenewu kuyambira pachiyambi popanda zoikamo zilizonse!

Kukhazikitsa malo oyesera

Mothandizidwa ndi malo oyesera, tidzatha kuyika katundu woyezedwa pa dongosolo lathu. Kuti muwunike, deta yokhudzana ndi magwiridwe antchito a intaneti idzapangidwa.

Katundu wamalonda

Chilengedwe ichi chimagwiritsa ntchito Vegeta kuti mupange makonda a pempho la HTTP mpaka itayimitsidwa:

$ make load-test LOAD_TEST_RATE=50
echo "POST http://localhost:8080" | vegeta attack -body tests/fixtures/age_no_match.json -rate=50 -duration=0 | tee results.bin | vegeta report

Kuwona

Zogulitsa zidzagwiritsidwa ntchito panthawi yokonzekera. Kuphatikiza pa kugwiritsa ntchito (chiwerengero cha zopempha, kuyankha latency) ndi makina ogwiritsira ntchito (memory, CPU, IOPS) metrics, mbiri ya pulogalamu idzayendetsedwa kuti imvetsetse komwe kuli ndi zovuta komanso momwe CPU ikugwiritsidwira ntchito.

Mbiri

Kulemba mbiri ndi mtundu wa muyeso womwe umakulolani kuti muwone komwe nthawi ya CPU ikupita pomwe pulogalamu ikugwira ntchito. Zimakuthandizani kuti mudziwe komwe ndi nthawi yochuluka bwanji ya purosesa:

Izi zitha kugwiritsidwa ntchito pakuwunika kuti mumvetsetse nthawi ya CPU yomwe yawonongeka komanso ntchito yosafunikira yomwe ikuchitika. Go (pprof) amatha kupanga mbiri ndikuwona ngati ma graph amoto pogwiritsa ntchito zida zokhazikika. Ndilankhula zakugwiritsa ntchito kwawo ndikuwongolera kalozera pambuyo pake m'nkhaniyi.

Kuchita, kuyang'anitsitsa, kusanthula.

Tiyeni tiyese kuyesa. Tidzachita, kuyang'ana ndi kusanthula mpaka titakhutira ndi ntchitoyo. Tiyeni tisankhe mtengo wamtengo wapatali wochepa kuti tigwiritse ntchito kuti tipeze zotsatira za zomwe taziwona poyamba. Pa sitepe iliyonse yotsatira tidzawonjezera katunduyo ndi chinthu china cha makulitsidwe, osankhidwa ndi zosiyana. Kuyesa kulikonse kumachitidwa ndi kuchuluka kwa zopempha zomwe zasinthidwa: make load-test LOAD_TEST_RATE=X.

Zopempha 50 pa sekondi iliyonse

Samalani ma graph awiri apamwamba. Kumanzere kumanzere kukuwonetsa kuti ntchito yathu imapanga zopempha 50 pamphindikati (ikuganiza) ndipo kumanja kumanja kukuwonetsa kutalika kwa pempho lililonse. Magawo awiriwa amatithandiza kuyang'ana ndikusanthula ngati tili m'malire athu kapena ayi. Mzere wofiira pa graph HTTP Pempho Latency akuwonetsa SLO pa 60ms. Mzerewu ukuwonetsa kuti tili pansi pa nthawi yathu yoyankha.

Tiyeni tiwone mbali ya mtengo:

Zopempha 10000 pa sekondi iliyonse / zopempha 50 pa seva = ma seva 200 + 1

Tikhozabe kuwongolera chiwerengerochi.

Zopempha 500 pa sekondi iliyonse

Zinthu zochititsa chidwi zimayamba kuchitika katundu akafika pazopempha 500 pamphindikati:

Apanso, pamwamba kumanzere graph inu mukhoza kuona kuti ntchito kujambula bwinobwino katundu. Ngati sizili choncho, pali vuto pa seva yomwe pulogalamuyo ikugwira ntchito. Kuyankha kwa latency graph kuli kumanja kumanja, kusonyeza kuti zopempha 500 pa sekondi iliyonse zidapangitsa kuchedwa kwa 25-40ms. 99th percentile ikukwanirabe bwino mu 60ms SLO yosankhidwa pamwambapa.

Pankhani ya mtengo:

Zopempha 10000 pa sekondi iliyonse / zopempha 500 pa seva = ma seva 20 + 1

Chilichonse chikhoza kuwongoleredwa.

Zopempha 1000 pa sekondi iliyonse

Kukhazikitsa kwakukulu! Ntchitoyi ikuwonetsa kuti idakonza zopempha za 1000 pamphindi, koma malire a latency adaphwanyidwa ndi SLO. Izi zitha kuwoneka pamzere p99 pa graph yakumanja yakumanja. Ngakhale kuti mzere wa p100 ndi wapamwamba kwambiri, kuchedwa kwenikweni kumakhala kwakukulu kuposa 60ms. Tiyeni tilowe mu mbiri kuti tidziwe zomwe pulogalamuyi imachita.

Mbiri

Polemba mbiri, timayika zopempha 1000 pamphindikati, kenako gwiritsani ntchito pprof kuti mugwire deta kuti mudziwe komwe pulogalamuyo ikugwiritsa ntchito nthawi ya CPU. Izi zitha kuchitika poyambitsa HTTP endpoint pprof, ndiyeno, mutanyamula, sungani zotsatira pogwiritsa ntchito ma curl:

$ curl http://localhost:8080/debug/pprof/profile?seconds=29 > cpu.1000_reqs_sec_no_optimizations.prof

Zotsatira zitha kuwonetsedwa motere:

$ go tool pprof -http=:12345 cpu.1000_reqs_sec_no_optimizations.prof

Grafu ikuwonetsa komwe pulogalamuyo imathera nthawi ya CPU komanso kuchuluka kwake. Kuchokera ku kufotokozera kuchokera Brendan Gregg:

X axis ndi kuchuluka kwambiri, zosankhidwa motsatira zilembo (ino si nthawi), axis Y imawonetsa kuya kwa stack, kuwerengera kuyambira ziro pa [pamwamba]. Rectangle iliyonse imakhala ndi chimango cha stack. Chotambasula chimango, nthawi zambiri chimakhala mumilu. Zomwe zili pamwamba zimayenda pa CPU, ndipo zomwe zili pansipa ndi zinthu zamwana. Mitundu nthawi zambiri sikutanthauza chilichonse, koma amangosankhidwa mwachisawawa kuti asiyanitse mafelemu.

Analysis - zongopeka

Pakusintha, timayang'ana kwambiri kuyesa kupeza nthawi yotayika ya CPU. Tidzayang'ana magwero akuluakulu a ndalama zopanda ntchito ndikuzichotsa. Chabwino, popeza kufotokozera kumawulula molondola kwambiri pomwe pulogalamuyo ikugwiritsira ntchito nthawi yake ya purosesa, mungafunike kutero kangapo, ndipo mudzafunikanso kusintha kachidindo kamene kamayambira, kuyesanso kuyesa ndikuwona kuti magwiridwe antchito akuyandikira chandamale.

Potsatira malingaliro a Brendan Gregg, tiwerenga tchati kuchokera pamwamba mpaka pansi. Mzere uliwonse ukuwonetsa chimango cha stack (kuyitana kwa ntchito). Mzere woyamba ndi malo olowera mu pulogalamuyi, kholo la mafoni ena onse (mwanjira ina, mafoni ena onse adzakhala nawo pamndandanda wawo). Mzere wotsatira ndi wosiyana kale:

Ngati muyang'ana cholozera pa dzina la ntchito pa graph, nthawi yonse yomwe inali pa stack panthawi yokonza zolakwika idzawonetsedwa. Ntchito ya HTTPServe inalipo 65% ya nthawiyo, ntchito zina zothamanga runtime.mcall, mstart и gc, zinatenga nthawi yotsalayo. Zosangalatsa: 5% ya nthawi yonse imagwiritsidwa ntchito pa mafunso a DNS:

Maadiresi omwe pulogalamuyi imayang'ana ndi a Postgresql. Dinani pa FindByAge:

Chochititsa chidwi n'chakuti, pulogalamuyi ikuwonetsa kuti, kwenikweni, pali magwero atatu akuluakulu omwe amawonjezera kuchedwa: kutsegula ndi kutseka kugwirizana, kupempha deta, ndi kulumikiza ku database. Grafu ikuwonetsa kuti zopempha za DNS, kutsegulira ndi kutseka kulumikizana kumatenga pafupifupi 13% ya nthawi yonse yophedwa.

Zongoyerekeza: Kugwiritsanso ntchito maulumikizidwe pogwiritsa ntchito kuphatikiza kuyenera kuchepetsa nthawi ya pempho limodzi la HTTP, kulola kutulutsa kwapamwamba komanso kutsika kwachedwa..

Kukhazikitsa pulogalamu - kuyesa

Timasintha kachidindo kochokera, yesetsani kuchotsa kulumikizidwa kwa Postgresql pa pempho lililonse. Njira yoyamba ndiyo kugwiritsa ntchito dziwe lolumikizana pamlingo wofunsira. Mukuyesera uku ife tiyeni tiyime kulumikizana kolumikizana pogwiritsa ntchito sql driver kuti mupite:

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)

if err != nil {
   return nil, err
}

Kuchita, kuyang'anitsitsa, kusanthula

Pambuyo poyambitsanso mayeso ndi zopempha za 1000 pamphindikati, zikuwonekeratu kuti milingo ya latency ya p99 yabwerera mwakale ndi SLO ya 60ms!

Mtengo wake ndi wotani?

Zopempha 10000 pa sekondi iliyonse / zopempha 1000 pa seva = ma seva 10 + 1

Tiyeni tichite bwino!

Zopempha 2000 pa sekondi iliyonse

Kuwirikiza katunduyo kukuwonetsa zomwezo, chithunzi chakumanzere chakumanzere chikuwonetsa kuti pulogalamuyo imatha kukonza zopempha 2000 pamphindikati, p100 ndiyotsika kuposa 60ms, p99 imakwaniritsa SLO.

Pankhani ya mtengo:

Zopempha 10000 pa sekondi iliyonse / zopempha 2000 pa seva = ma seva 5 + 1

Zopempha 3000 pa sekondi iliyonse

Apa pulogalamuyo imatha kukonza zopempha 3000 ndi p99 latency yochepera 60ms. SLO sinaphwanyidwe, ndipo mtengo wake umavomerezedwa motere:

Zopempha 10000 pamphindikati / zopempha 3000 pa seva = ma seva 4 + 1 (wolemba anamaliza, pafupifupi. womasulira)

Tiyeni tiyese kuzungulira kwina.

Analysis - zongopeka

Timasonkhanitsa ndikuwonetsa zotsatira zakusintha pulogalamuyo pazopempha 3000 pamphindikati:

Komabe 6% ya nthawi imagwiritsidwa ntchito kukhazikitsa maulumikizidwe. Kukhazikitsa dziwe kwayenda bwino, koma mutha kuwona kuti pulogalamuyi ikupitilizabe kupanga maulumikizidwe atsopano ku database.

Zongoyerekeza: Malumikizidwe, ngakhale pali dziwe, amagwetsedwabe ndikutsukidwa, chifukwa chake pulogalamuyo iyenera kuyambiranso. Kukhazikitsa kuchuluka kwa maulumikizidwe omwe akudikirira kukula kwa dziwe kuyenera kuthandizira kuchedwa pochepetsa nthawi yomwe pulogalamu imathera popanga kulumikizana..

Kukhazikitsa pulogalamu - kuyesa

Kuyesa kukhazikitsa MaxIdleConns zofanana ndi kukula kwa dziwe (lofotokozedwanso apa):

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)
db.SetMaxIdleConns(8)
if err != nil {
   return nil, err
}

Kuchita, kuyang'anitsitsa, kusanthula

Zopempha 3000 pa sekondi iliyonse

p99 ndi yocheperapo 60ms yokhala ndi p100 yochepa kwambiri!

Kuyang'ana chithunzi chamoto kukuwonetsa kuti kulumikizana sikukuwonekeranso! Tiyeni tione mwatsatanetsatane pg(*conn).query - Sitikuzindikiranso kulumikizana kukhazikitsidwa pano.

Pomaliza

Kusanthula kagwiridwe ka ntchito ndikofunikira kuti timvetsetse kuti zomwe kasitomala amayembekeza ndi zomwe sizikugwira ntchito zikukwaniritsidwa. Kusanthula poyerekezera zowonera ndi zomwe kasitomala amayembekeza kungathandize kudziwa zomwe zili zovomerezeka ndi zosayenera. Go imapereka zida zamphamvu zomangidwa mulaibulale yokhazikika zomwe zimapangitsa kusanthula kukhala kosavuta komanso kupezeka.

Source: www.habr.com