SRE: Performance Analysis. Kugadzirisa nzira uchishandisa iri nyore webhu server muGo

Kuongorora kwekuita uye tuning chishandiso chine simba chekuona kutevedza kwekuita kwevatengi.

Kuongorora kwekuita kunogona kushandiswa kutarisa mabhodhoro muchirongwa nekushandisa nzira yesainzi pakuyedza tuning bvunzo. Ichi chinyorwa chinotsanangura nzira yakajairika yekuongororwa kwekuita uye kugadzirisa, uchishandisa Go webserver semuenzaniso.

Enda inonyanya kunaka pano nekuti ine maturusi ekunyora pprof muraibhurari yakajairika.

The Strategy

Ngatigadzirei rondedzero yepfupiso yekuongorora kwedu kwemaitiro. Tichaedza kushandisa imwe data kuita sarudzo pane kuita shanduko zvichienderana nekuziva kana kufungidzira. Kuti tiite izvi tichaita izvi:

• Isu tinosarudza optimization miganhu (zvinodiwa);
• Isu tinoverenga mutoro wekutengesa kune system;
• Isu tinoita bvunzo (gadzira data);
• Tinocherechedza;
• Isu tinoongorora - zvese zvinodiwa zvinosangana here?
• Tinozvimisa nesainzi, toita fungidziro;
• Isu tinoita kuyedza kuedza iyi hypothesis.

Yakareruka HTTP Server Architecture

Kune chinyorwa ichi tichashandisa diki HTTP server muGolang. Yese kodhi kubva kuchinyorwa ichi inogona kuwanikwa pano.

Chishandiso chiri kuongororwa iHTTP server inovhota Postgresql pachikumbiro chega chega. Pamusoro pezvo, kune Prometheus, node_exporter uye Grafana yekuunganidza uye kuratidza application uye system metrics.

Kurerutsa, isu tinofunga kuti kune yakachinjika kuyera (uye kurerutsa kuverenga) sevhisi yega yega uye dhatabhesi zvinoiswa pamwechete:

Kutsanangura zvinangwa

Padanho iri, tinosarudza pane chinangwa. Tiri kuedza kuongorora chii? Tinoziva sei kana yava nguva yekupedzisira? Muchinyorwa chino, isu tichafungidzira kuti isu tine vatengi uye kuti sevhisi yedu ichaita 10 zvikumbiro pasekondi.

В Google SRE Book Nzira dzekusarudza nekuenzanisa dzinokurukurwa zvakadzama. Ngatiite zvimwechete uye tivake mamodheru:

• Latency: 99% yezvikumbiro inofanira kupedzwa isingasviki 60ms;
• Mutengo: Sevhisi inofanirwa kushandisa mari shoma yatinofunga kuti inogoneka. Kuti tiite izvi, isu tinowedzera throughput;
• Kuronga kwehunyanzvi: Inoda kunzwisisa uye kunyora kuti mangani maekisheni echikumbiro achada kuitiswa, kusanganisira kuita kwese kuyera kuita, uye kuti mangani ezviitiko achadikanwa kusangana nekutanga mutoro uye zvinodiwa zvekupa. redundancy n+1.

Latency inogona kuda optimization mukuwedzera kuongororo, asi mabudiro anonyatsoda kuongororwa. Paunenge uchishandisa iyo SRE SLO maitiro, chikumbiro chekunonoka chinouya kubva kumutengi kana bhizinesi, rinomiririrwa nemuridzi wechigadzirwa. Uye sevhisi yedu inozadzisa chisungo ichi kubva pakutanga pasina chero marongero!

Kugadzira nzvimbo yekuedza

Nerubatsiro rwenzvimbo yekuyedza, isu tichakwanisa kuisa mutoro wakayerwa pane yedu system. Kuongorora, data pamusoro pekushanda kwewebhu sevhisi ichagadzirwa.

Transaction load

Iyi nharaunda inoshandisa Vegeta kugadzira chiyero chekukumbira cheHTTP kusvika chamira:

$ make load-test LOAD_TEST_RATE=50
echo "POST http://localhost:8080" | vegeta attack -body tests/fixtures/age_no_match.json -rate=50 -duration=0 | tee results.bin | vegeta report

Kuongorora

Transaction load ichashandiswa panguva yekumhanya. Pamusoro pekushandisa (nhamba yezvikumbiro, mhinduro latency) uye sisitimu yekushandisa (memory, CPU, IOPS) metrics, profiling yekushandisa ichamhanya kuti unzwisise paine matambudziko uye kuti CPU nguva iri kupedzwa sei.

Profileing

Profileing imhando yekuyera iyo inokutendera iwe kuti uone kwainoenda CPU nguva kana application iri kushanda. Iyo inokutendera iwe kuti uone chaizvo kuti kupi uye ingani processor nguva inoshandiswa:

Iyi data inogona kushandiswa panguva yekuongorora kuti uwane nzwisiso mukupambadza nguva yeCPU uye basa risingakoshi riri kuitwa. Go (pprof) inogona kugadzira maprofile uye kuaona semurazvo wemagirafu uchishandisa yakajairwa seti yezvishandiso. Ini ndichataura nezve mashandisiro avo uye setup gwara gare gare muchinyorwa.

Kuitwa, kucherechedza, kuongorora.

Ngatiitei ongororo. Tichaita, kucherechedza uye kuongorora kusvika tagutsikana nekuita. Ngatisarudzei mutengo wakaderera wakaderera kuti tiuise kuti tiwane mibairo yekutarisa kwekutanga. Pane imwe neimwe inotevera nhanho isu tichawedzera mutoro neimwe scaling factor, yakasarudzwa neimwe shanduko. Yese yekuyedzwa kwemutoro inomhanya inoitwa nehuwandu hwezvikumbiro zvakagadziriswa: make load-test LOAD_TEST_RATE=X.

50 zvikumbiro pasekondi

Teerera kune maviri epamusoro magirafu. Kumusoro kuruboshwe kunoratidza kuti application yedu inoita makumi mashanu zvikumbiro pasekondi (inofunga) uye kumusoro kurudyi kunoratidza nguva yechikumbiro chega chega. Maparamita ese ari maviri anotibatsira kutarisa uye kuongorora kuti tiri mukati memiganhu yedu yekuita here kana kuti kwete. Mutsara mutsvuku pagirafu HTTP Chikumbiro Latency inoratidza SLO pa60ms. Mutsara unoratidza kuti tiri pasi penguva yedu yepamusoro yekupindura.

Ngatitarisei kudivi remutengo:

10000 zvikumbiro pasekondi / makumi mashanu zvikumbiro pasevha = mazana maviri maseva + 50

Tinogona kuvandudza nhamba iyi.

500 zvikumbiro pasekondi

Zvimwe zvinhu zvinonakidza zvinotanga kuitika kana mutoro wasvika ku500 zvikumbiro pasekondi:

Zvekare, mune yekumusoro kuruboshwe girafu iwe unogona kuona kuti iyo application iri kurekodha yakajairika mutoro. Kana zvisiri izvo, pane dambudziko pane server iyo application iri kushanda. Mhinduro latency girafu iri kumusoro kurudyi, zvichiratidza kuti mazana mashanu zvikumbiro pasekondi zvakakonzera kunonoka kwekupindura kwe500-25ms. Iyo 40th percentile ichiri kukwana zvakanaka mu99ms SLO yakasarudzwa pamusoro.

Maererano nemutengo:

10000 zvikumbiro pasekondi / makumi mashanu zvikumbiro pasevha = mazana maviri maseva + 500

Zvose zvinogona kuvandudzwa.

1000 zvikumbiro pasekondi

Kutanga kukuru! Chikumbiro chinoratidza kuti chakagadzirisa zvikumbiro zve1000 pasekondi, asi muganho wekunonoka wakatyorwa neSLO. Izvi zvinogona kuoneka mumutsara p99 mune yepamusoro kurudyi girafu. Pasinei nokuti mutsara wep100 wakakwirira zvikuru, kunonoka kwechokwadi kwakakwirira kudarika huwandu hwe60ms. Ngatinyure muprofiling kuti tione kuti application yacho inomboitei.

Profileing

Kunyora, tinoisa mutoro kune 1000 zvikumbiro pasekondi, wozoshandisa pprof kubata data kuti uone kuti application iri kushandisa CPU nguva kupi. Izvi zvinogona kuitwa nekumisa iyo HTTP endpoint pprof, uyezve, pasi pemutoro, chengetedza mhedzisiro uchishandisa curl:

$ curl http://localhost:8080/debug/pprof/profile?seconds=29 > cpu.1000_reqs_sec_no_optimizations.prof

Migumisiro inogona kuratidzwa seizvi:

$ go tool pprof -http=:12345 cpu.1000_reqs_sec_no_optimizations.prof

Girafu inoratidza kupi uye yakawanda sei iyo application inoshandisa CPU nguva. Kubva pane tsananguro kubva Brendan Gregg:

Iyo X axis ihuwandu hwehuwandu hwehuwandu, hwakarongwa nealfabheti (ino haisi nguva), iyo Y axis inoratidza kudzika kwestack, kuverenga kubva zero pa [pamusoro]. Rectangle imwe neimwe ine stack furemu. Iyo yakafara iyo furemu, iyo inowanzove iripo mumatanda. Izvo zviri pamusoro zvinomhanya paCPU, uye zviri pazasi zvinhu zvemwana. Iwo mavara kazhinji haarevi chero chinhu, asi anongosarudzwa chero kuti asiyanise mafuremu.

Analysis - hypothesis

Nekugadzirisa, isu tichatarisa pakuedza kutsvaga kutambisa CPU nguva. Tichatsvaga zvitubu zvakakura zvekushandisa zvisina basa tozvibvisa. Zvakanaka, zvichipihwa iyo profiling inoratidza chaizvo chaizvo iko iko iko iko iko iko kushandisa iko kushandisa kwayo processor nguva, iwe unogona kuzviita kakati wandei, uye iwe zvakare uchafanirwa kushandura iyo sosi kodhi yekushandisa, mhanyisa bvunzo uye woona kuti kuita kunosvika kune chinangwa.

Tichitevera kurudziro dzaBrendan Gregg, tichaverenga chati kubva kumusoro kusvika pasi. Mutsara wega wega unoratidza stack frame (function call). Mutsara wekutanga ndiyo nzvimbo yekupinda muchirongwa, mubereki wedzimwe dzese mafoni (nemamwe mazwi, mamwe ese mafoni anozove nawo pane yavo stack). Mutsetse unotevera wakatosiyana:

Kana ukasimudza chitubu pamusoro pezita rebasa riri pagirafu, nguva yese yayanga iri pachimedu panguva yekugadzirisa inozoratidzwa. Iyo HTTPServe basa yaivepo 65% yenguva, mamwe mabasa ekumhanya runtime.mcall, mstart и gc, akatora nguva yasara. Chokwadi chinonakidza: 5% yenguva yakazara inoshandiswa paDNS mibvunzo:

Makero anotsvagwa nechirongwa ndeePostgresql. Dzvanya pa FindByAge:

Sezvineiwo, chirongwa ichi chinoratidza kuti, mumusimboti, kune matatu makuru masosi anowedzera kunonoka: kuvhura nekuvhara makubatanidza, kukumbira data, nekubatanidza kune dhatabhesi. Girafu inoratidza kuti zvikumbiro zveDNS, kuvhura nekuvhara zvinongedzo zvinotora inenge 13% yenguva yakazara yekuuraya.

Hypothesis: Kushandisazve kubatanidza uchishandisa kubatanidza kunofanira kuderedza nguva yechikumbiro chimwe chete cheHTTP, ichibvumira kupinza kwepamusoro uye kuderera latency..

Kumisikidza application - kuyedza

Isu tinogadziridza iyo kodhi kodhi, edza kubvisa chinongedzo kuPostgresql pachikumbiro chega chega. Sarudzo yekutanga ndeye kushandisa dziva rekubatanidza padanho rekushandisa. Mukuedza uku isu ngatiimise kubatana kwekubatanidza uchishandisa sql mutyairi kuenda:

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)

if err != nil {
   return nil, err
}

Kuitwa, kucherechedza, kuongorora

Mushure mekutangazve bvunzo nezvikumbiro zve1000 pasekondi, zviri pachena kuti p99's latency level yadzokera kune yakajairika neSLO ye60ms!

Muripo wei?

10000 zvikumbiro pasekondi / makumi mashanu zvikumbiro pasevha = mazana maviri maseva + 1000

Ngatiite zvirinani!

2000 zvikumbiro pasekondi

Kupeta kaviri mutoro kunoratidza chinhu chimwe chete, iyo yekumusoro kuruboshwe girafu inoratidza kuti application inokwanisa kugadzirisa zvikumbiro mazana maviri pasekondi, p2000 yakaderera pane 100ms, p60 inogutsa iyo SLO.

Maererano nemutengo:

10000 zvikumbiro pasekondi / makumi mashanu zvikumbiro pasevha = mazana maviri maseva + 2000

3000 zvikumbiro pasekondi

Pano chishandiso chinogona kugadzirisa zviuru zvitatu zvikumbiro nep3000 latency isingasviki makumi matanhatu. Iyo SLO haina kutyorwa, uye mutengo unogamuchirwa sezvinotevera:

10000 zvikumbiro pasekondi / pa3000 zvikumbiro pa server = 4 maseva + 1 (munyori akaunganidza, approx. mushanduri)

Ngatiedze imwe denderedzwa rekuongorora.

Analysis - hypothesis

Isu tinounganidza uye tinoratidza mhedzisiro yedebugging application pazvikumbiro zviuru mazana matatu pasekondi:

Zvakadaro 6% yenguva inopedzerwa pakugadzira zvinongedzo. Kumisikidza dziva kwakavandudza mashandiro, asi iwe uchiri kuona kuti iyo application inoramba ichishanda mukugadzira mitsva yekubatanidza kune database.

Hypothesis: Zvisungo, kunyangwe kuvepo kwedziva, zvichiri kudonhedzwa uye kucheneswa, saka application inoda kuimisazve. Kuseta nhamba yakamirira yekubatanidza kune saizi yedziva kunofanirwa kubatsira nekunonoka nekudzikisa nguva inoshandiswa neapp kugadzira chinongedzo..

Kumisikidza application - kuyedza

Kuedza kuisa MaxIdleConns yakaenzana nesaizi yedziva (yakatsanangurwa zvakare pano):

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)
db.SetMaxIdleConns(8)
if err != nil {
   return nil, err
}

Kuitwa, kucherechedza, kuongorora

3000 zvikumbiro pasekondi

p99 iri pasi pe60ms ine p100 shoma!

Kutarisa girafu remoto kunoratidza kuti kubatana hakuchaonekwe! Ngationgororei zvakadzama pg(*conn).query - isu hationewo kubatana kuri kusimbiswa pano.

mhedziso

Kuongorora kwekuita kwakakosha kuti unzwisise kuti zvinotarisirwa nevatengi uye zvisiri zvekushanda zvinodiwa zviri kuzadzikiswa. Ongororo nekuenzanisa zvinoonekwa nezvinotarisirwa nevatengi zvinogona kubatsira kuona kuti ndezvipi zvinotenderwa nezvisingabvumirwe. Go inopa maturusi ane simba akavakirwa muraibhurari yakajairwa inoita kuti kuongororwa kuve nyore uye kuwanikwa.

Source: www.habr.com