Monga zolemba zambiri, pali vuto ndi ntchito yogawidwa, tiyeni tiyitane msonkhanowu Alvin. Nthawi iyi sindinapeze vuto ndekha, anyamata ochokera kumbali ya kasitomala adandidziwitsa.

Tsiku lina ndinadzuka ndikupeza imelo yokhumudwa chifukwa chochedwetsa ndi Alvin, zomwe tinkafuna kuziyambitsa posachedwa. Makamaka, kasitomala adakumana ndi 99th percentile latency m'chigawo cha 50ms, pamwamba pa bajeti yathu yochedwa. Izi zinali zodabwitsa pamene ndimayesa ntchitoyo mozama, makamaka pa latency, yomwe ndi madandaulo wamba.

Ndisanayike Alvin kuyesa, ndinayesa zoyesa zambiri pa mafunso a 40k pamphindi (QPS), zonse zikuwonetsa latency zosakwana 10ms. Ndinali wokonzeka kulengeza kuti sindinagwirizane ndi zotsatira zawo. Koma ndikuyang'ananso kalatayo, ndidawona china chatsopano: sindinayese ndendende zomwe adazitchula, QPS yawo inali yotsika kwambiri kuposa yanga. Ndinayesa pa 40k QPS, koma iwo okha pa 1k. Ndidayesanso kuyesa kwina, nthawi ino ndi QPS yotsika, kuti ndingowasangalatsa.

Popeza ndikulemba mabulogu za izi, mwina mwazindikira kale kuti manambala awo anali olondola. Ndinayesa kasitomala wanga mobwerezabwereza ndi zotsatira zofanana: zopempha zochepa sizimangowonjezera latency, koma zimawonjezera chiwerengero cha zopempha ndi latency yaikulu kuposa 10ms. Mwa kuyankhula kwina, ngati pa 40k QPS panali zopempha za 50 pamphindi pamwamba pa 50ms, ndiye pa 1k QPS panali zopempha 100 pamwamba pa 50ms sekondi iliyonse. Zodabwitsa!

Kuchepetsa kusaka

Mukakumana ndi vuto la latency mu dongosolo logawidwa ndi zigawo zambiri, sitepe yoyamba ndiyo kupanga mndandanda wafupipafupi wa okayikira. Tiyeni tifufuze mozama muzomangamanga za Alvin:

Malo abwino oyambira ndi mndandanda wa kusintha kwa I / O komwe kunachitika (ma foni a pa netiweki / kuyang'ana kwa disk, ndi zina). Tiyeni tiyese kudziwa komwe kuchedwa kuli. Kupatula I / O yodziwikiratu ndi kasitomala, Alvin amatenga gawo lowonjezera: amapeza sitolo ya data. Komabe, kusungirako uku kumagwira ntchito m'gulu lomwelo monga Alvin, kotero kuti latency iyenera kukhala yochepa kusiyana ndi kasitomala. Choncho, mndandanda wa okayikira:

1. Kuyimba foni kuchokera kwa kasitomala kupita kwa Alvin.
2. Imbani foni kuchokera kwa Alvin kupita ku data store.
3. Sakani pa disk mu data store.
4. Kuyimba foni kuchokera kumalo osungiramo data kupita kwa Alvin.
5. Kuyimba foni kwa Alvin kupita kwa kasitomala.

Tiyeni tiyese kumvetsetsa mfundo zina.

Kusungirako deta kulibe chochita nazo

Chinthu choyamba chomwe ndidachita ndikusinthira Alvin kukhala seva ya ping-ping yomwe simayankha zopempha. Ikalandira pempho, imabweza yankho lopanda kanthu. Ngati latency ikuchepa, ndiye kuti cholakwika mu Alvin kapena kusungitsa malo osungiramo data sichinamveke. Pakuyesa koyamba timapeza graph iyi:

Monga mukuwonera, palibe kusintha mukamagwiritsa ntchito seva ya ping-ping. Izi zikutanthauza kuti malo osungiramo data samachulukitsa latency, ndipo mndandanda wa okayikira wadulidwa pakati:

1. Kuyimba foni kuchokera kwa kasitomala kupita kwa Alvin.
2. Kuyimba foni kwa Alvin kupita kwa kasitomala.

Zabwino! Mndandandawu ukucheperachepera. Ndinkaganiza kuti ndatsala pang'ono kuzindikira chifukwa chake.

gRPC

Ino ndi nthawi yoti ndikuwonetseni wosewera watsopano: gRPC. Ili ndi laibulale yotseguka yochokera ku Google yolumikizirana CPR... Ngakhale gRPC zokongoletsedwa bwino komanso zogwiritsidwa ntchito kwambiri, iyi inali nthawi yanga yoyamba kuzigwiritsa ntchito pamakina akukula uku ndipo ndimayembekezera kuti kukhazikitsidwa kwanga kudzakhala kocheperako - kunena pang'ono.

kupezeka gRPC pagululi zidabweretsa funso latsopano: mwina ndikukhazikitsa kwanga kapena ine ndekha gRPC kumayambitsa vuto la latency? Kuwonjeza wokayikira watsopano pamndandanda:

1. Wothandizira amayimbira laibulale gRPC
2. Library gRPC imapanga foni yamtaneti ku library pa kasitomala gRPC pa seva
3. Library gRPC kulumikizana ndi Alvin (palibe ntchito ngati seva ya ping-pong)

Kuti ndikupatseni lingaliro la momwe code imawonekera, kasitomala wanga / Alvin kukhazikitsa sikusiyana kwambiri ndi kasitomala-seva. async zitsanzo.

Zindikirani: Mndandanda womwe uli pamwambawu ndi wosavuta chifukwa gRPC zimapangitsa kuti zitheke kugwiritsa ntchito yanu (chitsanzo?) chopangira ulusi, momwe mulingo wakupha umalumikizidwa gRPC ndi kukhazikitsa kwa ogwiritsa ntchito. Pofuna kuphweka, tidzamamatira ku chitsanzo ichi.

Kufotokozera kudzakonza zonse

Nditadutsa m'malo ogulitsira, ndimaganiza kuti ndatsala pang'ono kumaliza: "Tsopano ndi zophweka! Tiyeni tigwiritse ntchito mbiriyo ndikuwona komwe kuchedwerako kumachitika. ” Ine wokonda kwambiri mbiri yolondola, chifukwa ma CPU amathamanga kwambiri ndipo nthawi zambiri sakhala olepheretsa. Nthawi zambiri kuchedwa kumachitika pamene purosesa iyenera kusiya kukonza kuti ichite zina. Kulondola Kwambiri kwa CPU kumachita izi: imalemba zonse molondola kusintha kwa nkhani ndipo imafotokoza momveka bwino pamene kuchedwa kumachitika.

Ndinatenga mbiri zinayi: ndi QPS yapamwamba (low latency) ndi seva ya ping-pong yokhala ndi QPS yochepa (high latency), onse kumbali ya kasitomala komanso kumbali ya seva. Ndipo pokhapokha, ndinatenganso chitsanzo cha purosesa. Poyerekeza mbiri, nthawi zambiri ndimayang'ana ma call osadziwika bwino. Mwachitsanzo, kumbali yoyipa yokhala ndi latency yayikulu pali masiwichi ochulukirapo (nthawi 10 kapena kupitilira apo). Koma kwa ine, kuchuluka kwa masinthidwe amtunduwu kunali kofanana. Ndinadabwa kuona kuti panalibe chilichonse chochititsa chidwi kumeneko.

Zowonjezera Debugging

Ndinali wosimidwa. Sindinadziwe kuti ndi zida ziti zomwe ndingagwiritse ntchito, ndipo dongosolo langa lotsatira linali kubwereza zoyesererazo mosiyanasiyana m'malo mozindikira vutolo.

Zingatani Zitati

Ndinali ndi nkhawa za 50ms latency kuyambira pachiyambi. Iyi ndi nthawi yayikulu kwambiri. Ndinaganiza zodula ma code mpaka nditazindikira kuti ndi gawo liti lomwe likuyambitsa cholakwikacho. Kenako panabwera kuyesa komwe kunathandiza.

Monga mwachizolowezi, poyang'ana kumbuyo zikuwoneka kuti zonse zinali zoonekeratu. Ndinayika kasitomala pamakina omwewo monga Alvin - ndikutumiza pempho kwa localhost. Ndipo kuwonjezeka kwa latency kwapita!

Chinachake chalakwika ndi netiweki.

Kuphunzira luso la injiniya pa intaneti

Ndiyenera kuvomereza: chidziwitso changa chaukadaulo wapaintaneti ndi choyipa, makamaka poganizira kuti ndimagwira nawo ntchito tsiku lililonse. Koma netiweki inali yokayikira kwambiri, ndipo ndimayenera kuphunzira momwe ndingathetsere vutoli.

Mwamwayi, intaneti imakonda omwe akufuna kuphunzira. Kuphatikizika kwa ping ndi tracert kumawoneka ngati koyambira kokwanira kuthetsa zovuta zoyendera ma netiweki.

Choyamba, ndinayambitsa PsPing kupita ku doko la Alvin la TCP. Ndinagwiritsa ntchito makonda osasintha - palibe chapadera. Pa ma pings opitilira chikwi, palibe yomwe idapitilira 10ms, kupatula yoyamba kutenthetsa. Izi ndi zotsutsana ndi kuwonjezeka kwa latency ya 50ms pa 99th percentile: pamenepo, pazopempha 100 zilizonse, tikadayenera kuwona pempho limodzi ndi latency ya 50ms.

Kenako ndinayesa tracer: Pakhoza kukhala vuto pa imodzi mwa mfundo zomwe zili panjira pakati pa Alvin ndi kasitomala. Koma wofufuza uja anabwereranso chimanjamanja.

Chifukwa chake sinali khodi yanga, kukhazikitsa kwa gRPC, kapena netiweki yomwe imayambitsa kuchedwa. Ndinayamba kuda nkhawa kuti sindingamvetse izi.

Tsopano tili pa OS yanji

gRPC amagwiritsidwa ntchito kwambiri pa Linux, koma zachilendo pa Windows. Ndinaganiza zoyesa kuyesa, komwe kunathandiza: Ndinapanga makina enieni a Linux, ndinapanga Alvin kwa Linux, ndikuyiyika.

Ndipo izi ndi zomwe zidachitika: seva ya Linux ping-pong inalibe kuchedwa kofanana ndi Windows host host yofananira, ngakhale gwero la data silinali losiyana. Zikuoneka kuti vuto lili mu kukhazikitsa gRPC kwa Windows.

Algorithm ya Nagle

Nthawi yonseyi ndimaganiza kuti ndikuphonya mbendera gRPC. Tsopano ndikumvetsa chomwe chiri kwenikweni gRPC Windows mbendera ikusowa. Ndinapeza laibulale yamkati ya RPC yomwe ndinali ndi chidaliro kuti idzagwira ntchito bwino pamabendera onse Kupambana. Kenako ndidawonjeza mbendera zonsezi ku gRPC ndikuyika Alvin pa Windows, mu seva ya ping-pong ya Windows!

Pafupifupi Zachitika: Ndinayamba kuchotsa mbendera zowonjezeredwa imodzi imodzi mpaka kubwereranso kubwerera kuti ndidziwe chomwe chinayambitsa. Zinali zoipa TCP_NODELAY, Kusintha kwa algorithm ya Nagle.

Algorithm ya Nagle kuyesa kuchepetsa kuchuluka kwa mapaketi omwe amatumizidwa pa netiweki pochedwetsa kutumiza mauthenga mpaka kukula kwa paketi kupitilira kuchuluka kwa ma byte. Ngakhale izi zitha kukhala zabwino kwa ogwiritsa ntchito wamba, zimawononga ma seva anthawi yeniyeni chifukwa OS imachedwetsa mauthenga ena, ndikupangitsa kuti pakhale QPS yotsika. U gRPC mbendera iyi idakhazikitsidwa pakukhazikitsa kwa Linux kwa soketi za TCP, koma osati pa Windows. Ine ndine uyu kukonzedwa.

Pomaliza

Kuchedwa kwapamwamba pa QPS yotsika kudachitika chifukwa cha kukhathamiritsa kwa OS. Poyang'ana m'mbuyo, kufotokozera sikunapeze latency chifukwa kunachitika mu kernel mode osati mu wosuta mode. Sindikudziwa ngati algorithm ya Nagle imatha kuwonedwa kudzera pazithunzi za ETW, koma zingakhale zosangalatsa.

Ponena za kuyesa komweko, mwina sikunakhudze nambala yeniyeni yapaintaneti ndipo ma algorithm a Nagle sanayendetse, chifukwa chake zovuta za latency zidachoka pomwe kasitomala adafika ku Alvin kudzera pagulu.

Nthawi ina mukadzawona kuwonjezeka kwa latency pomwe kuchuluka kwa zopempha pamphindikati kucheperachepera, algorithm ya Nagle iyenera kukhala pamndandanda wanu wa omwe akukayikira!

Source: www.habr.com