Ndiye mumatolera ma metric. Monga ife tiri. Timasonkhanitsanso ma metric. Inde, zofunika kwa bizinesi. Lero tikambirana za ulalo woyamba wa dongosolo lathu lowunikira - seva yolumikizana ndi statsd-compatible aggregation , chifukwa chake tinalembera ndi chifukwa chake tinamusiyira brubeck.
Kuchokera m'nkhani zathu zam'mbuyomu (, ) mutha kupeza kuti mpaka nthawi yomwe tidasonkhanitsa ma marks pogwiritsa ntchito . Zalembedwa mu C. Kuchokera pamawonedwe a code, ndizosavuta ngati pulagi (izi ndizofunikira pamene mukufuna kupereka) ndipo, chofunika kwambiri, zimagwira ntchito zathu za 2 miliyoni metrics pamphindi (MPS) pachimake. popanda mavuto. Zolembazo zimathandizira 4 miliyoni MPS yokhala ndi nyenyezi. Izi zikutanthauza kuti mupeza chiwerengerocho ngati mukonza maukonde molondola pa Linux. (Sitikudziwa kuti mungapeze a MPS angati ngati mutasiya netiweki momwe zilili). Ngakhale kuti anali ndi ubwino umenewu, tinali ndi madandaulo angapo aakulu ponena za brubeck.
Ndemanga 1. Github, woyambitsa pulojekitiyi, anasiya kuichirikiza: kusindikiza zigamba ndi kukonza, kuvomereza zathu ndi (osati zathu zokha) PR. M'miyezi ingapo yapitayo (kwinakwake kuyambira February-March 2018), ntchito yayambiranso, koma izi zisanachitike panali pafupifupi zaka 2 za bata lathunthu. Kuphatikiza apo, ntchitoyi ikukonzedwa , zomwe zingakhale cholepheretsa kwambiri kuyambitsa zatsopano.
Ndemanga 2. Kulondola kwa masamu. Brubeck amasonkhanitsa zikhalidwe zonse za 65536 pakuphatikiza. Kwa ife, pama metrics ena, munthawi yophatikizira (masekondi 30), zochulukirapo zitha kufika (1 pachimake). Chifukwa cha zitsanzo izi, zotsika kwambiri komanso zochepa zimawoneka ngati zopanda ntchito. Mwachitsanzo, monga chonchi:
Monga izo zinali
Zikadayenera kukhala bwanji
Pachifukwa chomwecho, ndalama zambiri zimawerengedwa molakwika. Onjezani apa cholakwika chokhala ndi kusefukira kwa 32-bit, komwe kumatumiza seva ku segfault ikalandira metric yowoneka ngati yosalakwa, ndipo chilichonse chimakhala chabwino. Vutoli, mwa njira, silinakonzedwe.
Ndipo potsiriza Funsani X. Panthawi yolemba, ndife okonzeka kuziwonetsa kuzinthu zonse 14 zogwirira ntchito kapena zochepa zomwe tidapeza. Tiyerekeze kuti zomangamanga zina zakula kwambiri kotero kuti kuvomereza 4 miliyoni MPS sikukwanira. Kapena ngakhale sichinakule, koma ma metrics ndi ofunikira kale kwa inu kotero kuti ngakhale aifupi, 2-3 mphindi kuviika mu matchati akhoza kale kukhala ovuta ndi kuchititsa kuvutika maganizo kosatheka pakati pa oyang'anira. Popeza kuchiza matenda ovutika maganizo ndi ntchito yosayamika, pakufunika njira zothetsera vutoli.
Choyamba, kulolerana kolakwa, kotero kuti vuto ladzidzidzi pa seva silimayambitsa matenda a zombie apocalypse muofesi. Kachiwiri, kukulitsa kuti muthe kuvomereza ma MPS opitilira 4 miliyoni, osakumba mozama mu network ya Linux ndikukula modekha "m'lifupi" mpaka kukula kofunikira.
Popeza tinali ndi malo owonjezera, tinaganiza zoyamba ndi kulolera zolakwa. "ZA! Kulekerera zolakwa! Ndizosavuta, titha kuchita, "tidaganiza ndikuyambitsa ma seva a 2, ndikukweza brubeck iliyonse. Kuti tichite izi, tidayenera kutengera kuchuluka kwa magalimoto ndi ma metric ku maseva onse awiri ndikulembera izi . Tinathetsa vuto la kulekerera zolakwika ndi izi, koma ... osati bwino kwambiri. Poyamba zonse zinkawoneka bwino: brubeck aliyense amasonkhanitsa mtundu wake wa aggregation, amalemba deta ku Graphite kamodzi pa masekondi 30, ndikulemba nthawi yakale (izi zimachitika kumbali ya Graphite). Ngati seva imodzi yalephera mwadzidzidzi, nthawi zonse timakhala ndi yachiwiri yokhala ndi deta yakeyake. Koma apa pali vuto: ngati seva ikulephera, "mawona" amawonekera pazithunzi. Izi ndichifukwa choti nthawi za brubeck 30-sekondi sizimalumikizidwa, ndipo panthawi ya ngozi imodzi mwazo sinalembedwe. Seva yachiwiri ikayamba, zomwezo zimachitika. Zolekerera, koma ndikufuna bwino! Vuto la scalability silinathenso. Ma metrics onse "amawulukira" ku seva imodzi, chifukwa chake timangokhala ndi 2-4 miliyoni MPS, kutengera kuchuluka kwa maukonde.
Ngati mumaganizira pang'ono za vutoli ndipo panthawi imodzimodziyo kukumba chisanu ndi fosholo, ndiye kuti lingaliro lodziwikiratu lotsatira likhoza kubwera m'maganizo: muyenera statsd yomwe ingagwire ntchito yogawidwa. Ndiko kuti, yomwe imagwiritsa ntchito kulumikizana pakati pa node mu nthawi ndi ma metrics. "Zowona, yankho lotere mwina lilipo kale," tidatero ndikupita ku Googleβ¦. Ndipo sanapeze kanthu. Pambuyo podutsa zolemba za statsd zosiyanasiyana ( kuyambira pa Disembala 11.12.2017, XNUMX), sitinapeze chilichonse. Mwachiwonekere, ngakhale opanga kapena ogwiritsa ntchito mayankhowa sanakumanepo ndi ma metric ochuluka, apo ayi iwo angabwere ndi chinachake.
Kenako tidakumbukira za "chidole" statsd - bioyino, yomwe idalembedwa ku Just for Fun hackathon (dzina la polojekitiyi lidapangidwa ndi script isanayambe hackathon) ndipo tidazindikira kuti timafunikira ma statsd athu mwachangu. Zachiyani?
- chifukwa pali ma statsd ochepa kwambiri padziko lapansi,
- chifukwa ndizotheka kupereka zomwe mukufuna kapena pafupi ndi zomwe mukufuna kulolerana komanso scalability (kuphatikiza ma metric ophatikizika pakati pa ma seva ndikuthetsa vuto la kutumiza mikangano),
- chifukwa ndizotheka kuwerengera ma metric molondola kuposa momwe brubeck amachitira,
- chifukwa mutha kusonkhanitsa ziwerengero zambiri nokha, zomwe brubeck sanatipatse,
- chifukwa ndinali ndi mwayi wopanga hyperperformance kugawira labu ntchito yanga, zomwe sizingabwereze kwathunthu kamangidwe ka hyperfor ina yofananira ... chabwino, ndi momwemo.
Zolemba pa chiyani? Inde, mu Dzimbiri. Chifukwa chiyani?
- chifukwa panali kale prototype yankho,
- chifukwa wolemba nkhaniyi ankadziwa kale Rust panthawiyo ndipo anali wofunitsitsa kulemba chinachake mmenemo kuti apange ndi mwayi wochiyika poyera,
- chifukwa zilankhulo zomwe zili ndi GC sizoyenera kwa ife chifukwa cha kuchuluka kwa magalimoto omwe alandilidwa (pafupifupi nthawi yeniyeni) komanso kuyimitsa kwa GC ndikosavomerezeka,
- chifukwa mukufunikira magwiridwe antchito apamwamba kwambiri ngati C
- chifukwa dzimbiri amatipatsa concurrency mopanda mantha, ndipo tikadayamba kulemba mu C/C ++, tikanati rakeke mu chiwopsezo kwambiri, buffer kusefukira, mikhalidwe mtundu ndi mawu ena oopsa kuposa brubeck.
Panalinso mkangano wotsutsana ndi Dzimbiri. Kampaniyo inalibe chidziwitso chopanga ma projekiti ku Rust, ndipo tsopano sitikukonzekeranso kuzigwiritsa ntchito pantchito yayikulu. Chifukwa chake, panali mantha akulu kuti palibe chomwe chingachitike, koma tinaganiza zochita mwayi ndikuyesa.
Nthawi inapita...
Potsirizira pake, pambuyo poyesera kangapo kulephera, mtundu woyamba wogwirira ntchito unali wokonzeka. Chinachitika ndi chiyani? Izi nβzimene zinachitika.
Node iliyonse imalandira seti yakeyake ya ma metrics ndikuwaunjikira, ndipo samaphatikiza ma metric amitundu yomwe seti yawo yonse imafunikira pakuphatikiza komaliza. Ma node amalumikizidwa kwa wina ndi mzake ndi mtundu wina wa protocol yogawidwa, yomwe imakulolani kuti musankhe pakati pawo imodzi yokha (pano tinalira) yomwe ili yoyenera kutumiza ma metrics kwa Wamkulu. Vutoli likuthetsedwa ndi , koma mtsogolomo zokhumba za wolembayo zimafikira Raft, komwe woyenera kwambiri adzakhala, ndithudi, kukhala mtsogoleri wogwirizana. Kuphatikiza pa kuvomerezana, ma node nthawi zambiri (kamodzi pa sekondi imodzi mwachisawawa) amatumiza kwa anansi awo zigawo za ma metric ophatikiza omwe adakwanitsa kutolera mu sekondi imeneyo. Zikuoneka kuti makulitsidwe ndi kulolerana zolakwa zimasungidwa - mfundo iliyonse ikadali ndi seti yonse ya ma metrics, koma ma metrics amatumizidwa kale ataphatikizika, kudzera pa TCP ndikuyikidwa mu protocol ya binary, kotero ndalama zobwereza zimachepetsedwa kwambiri poyerekeza ndi UDP. Ngakhale pali ma metric ambiri omwe akubwera, kudzikundikira kumafuna kukumbukira pang'ono komanso CPU yochepa. Kwa ma mertics athu olimba kwambiri, awa ndi ma megabytes ochepa chabe a data. Monga bonasi yowonjezera, sitipeza zolemba zosafunikira ku Graphite, monga momwe zinalili ndi burbeck.
Mapaketi a UDP okhala ndi ma metrics ndi osagwirizana pakati pa node pazida zama network kudzera pa Round Robin yosavuta. Zachidziwikire, zida zapaintaneti sizimawerengera zomwe zili m'mapaketi motero zimatha kukoka mapaketi opitilira 4M pamphindikati, osatchulanso ma metric omwe sadziwa chilichonse. Ngati tiganizira kuti ma metrics samabwera imodzi imodzi pa paketi iliyonse, ndiye kuti sitiwoneratu vuto lililonse la magwiridwe antchito pamalo ano. Seva ikasweka, chipangizo cha netiweki mwachangu (mkati mwa masekondi 1-2) chimazindikira izi ndikuchotsa seva yosweka pozungulira. Chifukwa cha izi, ma node (osakhala atsogoleri) amatha kuyatsidwa ndi kuzimitsidwa mosazindikira ma chart. Kuchuluka komwe timataya ndi gawo la ma metric omwe adabwera sekondi yomaliza. Kutayika kwadzidzidzi / kutseka / kusintha kwa mtsogoleri kudzapangabe vuto laling'ono (nthawi yachiwiri ya 30 ikadali yosagwirizana), koma ngati pali kulankhulana pakati pa node, mavutowa akhoza kuchepetsedwa, mwachitsanzo, potumiza mapaketi ogwirizanitsa. .
Pang'ono za kapangidwe ka mkati. Ntchitoyi, ndithudi, imakhala yochuluka, koma zomangamanga ndizosiyana ndi zomwe zimagwiritsidwa ntchito mu brubeck. Ulusi mu brubeck ndi wofanana - aliyense wa iwo ali ndi udindo wosonkhanitsa zidziwitso komanso kuphatikiza. Mu bioyino, ogwira ntchito amagawidwa m'magulu awiri: omwe ali ndi udindo pa intaneti ndi omwe ali ndi udindo wogwirizanitsa. Gawoli limakupatsani mwayi wowongolera pulogalamuyo mosinthika kutengera mtundu wa ma metric: komwe kumafunika kuphatikizika kwakukulu, mutha kuwonjezera ma aggregator, pomwe pali magalimoto ambiri pamaneti, mutha kuwonjezera kuchuluka kwa maukonde. Pakalipano, pa maseva athu timagwira ntchito mu 8 network ndi 4 aggregation flows.
Gawo la kuwerengera (lomwe limayang'anira kuphatikizira) ndilotopetsa. Ma buffers odzazidwa ndi ma network amagawidwa pakati pa mawerengedwe owerengera, pomwe amagawidwa ndikuphatikizidwa. Akafunsidwa, ma metrics amaperekedwa kuti atumizidwe ku ma node ena. Zonsezi, kuphatikizapo kutumiza deta pakati pa ma node ndi kugwira ntchito ndi Consul, zimachitidwa asynchronously, kuthamanga pa chimango. .
Mavuto ochulukirapo panthawi ya chitukuko adayambitsidwa ndi gawo la netiweki lomwe limayang'anira kulandira ma metric. Cholinga chachikulu cholekanitsa ma network akuyenda m'mabungwe osiyana chinali kufuna kuchepetsa nthawi yomwe kuyenda kumawononga osati kuti muwerenge deta kuchokera ku socket. Zosankha zogwiritsa ntchito asynchronous UDP ndi recvmsg wamba zidasowa mwachangu: yoyamba imawononga CPU yochulukirapo kuti ikonze zochitika, yachiwiri imafuna masinthidwe ambiri. Chifukwa chake imagwiritsidwa ntchito okhala ndi zotchingira zazikulu (ndi zotchingira, maofesala njonda, palibe kanthu kwa inu!). Thandizo la UDP lokhazikika limasungidwa pamilandu yopepuka pomwe recvmmsg sikufunika. Mumawonekedwe a multimessage, ndizotheka kukwaniritsa chinthu chachikulu: nthawi zambiri, ulusi wa netiweki umathamangitsa mzere wa OS - umawerenga zambiri kuchokera pasoketi ndikusamutsira ku buffer yogwiritsa ntchito, ndikungosintha nthawi zina kuti apatse chosungira chodzaza. ophatikiza. Mzere mu socket siunjikana, kuchuluka kwa mapaketi otsika sikukula.
ndemanga
M'malo osasinthika, kukula kwa buffer kumayikidwa kukhala kwakukulu. Ngati mwadzidzidzi mwaganiza kuyesa seva nokha, mungakumane ndi mfundo yakuti mutatha kutumiza ma metrics ochepa, iwo sadzafika ku Graphite, otsalira mu buffer ya network. Kuti mugwire ntchito ndi ma metric ochepa, muyenera kuyika bufsize ndi kukula kwa mzere wantchito kukhala zing'onozing'ono mu config.
Pomaliza, ma chart ena okonda ma chart.
Ziwerengero za kuchuluka kwa ma metric omwe akubwera pa seva iliyonse: oposa 2 miliyoni MPS.
Kuyimitsa imodzi mwa node ndikugawanso ma metric omwe akubwera.
Ziwerengero zamametric otuluka: nodi imodzi yokha imatumiza nthawi zonse - abwana akuwukira.
Ziwerengero za magwiridwe antchito a node iliyonse, poganizira zolakwika m'ma module osiyanasiyana.
Tsatanetsatane wa ma metric omwe akubwera (mametric mayina amabisika).
Kodi tikukonzekera kuchita chiyani ndi zonsezi? Inde, lembani code, damn...! Ntchitoyi idakonzedweratu kuti ikhale yotseguka ndipo ikhalabe choncho moyo wake wonse. Zolinga zathu zaposachedwa zikuphatikiza kusinthira ku mtundu wathu wa Raft, kusintha ma protocol a anzanu kukhala osunthika kwambiri, kuyambitsa ziwerengero zamkati, mitundu yatsopano yama metric, kukonza zolakwika ndi kukonza kwina.
Inde, aliyense ndi wolandiridwa kuti athandize pa chitukuko cha polojekiti: pangani PR, Nkhani, ngati n'kotheka tidzayankha, kukonza, ndi zina zotero.
Ndi zomwe zikunenedwa, ndizo zonse anthu, gulani njovu zathu!
Source: www.habr.com