Pazaka zingapo zapitazi, nkhokwe zotsatizana za nthawi zasintha kuchokera kuzinthu zachilendo (zapadera zomwe zimagwiritsidwa ntchito poyang'anira zowunikira (komanso zolumikizidwa ndi mayankho enaake) kapena mapulojekiti a Big Data) kukhala "zogulitsa zogula". Pa gawo la Russian Federation, kuthokoza kwapadera kuyenera kuperekedwa kwa Yandex ndi ClickHouse chifukwa cha izi. Mpaka pano, ngati mukufunikira kusunga zambiri zanthawi yayitali, mumayenera kuzindikira kufunika kopanga stack yoyipa ya Hadoop ndikuyisunga, kapena kulumikizana ndi ma protocol amtundu uliwonse.
Zitha kuwoneka kuti mu 2019 nkhani yomwe TSDB ndiyofunika kugwiritsa ntchito ikhala ndi chiganizo chimodzi chokha: "ingogwiritsani ntchito ClickHouse." Koma ... pali ma nuances.
Zowonadi, ClickHouse ikukula mwachangu, malo ogwiritsira ntchito akukula, ndipo chithandizo chikugwira ntchito kwambiri, koma kodi takhala olandirira kupambana kwapagulu kwa ClickHouse, yomwe yaphimba ena, mwina mayankho ogwira mtima / odalirika?
Kumayambiriro kwa chaka chatha, tinayamba kukonzanso dongosolo lathu loyang'anira, pomwe funso lidabuka losankha nkhokwe yoyenera yosungiramo deta. Ndikufuna kulankhula za mbiri ya chisankho ichi apa.
Kupanga kwa vuto
Choyamba, chofunikira choyambirira. Chifukwa chiyani timafunikira dongosolo lathu lowunikira komanso linapangidwa bwanji?
Tidayamba kupereka chithandizo ku 2008, ndipo pofika 2010 zidawonekeratu kuti zidakhala zovuta kuphatikizira zambiri zazomwe zikuchitika mumayendedwe a kasitomala ndi mayankho omwe adalipo panthawiyo (tikunena, Mulungu andikhululukire, Cacti, Zabbix ndi Graphite wotuluka).
Zofunikira zathu zazikulu zinali:
- thandizo (panthawi imeneyo - ambiri, ndipo m'tsogolomu - mazana) a makasitomala mkati mwa dongosolo limodzi ndipo panthawi imodzimodziyo kukhalapo kwa dongosolo loyang'anira tcheru;
- kusinthasintha pakuwongolera dongosolo lochenjeza (kuwonjezeka kwa zidziwitso pakati pa oyang'anira ntchito, ndandanda, maziko a chidziwitso);
- Kutha kufotokoza mozama ma graph (Zabbix panthawiyo amamasulira ma graph ngati zithunzi);
- kusungidwa kwa nthawi yayitali kwa deta yambiri (chaka chimodzi kapena kuposerapo) ndikutha kuzipeza mwamsanga.
M'nkhaniyi tili ndi chidwi ndi mfundo yomaliza.
Ponena za kusungirako, zofunikira zinali motere:
- dongosolo liyenera kugwira ntchito mofulumira;
- ndizofunika kuti dongosololi likhale ndi mawonekedwe a SQL;
- dongosololi liyenera kukhala lokhazikika komanso lokhala ndi ogwiritsa ntchito ndi chithandizo (titakumana ndi kufunikira kothandizira machitidwe monga MemcacheDB, omwe sanapangidwenso, kapena malo osungiramo a MooseFS, cholozera cha bug chomwe chinasungidwa mu Chinese: tikubwerezanso nkhaniyi kuti polojekiti yathu sinafune);
- kutsata chiphunzitso cha CAP: Consitency (chofunikira) - deta iyenera kukhala yamakono, sitikufuna kuti dongosolo loyang'anira tcheru lisamalandire deta yatsopano ndikulavulira machenjezo okhudza kusafika kwa deta pama projekiti onse; Kulekerera Kugawanika (kofunikira) - sitikufuna kupeza dongosolo la Split Brain; Kupezeka (osati kofunikira, ngati pali chofanana) - titha kusinthira tokha tokha pakachitika ngozi, pogwiritsa ntchito code.
Zodabwitsa ndizakuti, panthawiyo MySQL idakhala yankho labwino kwa ife. Mapangidwe athu a data anali osavuta kwambiri: id ya seva, id yowerengera, sitampu yanthawi ndi mtengo; Sampuli yachangu ya data yotentha idatsimikiziridwa ndi dziwe lalikulu, ndipo zitsanzo za mbiri yakale zidatsimikiziridwa ndi SSD.
Chifukwa chake, tidapeza zitsanzo zatsopano zamasabata awiri, mwatsatanetsatane mpaka 200 ms yachiwiri deta isanatulutsidwe kwathunthu, ndipo tidakhala m'dongosolo lino kwa nthawi yayitali.
Panthawiyi, nthawi inadutsa ndipo kuchuluka kwa deta kunakula. Pofika chaka cha 2016, kuchuluka kwa data kudafika makumi a ma terabytes, zomwe zinali zowononga kwambiri pakusungirako lendi SSD.
Panthawiyi, nkhokwe zachitukuko zinali zofala kwambiri, zomwe tinayamba kuziganizira mozama: m'mabuku a columnar, deta imasungidwa, monga momwe mungamvetsetse, m'mipingo, ndipo ngati muyang'ana deta yathu, n'zosavuta kuona zambiri. chiwerengero cha zobwereza zomwe zingathe, mu Ngati mugwiritsa ntchito nkhokwe ya columnar, ikanikiza pogwiritsa ntchito compression.
Komabe, makina ofunikira a kampaniyo adapitilizabe kugwira ntchito mokhazikika, ndipo sindinkafuna kuyesa kusintha china.
Mu 2017, pamsonkhano wa Percona Live ku San Jose, opanga Clickhouse mwina adalengeza okha kwa nthawi yoyamba. Poyang'ana koyamba, dongosololi linali lokonzekera kupanga (chabwino, Yandex.Metrica ndi njira yopangira nkhanza), chithandizo chinali chofulumira komanso chosavuta, ndipo, chofunika kwambiri, ntchito inali yosavuta. Kuyambira 2018, tayamba ntchito yosinthira. Koma pofika nthawi imeneyo, panali machitidwe ambiri a "akuluakulu" komanso oyesedwa nthawi ya TSDB, ndipo tinaganiza zopatula nthawi yochuluka ndikufanizira njira zina kuti tiwonetsetse kuti palibe njira zina zothetsera Clickhouse, malinga ndi zofunikira zathu.
Kuphatikiza pa zofunika zomwe zafotokozedwa kale, zatsopano zawonekera:
- dongosolo latsopanolo liyenera kupereka osachepera ntchito yofanana ndi MySQL pa kuchuluka kwa hardware;
- kusungidwa kwa dongosolo latsopano kuyenera kutenga malo ochepa kwambiri;
- DBMS iyenera kukhala yosavuta kuyendetsa;
- Ndinkafuna kusintha pulogalamuyo pang'ono posintha DBMS.
Ndi machitidwe ati omwe tayamba kuwaganizira?
Apache Hive / Apache Impala
Zolemba zakale, zoyesedwa pankhondo za Hadoop. Kwenikweni, ndi mawonekedwe a SQL omangidwa pamwamba posungira deta m'mawonekedwe amtundu wa HDFS.
Ubwino.
- Ndi ntchito yokhazikika, ndikosavuta kukulitsa deta.
- Pali njira zothetsera magawo osungira deta (malo ochepa).
- Kuchita mofulumira kwambiri kwa ntchito zofananira pamene zothandizira zilipo.
Chidwi
- Ndi Hadoop, ndipo ndizovuta kugwiritsa ntchito. Ngati sitinakonzekere kutenga yankho lokonzekera mumtambo (ndipo sitinakonzekere malinga ndi mtengo), mulu wonsewo uyenera kusonkhanitsidwa ndikuthandizidwa ndi manja a admins, ndipo sitikufuna kwenikweni. izi.
- Zambiri zaphatikizidwa .
Komabe:
Kuthamanga kumatheka ndikukulitsa kuchuluka kwa ma seva apakompyuta. Mwachidule, ngati ndife kampani yayikulu, yomwe imagwira ntchito zowunikira, ndipo ndikofunikira kuti bizinesi iwonjezere zambiri mwachangu (ngakhale pamtengo wogwiritsa ntchito zinthu zambiri zamakompyuta), izi zitha kukhala zosankha zathu. Koma sitinali okonzeka kuchulukitsa zombo za hardware kuti tifulumire ntchito.
Druid / Pinot
Pali zambiri za TSDB makamaka, koma kachiwiri, Hadoop stack.
pali .
M'mawu ochepa: Druid/Pinot amawoneka bwino kuposa Clickhouse pomwe:
- Muli ndi mitundu yosiyanasiyana ya data (kwa ife, timangolemba mndandanda wanthawi ya ma metric a seva, ndipo, kwenikweni, ili ndi tebulo limodzi. kapangidwe kake komwe kamayenera kuphatikizidwa ndikukonzedwa).
- Komanso, pali zambiri za deta.
- Matebulo ndi zidziwitso zokhala ndi nthawi zimawonekera ndikuzimiririka (ndiko kuti, seti ina ya data idafika, idawunikidwa ndikuchotsedwa).
- Palibe njira yodziwikiratu yomwe deta ingagawidwe.
Nthawi zina, ClickHouse imachita bwino, ndipo ndi nkhani yathu.
Dinani Nyumba
- SQL ngati
- Zosavuta kusamalira.
- Anthu amati zimagwira ntchito.
Amasankhidwa kuti ayesedwe.
InfluxDB
Njira ina yakunja kwa ClickHouse. Za minuses: Kupezeka Kwapamwamba kumangopezeka muzamalonda, koma kuyenera kufananizidwa.
Amasankhidwa kuti ayesedwe.
Cassandra
Kumbali imodzi, tikudziwa kuti imagwiritsidwa ntchito kusungira nthawi ya metric ndi machitidwe owunikira monga, mwachitsanzo, kapena OkMeter. Komabe, pali zenizeni.
Cassandra si malo osungirako zinthu zakale. Zikuwoneka ngati mawonedwe a mzere, koma mzere uliwonse ukhoza kukhala ndi chiwerengero chosiyana cha mizati, zomwe zimapangitsa kuti zikhale zosavuta kupanga mawonekedwe a columnar. M'lingaliro limeneli, zikuwonekeratu kuti ndi malire a mizati 2 biliyoni, ndizotheka kusunga deta ina muzitsulo (ndi nthawi yomweyo). Mwachitsanzo, mu MySQL pali malire a mizati 4096 ndipo n'zosavuta kupunthwa cholakwika ndi code 1117 ngati muyesera kuchita chimodzimodzi.
Injini ya Cassandra ikuyang'ana kwambiri kusunga deta yambiri mu dongosolo logawidwa popanda mbuye, ndipo Cassandra CAP theorem yomwe tatchula pamwambapa ikukhudza AP, ndiko kuti, za kupezeka kwa deta ndi kukana kugawa. Chifukwa chake, chida ichi chikhoza kukhala chabwino ngati mungofunika kulembera ku Nawonsokitale iyi ndipo simumawerenga kawirikawiri. Ndipo apa ndizomveka kugwiritsa ntchito Cassandra ngati malo osungira "ozizira". Ndiko kuti, monga nthawi yayitali, malo odalirika osungiramo zambiri za mbiri yakale zomwe sizikufunika kawirikawiri, koma zikhoza kubwezedwa ngati kuli kofunikira. Komabe, chifukwa cha kukwanira, tidzayesanso. Koma, monga ndanenera kale, palibe chikhumbo cholembanso kachidindo kameneka kamene kamasankhidwa, kotero tidzayesa pang'ono - popanda kusintha ndondomeko ya database kuti ikhale yeniyeni ya Cassandra.
Prometheus
Chabwino, chifukwa cha chidwi, tinaganiza zoyesa ntchito ya Prometheus yosungirako - kuti timvetse ngati tili othamanga kapena ochedwa kusiyana ndi mayankho omwe alipo komanso kuchuluka kwake.
Njira zoyesera ndi zotsatira
Chifukwa chake, tidayesa nkhokwe 5 pamasinthidwe 6 otsatirawa: ClickHouse (1 node), ClickHouse (tebulo logawidwa la 3 node), InfluxDB, Mysql 8, Cassandra (3 node) ndi Prometheus. Dongosolo la mayeso lili motere:
- kwezani mbiri yakale kwa sabata (840 miliyoni zamtengo patsiku; 208 metrics);
- timapanga chojambula chojambulira (mitundu 6 yolemetsa idaganiziridwa, onani pansipa);
- Kufanana ndi kujambula, nthawi ndi nthawi timasankha, kutengera zopempha za wogwiritsa ntchito ma chart. Kuti tisasokoneze zinthu kwambiri, tidasankha ma data a 10 metrics (ndiwo ndendende kuchuluka komwe kuli pa graph ya CPU) kwa sabata.
Timadzaza potengera machitidwe a wothandizira athu, omwe amatumiza milingo ku metric iliyonse kamodzi masekondi 15 aliwonse. Pa nthawi yomweyo, timakonda zosiyanasiyana:
- chiwerengero chonse cha ma metric omwe deta imalembedwa;
- nthawi yotumizira zinthu ku metric imodzi;
- kukula kwa gulu.
Za kukula kwa batch. Popeza sikovomerezeka kukweza pafupifupi nkhokwe zathu zonse zoyesera ndi zoyikapo kamodzi, tidzafunika kutumiza komwe kumasonkhanitsa ma metric omwe akubwera ndikuwagawa m'magulu ndikumalemba ku database ngati batch insert.
Komanso, kuti mumvetse bwino momwe mungatanthauzire zomwe mwalandira, tiyeni tiyerekeze kuti sitikutumiza ma metric, koma ma metrics amapangidwa kukhala ma seva - ma metric 125 pa seva. Apa seva ndi chinthu chenicheni - kungomvetsetsa kuti, mwachitsanzo, ma metric 10000 amafanana ndi ma seva pafupifupi 80.
Ndipo apa, poganizira zonsezi, ndi njira zathu 6 zolembera zolemba:
Pali mfundo ziwiri apa. Choyamba, kwa Cassandra makulidwe awa adakhala akulu kwambiri, pamenepo tidagwiritsa ntchito mfundo za 50 kapena 100. Ndipo chachiwiri, popeza Prometheus amagwira ntchito mosamalitsa kukoka, i.e. iyo yokha imapita ndikusonkhanitsa deta kuchokera kumagwero a metrics (ndipo ngakhale pushgateway, ngakhale dzina lake, silisintha kwenikweni), katundu wofananawo adakhazikitsidwa pogwiritsa ntchito ma configs osakanikirana.
Zotsatira za mayeso ndi izi:
Choyenera kuzindikira: zitsanzo zofulumira kwambiri kuchokera ku Prometheus, zitsanzo zochedwa kwambiri kuchokera ku Cassandra, zitsanzo zochepetsetsa mosadziwika kuchokera ku InfluxDB; Pankhani ya liwiro lojambulira, ClickHouse adapambana aliyense, ndipo Prometheus satenga nawo gawo pampikisano, chifukwa amadzipangira okha ndipo sitiyesa chilichonse.
Pamapeto pake: ClickHouse ndi InfluxDB adadziwonetsera okha kuti ndiabwino kwambiri, koma gulu lochokera ku Influx likhoza kumangidwa kokha pamaziko a Enterprise version, yomwe imawononga ndalama, pamene ClickHouse sichiwononga chilichonse ndipo imapangidwa ku Russia. Ndizomveka kuti ku USA chisankhocho chikukomera inInfluxDB, ndipo m'dziko lathu ndichokomera ClickHouse.
Source: www.habr.com