HighLoad++ Moscow 2018, Congress Hall. Novemba 9, 15:00
Izifinyezo nezethulo:
U-Yuri Nasretdinov (VKontakte): umbiko uzokhuluma ngesipiliyoni sokuqalisa i-ClickHouse enkampanini yethu - kungani siyidinga, sigcina idatha engakanani, siyibhala kanjani, njalonjalo.
Izinto ezingeziwe:
U-Yuri Nasretdinov: - Sanibonani nonke! Igama lami nginguYuri Nasretdinov, njengoba sengethulwe kakade. Ngisebenza ku-VKontakte. Ngizokhuluma ngendlela esifaka ngayo idatha ku-ClickHouse kusuka kumikhumbi yethu yeseva (amashumi ezinkulungwane).
Ziyini izingodo futhi kungani ziqoqwa?
Esizokutshela kona: esikwenzile, kungani besidinga i-“ClickHouse”, ngokulandelana, kungani siyikhethile, hlobo luni lokusebenza ongaluthola cishe ngaphandle kokumisa noma yini ngokukhethekile. Ngizokutshela ngokuqhubekayo mayelana namathebula e-buffer, ngezinkinga esasinazo ngawo kanye nezisombululo zethu esizithuthukise ngomthombo ovulekile - i-KittenHouse ne-Lighthouse.
Kungani kwakudingeka senze noma yini (konke kuhlale kukuhle ku-VKontakte, akunjalo?). Besifuna ukuqoqa amalogi okususa iphutha (futhi bekunamakhulu ama-terabytes edatha lapho), mhlawumbe ngandlela thile bekungaba lula kakhulu ukubala izibalo; futhi sineqoqo lamashumi ezinkulungwane zamaseva lapho konke lokhu kudinga ukwenziwa khona.
Kungani sinqume? Cishe besinezixazululo zokugcina izingodo. Lapha - kukhona "Bakend VK" yomphakathi enjalo. Ngincoma kakhulu ukubhalisa kuyo.
Ziyini izingodo? Lena injini ebuyisela amalungu afanayo angenalutho. Izinjini ku-VK yilokho abanye abakubiza ngama-microservices. Futhi nasi isitika esimamathekayo (okuningi okuthandwayo). Kanjani? Nokho, lalela ngokuqhubekayo!
Yini engasetshenziswa ukugcina izingodo? Akunakwenzeka ingasaphathwa iHadup. Bese, isibonelo, i-Rsyslog (ukugcina lawa malogi kumafayela). I-LSD. Ubani owaziyo ukuthi iyini i-LSD? Cha, hhayi le LSD. Gcina amafayela, ngokulandelana, futhi. Yebo, i-ClickHouse inketho engavamile.
Clickhouse kanye nezimbangi: izidingo kanye namathuba
Sifunani? Sifuna ukuqinisekisa ukuthi akudingeki sikhathazeke kakhulu mayelana nokusebenza, ukuze kusebenze ngaphandle kwebhokisi, okungcono ngokucushwa okuncane. Sifuna ukubhala kakhulu, futhi sibhale ngokushesha. Futhi sifuna ukuyigcina kuzo zonke izinhlobo zezinyanga, iminyaka, okungukuthi, isikhathi eside. Singase sifune ukuqonda inkinga ethile abeze nayo kithi bathi, "Kukhona okungasebenzi lapha," futhi lokho kwenzeka ezinyangeni ezi-3 ezedlule), futhi sifuna ukwazi ukubona ukuthi kwenzekeni ezinyangeni ezi-3 ezedlule " Ukucindezelwa kwedatha - kuyacaca ukuthi kungani kungaba ukuhlanganisa - ngoba kunciphisa inani lesikhala elisithathayo.
Futhi sinesidingo esinjalo esithakazelisayo: ngezinye izikhathi sibhala umphumela weminye imiyalo (isibonelo, izingodo), kungaba ngaphezu kwamakhilobhayithi angu-4 kalula. Futhi uma le nto isebenza ngaphezu kwe-UDP, ngakho-ke ayidingi ukuchitha ... ngeke ibe nayo "i-overhead" yokuxhumeka, futhi ngenani elikhulu lamaseva lokhu kuzoba ukuhlanganisa.
Ake sibone ukuthi umthombo ovulekile usinikeza ini. Okokuqala, sine-Logs Engine - lena injini yethu; Empeleni, angenza yonke into, angabhala ngisho nemigqa emide. Nokho, ayiminyanisi ngokusobala idatha - singacindezela amakholomu amakhulu ngokwethu uma sifuna... thina, yebo, asifuni (uma kungenzeka). Inkinga nje ukuthi anganikeza kuphela lokho okumfanelayo enkumbulo yakhe; Ukuze ufunde okusele, udinga ukuthola i-binlog yale injini futhi, ngokufanele, kuthatha isikhathi eside impela.
Yiziphi ezinye izinketho ezikhona? Ngokwesibonelo, "Hadup". Ukusebenza kalula... Ubani ocabanga ukuthi i-Hadup kulula ukuyimisa? Yiqiniso, azikho izinkinga ngokurekhoda. Lapho ufunda, ngezinye izikhathi kuphakama imibuzo. Empeleni, ngingathi mhlawumbe cha, ikakhulukazi izingodo. Ukugcinwa kwesikhathi eside - kunjalo, yebo, ukucindezelwa kwedatha - yebo, izintambo ezinde - kusobala ukuthi ungarekhoda. Kodwa ukurekhoda kusuka kwinani elikhulu lamaseva... Kusafanele wenze okuthile ngokwakho!
I-Rsyslog. Eqinisweni, siyisebenzise njengenketho yokusekelayo ukuze sikwazi ukuyifunda ngaphandle kokulahla i-binlog, kodwa ayikwazi ukubhala imigqa emide; empeleni, ayikwazi ukubhala ngaphezu kwamakhilobhayithi angu-4. Kufanele wenze ukuminyanisa idatha ngendlela efanayo wena. Ukufunda kuzovela kumafayela.
Bese kuba khona "badushka" ukuthuthukiswa LSD. Ngokuyisisekelo ifana ne-“Rsyslog”: isekela izintambo ezinde, kodwa ayikwazi ukusebenza nge-UDP futhi, eqinisweni, ngenxa yalokhu, ngeshwa, izinto eziningi impela zidinga ukubhalwa kabusha lapho. I-LSD idinga ukuklanywa kabusha ukuze ikwazi ukurekhoda kusukela emashumini ezinkulungwane zamaseva.
Futhi lapha! Inketho ehlekisayo i-ElasticSearch. Ungasho kanjani? Wenza kahle ngokufunda, okungukuthi, ufunda ngokushesha, kodwa hhayi kahle kakhulu ngokubhala. Okokuqala, uma icindezela idatha, ibuthakathaka kakhulu. Cishe, ukusesha okugcwele kudinga izakhiwo zedatha ezinkulu kunevolumu yoqobo. Kunzima ukusebenza futhi izinkinga zivame ukuvela ngakho. Futhi, futhi, ukuqopha nge-Elastic - kufanele sizenzele yonke into.
Lapha i-ClickHouse iyindlela ekahle, yebo. Okuwukuphela kwento ukuthi ukuqopha kusuka emashumini ezinkulungwane zamaseva kuyinkinga. Kodwa okungenani kukhona inkinga eyodwa, singazama ukuyixazulula ngandlela-thile. Futhi umbiko osele umayelana nale nkinga. Hlobo luni lokusebenza ongalulindela ku-ClickHouse?
Sizowufaka kanjani? I-MergeTree
Ubani phakathi kwenu ongakaze ezwe noma azi mayelana ne-"ClickHouse"? Ngidinga ukukutshela, akunjalo? Ngokushesha. Ukufakwa lapho - amagigabhithi angu-1-2 ngomzuzwana, ukuqhuma okungafika ku-10 gigabits ngomzuzwana empeleni kungamelana nalokhu kumiswa - kukhona ama-Xeon amabili angu-6-core (okungukuthi, ngisho nanamandla kakhulu), amagigabhayithi angu-256 e-RAM, ama-terabytes angu-20. ku-RAID (akekho omisiwe, izilungiselelo ezizenzakalelayo). U-Alexey Milovidov, umthuthukisi we-ClickHouse, cishe uhlezi lapho ekhala ngoba asizange silungise lutho (konke kusebenze kanjalo kithi). Ngakho-ke, isivinini sokuskena, sithi, cishe imigqa eyizigidi eziyizinkulungwane ezingu-6 ngomzuzwana ingatholakala uma idatha icindezelwe kahle. Uma uthanda % ochungechungeni lombhalo - imigqa eyizigidi eziyi-100 ngomzuzwana, okungukuthi, ibonakala ishesha kakhulu.
Sizowufaka kanjani? Nokho, uyazi ukuthi i-VK isebenzisa i-PHP. Sizofaka kusuka kusisebenzi ngasinye se-PHP nge-HTTP siye kokuthi “ClickHouse”, kuthebula le-MergeTree ngerekhodi ngalinye. Ubani obona inkinga ngalolu hlelo? Ngesizathu esithile, akubona bonke abaphakamisa izandla zabo. Ake ngikutshele.
Okokuqala, kunamaseva amaningi - ngokufanele, kuzoba nokuxhumana okuningi (okubi). Bese kuba ngcono ukufaka idatha ku-MergeTree izikhathi ezingaphezu kwesisodwa ngomzuzwana. Futhi ubani owaziyo ukuthi kungani? Kulungile, kulungile. Ngizokutshela okwengeziwe ngalokhu. Omunye umbuzo othokozisayo ukuthi asenzi izibalo, asidingi ukucebisa idatha, asidingi amaseva aphakathi nendawo, sifuna ukufaka ngqo ku-"ClickHouse" (okungcono - uma kuqondile, kungcono).
Ngokufanelekile, ukufakwa kwenziwa kanjani ku-MergeTree? Kungani kungcono ukufaka kuwo izikhathi ezingaphezu kwesisodwa ngomzuzwana noma ngaphansi? Iqiniso liwukuthi i-"ClickHouse" iyisizindalwazi sekholomu futhi ihlunga idatha ngendlela ekhuphukayo yokhiye oyinhloko, futhi uma ufaka, inani lamafayela lenziwa okungenani elilingana nenani lamakholomu lapho idatha ihlelwa khona. ngohlelo olukhuphukayo lokhiye oyinhloko (kudalwa uhla lwemibhalo oluhlukile, isethi yamafayela kudiski okufakwayo ngakunye). Khona-ke ukufakwa okulandelayo kuyeza, futhi ngemuva kuhlanganiswe "ama-partitions" amakhulu. Njengoba idatha ihlungiwe, kungenzeka "ukuhlanganisa" amafayela amabili ahlungiwe ngaphandle kokusebenzisa inkumbulo eningi.
Kodwa, njengoba ungase uqagele, uma ubhala amafayela angu-10 ngokufaka ngakunye, i-ClickHouse (noma iseva yakho) izophela ngokushesha, ngakho-ke kunconywa ukuthi ufake ngamaqoqo amakhulu. Ngakho-ke, asizange sethule uhlelo lokuqala ekukhiqizeni. Ngokushesha sethule eyodwa, ethi lapha No. 2 ine:
Lapha cabanga ukuthi kukhona cishe amaseva ayinkulungwane esethule kuwo, kukhona i-PHP nje. Futhi kuseva ngayinye kukhona i-ejenti yethu yendawo, esiyibiza ngokuthi "Kittenhouse", egcina uxhumano olulodwa ne-"ClickHouse" futhi ifaka idatha njalo ngemizuzwana embalwa. Ayifaki idatha ku-MergeTree, kodwa kuthebula le-buffer, elisebenza ngokunembile ukugwema ukufaka ngqo ku-MergeTree ngokushesha.
Ukusebenza ngamatafula e-buffer
Yini? Amathebula ebhafa awucezu lwenkumbulo oluhlukaniswayo (okungukuthi, lungafakwa kulo njalo). Ziqukethe izingcezu eziningana, futhi ngayinye yezingcezu zisebenza njenge-buffer ezimele, futhi zishaywa ngokuzimela (uma unezicucu eziningi ku-buffer, khona-ke kuzoba nokufakwa okuningi ngomzuzwana). Kungenzeka ukufunda kusuka kulawa mathebula - bese ufunda inyunyana yokuqukethwe kwe-buffer kanye netafula lomzali, kodwa okwamanje ukubhala kuvinjiwe, ngakho-ke kungcono ukuthi ungafundi kusukela lapho. Futhi amathebula e-buffer abonisa i-QPS enhle kakhulu, okungukuthi, kufika ku-QPS eyizinkulungwane ezi-3 ngeke ube nezinkinga nhlobo lapho ufaka. Kuyacaca ukuthi uma iseva ilahlekelwa amandla, khona-ke idatha ingalahleka, ngoba igcinwe kuphela kumemori.
Ngasikhathi sinye, uhlelo olune-buffer luhlanganisa i-ALTER, ngoba okokuqala udinga ukulahla ithebula elidala le-buffer nohlelo oludala (idatha ngeke ishabalale noma kuphi, ngoba izosulwa ngaphambi kokuthi ithebula lisuswe). Bese "ushintsha" ithebula olidingayo bese udala ithebula lebhafa futhi. Ngokufanelekile, nakuba lingekho ithebula le-buffer, idatha yakho ngeke igeleze noma yikuphi, kodwa ungaba nayo kudiski okungenani endaweni.
Iyini i-Kittenhouse futhi isebenza kanjani?
Iyini i-KittenHouse? Lona ummeleli. Qagela ukuthi yiluphi ulimi? Ngiqoqe izihloko ezinhle kakhulu embikweni wami - "Clickhouse", Hamba, mhlawumbe ngizokhumbula enye into. Yebo, lokhu kubhalwe ku-Go, ngoba angazi ngempela ukubhala ngo-C, angifuni.
Ngokufanelekile, igcina ukuxhumana neseva ngayinye futhi ingabhalela kumemori. Isibonelo, uma sibhala amalogi wephutha ku-Clickhouse, khona-ke uma i-Clickhouse ingenaso isikhathi sokufaka idatha (ngemuva kwakho konke, uma kubhalwe kakhulu), ngakho-ke asivuvukali inkumbulo - sivele silahle konke okunye. Ngoba uma sibhala ama-gigabits ambalwa ngomzuzwana wamaphutha, khona-ke singaphonsa amanye ngaphandle. I-Kittenhouse ingakwenza lokhu. Futhi, ingakwazi ukulethwa okuthembekile, okungukuthi, ukubhala kudiski emshinini wendawo futhi kanye ngaso sonke isikhathi (lapho, kanye njalo ngemva kwemizuzwana embalwa) izama ukuletha idatha kusuka kuleli fayela. Futhi ekuqaleni sasebenzisa ifomethi ye-Values evamile - hhayi ifomethi ethile kanambambili, ifomethi yombhalo (njengaku-SQL evamile).
Kodwa-ke lokhu kwenzeka. Sasebenzisa ukulethwa okuthembekile, sabhala izingodo, sabe sesinquma (kwakuyiqoqo lokuhlola elinemibandela)... Yakhishwa amahora ambalwa futhi yabuyiselwa emuva, futhi ukufakwa kwaqala kusuka kumaseva ayinkulungwane - kwavela ukuthi i-Clickhouse isenayo "Uchungechunge ekuxhumekeni" - ngokufanele, ekuxhumekeni okuyinkulungwane, ukufakwa okusebenzayo kuholela esilinganisweni somthwalo kuseva cishe inkulungwane eyodwa nengxenye. Ngokumangalisayo, iseva yamukele izicelo, kodwa idatha yayisafakwa ngemva kwesikhathi esithile; kodwa bekunzima kakhulu ukuthi iseva isebenze...
Engeza nginx
Isixazululo esinjalo semodeli ye-Thread per uxhumano yi-nginx. Sifake i-nginx phambi kweClickhouse, ngesikhathi esifanayo setha ukulinganisa kwe-replicas ezimbili (isivinini sethu sokufaka sikhuphuke izikhathi ezingu-2, nakuba kungelona iqiniso ukuthi lokhu kufanele kube njalo) futhi sinciphise inani lokuxhumeka ku-Clickhouse, ku- phezulu nomfula futhi, ngokufanele, okuningi, kunokuxhumana okungu-50, kubonakala sengathi asikho isidingo sokufaka.
Sabe sesibona ukuthi lolu hlelo luvamise ukunezinkinga, ngoba sinenginx eyodwa kuphela lapha. Ngokufanelekile, uma le nginx iphahlazeka, naphezu kokuba khona kwe-replicas, silahlekelwa idatha noma, okungenani, singabhali ndawo. Yingakho sizenzele okwethu ukulinganisa umthwalo. Siphinde saqaphela ukuthi "Clickhouse" kusafanele izingodo, futhi "idemoni" futhi waqala ukubhala izingodo zakhe "Clickhouse" - elula kakhulu, ukwethembeka. Sisayisebenzisela amanye “amademoni”.
Sabe sesithola le nkinga ethokozisayo: uma usebenzisa indlela engajwayelekile yokufaka kumodi ye-SQL, iphoqa isihlaluli esigcwele se-AST-based SQL, esinensa impela. Ngokufanelekile, sengeze izilungiselelo ukuqinisekisa ukuthi lokhu akwenzeki. Senze i-load balancing, sahlola impilo, ukuze kuthi uma umuntu efa, sisawashiya ama-data. Manje sesinenqwaba yamatafula esiwadingayo ukuze sibe namaqoqo ahlukene e-Clickhouse. Futhi siqale ukucabanga ngokunye ukusetshenziswa - ngokwesibonelo, besifuna ukubhala izingodo kusuka kumamojula we-nginx, kodwa abazi ukuthi bangaxhumana kanjani sisebenzisa i-RPC yethu. Hhayi-ke, ngingathanda ukubafundisa ukuthi bathumela kanjani okungenani ngandlela thile - ngokwesibonelo, ukwamukela imicimbi ku-localhost nge-UDP bese uyidlulisela kuClickhouse.
Isinyathelo esisodwa kude nesixazululo
Uhlelo lokugcina lwaqala ukubukeka kanje (inguqulo yesine yalolu hlelo): kuseva ngayinye ephambi kwe-Clickhouse kukhona nginx (kuseva efanayo) futhi imane ifake izicelo zokubamba indawo ngomkhawulo enanini lokuxhunywa kuka-50. izingcezu. Futhi lolu hlelo lwaluvele lusebenza, yonke into yayihamba kahle ngalo.
Sahlala kanje isikhathi esingangenyanga. Wonke umuntu wayejabule, bangeza amatafula, banezela, banezela... Ngokuvamile, kwavela ukuthi indlela esengeze ngayo amatafula e-buffer yayingeyona elungile kakhulu (ake siyibeke kanjalo). Senze izingcezu ezingu-16 etafuleni ngalinye kanye ne-flash interval yemizuzwana embalwa; sasinamatafula angu-20 futhi itafula ngalinye lathola ukufakwa okungu-8 ngomzuzwana - futhi kuleli phuzu elithi “Clickhouse” laqala... amarekhodi aqala ukuncipha. Hhayi nje kuphela ukuthi abadlulanga... Ngokuzenzakalelayo, i-nginx yayinento ethokozisayo yokuthi uma ukuxhumeka kuphelela enhla nomfula, ivele ibuyisele u-“502” kuzo zonke izicelo ezintsha.
Futhi lapha sinakho (ngisanda kubuka izingodo ku-Clickhouse ngokwayo) cishe ingxenye yephesenti yezicelo zehlulekile. Ngakho-ke, ukusetshenziswa kwediski bekuphezulu, bekunokuhlangana okuningi. Pho, ngenzeni? Ngokwemvelo, angizange ngizihluphe ngokuthola ukuthi kungani ngempela uxhumano nokukhuphuka komfula kuphelile.
Ukushintsha i-nginx ngommeleli ohlanekezelwe
Nginqume ukuthi sidinga ukuphatha lokhu ngokwethu, asikho isidingo sokuyishiya ku-nginx - i-nginx ayazi ukuthi yimaphi amatafula akhona ku-Clickhouse, futhi ngishintshe i-nginx ngommeleli ophambene, engizibhalele nami.
Wenzani? Isebenza ngokususelwe kumtapo wezincwadi we-fasthttp "goshnoy", okungukuthi, ngokushesha, cishe ngokushesha njenge-nginx. Uxolo, Igor, uma ukhona lapha (qaphela: U-Igor Sysoev ungumhleli wezinhlelo waseRussia odale iseva yewebhu ye-nginx). Ingakwazi ukuqonda ukuthi hlobo luni lwemibuzo lena - FAKA noma KHETHA - ngokufanele, ibamba amachibi okuxhumana ahlukene ezinhlobo ezahlukene zemibuzo.
Ngokufanelekile, ngisho noma singenaso isikhathi sokuqedela izicelo zokufaka, "okukhethiwe" kuzodlula, futhi ngokuphambene nalokho. Futhi ihlanganisa idatha ibe amatafula e-buffer - nge-buffer encane: uma bekukhona amaphutha, amaphutha e-syntax, nokunye - ukuze angathinti kakhulu yonke idatha, ngoba lapho sivele sishutheka kumathebula e-buffer, yayine-"bachi" encane, futhi wonke amaphutha e-syntax athinte kuphela lesi siqeshana esincane; futhi lapha zizovele zithinte isigcinalwazi esikhulu. Okuncane kungu-1 megabyte, okusho ukuthi, akuyona encane kakhulu.
Ukufaka ukuvumelanisa kanye nokufaka esikhundleni se-nginx, empeleni kwenza into efanayo naleyo i-nginx eyayenza ngaphambili - awudingi ukushintsha "i-Kittenhouse" yendawo kulokhu. Futhi njengoba isebenzisa i-fasthttp, ishesha kakhulu - ungenza izicelo ezingaphezu kwezinkulungwane eziyi-100 ngomzuzwana ngokufaka okukodwa ngokusebenzisa ummeleli ohlanekezelwe. Ngokwethiyori, ungafaka umugqa owodwa ngesikhathi kummeleli we-kittenhouse reverse, kodwa-ke asikwenzi lokho.
Uhlelo lwaqala ukubukeka kanje: "I-Kittenhouse", amaqembu e-proxy ehlanekile acela izicelo eziningi ematafuleni futhi, amatafula e-buffer awafaka kwabayinhloko.
UKiller uyisixazululo sesikhashana, iKitten ingunaphakade
Lena inkinga ethokozisayo... Ngabe ukhona kini oke wasebenzisa fasthttp? Ubani osebenzise i-fasthttp ngezicelo ze-POST? Mhlawumbe, lokhu bekungafanele ngempela kwenziwe, ngoba kuphazamisa indikimba yesicelo ngokuzenzakalelayo, futhi usayizi webhafa wethu usethelwe ku-16 megabytes. Ukufakwa kuyekile ukuhambisana ngesikhathi esithile, futhi izingxenye ze-16-megabyte zaqala ukufika zivela kuwo wonke amashumi ezinkulungwane zamaseva, futhi wonke agcinwa enkumbulweni ngaphambi kokuba athunyelwe ku-Clickhouse. Ngokuvumelana nalokho, inkumbulo yaphela, Umbulali Ongaphandle KweMemori weza wabulala ummeleli ongemuva (noma i-“Clickhouse”, okungenzeka ukuthi “idle” ngaphezu kommeleli ongemuva). Umjikelezo waziphinda. Hhayi inkinga emnandi kakhulu. Nakuba sakhubeka kulokhu kuphela ngemva kwezinyanga ezimbalwa sisebenza.
Ngenzeni? Futhi, angithandi ngempela ukuqonda ukuthi kwenzekeni ngempela. Ngicabanga ukuthi kusobala kakhulu ukuthi akufanele ungene enkumbulweni. Angikwazanga ukuchibiyela ngokusheshahttp, nakuba ngizamile. Kodwa ngithole indlela yokuyenza ukuze kungabikho isidingo sokunamathisela noma yini, futhi ngaqhamuka nendlela yami ku-HTTP - ngayibiza ngokuthi KITTEN. Nokho, kunengqondo - "VK", "Kitten"... Yini enye? ..
Uma isicelo siza kuseva ngendlela yeKitten, iseva kufanele iphendule ngokuthi “meow” - ngokunengqondo. Uma ephendula kulokhu, khona-ke kubhekwa ukuthi uyayiqonda le nqubo, bese nginqamula uxhumano (i-fasthttp inendlela enjalo), futhi uxhumano lungena kumodi "eluhlaza". Kungani ngiyidinga? Ngifuna ukulawula ukuthi ukufunda kusuka ekuxhumekeni kwe-TCP kwenzeka kanjani. I-TCP inempahla emangalisayo: uma kungekho muntu ofunda ngakolunye uhlangothi, khona-ke ukubhala kuqala ukulinda, futhi inkumbulo ayisetshenziswanga ngokukhethekile kulokhu.
Futhi ngakho ngifunde kusuka kumakhasimende angaba ngu-50 ngesikhathi (kusukela kumashumi amahlanu ngoba amashumi amahlanu kufanele nakanjani abe ngokwanele, ngisho noma izinga livela kwenye i-DC)... Ukusetshenziswa kwehle ngale ndlela okungenani izikhathi ezingu-20, kodwa mina, ukwethembeka , angikwazanga ukulinganisa kahle ukuthi yisiphi isikhathi, ngoba isivele ingenangqondo (sesivele sifinyelele izinga lephutha). Iphrothokholi iyinambambili, okungukuthi, iqukethe igama lethebula nedatha; azikho izihloko ze-http, ngakho-ke angizange ngisebenzise isokhethi yewebhu (angidingi ukuxhumana neziphequluli - ngenze iphrothokholi evumelana nezidingo zethu). Futhi konke kwahamba kahle naye.
Ithebula lebhafa liyadabukisa
Muva nje sihlangane nesinye isici esithokozisayo samathebula e-buffer. Futhi le nkinga isivele ibuhlungu kakhulu kunezinye. Ake sicabange lesi simo: usuvele usebenzisa i-Clickhouse ngenkuthalo, unenqwaba yamaseva e-Clickhouse, futhi unezicelo ezithatha isikhathi eside kakhulu ukufunda (ake sithi, ngaphezu kwemizuzwana engama-60); bese ufika wenze okuthi Alter ngalo mzuzu... Okwamanje, “khetha” okuqale ngaphambi kokuthi “Alter” ngeke kufakwe kuleli thebula, “Alter” ngeke kuqale - mhlawumbe ezinye izici zendlela i-“Clickhouse” esebenza ngayo. lendawo. Mhlawumbe lokhu kungalungiswa? Noma akunakwenzeka?
Ngokuvamile, kusobala ukuthi empeleni lokhu akuyona inkinga enkulu kangaka, kodwa ngamatafula e-buffer kuba buhlungu kakhulu. Ngoba, uma, ake sithi, “Shintsha” ukuphela kwakho kwesikhathi (futhi kungase kuphele komunye umsingathi - hhayi kweyakho, kodwa efanekiseni, isibonelo), bese... Ususe ithebula lebhafa, “Alter” yakho ( noma omunye umsingathi) kuphelelwe yisikhathi. bese kwenzeke iphutha elithi “Alter”) - usadinga ukuqinisekisa ukuthi idatha iyaqhubeka nokubhalwa: udala amathebula ebhafa emuva (ngokohlelo olufanayo nethebula lomzali), bese "Ukushintsha" kudlula, kuyaphela phela, futhi isigcinalwazi ithebula liqala ukuhluka ku-schema kumzali. Kuya ngokuthi “i-Alter” ibiyini, okokufaka kungase kungasayi kuleli tafula lebhafa - lokhu kudabukisa kakhulu.
Kukhona nophawu olunjalo (mhlawumbe othile ulubonile) - lubizwa ngokuthi query_thread_log ezinguqulweni ezintsha zeClickhouse. Ngokuzenzakalelayo, kwenye inguqulo kwakukhona eyodwa. Lapha siqoqe amarekhodi ayizigidi ezingu-840 ezinyangeni ezimbalwa (amagigabhayithi angu-100). Lokhu kungenxa yokuthi "ukufakwa" kwabhalwa lapho (mhlawumbe manje, ngendlela, akubhaliwe). Njengoba nginitshelile, "okufakwayo" kwethu kuncane - sibe "nokufakwa" okuningi kumatafula ebhafa. Kuyacaca ukuthi lokhu kukhutshaziwe - ngikutshela nje engikubone kuseva yethu. Kungani? Lena enye ingxabano ephikisana nokusebenzisa amathebula e-buffer! Spotty kudabukisa kakhulu.
Ubani owayazi ukuthi igama lalo mfana kwakunguSpotty? Abasebenzi bakwa-VK baphakamise izandla. KULUNGILE.
Mayelana nezinhlelo ze-“KitttenHouse”
Izinhlelo ngokuvamile azabiwe, akunjalo? Ngokungazelelwe ngeke uzifeze futhi ngeke ubukeke kahle kakhulu emehlweni abanye abantu. Kodwa ngizozifaka engozini! Sifuna ukwenza lokhu okulandelayo: amathebula e-buffer, kubonakala kimi, aseyi-crutch futhi sidinga ukufaka kubhafa ngokwethu. Kodwa asikafuni ukuyigcina ku-disk, ngakho-ke sizofaka ku-buffer kumemori.
Ngakho-ke, lapho "ukufaka" kwenziwa, ngeke kusavunyelaniswa - sekuzosebenza njengethebula le-buffer, lizofaka etafuleni labazali (kahle, ngolunye usuku kamuva) futhi libike ngesiteshi esihlukile okufakiwe okudlulile futhi okufakiwe. anginayo.
Kungani ngingakwazi ukushiya okokufaka okuvumelanayo? Kulula kakhulu. Iqiniso liwukuthi uma ufaka kusuka kubabungazi abayizinkulungwane ezingu-10, khona-ke konke kuhamba kahle - uzothola kancane kusuka kumsingathi ngamunye, ufaka lapho kanye ngomzuzwana, konke kuhamba kahle. Kodwa ngingathanda ukuthi lolu hlelo lusebenze, isibonelo, kusuka emishinini emibili, ukuze ukwazi ukulanda ngesivinini esikhulu - mhlawumbe ungatholi ubuningi be-Clickhouse, kodwa bhala okungenani ama-megabytes angu-100 ngomzuzwana kusuka kumshini owodwa ngokusebenzisa ummeleli ohlanekezelwe - lesi sikimu kufanele silinganisele kuzo zombili izilinganiso ezinkulu nezincane, ngakho-ke asikwazi ukulinda isekhondi ngokufaka ngakunye, ngakho-ke kufanele kuhambisane. Futhi ngendlela efanayo, ukuqinisekiswa kwe-asynchronous kufanele kufike ngemva kokuqedwa kokufakwa. Sizokwazi ukuthi iphumelele noma cha.
Okubaluleke kakhulu ukuthi kulolu hlelo sazi ngokuqinisekile ukuthi ukufakwa kwenzeka noma cha. Cabanga ngalesi simo: unetafula le-buffer, ubhale okuthile kulo, bese-ke, ake sithi, ithebula lingene kumodi yokufunda kuphela bese lizama ukusula isigcinalwazi. Izoya kuphi idatha? Bazohlala ku-buffer. Kodwa asikwazi ukuqiniseka ngalokhu - kuthiwani uma kukhona elinye iphutha, ngenxa yokuthi idatha ngeke ihlale ku-buffer... (Amakheli u-Alexey Milovidov, Yandex, unjiniyela we-ClickHouse) Noma ingabe izohlala? Njalo? U-Alexey usiqinisekisa ukuthi konke kuzohamba kahle. Asinaso isizathu sokungamkholwa. Kodwa kuyafana: uma singasebenzisi amatafula e-buffer, ngeke kube nezinkinga ngawo. Ukudala amatafula amaningi aphindwe kabili nakho akusizi, nakuba ngokomthetho azikho izinkinga ezinkulu. Lolu wuhlelo.
Ake sikhulume ngokufunda
Manje ake sikhulume ngokufunda. Siphinde sabhala ithuluzi lethu lapha. Kungabonakala, kungani ubhala insimbi yakho lapha?.. Futhi ubani osebenzise i-Tabix? Ngandlela-thile bambalwa abantu abaphakamisa izandla zabo ... Futhi ubani owanelisekile ngokusebenza kwe-Tabix? Hhayi-ke, asijabule ngakho, futhi akulula kakhulu ukubuka idatha. Ilungele ukuhlaziya, kodwa ukuyibuka nje ngokusobala ayithuthukisiwe. Ngakho ngabhala okwami, okwami ukuxhumana.
Ilula kakhulu - ingakwazi ukufunda idatha kuphela. Akakwazi ukukhombisa imidwebo, akazi ukwenza noma yini. Kodwa ingabonisa esikudingayo: isibonelo, mingaki imigqa esetafuleni, singakanani isikhala esithathayo (ngaphandle kokuyihlephula sibe amakholomu), okungukuthi, isikhombimsebenzisi esiyisisekelo yikho esikudingayo.
Futhi kubukeka kufana kakhulu ne-Sequel Pro, kodwa kwenziwa kuphela ku-Bootstrap ye-Twitter, kanye nenguqulo yesibili. Uyabuza: "Yuri, kungani enguqulweni yesibili?" Ngamuphi unyaka? 2018? Sekukonke, ngikwenzele lokhu kudala kakhulu nge- "Muscle" (MySQL) futhi ngisanda kushintsha imigqa embalwa emibuzweni lapho, futhi yaqala ukusebenzela i-"Clickhouse", engibonga kakhulu ngayo! Ngoba umhlaseli ufana kakhulu ne "imisipha", futhi imibuzo ifana kakhulu - ilula kakhulu, ikakhulukazi ekuqaleni.
Hhayi-ke, ingahlunga amatafula, ingakhombisa ukwakheka nokuqukethwe kwethebula, ikuvumela ukuthi uhlele, uhlunge ngamakholomu, ubonise umbuzo obangele umphumela, imigqa ethintekile (bangakhi umphumela), okungukuthi, izinto eziyisisekelo zokubuka idatha. Ngokushesha kakhulu.
Kukhona nomhleli. Ngazama ngokweqiniso ukweba umhleli wonke ku-Tabix, kodwa ngehluleka. Kodwa ngandlela-thile kuyasebenza. Empeleni, yilokho kuphela.
"Clickhouse" ilungele imihume
Ngifuna ukukutshela ukuthi i-Clickhouse, naphezu kwazo zonke izinkinga ezichazwe, ifaneleka kakhulu izingodo. Okubaluleke kakhulu, ixazulula inkinga yethu - iyashesha kakhulu futhi ikuvumela ukuthi uhlunge izingodo ngamakholomu. Empeleni, amatafula e-buffer awazange enze kahle, kodwa ngokuvamile akekho owaziyo ukuthi kungani... Mhlawumbe manje wazi kangcono lapho uzoba nezinkinga.
I-TCP? Ngokuvamile, ku-VK kuwumkhuba ukusebenzisa i-UDP. Futhi lapho ngisebenzisa i-TCP... Yiqiniso, akekho owangitshela: “Yuri, ukhuluma ngani! Awukwazi, udinga i-UDP. " Kwavela ukuthi i-TCP ayesabi kangako. Into kuphela ukuthi, uma unamashumi ezinkulungwane zamakhompiyutha asebenzayo owabhalayo, udinga ukukulungisa kancane kancane; kodwa kungenzeka, futhi kulula kakhulu.
Ngethembise ukuthumela i-“Kittenhouse” kanye ne-“Lighthouse” ku-HighLoad Siberia uma wonke umuntu ebhalisele “i-VK backend” yethu yomphakathi... Futhi uyazi, akubona bonke ababhalisile... Yiqiniso, ngeke ngikufune ukuthi ubhalisele uhlelo lwethu lokusebenza. umphakathi. Nisebaningi kakhulu, omunye angase acasuke, kodwa noma kunjalo, ngicela ubhalise (futhi lapha kufanele ngenze amehlo afana nalawo ekati). Yilokho . Ngibona kakhulu! I-Github ingeyethu . Nge-Clickhouse izinwele zakho zizoba thambile futhi zibe silky.
Okuhamba phambili: - Abangane, manje ngemibuzo. Ngemva nje kokuba sethule isitifiketi sokubonga kanye nombiko wakho nge-VHS.
U-Yuri Nasretdinov (ozobizwa kamuva ngokuthi YN): - Ukwazi kanjani ukuqopha umbiko wami nge-VHS uma usanda kuphela?
Okuhamba phambili: - Nawe, awukwazi ukucacisa ngokugcwele ukuthi i-"Clickhouse" izosebenza kanjani noma cha! Bangani, imizuzu emi-5 yemibuzo!
Imibuzo yakho
Umbuzo ovela ezithamelini (lapha kuzobizwa ngo-Q): - Sawubona. Siyabonga kakhulu ngombiko. Nginemibuzo emibili. Ngizoqala ngento engenangqondo: ingabe inombolo yezinhlamvu t egameni elithi "Kittenhouse" emidwebeni (3, 4, 7 ...) ithinta ukwaneliseka kwamakati?
YN: - Ubuningi bani?
Z: – Incwadi t. Kukhona o-t abathathu, endaweni ethile eduze no-t abathathu.
YN: - Angizange ngikulungise? Yebo kunjalo! Lena imikhiqizo ehlukene - bengikukhohlisa nje sonke lesi sikhathi. Kulungile, ngiyadlala - akunandaba. Ah, khona lapha! Cha, yinto efanayo, ngenze iphutha.
Z: - Ngiyabonga. Umbuzo wesibili ubalulekile. Ngokwazi kwami, ku-Clickhouse, amatafula e-buffer ahlala enkumbulo kuphela, awakhihliwe kudiski futhi, ngokufanelekile, awaphikelele.
YN: - Yebo.
Z: - Futhi ngesikhathi esifanayo, iklayenti lakho ligcina kudiski, okusho isiqinisekiso esithile sokulethwa kwala malogi afanayo. Kodwa lokhu akuqinisekisiwe neze kwaClickhouse. Chaza ukuthi isiqinisekiso senziwa kanjani, ngenxa yani?.. Nansi le nqubo ngokuningiliziwe
YN: - Yebo, ngokwethiyori akukho ukuphikisana lapha, ngoba uma i-Clickhouse iwa, ungakwazi ukuyibona ngezindlela ezahlukene eziyisigidi. Uma i-Clickhouse iphahlazeka (uma iphetha ngokungalungile), ungakwazi, uma ukhuluma nje, ubuyisele emuva kancane ilogi yakho oyibhale phansi bese uqala kusukela lapho yonke into yayihamba kahle. Ake sithi uhlehlisa umzuzu, okusho ukuthi, kuthathwa ngokuthi ukhiphe yonke into ngomzuzu.
Z: - Okusho ukuthi, "I-Kittenhouse" ibamba iwindi isikhathi eside futhi, uma kwenzeka ukuwa, ingakwazi ukuyibona futhi ibuyisele emuva?
YN: - Kodwa lokhu kuwumbono. Empeleni, asikwenzi lokhu, futhi ukulethwa okuthembekile kuvela ku-zero kuya ezikhathini ezingapheli. Kodwa ngokwesilinganiso esisodwa. Sanelisekile ukuthi uma i-Clickhouse iphahlazeka ngesizathu esithile noma amaseva "eqala kabusha," sizobe silahlekelwa kancane. Kuzo zonke ezinye izimo, akukho okuzokwenzeka.
Z: - Sawubona. Kusukela ekuqaleni kwabonakala kimi ukuthi ngempela uzosebenzisa i-UDP kusukela ekuqaleni kombiko. Unayo i-http, konke lokho... Futhi iningi lezinkinga ozichazile, njengoba ngikuqonda, zidalwe yilesi sixazululo...
YN: - Sisebenzisa ini i-TCP?
Z: - Empeleni yebo.
YN: - Cha.
Z: - Kwaba nge-fasthttp lapho ube nezinkinga, ngoxhumo ube nezinkinga. Ukube usanda kusebenzisa i-UDP ngabe uzigcine isikhathi esithile. Hhayi-ke, kungaba nezinkinga ngemiyalezo emide noma enye into...
YN: - Ngani?
Z: - Ngemilayezo emide, njengoba ingase ingangeni ku-MTU, enye into ... Hhayi-ke, kungase kube nezinkinga zabo siqu. Umbuzo uwukuthi: kungani kungenjalo i-UDP?
YN: - Ngikholelwa ukuthi ababhali abathuthukise i-TCP/IP bahlakaniphe kakhulu kunami futhi bazi kangcono kunami ukuthi bangawahlela kanjani amaphakethe (ukuze ahambe), ngesikhathi esifanayo lungisa iwindi lokuthumela, ungalayishi inethiwekhi ngokweqile, unikeze impendulo kulokho ayifundiwe, ingabalwa ngakolunye uhlangothi ... Zonke lezi zinkinga, ngokubona kwami, zizoba khona ku-UDP, kuphela kuzodingeka ngibhale ikhodi engaphezu kwalokho engangivele ngibhalile ukuze ngifeze into efanayo ngokwami futhi cishe kabi. Angithandi ngisho nokubhala ngo-C, ngingasasho lapho...
Z: - Kulula nje! Kuthunyelwe kulungile futhi ungalindi lutho - akuhambisani ngokuphelele. Kwabuya isaziso sokuthi konke kuhamba kahle - lokho kusho ukuthi isifikile; Uma ingafiki, kusho ukuthi yimbi.
YN: - Ngidinga kokubili - ngidinga ukukwazi ukuthumela kokubili ngesiqinisekiso sokulethwa futhi ngaphandle kwesiqinisekiso sokulethwa. Lezi yizimo ezimbili ezahlukene. Ngidinga ukungalahlekelwa amalogi athile noma ngingawalahleki ngesizathu.
Z: - Ngeke ngichithe isikhathi. Lokhu kumele kuxoxwe kakhudlwana. Ngiyabonga.
Okuhamba phambili: - Ubani onemibuzo - izandla eziya esibhakabhakeni!
Z: - Sawubona, nginguSasha. Endaweni ethile phakathi kombiko, kwavela umuzwa wokuthi, ngaphezu kwe-TCP, kwakungenzeka ukusebenzisa isisombululo esenziwe ngomumo - uhlobo oluthile lwe-Kafka.
YN: - Hhayi-ke ... Ngikutshelile ukuthi angifuni ukusebenzisa amaseva aphakathi, ngoba ... e-Kafka, kuvela ukuthi sinezinkampani eziyizinkulungwane eziyishumi; empeleni, sinokuningi - amashumi ezinkulungwane zabasingathi. Kungase futhi kube buhlungu ukwenza ne-Kafka ngaphandle kwama-proxies. Ngaphezu kwalokho, okubaluleke kakhulu, isanikeza "i-latency", inikeza ababungazi abengeziwe okudingeka ube nabo. Kepha angifuni ukuba nazo - ngifuna...
Z: "Kodwa ekugcineni kwavela kanjalo."
YN: – Cha, abekho ababungazi! Konke lokhu kusebenza kubasingathi be-Clickhouse.
Z: - Yebo, futhi "Kittenhouse", okuyinto ephambene - uhlala kuphi?
YN: - Ku-Clickhouse host, ayibhali lutho kudiski.
Z: - Ake sithi.
Okuhamba phambili: – Ingabe wanelisekile? Singakunikeza iholo?
Z: - Yebo, ungakwazi. Eqinisweni, kunezinduku eziningi ukuze uthole into efanayo, futhi manje - impendulo yangaphambilini ngesihloko se-TCP iyaphikisana, ngombono wami, lesi simo. Kuzwakala sengathi konke bekungenziwa emadolweni ami ngesikhathi esincane kakhulu.
YN: Futhi futhi kungani ngingafuni ukusebenzisa i-Kafka, ngoba bekunezikhalazo eziningi engxoxweni ye-Clickhouse Telegram ukuthi, ngokwesibonelo, imilayezo evela e-Kafka ilahlekile. Hhayi kusukela eKafka ngokwayo, kodwa ekuhlanganisweni kweKafka neClikhaus; noma okuthile akuzange kuxhumeke lapho. Uma sikhuluma nje, kungadingeka ukuthi ubhalele i-Kafka iklayenti ngaleso sikhathi. Angicabangi ukuthi kungaba nesixazululo esilula noma esithembeke kakhulu.
Z: - Ngitshele, kungani ungazange uzame noma yimiphi imigqa noma uhlobo oluthile lwebhasi elivamile? Njengoba uthi nge-asynchrony ungathumela izingodo ngokwazo kulayini futhi uthole impendulo ngokulinganayo ngomugqa?
YN: - Sicela uphakamise ukuthi yimiphi imigqa engasetshenziswa?
Z: - Noma yikuphi, ngisho nangaphandle kwesiqinisekiso sokuthi zihlelekile. Uhlobo oluthile lwe-Redis, i-RMQ...
YN: - Nginomuzwa wokuthi i-Redis cishe ngeke ikwazi ukudonsa umthamo onjalo wokufaka ngisho nakusokhaya oyedwa (ngomqondo wamaseva amaningana) ekhipha i-Clickhouse. Angikwazi ukusekela lokhu nganoma yibuphi ubufakazi (angikabulinganisi), kodwa kubonakala kimi ukuthi iRedis ayisona isisombululo esingcono kakhulu lapha. Empeleni, le sistimu ingabhekwa njengomugqa womlayezo othuthukisiwe, kodwa oklanyelwe kuphela "I-Clickhouse"
Okuhamba phambili: - Yuri, ngiyabonga kakhulu. Ngiphakamisa ukuthi kuqedwe imibuzo nezimpendulo lapha bese ngithi kulabo ababuze umbuzo sizomnika bani incwadi.
YN: - Ngingathanda ukunikeza incwadi kumuntu wokuqala obuze umbuzo.
Okuhamba phambili: - Wonderful! Kuhle! Okumangalisayo! Ngibonga kakhulu!
Ezinye izikhangiso 🙂
Siyabonga ngokuhlala nathi. Uyazithanda izindatshana zethu? Ufuna ukubona okuqukethwe okuthakaselayo okwengeziwe? Sisekele ngokufaka i-oda noma ngokuncoma kubangani, , i-analogue ehlukile yamaseva ezinga lokungena, esungulwe yithi ngenxa yakho: (itholakala nge-RAID1 kanye ne-RAID10, kufika kuma-cores angu-24 kuze kufike ku-40GB DDR4).
I-Dell R730xd 2x ishibhile esikhungweni sedatha se-Equinix Tier IV e-Amsterdam? Lapha kuphela eNetherlands! I-Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - isuka ku-$99! Funda mayelana
Source: www.habr.com