{"id":92508,"date":"2020-08-28T07:42:10","date_gmt":"2020-08-28T05:42:10","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak"},"modified":"2020-08-28T07:42:10","modified_gmt":"2020-08-28T05:42:10","slug":"kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","title":{"rendered":"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>\u017dijeme v \u00fa\u017easnej dobe, ke\u010f je mo\u017en\u00e9 r\u00fdchlo a jednoducho skombinova\u0165 nieko\u013eko hotov\u00fdch otvoren\u00fdch n\u00e1strojov, nastavi\u0165 ich s \u201evypnut\u00fdm vedom\u00edm\u201c pod\u013ea r\u00e1d stackoverflow, bez toho, aby sme sa zaoberali \u201emnoh\u00fdmi p\u00edsmenami\u201c, a spusti\u0165 ich do komer\u010dnej prev\u00e1dzky. A ke\u010f bude potrebn\u00e9 aktualizova\u0165\/roz\u0161irova\u0165 alebo niekto n\u00e1hodou re\u0161tartuje p\u00e1r serverov \u2014 uvedomi\u0165 si, \u017ee za\u010dal ak\u00fdsi vnucuj\u00faci sa zl\u00fd sen, v\u0161etko sa n\u00e1hle skomplikovalo do nepoznania, cesta sp\u00e4\u0165 nie je, bud\u00facnos\u0165 je neist\u00e1 a bezpe\u010dnej\u0161ie je, namiesto programovania, chova\u0165 v\u010dely a vyr\u00e1ba\u0165 syr.<\/p>\n<p>Nie nadarmo sk\u00fasenej\u0161\u00ed kolegovia, s hlavy zasypanej chybami a z toho u\u017e \u0161ediv\u00fdmi vlasmi, pozoruj\u00fac neuverite\u013ene r\u00fdchle nasadenie bal\u00edkov \u201ekontajnerov\u201c v \u201ekock\u00e1ch\u201c na desiatkach serverov na \u201em\u00f3dnych jazykoch\u201c s vstavanou podporou asynchr\u00f3nneho-neblokuj\u00faceho vstupu-v\u00fdstupu \u2014 skromne sa usmievaj\u00fa. A ml\u010dky na\u010falej pre\u010d\u00edtavaj\u00fa \u201eman ps\u201c, pon\u00e1raj\u00fa sa do zdrojov\u00fdch k\u00f3dov \u201enginx\u201c a p\u00ed\u0161u-p\u00ed\u0161u-p\u00ed\u0161u unit testy. Kolegovia vedia, \u017ee to najzauj\u00edmavej\u0161ie ich \u010dak\u00e1 vpred, ke\u010f sa \u201eto v\u0161etko\u201c raz stane no\u010dnou morou pod Nov\u00fd rok. A pom\u00f4\u017ee im len hlbok\u00e9 ch\u00e1panie podstaty unixu, zapam\u00e4tan\u00e1 tabu\u013eka stavov TCP\/IP a z\u00e1kladn\u00e9 algoritmy triedenia a vyh\u013ead\u00e1vania. Aby pod \u00faderom hod\u00edn o\u017eili syst\u00e9m znova.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nAch \u00e1no, trochu som sa odklonil, ale d\u00fafam, \u017ee sa mi podarilo prenies\u0165 stav o\u010dak\u00e1vania.<br \/>\nDnes chcem zdie\u013ea\u0165 na\u0161e sk\u00fasenosti s nasaden\u00edm pohodln\u00e9ho a cenovo dostupn\u00e9ho stacku pre DataLake, ktor\u00fd rie\u0161i v\u00e4\u010d\u0161inu analytick\u00fdch \u00faloh v spolo\u010dnosti pre \u00faplne r\u00f4zne organiza\u010dn\u00e9 jednotky.<\/p>\n<p>Ned\u00e1vno sme pri\u0161li na to, \u017ee spolo\u010dnosti potrebuj\u00fa \u010doraz viac plody ako produktovej, tak aj technickej analytiky (nepovediac o \u010dere\u0161ni\u010dk\u00e1ch na torte v podobe machine learning) a na pochopenie trendov a riz\u00edk je potrebn\u00e9 zbiera\u0165 a analyzova\u0165 \u010doraz viac metrik.<\/p>\n<h3>Z\u00e1kladn\u00e1 technick\u00e1 analytika v \u201eBitrix24\u201c<\/h3>\n<p>\nPred nieko\u013ek\u00fdmi rokmi, s\u00fa\u010dasne so spusten\u00edm slu\u017eby \u201eBitrix24\u201c, sme akt\u00edvne investovali \u010das a zdroje do vytvorenia jednoduch\u00e9 a spo\u013eahlivej analytickej platformy, ktor\u00e1 by pom\u00e1hala r\u00fdchlo identifikova\u0165 probl\u00e9my v infra\u0161trukt\u00fare a napl\u00e1nova\u0165 \u010fal\u0161ie kroky. Samozrejme, preferovali sme hotov\u00e9 n\u00e1stroje, \u010do najjednoduch\u0161ie a najzrozumite\u013enej\u0161ie. V d\u00f4sledku toho bol vybran\u00fd nagios na monitorovanie a munin na anal\u00fdzu a vizualiz\u00e1ciu. Teraz m\u00e1me tis\u00edce kontrol v nagios, stovky grafov v munin a kolegovia ich ka\u017ed\u00fd de\u0148 \u00faspe\u0161ne vyu\u017e\u00edvaj\u00fa. Met\u00f3dy s\u00fa zrozumite\u013en\u00e9, grafy jasn\u00e9, syst\u00e9m funguje spo\u013eahlivo u\u017e nieko\u013eko rokov a pravidelne sa prid\u00e1vaj\u00fa nov\u00e9 testy a grafy: uvedieme nov\u00fa slu\u017ebu do prev\u00e1dzky \u2014 prid\u00e1me nieko\u013eko testov a grafov. Na zdravie.<\/p>\n<h3>Ruka na pulze \u2014 roz\u0161\u00edren\u00e1 technick\u00e1 analytika<\/h3>\n<p>\nT\u00fa\u017eba z\u00edska\u0165 inform\u00e1cie o probl\u00e9moch \u201e\u010do najsk\u00f4r\u201c n\u00e1s priviedla k akt\u00edvnym experimentom so jednoduch\u00fdmi a zrozumite\u013en\u00fdmi n\u00e1strojmi \u2014 pinba a xhprof.<\/p>\n<p>Pinba n\u00e1m posielala v UDP paketoch \u0161tatistiky o r\u00fdchlosti fungovania \u010dast\u00ed webov\u00fdch str\u00e1nok na PHP a bolo mo\u017en\u00e9 v re\u00e1lnom \u010dase vidie\u0165 v \u00falo\u017eisku MySQL (s pinbou prich\u00e1dza vlastn\u00fd motor MySQL na r\u00fdchlu anal\u00fdzu udalost\u00ed) kr\u00e1tky zoznam probl\u00e9mov a reagova\u0165 na ne. A xhprof v automatickom re\u017eime umo\u017e\u0148oval zbiera\u0165 grafy vykon\u00e1vania najpomal\u0161\u00edch PHP str\u00e1nok u klientov a analyzova\u0165, \u010do to mohlo sp\u00f4sobi\u0165 \u2014 pokojne, s \u0161\u00e1lkou \u010daju alebo nie\u010do silnej\u0161ie.<\/p>\n<p>Pred nieko\u013ek\u00fdmi mesiacmi bol n\u00e1stroj doplnen\u00fd o \u010fal\u0161\u00ed dos\u0165 jednoduch\u00fd a zrozumite\u013en\u00fd motor zalo\u017een\u00fd na algoritme sp\u00e4tn\u00e9ho indexovania, skvele realizovanom v legend\u00e1rnej kni\u017enici Lucene \u2014 Elastic\/Kibana. Jednoduch\u00e1 my\u0161lienka viacer\u00fdch vl\u00e1kien na z\u00e1pis dokumentov do inverzn\u00e9ho indexu Lucene na z\u00e1klade udalost\u00ed v logoch a r\u00fdchle vyh\u013ead\u00e1vanie pomocou fasetn\u00e9ho delenia sa uk\u00e1zala by\u0165 naozaj u\u017eito\u010dn\u00e1.<\/p>\n<p>Napriek dos\u0165 technick\u00e9mu vzh\u013eadu vizualiz\u00e1ci\u00ed v Kibana s \u201epretekan\u00edm hore\u201c n\u00edzko\u00farov\u0148ov\u00fdch konceptov ako \u201ebucket\u201c a znovu vyn\u00e1jden\u00e9mu jazyku, ktor\u00fd e\u0161te nie je zabudnut\u00fd \u2014 n\u00e1stroj n\u00e1m dobre pom\u00e1hal v nasleduj\u00facich \u00faloh\u00e1ch:<\/p>\n<ul>\n<li>Ko\u013eko ch\u00fdb PHP malo klient z Bitrix24 na port\u00e1li p1 za posledn\u00fa hodinu a ak\u00e9? Pochopi\u0165, odpusti\u0165 a r\u00fdchlo opravi\u0165.<\/li>\n<li>Ko\u013eko video-hovorov sa uskuto\u010dnilo na port\u00e1loch v Nemecku za predch\u00e1dzaj\u00facich 24 hod\u00edn, ak\u00fa mali kvalitu a boli nejak\u00e9 probl\u00e9my s kan\u00e1lom\/sie\u0165ou?<\/li>\n<li>Ak\u00e9 je fungovanie syst\u00e9movej funkcionality (n\u00e1\u0161 roz\u0161\u00edrenie na C pre PHP), skompilovan\u00e9 z p\u00f4vodn\u00fdch zdrojov v poslednej aktualiz\u00e1cii slu\u017eby a distribuovan\u00e9 klientom? Nie s\u00fa tam \u017eiadne segfaulty?<\/li>\n<li>Ukladaj\u00fa sa \u00fadaje klientov do pam\u00e4te PHP? Nie s\u00fa tam chyby presahuj\u00face vyhraden\u00fa pam\u00e4\u0165 procesov: \u201eout of memory\u201c? N\u00e1jdi a neutralizuj.<\/li>\n<\/ul>\n<p>\nTu je konkr\u00e9tny pr\u00edklad. Napriek d\u00f4kladn\u00e9mu a viac\u00farov\u0148ov\u00e9mu testovaniu sa u klienta pri ve\u013emi netradi\u010dnom pr\u00edpade a po\u0161koden\u00fdch vstupn\u00fdch \u00fadajoch objavila nepr\u00edjemn\u00e1 a neo\u010dak\u00e1van\u00e1 chyba, zaznela sir\u00e9na a za\u010dal sa proces jej r\u00fdchlej opravy:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/8a802dba41b5d1a85c0dc41dfbf8b84e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nOkrem toho, kibana umo\u017e\u0148uje organizova\u0165 upozornenia na vybran\u00e9 udalosti a v kr\u00e1tkom \u010dase sa tento n\u00e1stroj stal v spolo\u010dnosti pou\u017e\u00edvan\u00fd desiatkami zamestnancov z r\u00f4znych oddelen\u00ed \u2013 od technickej podpory a v\u00fdvoja po QA.<\/p>\n<p>Aktivitu ak\u00e9hoko\u013evek oddelenia v r\u00e1mci spolo\u010dnosti bolo mo\u017en\u00e9 pohodlne sledova\u0165 a mera\u0165 \u2013 namiesto ru\u010dn\u00e9ho anal\u00fdzy logov na serveroch sta\u010dilo raz nastavi\u0165 parsovanie logov a ich odoslanie do klastru elastic, aby ste si mohli vychutn\u00e1va\u0165, napr\u00edklad, poh\u013ead na dashboarde kibana so stovkami predan\u00fdch dvojhlav\u00fdch ma\u010diatok, vytla\u010den\u00fdch na 3D tla\u010diarni za minul\u00fd lun\u00e1rny mesiac.<\/p>\n<h3>Z\u00e1kladn\u00e1 obchodn\u00e1 anal\u00fdza<\/h3>\n<p>\nV\u0161etci vedia, \u017ee obchodn\u00e1 anal\u00fdza vo firm\u00e1ch \u010dasto za\u010d\u00edna extr\u00e9mne akt\u00edvnym vyu\u017e\u00edvan\u00edm, \u00e1no, \u00e1no, Excelu. Ale najd\u00f4le\u017eitej\u0161ie je, aby tam neskon\u010dila. Olej do oh\u0148a e\u0161te dobre prilieva cloudov\u00fd Google Analytics \u2013 na dobr\u00e9 sa r\u00fdchlo zvyk\u00e1.<\/p>\n<p>V na\u0161ej, harmonicky sa rozv\u00edjaj\u00facej spolo\u010dnosti, sa tu a tam objavovali \u201eproroci\u201c intenz\u00edvnej\u0161ej pr\u00e1ce s v\u00e4\u010d\u0161\u00edmi \u00fadajmi. Pravidelne sa objavovali po\u017eiadavky na hlb\u0161ie a komplexnej\u0161ie spr\u00e1vy a \u00fasil\u00edm chalanov z r\u00f4znych oddelen\u00ed bolo pred nieko\u013ek\u00fdmi mesiacmi zorganizovan\u00e9 jednoduch\u00e9 a praktick\u00e9 rie\u0161enie \u2013 prepojenie ClickHouse a PowerBI.<\/p>\n<p>Celkom dlh\u00fd \u010das toto flexibiln\u00e9 rie\u0161enie vynikaj\u00faco pom\u00e1halo, ale postupne sa za\u010dalo ch\u00e1pa\u0165, \u017ee ClickHouse nie je gumen\u00fd a nem\u00f4\u017eeme sa n\u00edm tak zle zaobch\u00e1dza\u0165.<\/p>\n<p>Tu je d\u00f4le\u017eit\u00e9 dobre pochopi\u0165, \u017ee ClickHouse, rovnako ako Druid, Vertica a Amazon RedShift (ktor\u00fd je zalo\u017een\u00fd na Postgres), s\u00fa analytick\u00e9 enginy optimalizovan\u00e9 na pomerne pohodln\u00fa analytiku (s\u00fahrny, agreg\u00e1cie, minimum maximum v st\u013apci a len nieko\u013eko spojen\u00ed), preto\u017ee s\u00fa organizovan\u00e9 na efekt\u00edvne ulo\u017eenie st\u013apcov rela\u010dn\u00fdch tabuliek, na rozdiel od zn\u00e1meho MySQL a \u010fal\u0161\u00edch (b\u00e1z orientovan\u00fdch na riadky).<\/p>\n<p>V podstate je ClickHouse len priestrannej\u0161ia 'baza' d\u00e1t, s nie ve\u013emi pohodln\u00fdm bodov\u00fdm vkladom (tak to bolo zam\u00fd\u0161\u013ean\u00e9, v\u0161etko v poriadku), ale s pr\u00edjemnou analytikou a zauj\u00edmavou sadou mocn\u00fdch funkci\u00ed na pr\u00e1cu s d\u00e1tami. \u00c1no, m\u00f4\u017eete dokonca vytvori\u0165 klaster \u2014 ale, ch\u00e1pete, \u017ee zat\u013aka\u0165 klince mikroskopom nie je \u00faplne spr\u00e1vne, a za\u010dali sme h\u013eada\u0165 in\u00e9 rie\u0161enia.<\/p>\n<h3>Dopyt po Pythone a analytikoch<\/h3>\n<p>\nV na\u0161ej spolo\u010dnosti je ve\u013ea v\u00fdvoj\u00e1rov, ktor\u00ed p\u00ed\u0161u k\u00f3d takmer ka\u017ed\u00fd de\u0148 u\u017e 10-20 rokov v PHP, JavaScripte, C#, C\/C++, Jave, Go, Rust, Pythone, Bashi. Taktie\u017e m\u00e1me mnoho sk\u00fasen\u00fdch syst\u00e9mov\u00fdch administr\u00e1torov, ktor\u00ed pre\u017eili nie jednu \u00faplne neuverite\u013en\u00fa katastrofu, ktor\u00e1 sa nezmestila do z\u00e1konov \u0161tatistiky (napr\u00edklad, ke\u010f je zni\u010den\u00e1 v\u00e4\u010d\u0161ina diskov v raid-10 pri silnom \u00fadere blesku). V tak\u00fdch podmienkach dlho nebolo jasn\u00e9, \u010do to vlastne znamen\u00e1 'analytik na Pythone'. Python je predsa len ako PHP, iba n\u00e1zov je o nie\u010do dlh\u0161\u00ed a stopy l\u00e1tok, ktor\u00e9 menia vedomie, v zdrojovom k\u00f3de interpretera je o nie\u010do menej. Av\u0161ak s prib\u00fadaj\u00facimi analytick\u00fdmi spr\u00e1vami sa sk\u00fasen\u00ed v\u00fdvoj\u00e1ri \u010doraz hlb\u0161ie zam\u00fd\u0161\u013eali nad d\u00f4le\u017eitos\u0165ou \u00fazkej \u0161pecializ\u00e1cie v n\u00e1strojoch ako numpy, pandas, matplotlib a seaborn.<br \/>\nRozhoduj\u00facu \u00falohu, pravdepodobne, hrali n\u00e1hle mdloby zamestnancov od kombin\u00e1cie slov 'logistick\u00e1 regresia' a prezent\u00e1cia efekt\u00edvneho zostavovania spr\u00e1v z objemov\u00fdch d\u00e1t pomocou \u00e1no, \u00e1no, pyspark.<\/p>\n<p>Apache Spark, jeho funk\u010dn\u00e1 paradigma, na ktor\u00fa skvele zapad\u00e1 rela\u010dn\u00e1 algebra a mo\u017enosti zap\u00f4sobila na v\u00fdvoj\u00e1rov zvyknut\u00fdch na MySQL tak, \u017ee potreba posilni\u0165 bojov\u00e9 rady sk\u00fasen\u00fdmi analytikmi sa stala jasnou ako de\u0148.<\/p>\n<h3>\u010eal\u0161ie pokusy Apache Spark\/Hadoop vzlietnu\u0165 a \u010do ne\u0161lo celkom pod\u013ea pl\u00e1nu<\/h3>\n<p>\nAv\u0161ak \u010doskoro sa uk\u00e1zalo, \u017ee so Sparkom je zrejme nie\u010do syst\u00e9movo nespr\u00e1vne, alebo je potrebn\u00e9 lep\u0161ie si um\u00fdva\u0165 ruky. K\u00fdm stack Hadoop\/MapReduce\/Lucene vytv\u00e1rali dostato\u010dne sk\u00fasen\u00ed program\u00e1tori, \u010do je o\u010dividn\u00e9, ak sa pozrieme na zdrojov\u00e9 k\u00f3dy v Jave alebo my\u0161lienky Duga Cuttinga v Lucene, Spark je n\u00e1hle nap\u00edsan\u00fd v ve\u013emi kontroverznom jazyku Scala, z praktick\u00e9ho h\u013eadiska nezrozumite\u013enom a moment\u00e1lne sa nerozv\u00edjaj\u00facom. Zos\u00faladen\u00e9 v\u00fdpo\u010dty v klastri Spark, kv\u00f4li nelogickej a nie ve\u013emi jasnej pr\u00e1ci s alok\u00e1ciu pam\u00e4te pre oper\u00e1cie reduce (prich\u00e1dza hne\u010f ve\u013ea k\u013e\u00fa\u010dov) \u2014 vytvorilo okolo neho aureolu nie\u010doho, \u010do m\u00e1 kam r\u00e1s\u0165. \u010eal\u0161ie zhor\u0161ovanie situ\u00e1cie sp\u00f4sobovalo mno\u017estvo podivn\u00fdch otvoren\u00fdch portov, do\u010dasn\u00fdch s\u00faborov rast\u00facich na t\u00fdch najnen\u00e1padnej\u0161\u00edch miestach a administr\u00e1cie jar-z\u00e1vislost\u00ed \u2014 \u010do vyvol\u00e1valo u syst\u00e9mov\u00fdch administr\u00e1torov jedno zn\u00e1me detstvo pocit: nepriate\u013esk\u00fa nen\u00e1vis\u0165 (a mo\u017eno bolo potrebn\u00e9 umy\u0165 si ruky mydlom).<\/p>\n<p>My sme, v d\u00f4sledku toho, \"pre\u017eili\" nieko\u013eko intern\u00fdch analytick\u00fdch projektov, ktor\u00e9 akt\u00edvne pou\u017e\u00edvali Apache Spark (vr\u00e1tane Spark Streaming, Spark SQL) a ekosyst\u00e9m Hadoop (a \u010fal\u0161ie a \u010fal\u0161ie). Napriek tomu, \u017ee sme sa v priebehu \u010dasu nau\u010dili \"to\" celkom dobre pripravova\u0165 a monitorova\u0165 a \"predmet\" prakticky prestal n\u00e1hle pada\u0165 kv\u00f4li zmene charakteru \u00fadajov a nevyv\u00e1\u017eenosti rovnomern\u00e9ho hashovania RDD, t\u00fa\u017eba z\u00edska\u0165 nie\u010do u\u017e hotov\u00e9, aktualizovan\u00e9 a spravovan\u00e9 niekde v cloude sa st\u00e1le viac a viac zintenz\u00edvnila. Presne v tomto \u010dase sme sa pok\u00fasili vyu\u017ei\u0165 hotov\u00e9 cloudov\u00e9 rie\u0161enie Amazon Web Services \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/emr\/\">EMR<\/a><\/noindex> a nesk\u00f4r sme sa sna\u017eili u\u017e rie\u0161i\u0165 \u00falohy na \u0148om. EMR je pripraven\u00fd Amazonom Apache Spark s dodato\u010dn\u00fdm softv\u00e9rom z ekosyst\u00e9mu, pribli\u017ene ako Cloudera\/Hortonworks zostavy.<\/p>\n<h3>\"Gumov\u00e9\" \u00falo\u017eisko pre analytiku \u2014 naliehav\u00e1 potreba<\/h3>\n<p>\nSk\u00fasenos\u0165 so \"pr\u00edpravou\" Hadoop\/Spark s pop\u00e1leninami r\u00f4znych \u010dast\u00ed tela nepre\u0161la nadarmo. Za\u010dala sa zrete\u013enej\u0161ie vykres\u013eova\u0165 potreba vytvorenia jednotn\u00e9ho, lacn\u00e9ho a spo\u013eahliv\u00e9ho \u00falo\u017eiska s\u00faborov, ktor\u00e9 by odol\u00e1valo hardv\u00e9rov\u00fdm hav\u00e1ri\u00e1m a v ktorom by bolo mo\u017en\u00e9 uklada\u0165 s\u00fabory v r\u00f4znych form\u00e1toch z r\u00f4znych syst\u00e9mov a robi\u0165 efekt\u00edvne a v rozumnom \u010dase vykon\u00e1van\u00e9 v\u00fdbery pre spr\u00e1vy.<\/p>\n<p>R\u00e1di bychom, aby aktualizace softwaru na t\u00e9to platform\u011b nebyla no\u010dn\u00ed m\u016frou, kdy bychom museli \u010d\u00edst 20str\u00e1nkov\u00e9 Java stack trace a analyzovat kilometry podrobn\u00e9ho logov\u00e1n\u00ed provozu clusteru pomoc\u00ed Spark History Server a lupy se zv\u00fdrazn\u011bn\u00edm. Cht\u011bli bychom m\u00edt jednoduch\u00fd a transparentn\u00ed n\u00e1stroj, kter\u00fd nevy\u017eaduje pravideln\u00e9 nahl\u00ed\u017een\u00ed pod kapotu, kdyby u v\u00fdvoj\u00e1\u0159e p\u0159estala fungovat standardn\u00ed MapReduce dotaz p\u0159i vyb\u011bhnut\u00ed z pam\u011bti pracovn\u00edka reduce dat kv\u016fli \u0161patn\u011b zvolen\u00e9mu algoritmu partitionov\u00e1n\u00ed p\u016fvodn\u00edch dat.<\/p>\n<h3>Je Amazon S3 uchaze\u010dem o DataLake?<\/h3>\n<p>\nZku\u0161enosti s Hadoop\/MapReduce m\u011b nau\u010dily, \u017ee je pot\u0159ebn\u00e9 m\u00edt tak\u00e9 \u0161k\u00e1lovateln\u00fd a spolehliv\u00fd souborov\u00fd syst\u00e9m a nad n\u00edm \u0161k\u00e1lovateln\u00e9 pracovn\u00edky, kte\u0159\u00ed p\u0159ich\u00e1zej\u00ed co nejbl\u00ed\u017e k dat\u016fm, aby nebylo nutn\u00e9 pos\u00edlat data p\u0159es s\u00ed\u0165. Pracovn\u00edci by m\u011bli b\u00fdt schopni \u010d\u00edst data v r\u016fzn\u00fdch form\u00e1tech, ale ide\u00e1ln\u011b by nem\u011bli \u010d\u00edst zbyte\u010dn\u00e9 informace a m\u011blo by b\u00fdt mo\u017en\u00e9 p\u0159edem ukl\u00e1dat data v pracovn\u00edky pohodln\u00fdch form\u00e1tech.<\/p>\n<p><b>Je\u0161t\u011b jednou \u2014 hlavn\u00ed my\u0161lenka.<\/b> Nem\u00e1me z\u00e1jem \"nahr\u00e1vat\" velk\u00e1 data do jednoho clusterov\u00e9ho analytick\u00e9ho stroje, kter\u00fd se nakonec stejn\u011b utop\u00ed a budeme ho muset nehezky shardovat. Chceme ukl\u00e1dat soubory, prost\u011b soubory, v srozumiteln\u00e9ho form\u00e1tu a prov\u00e1d\u011bt na nich efektivn\u00ed analytick\u00e9 dotazy r\u016fzn\u00fdmi, ale srozumiteln\u00fdmi n\u00e1stroji. A soubor\u016f v r\u016fzn\u00fdch form\u00e1tech bude st\u00e1le v\u00edce. A l\u00e9pe shardovat ne stroj, ale p\u016fvodn\u00ed data. Pot\u0159ebujeme roz\u0161i\u0159iteln\u00fd a univerz\u00e1ln\u00ed DataLake, rozhodli jsme se\u2026<\/p>\n<p>A co kdybychom ukl\u00e1dali soubory v tradi\u010dn\u00edm a mnoha lidem zn\u00e1m\u00e9m \u0161k\u00e1lovateln\u00e9m cloudov\u00e9m \u00falo\u017ei\u0161ti Amazon S3, ani\u017e bychom se museli zab\u00fdvat p\u0159\u00edpravou \"od\u0159\u00edznut\u00fdch\" variant z Hadoop?<\/p>\n<p>Jasn\u011b, osobn\u00ed \u00fadaje \"nep\u0159ich\u00e1z\u00ed v \u00favahu\", ale co ostatn\u00ed data, pokud by se tam p\u0159esunula a \"efektivn\u011b poh\u00e1n\u011bla\"?<\/p>\n<h3>Klastrov\u00fd big data analytick\u00fd ekosyst\u00e9m Amazon Web Services \u2014 velmi jednodu\u0161e \u0159e\u010deno<\/h3>\n<p>\nPodle na\u0161ich zku\u0161enost\u00ed s AWS se tam u\u017e dlouho a aktivn\u011b pou\u017e\u00edv\u00e1 Apache Hadoop\/MapReduce pod r\u016fzn\u00fdmi om\u00e1\u010dkami, nap\u0159\u00edklad v slu\u017eb\u011b DataPipeline (z\u00e1vid\u00edm koleg\u016fm, nau\u010dili se ho spr\u00e1vn\u011b p\u0159ipravovat). Tady jsme nastavili z\u00e1lohy z r\u016fzn\u00fdch slu\u017eeb z tabulek DynamoDB:<br \/>\n<img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/058dc54ed032a7bf3e9e129646202440.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA pravideln\u011b se prov\u00e1d\u011bj\u00ed na vestav\u011bn\u00fdch clusterech Hadoop\/MapReduce jako hodinky u\u017e n\u011bkolik let. \"Nastavit a zapomenout\":<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/a6569da8cafdb96c63250bb32bf51704.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTaktie\u017e je mo\u017en\u00e9 efekt\u00edvne sa zaobera\u0165 d\u00e1tov\u00fdm satanizmom, spusti\u0165 pre analytikov Jupiter-notebooky v cloude a vyu\u017ei\u0165 na u\u010denie a nasadenie modelov AI slu\u017ebu AWS SageMaker. Takto to vyzer\u00e1 u n\u00e1s:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/c825d979c9278a8edf8e1e747ef6def8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA \u00e1no, m\u00f4\u017eete si spusti\u0165 notebook v cloude alebo pre analytiku a pripoji\u0165 ho k Hadoop\/Spark klastru, vypo\u010d\u00edta\u0165 a potom v\u0161etko \u201epribi\u0165\u201c:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/73cea18c54d2a9ce8d0441463991808b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNaozaj pohodln\u00e9 pre jednotliv\u00e9 analytick\u00e9 projekty a pre niektor\u00e9 sme \u00faspe\u0161ne pou\u017eili slu\u017ebu EMR na rozsiahle prepo\u010dty a anal\u00fdzy. A \u010do syst\u00e9mov\u00e9 rie\u0161enie pre DataLake, bude to fungova\u0165? V tom momente sme boli na pokraji n\u00e1deje a bezn\u00e1deje a pokra\u010dovali sme v h\u013eadan\u00ed.<\/p>\n<h3>AWS Glue \u2013 elegantne zabalen\u00fd Apache Spark \u201ena stern\u00e1dy\u201c<\/h3>\n<p>\nUk\u00e1zalo sa, \u017ee AWS m\u00e1 \u201esvoju\u201c verziu stacku \u201eHive\/Pig\/Spark\u201c. Rolu Hive, t.j. katal\u00f3g s\u00faborov a ich typov v DataLake, pln\u00ed slu\u017eba \u201eData catalog\u201c, ktor\u00e1 sa ani neskr\u00fdva o svojej kompatibilite s form\u00e1tom Apache Hive. Do tejto slu\u017eby treba prida\u0165 inform\u00e1cie o tom, kde s\u00fa va\u0161e s\u00fabory a v akom s\u00fa form\u00e1te. \u00dadaje m\u00f4\u017eu by\u0165 nielen v s3, ale aj v datab\u00e1ze, ale o tom nie je tento pr\u00edspevok. Takto je u n\u00e1s organizovan\u00fd katal\u00f3g d\u00e1t DataLake:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/adb45d09698fdacbf41c86bbadde8bb2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nS\u00fabory s\u00fa registrovan\u00e9, v\u00fdborne. Ak sa s\u00fabory aktualizovali \u2013 sp\u00fa\u0161\u0165ame bu\u010f manu\u00e1lne alebo pod\u013ea rozvrhu crawlers, ktor\u00e9 aktualizuj\u00fa inform\u00e1cie o nich z jazera a ulo\u017eia. Potom je mo\u017en\u00e9 \u00fadaje z jazera spracov\u00e1va\u0165 a v\u00fdsledky niekam exportova\u0165. V najjednoduch\u0161om pr\u00edpade \u2013 exportujeme tie\u017e do s3. Spracovanie \u00fadajov je mo\u017en\u00e9 robi\u0165 kdeko\u013evek, ale odpor\u00fa\u010da sa nastavi\u0165 proces spracovania na klastri Apache Spark s vyu\u017eit\u00edm roz\u0161\u00edren\u00fdch funkci\u00ed prostredn\u00edctvom API AWS Glue. V podstate je mo\u017en\u00e9 vzia\u0165 star\u00fd dobr\u00fd a zn\u00e1my k\u00f3d v pythone s pou\u017eit\u00edm kni\u017enice pyspark a nastavi\u0165 jeho vykon\u00e1vanie na N uzloch klastra nejakej kapacity s monitorovan\u00edm, bez hrabania sa v \u00fatrob\u00e1ch Hadoop a \u0165ahania docker-mokr\u00e9 kontajnerov a odstra\u0148ovania konfliktov z\u00e1vislost\u00ed.<\/p>\n<p><b>E\u0161te raz \u2013 jednoduch\u00e1 my\u0161lienka.<\/b> Nie je potrebn\u00e9 nastavova\u0165 Apache Spark, sta\u010d\u00ed nap\u00edsa\u0165 k\u00f3d v pythone pre pyspark, otestova\u0165 ho lok\u00e1lne na pracovnej ploche a potom ho spusti\u0165 na ve\u013ekom klastri v cloude, \u0443\u043a\u0430\u0437\u0430\u0432, kde sa nach\u00e1dzaj\u00fa zdrojov\u00e9 \u00fadaje a kam ulo\u017ei\u0165 v\u00fdsledok. Niekedy je to potrebn\u00e9 a u\u017eito\u010dn\u00e9 a takto to m\u00e1me nastaven\u00e9 u n\u00e1s:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/dc03181573bb3f5cfcc3a8760bc7e07b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTak\u017ee, ak potrebujete nie\u010do prepo\u010d\u00edta\u0165 na Spark klastri na \u00fadajoch v s3 \u2013 nap\u00ed\u0161eme k\u00f3d v python\/pyspark, otestujeme a do dobr\u00e9ho smeru do cloudu.<\/p>\n<p>A \u010do s orchestr\u00e1ciou? A \u010do ak \u00faloha spadla a zmizla? \u00c1no, je navrhnut\u00e9 vytvori\u0165 pekn\u00fd pipeline v \u0161t\u00fdle Apache Pig a aj sme sa o to pok\u00fasili, ale zatia\u013e sme sa rozhodli pou\u017ei\u0165 na\u0161u hlboko prisp\u00f4soben\u00fa orchestr\u00e1ciu na PHP a JavaScript (ch\u00e1pem, \u017ee vznik\u00e1 kognit\u00edvny disonanc, ale funguje to, roky a bez ch\u00fdb).<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/e00ed2047c5c6492e36fccc82e7278a3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Form\u00e1t s\u00faborov ulo\u017een\u00fdch v jazere \u2014 k\u013e\u00fa\u010d k v\u00fdkonnosti<\/h3>\n<p>\nJe naozaj ve\u013emi d\u00f4le\u017eit\u00e9 pochopi\u0165 e\u0161te dva k\u013e\u00fa\u010dov\u00e9 momenty. Aby sa dopyty na d\u00e1ta s\u00faborov v jazere vykon\u00e1vali maxim\u00e1lne r\u00fdchlo a v\u00fdkonnos\u0165 sa nezni\u017eovala pri prid\u00e1van\u00ed nov\u00fdch inform\u00e1ci\u00ed, je potrebn\u00e9:<\/p>\n<ul>\n<li>St\u013apce s\u00faborov uchov\u00e1va\u0165 zvl\u00e1\u0161\u0165 (aby nebolo potrebn\u00e9 pre\u010d\u00edta\u0165 v\u0161etky riadky, aby sme pochopili, \u010do je v st\u013apcoch). Na to sme vzali form\u00e1t parquet so zhlukovan\u00edm<\/li>\n<li>Je ve\u013emi d\u00f4le\u017eit\u00e9 shardova\u0165 s\u00fabory pod\u013ea prie\u010dinkov v duchu: jazyk, rok, mesiac, de\u0148, t\u00fd\u017ede\u0148. Enginy, ktor\u00e9 rozumej\u00fa tomuto typu shardovania, sa bud\u00fa pozera\u0165 iba do potrebn\u00fdch prie\u010dinkov, bez toho, aby preh\u013ead\u00e1vali v\u0161etky d\u00e1ta naraz.<\/li>\n<\/ul>\n<p>\nV podstate t\u00fdmto sp\u00f4sobom vystavujete p\u00f4vodn\u00e9 d\u00e1ta v najefekt\u00edvnej\u0161ej forme pre analytick\u00e9 enginy, ktor\u00e9 dok\u00e1\u017eu selekt\u00edvne vstupova\u0165 do shardovan\u00fdch prie\u010dinkov a \u010d\u00edta\u0165 iba potrebn\u00e9 st\u013apce. Niekam ich \u201enalieva\u0165\u201c netreba (\u00falo\u017eisko by predsa jednoducho prasklo) \u2014 jednoducho ich hne\u010f rozumne umiestnite do s\u00faborov\u00e9ho syst\u00e9mu v spr\u00e1vnom form\u00e1te. Samozrejme, tu by malo by\u0165 jasn\u00e9, \u017ee uchov\u00e1va\u0165 v DataLake obrovsk\u00fd csv s\u00fabor, ktor\u00fd mus\u00ed by\u0165 najprv cel\u00fd riadok po riadku pre\u010d\u00edtan\u00fd klastrom, aby sa extrahovali st\u013apce \u2014 nie je ve\u013emi rozumn\u00e9. Znovu si premyslite dva vy\u0161\u0161ie uveden\u00e9 body, ak to zatia\u013e nie je jasn\u00e9, pre\u010do je toto v\u0161etko d\u00f4le\u017eit\u00e9.<\/p>\n<h3>AWS Athena \u2014 \u201e\u010dert\u201c z tabatierky<\/h3>\n<p>\nA tu, pri vytv\u00e1ran\u00ed jazera, sme, akosi mimochodom, narazili na Amazon Athena. Zrazu sa uk\u00e1zalo, \u017ee ak spr\u00e1vne usporiadame na\u0161e obrovsk\u00e9 logy do shardov-prie\u010dinkov v spr\u00e1vnom (parquet) st\u013apcovom form\u00e1te \u2014 d\u00e1 sa ve\u013emi r\u00fdchlo robi\u0165 mimoriadne informat\u00edvne v\u00fdbery a vytv\u00e1ra\u0165 spr\u00e1vy BEZ apache Spark\/Glue klastra.<\/p>\n<p>Engin Athena, ktor\u00fd pracuje s d\u00e1tami v s3, je zalo\u017een\u00fd na legend\u00e1rnom <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/big-data\/what-is-presto\/\">Presto<\/a><\/noindex> \u2014 reprezentant rodiny MPP (massive parallel processing) pr\u00edstupov k spracovaniu \u00fadajov, ber\u00faci \u00fadaje tam, kde sa nach\u00e1dzaj\u00fa, od s3 a Hadoop po Cassandra a be\u017en\u00e9 textov\u00e9 s\u00fabory. Sta\u010d\u00ed po\u017eiada\u0165 Athena, aby vykonala SQL dotaz, a v\u0161etko \"pracuje r\u00fdchlo a samo\". Je d\u00f4le\u017eit\u00e9 poznamena\u0165, \u017ee Athena je \"inteligentn\u00e1\", ide len do potrebn\u00fdch rozdelen\u00fdch prie\u010dinkov a na\u010d\u00edtava len potrebn\u00e9 st\u013apce v dopyte.<\/p>\n<p>Sp\u00f4sob \u00fa\u010dtovania dotazov do Atheny je tie\u017e zauj\u00edmav\u00fd. Plat\u00edme za <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/athena\/pricing\/\">objem preh\u013eadan\u00fdch \u00fadajov<\/a><\/noindex>. T.j. nie za po\u010det strojov v klastri za min\u00fatu, ale\u2026 za naozaj preh\u013eadan\u00e9 \u00fadaje len potrebn\u00e9 na vykonanie dotazu na 100-500 strojoch.<\/p>\n<p>A ke\u010f po\u017eadujeme len potrebn\u00e9 st\u013apce z spr\u00e1vne rozdelen\u00fdch prie\u010dinkov, uk\u00e1zalo sa, \u017ee slu\u017eba Athena n\u00e1s stoj\u00ed desiatky dol\u00e1rov mesa\u010dne. No predsa skvele, takmer zadarmo v porovnan\u00ed s analytikou na klastraoch!<\/p>\n<p>Tu, mimochodom, ako rozde\u013eujeme svoje \u00fadaje v s3:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/00bd9ae48c1cd13f3c4f7d32692c9209.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fdsledkom je, \u017ee za kr\u00e1tky \u010das sa v spolo\u010dnosti \u00faplne r\u00f4zne oddelenia, od informa\u010dnej bezpe\u010dnosti po analytiku, za\u010dali akt\u00edvne p\u00fdta\u0165 na Athenou a r\u00fdchlo, za sekundy, dost\u00e1va\u0165 u\u017eito\u010dn\u00e9 odpovede z \"ve\u013ek\u00fdch\" \u00fadajov za pomerne dlh\u00e9 \u010dasov\u00e9 obdobia: mesiace, polrok a pod.<\/p>\n<p>Ale \u0161li sme \u010falej a za\u010dali sa p\u00fdta\u0165 na odpovede v oblaku <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/connect-with-odbc.html\">cez ODBC ovl\u00e1da\u010d<\/a><\/noindex>: analytik v obvyklej konzole p\u00ed\u0161e SQL dotaz, ktor\u00fd na 100-500 strojoch \"za mal\u00e9 peniaze\" preh\u013ead\u00e1va \u00fadaje v s3 a zvy\u010dajne vracia odpove\u010f za nieko\u013eko sek\u00fand. Pohodln\u00e9. A r\u00fdchle. Doteraz tomu nem\u00f4\u017eem uveri\u0165.<\/p>\n<p>Nakoniec, rozhodnut\u00edm uchov\u00e1va\u0165 \u00fadaje v s3, v efekt\u00edvnom st\u013apcovom form\u00e1te a s rozumn\u00fdm delen\u00edm \u00fadajov po prie\u010dinkoch\u2026 sme dostali DataLake a r\u00fdchly a lacn\u00fd analytick\u00fd engine \u2014 zadarmo. A stal sa ve\u013emi popul\u00e1rnym v spolo\u010dnosti, preto\u017ee rozumie SQL a pracuje r\u00e1dovo r\u00fdchlej\u0161ie, ako zadan\u00edm\/pozastaven\u00edm\/nastaven\u00edm klastra. \"A ak je v\u00fdsledok rovnak\u00fd, pre\u010do plati\u0165 viac?\"<\/p>\n<p>Dotaz do Atheny vyzer\u00e1 pribli\u017ene takto. Ak si \u017eel\u00e1te, samozrejme, m\u00f4\u017eete vytvori\u0165 dos\u0165 <noindex><a rel=\"nofollow\" href=\"https:\/\/prestodb.io\/docs\/0.172\/index.html\">zlo\u017eit\u00fd a viacstranov\u00fd SQL dopyt<\/a><\/noindex>, ale obmedz\u00edme sa na jednoduch\u00e9 agregovanie. Pozrieme sa, ak\u00e9 chybov\u00e9 k\u00f3dy mal klient pred nieko\u013ek\u00fdmi t\u00fd\u017ed\u0148ami v logoch pr\u00e1ce webov\u00e9ho servera a presved\u010d\u00edme sa, \u017ee nie s\u00fa \u017eiadne chyby:<\/p>\n<p><img decoding=\"async\" alt=\"Ako sme zorganizovali vysokoefekt\u00edvny a lacn\u00fd DataLake a pre\u010do pr\u00e1ve takto\" src=\"\/wp-content\/uploads\/2020\/08\/30028991467b9e52f597faa617d374b5.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Zhrnutie<\/h3>\n<p>\nPo ceste, ktor\u00e1 nebola dlh\u00e1, ale bolestn\u00e1, sme neust\u00e1le adekv\u00e1tne hodnotili rizik\u00e1, \u00farove\u0148 zlo\u017eitosti a n\u00e1klady na \u00fadr\u017ebu a na\u0161li sme rie\u0161enie pre DataLake a analytiku, ktor\u00e9 n\u00e1s neprest\u00e1va te\u0161i\u0165 ako r\u00fdchlos\u0165ou, tak aj n\u00e1kladmi na vlastn\u00edctvo.<\/p>\n<p>Uk\u00e1zalo sa, \u017ee vybudova\u0165 efekt\u00edvny, r\u00fdchly a lacn\u00fd na prev\u00e1dzku DataLake pre potreby \u00faplne r\u00f4znych oddelen\u00ed spolo\u010dnosti je \u00faplne v sil\u00e1ch aj sk\u00fasen\u00fdch v\u00fdvoj\u00e1rov, ktor\u00ed nikdy nepracovali ako architekti a nevedia kresli\u0165 \u0161\u00edpky na \u0161tvorce a poznaj\u00fa 50 term\u00ednov z ekosyst\u00e9mu Hadoop.<\/p>\n<p>Na za\u010diatku cesty ma bolela hlava z mno\u017estva zlo\u017eit\u00fdch zverincov otvoren\u00e9ho a uzavret\u00e9ho softv\u00e9ru a zo\u0165a\u017eenia zodpovednos\u0165ou vo\u010di bud\u00facim gener\u00e1ci\u00e1m. Jednoducho za\u010dnite budova\u0165 svoj DataLake od jednoduch\u00fdch n\u00e1strojov: nagios\/munin -&gt; elastic\/kibana -&gt; Hadoop\/Spark\/s3 \u2026, zbierajte sp\u00e4tn\u00fa v\u00e4zbu a hlboko rozumejte fyzike prebiehaj\u00facich procesov. V\u0161etko zlo\u017eit\u00e9 a nejasn\u00e9 prenechajte nepriate\u013eom a konkurentom.<\/p>\n<p>Ak nechcete do cloudu a radi sa star\u00e1te, aktualizujete a patchujete otvoren\u00e9 projekty, m\u00f4\u017eete vybudova\u0165 anal\u00f3g na\u0161ej sch\u00e9my lok\u00e1lne, na kancel\u00e1rskych lacn\u00fdch strojoch s Hadoop a Presto navrch. Hlavn\u00e9 je nepresta\u0165 a \u00eds\u0165 dopredu, po\u010d\u00edta\u0165, h\u013eada\u0165 jednoduch\u00e9 a jasn\u00e9 rie\u0161enia a v\u0161etko sa ur\u010dite podar\u00ed! \u0160\u0165astie v\u0161etk\u00fdm a do nov\u00fdch stretnut\u00ed!<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/bitrix\/blog\/516374\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432 \u043a\u043e\u043c\u043c\u0435\u0440\u0447\u0435\u0441\u043a\u0443\u044e \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044e. \u0410 \u043a\u043e\u0433\u0434\u0430 \u043d\u0443\u0436\u043d\u043e \u0431\u0443\u0434\u0435\u0442 \u043e\u0431\u043d\u043e\u0432\u043b\u044f\u0442\u044c\u0441\u044f\/\u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c\u0441\u044f \u0438\u043b\u0438 \u043a\u0442\u043e-\u0442\u043e \u0441\u043b\u0443\u0447\u0430\u0439\u043d\u043e \u043f\u0435\u0440\u0435\u0437\u0430\u0433\u0440\u0443\u0437\u0438\u0442 \u043f\u0430\u0440\u0443 \u043c\u0430\u0448\u0438\u043d \u2014 \u043e\u0441\u043e\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043d\u0430\u0447\u0430\u043b\u0441\u044f \u043a\u0430\u043a\u043e\u0439-\u0442\u043e \u043d\u0430\u0432\u044f\u0437\u0447\u0438\u0432\u044b\u0439 \u0434\u0443\u0440\u043d\u043e\u0439 \u0441\u043e\u043d \u043d\u0430\u044f\u0432\u0443, \u0432\u0441\u0435 \u0440\u0435\u0437\u043a\u043e \u0443\u0441\u043b\u043e\u0436\u043d\u0438\u043b\u043e\u0441\u044c \u0434\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":92509,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-92508","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Ako sme organizovali vysoko efekt\u00edvny a cenovo dostupn\u00fd DataLake a pre\u010do pr\u00e1ve tak | ProHoster","description":"\u017dijeme vo fascinuj\u00facej dobe, ke\u010f m\u00f4\u017eeme r\u00fdchlo a jednoducho prepoji\u0165 nieko\u013eko hotov\u00fdch otvoren\u00fdch n\u00e1strojov, nastavi\u0165 ich s \u201evypnut\u00fdm vedom\u00edm\u201c pod\u013ea r\u00e1d stackoverflow, bez toho, aby sme sa zaoberali \u201emno\u017estvom p\u00edsmen\u201c, a spusti\u0165 ich.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster","og:description":"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-28T05:42:10+00:00","article:modified_time":"2020-08-28T05:42:10+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"92508","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:07:39","updated":"2022-10-01 09:50:53","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/92508","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=92508"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/92508\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/92509"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=92508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=92508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=92508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}