{"id":92508,"date":"2020-08-28T07:42:10","date_gmt":"2020-08-28T05:42:10","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak"},"modified":"2020-08-28T07:42:10","modified_gmt":"2020-08-28T05:42:10","slug":"kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","title":{"rendered":"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>\u017dijeme v \u00fa\u017easn\u00e9 dob\u011b, kdy je mo\u017en\u00e9 rychle a jednodu\u0161e propojit n\u011bkolik hotov\u00fdch otev\u0159en\u00fdch n\u00e1stroj\u016f, nastavit je s \"vypnut\u00fdm v\u011bdom\u00edm\" podle rad ze stackoverflow, ani\u017e bychom se zab\u00fdvali \"dlouh\u00fdmi v\u00fdrazy\", a uv\u00e9st je do komer\u010dn\u00edho provozu. A kdy\u017e bude pot\u0159eba se aktualizovat\/roz\u0161i\u0159ovat, nebo n\u011bkdo n\u00e1hodou restartuje p\u00e1r stroj\u016f \u2014 uv\u011bdom\u00edme si, \u017ee za\u010d\u00edn\u00e1 n\u011bjak\u00fd obt\u011b\u017euj\u00edc\u00ed zl\u00fd sen na\u017eivu, v\u0161e se n\u00e1hle zkomplikovalo do nepozn\u00e1n\u00ed, cesta zp\u011bt nen\u00ed, budoucnost je nejasn\u00e1 a bezpe\u010dn\u011bj\u0161\u00ed je m\u00edsto programov\u00e1n\u00ed chovat v\u010dely a vyr\u00e1b\u011bt s\u00fdr.<\/p>\n<p>Nen\u00ed divu, \u017ee zku\u0161en\u011bj\u0161\u00ed kolegov\u00e9, s \u0161edinami zp\u016fsoben\u00fdmi bugy, pozoruj\u00ed neuv\u011b\u0159iteln\u011b rychl\u00e9 nasazen\u00ed bal\u00ed\u010dk\u016f \"kontejner\u016f\" v \"kostk\u00e1ch\" na des\u00edtk\u00e1ch server\u016f na \"modern\u00edch jazyc\u00edch\" s vestav\u011bnou podporou asynchronn\u00edho-neblokovateln\u00e9ho vstupu-v\u00fdstupu \u2014 skromn\u011b se usm\u00edvaj\u00ed. A ti\u0161e pokra\u010duj\u00ed ve \u010dten\u00ed \"man ps\", soust\u0159ed\u00ed se do krve na zdrojov\u00e9 k\u00f3dy \"nginx\" a p\u00ed\u0161\u00ed-p\u00ed\u0161\u00ed-p\u00ed\u0161\u00ed unit testy. Kolegov\u00e9 v\u011bd\u00ed, \u017ee to nejzaj\u00edmav\u011bj\u0161\u00ed je teprve p\u0159ed n\u00e1mi, kdy\u017e se \"to v\u0161echno\" jednou stane noc\u00ed kolena pod Nov\u00fd rok. A pom\u016f\u017ee jim jen hlubok\u00e9 porozum\u011bn\u00ed povaze unixu, nau\u010den\u00e9 tabulky stav\u016f TCP\/IP a z\u00e1kladn\u00ed algoritmy t\u0159\u00edd\u011bn\u00ed-hled\u00e1n\u00ed. Aby mohli s \u00faderem hodin o\u017eivit syst\u00e9m.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nAch ano, trochu jsem se odklonil, ale douf\u00e1m, \u017ee se mi poda\u0159ilo p\u0159edat stav o\u010dek\u00e1v\u00e1n\u00ed.<br \/>\nDnes bych se cht\u011bl pod\u011blit o na\u0161i zku\u0161enost s nasazen\u00edm pohodln\u00e9ho a levn\u00e9ho stacku pro DataLake, kter\u00fd \u0159e\u0161\u00ed v\u011bt\u0161inu analytick\u00fdch \u00fakol\u016f ve firm\u011b pro \u00fapln\u011b r\u016fzn\u00e9 struktury.<\/p>\n<p>P\u0159ed n\u011bjak\u00fdm \u010dasem jsme dosp\u011bli k n\u00e1zoru, \u017ee firmy \u010d\u00edm d\u00e1l v\u00edce pot\u0159ebuj\u00ed plody jak produktov\u00e9, tak technick\u00e9 analytiky (o t\u0159e\u0161ink\u00e1ch na dortu v podob\u011b strojov\u00e9ho u\u010den\u00ed nemluv\u011b) a pro pochopen\u00ed trend\u016f a rizik \u2014 je pot\u0159eba sb\u00edrat a analyzovat st\u00e1le v\u00edce a v\u00edce metrik.<\/p>\n<h3>Z\u00e1kladn\u00ed technick\u00e1 analytika v \"Bitrix24\"<\/h3>\n<p>\nP\u0159ed n\u011bkolika lety, spolu se spu\u0161t\u011bn\u00edm slu\u017eby \u201eBitrix24\u201c, jsme intenzivn\u011b investovali \u010das a zdroje do vytvo\u0159en\u00ed jednoduch\u00e9 a spolehliv\u00e9 analytick\u00e9 platformy, kter\u00e1 by pom\u00e1hala rychle odhalit probl\u00e9my v infrastruktu\u0159e a napl\u00e1novat dal\u0161\u00ed kroky. Samoz\u0159ejm\u011b, \u017ee n\u00e1stroje bylo \u017e\u00e1douc\u00ed m\u00edt hotov\u00e9, co nejjednodu\u0161\u0161\u00ed a srozumiteln\u00e9. V\u00fdsledkem byl v\u00fdb\u011br Nagios pro monitoring a Munin pro analytiku a vizualizaci. Nyn\u00ed m\u00e1me tis\u00edce kontrol v Nagios, stovky graf\u016f v Munin a kolegov\u00e9 je denn\u011b \u00fasp\u011b\u0161n\u011b vyu\u017e\u00edvaj\u00ed. Metry jsou srozumiteln\u00e9, grafy jasn\u00e9, syst\u00e9m funguje spolehliv\u011b ji\u017e n\u011bkolik let a pravideln\u011b se do n\u011bj p\u0159id\u00e1vaj\u00ed nov\u00e9 testy a grafy: zav\u00e1d\u00edme novou slu\u017ebu do provozu \u2014 p\u0159id\u00e1v\u00e1me n\u011bkolik test\u016f a graf\u016f. \u0160\u0165astnou cestu.<\/p>\n<h3>Ruka na pulsu \u2014 roz\u0161\u00ed\u0159en\u00e1 technick\u00e1 analytika<\/h3>\n<p>\nTouha z\u00edsk\u00e1vat informace o probl\u00e9mech \u201eco nejrychleji\u201c n\u00e1s p\u0159ivedla k aktivn\u00edm experiment\u016fm se jednoduch\u00fdmi a srozumiteln\u00fdmi n\u00e1stroji \u2014 Pinba a XHProf.<\/p>\n<p>Pinba n\u00e1m odes\u00edlala statistiky o rychlosti fungov\u00e1n\u00ed \u010d\u00e1st\u00ed webov\u00fdch str\u00e1nek na PHP v UDP paketech, a my jsme mohli v re\u00e1ln\u00e9m \u010dase vid\u011bt v \u00falo\u017ei\u0161ti MySQL (s Pinba jde vlastn\u00ed engine MySQL pro rychlou anal\u00fdzu ud\u00e1lost\u00ed) kr\u00e1tk\u00fd seznam probl\u00e9m\u016f a reagovat na n\u011b. A XHProf automaticky shroma\u017e\u010foval grafy vykon\u00e1v\u00e1n\u00ed nejpomalej\u0161\u00edch PHP str\u00e1nek u klient\u016f a analyzoval, co to mohlo zp\u016fsobit \u2014 v klidu, p\u0159i \u0161\u00e1lku \u010daje nebo n\u011b\u010deho siln\u011bj\u0161\u00edho.<\/p>\n<p>P\u0159ed n\u011bjakou dobou byl n\u00e1stroj dopln\u011bn dal\u0161\u00edm pom\u011brn\u011b jednoduch\u00fdm a srozumiteln\u00fdm enginem na z\u00e1klad\u011b algoritmu obr\u00e1cen\u00e9ho indexu, skv\u011ble implementovan\u00e9ho v legend\u00e1rn\u00ed knihovn\u011b Lucene \u2014 Elastic\/Kibana. Jednoduch\u00e1 idea v\u00edcestup\u0148ov\u00e9ho zapisov\u00e1n\u00ed dokument\u016f do inverzn\u00edho indexu Lucene na z\u00e1klad\u011b ud\u00e1lost\u00ed v logech a rychl\u00e9 vyhled\u00e1v\u00e1n\u00ed pomoc\u00ed fasetn\u00edho d\u011blen\u00ed se uk\u00e1zala jako skute\u010dn\u011b u\u017eite\u010dn\u00e1.<\/p>\n<p>Navzdory pon\u011bkud technick\u00e9mu vzhledu vizualizac\u00ed v Kibana s \u201eprot\u00e9kaj\u00edc\u00edmi naho\u0159e\u201c n\u00edzko\u00farov\u0148ov\u00fdmi koncepty jako \u201ebucket\u201c a znovu vynalezen\u00e9mu jazyku d\u00e1vno zapomenut\u00e9 rela\u010dn\u00ed algebry \u2014 n\u00e1stroj se n\u00e1m za\u010dal dob\u0159e pom\u00e1hat p\u0159i n\u00e1sleduj\u00edc\u00edch \u00faloh\u00e1ch:<\/p>\n<ul>\n<li>Kolik bylo chyb PHP u klienta Bitrix24 na port\u00e1lu p1 za posledn\u00ed hodinu a jak\u00e9? Pochopit, odpustit a rychle opravit.<\/li>\n<li>Kolik video hovor\u016f prob\u011bhlo na port\u00e1lech v N\u011bmecku za posledn\u00edch 24 hodin, jak\u00e1 byla jejich kvalita a byly n\u011bjak\u00e9 probl\u00e9my s kan\u00e1lem\/s\u00edt\u00ed?<\/li>\n<li>Jak dob\u0159e funguje syst\u00e9mov\u00e1 funkce (na\u0161e roz\u0161\u00ed\u0159en\u00ed v C pro PHP), zkompilovan\u00e1 z zdrojov\u00fdch k\u00f3d\u016f v posledn\u00ed aktualizaci slu\u017eby a distribuovan\u00e1 klient\u016fm? Neobjevily se segfaulty?<\/li>\n<li>Jsou data klient\u016f ulo\u017eena v pam\u011bti PHP? Neobjevily se chyby p\u0159ekro\u010den\u00ed p\u0159id\u011blen\u00e9 pam\u011bti proces\u016fm: \u201eout of memory\u201c? Naj\u00edt a odstranit.<\/li>\n<\/ul>\n<p>\nZde je konkr\u00e9tn\u00ed p\u0159\u00edklad. I p\u0159es pe\u010dliv\u00e9 a v\u00edcestup\u0148ov\u00e9 testov\u00e1n\u00ed se u klienta objevila nep\u0159\u00edjemn\u00e1 a ne\u010dekan\u00e1 chyba p\u0159i velmi atypick\u00e9m p\u0159\u00edpadu a po\u0161kozen\u00fdch vstupn\u00edch datech, zazn\u011bl alarm a za\u010dal proces jej\u00edho rychl\u00e9ho opraven\u00ed:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/8a802dba41b5d1a85c0dc41dfbf8b84e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKrom\u011b toho umo\u017e\u0148uje Kibana uspo\u0159\u00e1dat upozorn\u011bn\u00ed na vybran\u00e9 ud\u00e1losti a b\u011bhem kr\u00e1tk\u00e9ho \u010dasu za\u010dalo t\u00edmto n\u00e1strojem pou\u017e\u00edvat des\u00edtky zam\u011bstnanc\u016f z r\u016fzn\u00fdch odd\u011blen\u00ed \u2013 od technick\u00e9 podpory a v\u00fdvoje a\u017e po QA.<\/p>\n<p>Aktivitu jak\u00e9hokoli odd\u011blen\u00ed uvnit\u0159 spole\u010dnosti bylo mo\u017en\u00e9 snadno sledovat a m\u011b\u0159it \u2013 m\u00edsto ru\u010dn\u00edho anal\u00fdzy log\u016f na serverech sta\u010dilo jednou nastavit parsov\u00e1n\u00ed log\u016f a jejich odes\u00edl\u00e1n\u00ed do elastic clusteru, aby si \u010dlov\u011bk mohl u\u017e\u00edvat pohled na dashboardu Kibana, kolik prodan\u00fdch dvouhlav\u00fdch ko\u0165at bylo vyti\u0161t\u011bno na 3D tisk\u00e1rn\u011b za minul\u00fd lun\u00e1rn\u00ed m\u011bs\u00edc.<\/p>\n<h3>Z\u00e1kladn\u00ed business analytika<\/h3>\n<p>\nV\u0161ichni v\u00ed, \u017ee \u010dasto business analytika ve spole\u010dnostech za\u010d\u00edn\u00e1 extr\u00e9mn\u011b aktivn\u00edm pou\u017e\u00edv\u00e1n\u00edm, ano, ano, Excelu. Ale hlavn\u011b, aby t\u00edmto nekon\u010dila. Olej do ohn\u011b je\u0161t\u011b dob\u0159e p\u0159il\u00e9v\u00e1 cloudov\u00fd Google Analytics \u2013 na dobr\u00e9m si zvykne\u0161 rychle.<\/p>\n<p>V na\u0161\u00ed harmonicky se rozv\u00edjej\u00edc\u00ed spole\u010dnosti se za\u010dali objevovat \"proroci\" intenzivn\u011bj\u0161\u00ed pr\u00e1ce s v\u011bt\u0161\u00edmi daty. Pravideln\u011b se objevuj\u00ed pot\u0159eby po hloubkov\u00fdch a mnohostrann\u00fdch reportech a \u00fasil\u00edm kluk\u016f z r\u016fzn\u00fdch odd\u011blen\u00ed bylo p\u0159ed n\u011bjak\u00fdm \u010dasem zorganizov\u00e1no jednoduch\u00e9 a praktick\u00e9 \u0159e\u0161en\u00ed \u2013 spojen\u00ed ClickHouse a PowerBI.<\/p>\n<p>Pom\u011brn\u011b dlouhou dobu toto flexibiln\u00ed \u0159e\u0161en\u00ed v\u00fdborn\u011b pom\u00e1halo, ale postupn\u011b za\u010dalo doch\u00e1zet k ch\u00e1p\u00e1n\u00ed, \u017ee ClickHouse nen\u00ed guma a nem\u011blo by se s n\u00edm takto zach\u00e1zet.<\/p>\n<p>Je d\u016fle\u017eit\u00e9 si dob\u0159e uv\u011bdomit, \u017ee ClickHouse, stejn\u011b jako Druid, Vertica a Amazon RedShift (kter\u00fd vych\u00e1z\u00ed z Postgresu), jsou analytick\u00e9 enginy optimalizovan\u00e9 pro pom\u011brn\u011b pohodlnou analytiku (s\u010d\u00edt\u00e1n\u00ed, aggregace, minimum a maximum sloupce a ob\u010dasn\u00e9 spojov\u00e1n\u00ed), proto\u017ee jsou navr\u017eeny pro efektivn\u00ed ukl\u00e1d\u00e1n\u00ed sloupcov\u00fdch rela\u010dn\u00edch tabulek, na rozd\u00edl od zn\u00e1m\u00e9ho MySQL a dal\u0161\u00edch (datab\u00e1z\u00ed orientovan\u00fdch na \u0159\u00e1dky).<\/p>\n<p>V podstat\u011b je ClickHouse jen prostorn\u011bj\u0161\u00ed \"datab\u00e1ze\", s trochu nepohodln\u00fdm bodov\u00fdm vkl\u00e1d\u00e1n\u00edm (tak to bylo zam\u00fd\u0161leno, v po\u0159\u00e1dku), ale p\u0159\u00edjemnou analytikou a sadou zaj\u00edmav\u00fdch siln\u00fdch funkc\u00ed pro pr\u00e1ci s daty. Ano, m\u016f\u017eete dokonce vytvo\u0159it klastr \u2014 ale ch\u00e1pete, \u017ee pou\u017e\u00edvat mikroskop na zatlouk\u00e1n\u00ed h\u0159eb\u00edk\u016f nen\u00ed \u00fapln\u011b spr\u00e1vn\u00e9 a za\u010dali jsme hledat jin\u00e1 \u0159e\u0161en\u00ed.<\/p>\n<h3>Popt\u00e1vka po Pythonu a analytik\u00e1ch<\/h3>\n<p>\nV na\u0161\u00ed spole\u010dnosti je spousta v\u00fdvoj\u00e1\u0159\u016f, kte\u0159\u00ed p\u00ed\u0161\u00ed k\u00f3d t\u00e9m\u011b\u0159 ka\u017ed\u00fd den po dobu 10-20 let v PHP, JavaScriptu, C#, C\/C++, Jav\u011b, Go, Rustu, Pythonu, Bash. Tak\u00e9 je zde mnoho zku\u0161en\u00fdch syst\u00e9mov\u00fdch administr\u00e1tor\u016f, kte\u0159\u00ed pro\u0161li nejednou zcela neuv\u011b\u0159itelnou katastrofou, kter\u00e1 se nevleze do z\u00e1kon\u016f statistiky (nap\u0159\u00edklad, kdy\u017e se p\u0159i siln\u00e9m \u00faderu blesku zni\u010d\u00ed v\u011bt\u0161ina disk\u016f v RAID-10). V takov\u00fdch podm\u00ednk\u00e1ch nebylo dlouho jasn\u00e9, co je \"analytik na Pythonu\". Python je jako PHP, jen jm\u00e9no je o n\u011bco del\u0161\u00ed a ne\u017e\u00e1douc\u00ed l\u00e1tky m\u011bn\u00edc\u00ed v\u011bdom\u00ed jsou v p\u016fvodn\u00edm k\u00f3du interpretru o n\u011bco m\u00e9n\u011b. Nicm\u00e9n\u011b, s vytvo\u0159en\u00edm st\u00e1le nov\u011bj\u0161\u00edch a nov\u011bj\u0161\u00edch analytick\u00fdch zpr\u00e1v si zku\u0161en\u00ed v\u00fdvoj\u00e1\u0159i za\u010dali st\u00e1le v\u00edce uv\u011bdomovat d\u016fle\u017eitost \u00fazk\u00e9 specializace v n\u00e1stroj\u00edch typu numpy, pandas, matplotlib, seaborn.<br \/>\nRozhoduj\u00edc\u00ed roli, pravd\u011bpodobn\u011b, sehr\u00e1ly n\u00e1hl\u00e9 mdloby zam\u011bstnanc\u016f p\u0159i spojen\u00ed slov \"logistick\u00e1 regrese\" a demonstrace efektivn\u00edho vytv\u00e1\u0159en\u00ed zpr\u00e1v na objemn\u00fdch datech pomoc\u00ed ano, ano, pyspark.<\/p>\n<p>Apache Spark, jeho funk\u010dn\u00ed paradigma, na kter\u00e9 se skv\u011ble hod\u00ed rela\u010dn\u00ed algebra a mo\u017enosti, vyvolalo takov\u00fd dojem na v\u00fdvoj\u00e1\u0159e zvykl\u00e9 na MySQL, \u017ee pot\u0159eba pos\u00edlen\u00ed bojov\u00fdch \u0159ad zku\u0161en\u00fdmi analytiky se stala jasnou jako den.<\/p>\n<h3>Dal\u0161\u00ed pokusy Apache Spark\/Hadoop vzl\u00e9tnout a co ne\u0161lo \u00fapln\u011b podle sc\u00e9n\u00e1\u0159e<\/h3>\n<p>\nBrzy se v\u0161ak uk\u00e1zalo, \u017ee se Sparkem, jak se zd\u00e1, n\u011bco syst\u00e9mov\u011b nen\u00ed v po\u0159\u00e1dku, nebo je t\u0159eba si prost\u011b l\u00e9pe um\u00fdt ruce. Pokud byl stack Hadoop\/MapReduce\/Lucene vytvo\u0159en dostate\u010dn\u011b zku\u0161en\u00fdmi program\u00e1tory, co\u017e je z\u0159ejm\u00e9, pokud se d\u016fkladn\u011b pod\u00edv\u00e1me na zdrojov\u00e9 k\u00f3dy v Jav\u011b nebo na my\u0161lenky Duga Cuttinga v Lucene, pak je Spark n\u00e1hle naps\u00e1n v velmi sporn\u00e9m z praktick\u00e9ho hlediska a nyn\u00ed ji\u017e nerozv\u00edjen\u00e9m exotick\u00e9m jazyce Scala. A pravideln\u00fd pokles v\u00fdpo\u010dt\u016f na clusteru Spark kv\u016fli nelogick\u00e9 a m\u00e1lo p\u0159ehledn\u00e9 pr\u00e1ci s alokac\u00ed pam\u011bti pro operace reduce (sou\u010dasn\u011b p\u0159ijde mnoho kl\u00ed\u010d\u016f) \u2014 vytvo\u0159il kolem n\u011bj aurou n\u011b\u010deho, co m\u00e1 prostor pro r\u016fst. Nav\u00edc situaci zhor\u0161ovalo velk\u00e9 mno\u017estv\u00ed podivn\u00fdch otev\u0159en\u00fdch port\u016f, do\u010dasn\u00fdch soubor\u016f, rostouc\u00edch na naprosto nepochopiteln\u00fdch m\u00edstech a z\u00e1vislost\u00ed na jar \u2014 co\u017e vyvol\u00e1valo u syst\u00e9mov\u00fdch administr\u00e1tor\u016f jedno a dob\u0159e zn\u00e1m\u00e9 od d\u011btstv\u00ed pocity: zu\u0159ivou nen\u00e1vist (mo\u017en\u00e1 bylo t\u0159eba si um\u00fdt ruce m\u00fddlem).<\/p>\n<p>Nakonec jsme \"p\u0159e\u017eili\" n\u011bkolik intern\u00edch analytick\u00fdch projekt\u016f, kter\u00e9 aktivn\u011b vyu\u017e\u00edvaly Apache Spark (v\u010detn\u011b Spark Streaming, Spark SQL) a ekosyst\u00e9m Hadoop (a dal\u0161\u00ed a dal\u0161\u00ed). I kdy\u017e jsme se \u010dasem nau\u010dili \"to\" docela dob\u0159e p\u0159ipravovat a monitorovat a \"to\" t\u00e9m\u011b\u0159 p\u0159estalo n\u00e1hle padat kv\u016fli zm\u011bn\u011b povahy dat a nevyv\u00e1\u017eenosti rovnom\u011brn\u00e9ho hashov\u00e1n\u00ed RDD, touha m\u00edt n\u011bco ji\u017e p\u0159ipraven\u00e9ho, aktualizovan\u00e9ho a spravovan\u00e9ho n\u011bkde v cloudu s\u00edlila \u010d\u00edm d\u00e1l v\u00edce. V tuto dobu jsme se pokusili pou\u017e\u00edt hotov\u00fd cloudov\u00fd bal\u00ed\u010dek Amazon Web Services \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/emr\/\">EMR<\/a><\/noindex> a v d\u016fsledku toho jsme se sna\u017eili \u0159e\u0161it \u00fakoly ji\u017e na n\u011bm. EMR je Amazonem p\u0159ipraven\u00fd Apache Spark s dal\u0161\u00edmi programy z ekosyst\u00e9mu, p\u0159ibli\u017en\u011b jako Cloudera\/Hortonworks bal\u00ed\u010dky.<\/p>\n<h3>\"Gumov\u00e9\" \u00falo\u017ei\u0161t\u011b pro anal\u00fdzu \u2014 nal\u00e9hav\u00e1 pot\u0159eba<\/h3>\n<p>\nZku\u0161enosti \"p\u0159\u00edpravy\" Hadoop\/Spark s pop\u00e1leninami r\u016fzn\u00fdch \u010d\u00e1st\u00ed t\u011bla nevy\u0161ly nadarmo. Stala se st\u00e1le jasn\u011bji z\u0159eteln\u00e1 pot\u0159eba vytvo\u0159it jednotn\u00e9, cenov\u011b dostupn\u00e9 a spolehliv\u00e9 \u00falo\u017ei\u0161t\u011b, kter\u00e9 by bylo odoln\u00e9 v\u016f\u010di hardwarov\u00fdm hav\u00e1ri\u00edm a umo\u017enilo ukl\u00e1dat soubory v r\u016fzn\u00fdch form\u00e1tech z r\u016fzn\u00fdch syst\u00e9m\u016f a efektivn\u011b vykon\u00e1vat dotazy na tato data v rozumn\u00e9m \u010dase pro zpr\u00e1vy.<\/p>\n<p>Cht\u011bli jsme, aby aktualizace softwaru t\u00e9to platformy nebyla no\u010dn\u00ed m\u016frou na Nov\u00fd rok, s \u010dten\u00edm 20str\u00e1nkov\u00fdch Java stack trace a anal\u00fdzou kilometr\u016f podrobn\u00fdch log\u016f provozu clustera pomoc\u00ed Spark History Server a lupy s podsv\u00edcen\u00edm. Pot\u0159ebujeme jednoduch\u00fd a transparentn\u00ed n\u00e1stroj, kter\u00fd nevy\u017eaduje pravidelnou diagnostiku pod kapotou, pokud p\u0159estane fungovat standardn\u00ed MapReduce dotaz v d\u016fsledku selh\u00e1n\u00ed pam\u011bti pracovn\u00edka pro vyrovn\u00e1vac\u00ed data, kv\u016fli nevhodn\u011b zvolen\u00e9mu algoritmu partitioningu vstupn\u00edch dat.<\/p>\n<h3>Amazon S3 \u2013 kandid\u00e1t na DataLake?<\/h3>\n<p>\nZku\u0161enosti s Hadoop\/MapReduce m\u011b nau\u010dily, \u017ee pot\u0159ebuji \u0161k\u00e1lovateln\u00fd a spolehliv\u00fd souborov\u00fd syst\u00e9m a nad n\u00edm \u0161k\u00e1lovateln\u00e9 pracovn\u00edky, kte\u0159\u00ed se \"p\u0159ibli\u017euj\u00ed\" k dat\u016fm, aby se nehnali data po s\u00edti. Pracovn\u00edci by m\u011bli um\u011bt \u010d\u00edst data v r\u016fzn\u00fdch form\u00e1tech, ale ide\u00e1ln\u011b by nem\u011bli \u010d\u00edst zbyte\u010dn\u00e9 informace a data by m\u011bla b\u00fdt ulo\u017eena v pro pracovn\u00edky vhodn\u00fdch form\u00e1tech.<\/p>\n<p><b>Je\u0161t\u011b jednou \u2013 hlavn\u00ed my\u0161lenka.<\/b> Nem\u00e1m z\u00e1jem \"nahr\u00e1vat\" velk\u00e1 data do jedin\u00e9ho clusterov\u00e9ho analytick\u00e9ho enginu, kter\u00fd se d\u0159\u00edve nebo pozd\u011bji stejn\u011b ucpou, a bude t\u0159eba jej ne\u0161ikovn\u011b rozd\u011blit. Chci ukl\u00e1dat soubory, prost\u011b soubory, v srozumiteln\u00e9m form\u00e1tu a efektivn\u011b prov\u00e1d\u011bt analytick\u00e9 dotazy r\u016fzn\u00fdmi, ale pochopiteln\u00fdmi n\u00e1stroji. A soubor\u016f v r\u016fzn\u00fdch form\u00e1tech bude st\u00e1le v\u00edce a v\u00edce. A je lep\u0161\u00ed rozd\u011blit ne engine, ale vstupn\u00ed data. Rozhodli jsme se, \u017ee pot\u0159ebujeme roz\u0161i\u0159iteln\u00fd a univerz\u00e1ln\u00ed DataLake...<\/p>\n<p>A co kdy\u017e ulo\u017e\u00edme soubory do zn\u00e1m\u00e9ho a mnoha lidmi pou\u017e\u00edvan\u00e9ho \u0161k\u00e1lovateln\u00e9ho cloudov\u00e9ho \u00falo\u017ei\u0161t\u011b Amazon S3, ani\u017e bychom se starali o vlastn\u00ed p\u0159\u00edpravu \u201ekotlet\u201c z Hadoop?<\/p>\n<p>Jasn\u011b, osobn\u00ed \u00fadaje \u201ene\u201d, ale co ostatn\u00ed data, kdy\u017e je tam vyvedeme a \u201eefektivn\u011b jimi manipulujeme\u201c?<\/p>\n<h3>Klastr-velk\u00e1 data-analytick\u00e1 ekosyst\u00e9m Amazon Web Services \u2013 velmi jednoduch\u00fdmi slovy<\/h3>\n<p>\nPodle na\u0161ich zku\u0161enost\u00ed s AWS se zde dlouho a aktivn\u011b pou\u017e\u00edv\u00e1 r\u016fzn\u00e9 varianty Apache Hadoop\/MapReduce, nap\u0159\u00edklad v slu\u017eb\u011b DataPipeline (z\u00e1vid\u00edm koleg\u016fm, jak se nau\u010dili jej spr\u00e1vn\u011b p\u0159ipravovat). Tady jsme nastavili z\u00e1lohy z r\u016fzn\u00fdch slu\u017eeb z tabulek DynamoDB:<br \/>\n<img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/058dc54ed032a7bf3e9e129646202440.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA ty se pravideln\u011b prov\u00e1d\u011bj\u00ed na vestav\u011bn\u00fdch clusterech Hadoop\/MapReduce jako hodinky ji\u017e n\u011bkolik let. \"Nastavil a zapomn\u011bl\":<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/a6569da8cafdb96c63250bb32bf51704.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTak\u00e9 je mo\u017en\u00e9 efektivn\u011b provozovat datasantismus t\u00edm, \u017ee pro analytiky spust\u00edme Jupiter notebooky v cloudu a pou\u017eijeme je pro tr\u00e9nink a nasazen\u00ed AI model\u016f v syst\u00e9mu AWS SageMaker. Takhle to vypad\u00e1 u n\u00e1s:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/c825d979c9278a8edf8e1e747ef6def8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nA ano, m\u016f\u017eete si spustit notebook v cloudu nebo pro analytika napojit na Hadoop\/Spark cluster, prov\u00e9st v\u00fdpo\u010dty a pak to v\u0161e \u201ep\u0159ib\u00edt\u201c:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/73cea18c54d2a9ce8d0441463991808b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nOpravdu pohodln\u00e9 pro jednotliv\u00e9 analytick\u00e9 projekty a pro n\u011bkter\u00e9 jsme \u00fasp\u011b\u0161n\u011b pou\u017e\u00edvali slu\u017ebu EMR pro rozs\u00e1hl\u00e9 v\u00fdpo\u010dty a anal\u00fdzu. A co syst\u00e9mov\u00e9 \u0159e\u0161en\u00ed pro DataLake, bude to mo\u017en\u00e9? V tuto chv\u00edli jsme byli na pokraji nad\u011bje a zoufalstv\u00ed a pokra\u010dovali v hled\u00e1n\u00ed.<\/p>\n<h3>AWS Glue \u2014 pe\u010dliv\u011b zabalen\u00fd Apache Spark \u201ena steroidech\u201c<\/h3>\n<p>\nUk\u00e1zalo se, \u017ee AWS m\u00e1 \u201esvou\u201c verzi stacku \u201eHive\/Pig\/Spark\u201c. \u00dalohu Hive, tedy katalogu soubor\u016f a jejich typ\u016f v DataLake, hraje slu\u017eba \u201eData catalog\u201c, kter\u00e1 se ani netaj\u00ed svou kompatibilitou s form\u00e1tem Apache Hive. Do t\u00e9to slu\u017eby je t\u0159eba p\u0159idat informace o tom, kde se nach\u00e1zej\u00ed va\u0161e soubory a v jak\u00e9m form\u00e1tu jsou. Data mohou b\u00fdt nejen v S3, ale i v datab\u00e1zi, ale o tom v tomto p\u0159\u00edsp\u011bvku nep\u00ed\u0161eme. Takhle je n\u00e1\u0161 katalog dat DataLake uspo\u0159\u00e1d\u00e1n:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/adb45d09698fdacbf41c86bbadde8bb2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSoubory jsou registrov\u00e1ny, skv\u011bl\u00e9. Pokud se soubory aktualizovaly \u2014 spou\u0161t\u00edme bu\u010f ru\u010dn\u011b, nebo podle pl\u00e1nu crawlers, kter\u00e9 obnov\u00ed informace o nich z jezera a ulo\u017e\u00ed je. D\u00e1le lze data z jezera zpracov\u00e1vat a v\u00fdsledky n\u011bkam exportovat. V nejjednodu\u0161\u0161\u00edm p\u0159\u00edpad\u011b \u2014 exportujeme tak\u00e9 do S3. Zpracov\u00e1n\u00ed dat lze prov\u00e1d\u011bt kdekoli, ale doporu\u010duje se nastavit proces zpracov\u00e1n\u00ed na Apache Spark clusterech s vyu\u017eit\u00edm roz\u0161\u00ed\u0159en\u00fdch mo\u017enost\u00ed prost\u0159ednictv\u00edm API AWS Glue. V podstat\u011b je mo\u017en\u00e9 vz\u00edt star\u00fd dobr\u00fd a zn\u00e1m\u00fd k\u00f3d v pythonu s pou\u017eit\u00edm knihovny pyspark a nastavit jeho spu\u0161t\u011bn\u00ed na N uzlech clustru s n\u011bjak\u00fdm v\u00fdkonem a monitoringem, ani\u017e bychom se museli hrabat v podrobnostech Hadoopu a tahat dockerov\u00e9 kontejnery a \u0159e\u0161it konflikty z\u00e1vislost\u00ed.<\/p>\n<p><b>Je\u0161t\u011b jednou \u2014 jednoduch\u00e1 my\u0161lenka.<\/b> Nen\u00ed t\u0159eba nastavovat Apache Spark, sta\u010d\u00ed napsat k\u00f3d v pythonu pro pyspark, otestovat ho lok\u00e1ln\u011b na pracovn\u00edm stole a pot\u00e9 ho spustit na velk\u00e9m clusteru v cloudu, uv\u00e9st, kde le\u017e\u00ed vstupn\u00ed data a kam ulo\u017eit v\u00fdsledek. N\u011bkdy to je pot\u0159eba a u\u017eite\u010dn\u00e9, a takto je to u n\u00e1s nastaveno:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/dc03181573bb3f5cfcc3a8760bc7e07b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTak\u017ee, pokud je pot\u0159eba n\u011bco spo\u010d\u00edtat na Spark clusteru pomoc\u00ed dat v S3 \u2014 nap\u00ed\u0161eme k\u00f3d v pythonu\/pyspark, otestujeme a hur\u00e1 do cloudu.<\/p>\n<p>A co orchestrace? A pokud \u00fakol spadne a zmiz\u00ed? Ano, navrhuje se vytvo\u0159it hezkou pipeline ve stylu Apache Pig a my jsme je dokonce vyzkou\u0161eli, ale rozhodli jsme se zat\u00edm pou\u017e\u00edt na\u0161i hluboce p\u0159izp\u016fsobenou orchestraci na PHP a JavaScriptu (ch\u00e1pu, vznik\u00e1 kognitivn\u00ed disonance, ale funguje to, roky a bez chyb).<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/e00ed2047c5c6492e36fccc82e7278a3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Form\u00e1t soubor\u016f ulo\u017een\u00fdch v jeze\u0159e \u2013 kl\u00ed\u010d k v\u00fdkonnosti<\/h3>\n<p>\nJe velmi, velmi d\u016fle\u017eit\u00e9 pochopit je\u0161t\u011b dva kl\u00ed\u010dov\u00e9 momenty. Aby byly dotazy na soubory v jeze\u0159e prov\u00e1d\u011bny co nejrychleji a v\u00fdkonnost se p\u0159i p\u0159id\u00e1v\u00e1n\u00ed nov\u00fdch informac\u00ed nezhor\u0161ovala, je t\u0159eba:<\/p>\n<ul>\n<li>Sloupce soubor\u016f uchov\u00e1vat odd\u011blen\u011b (aby nebylo nutn\u00e9 \u010d\u00edst v\u0161echny \u0159\u00e1dky pro zji\u0161t\u011bn\u00ed, co je ve sloupc\u00edch). K tomu jsme zvolili form\u00e1t parquet se zpracov\u00e1n\u00edm.<\/li>\n<li>Je velmi d\u016fle\u017eit\u00e9 shardovat soubory do slo\u017eek ve smyslu: jazyk, rok, m\u011bs\u00edc, den, t\u00fdden. Enginy, kter\u00e9 rozum\u00ed tomuto typu shardov\u00e1n\u00ed, budou prohled\u00e1vat pouze pot\u0159ebn\u00e9 slo\u017eky, ani\u017e by musely proch\u00e1zet v\u0161echny data.<\/li>\n<\/ul>\n<p>\nV podstat\u011b t\u00edmto zp\u016fsobem ukl\u00e1d\u00e1te vstupn\u00ed data v nejefektivn\u011bj\u0161\u00edm form\u00e1tu pro analytick\u00e9 enginy, kter\u00e9 um\u00ed selektivn\u011b p\u0159istupovat do shardovan\u00fdch slo\u017eek a \u010d\u00edst ze soubor\u016f pouze pot\u0159ebn\u00e9 sloupce. Nen\u00ed t\u0159eba nikam \"nahr\u00e1vat\" data (\u00falo\u017ei\u0161t\u011b by se prost\u011b roztrhlo) \u2013 jednodu\u0161e je rozumn\u011b ulo\u017ete do souborov\u00e9ho syst\u00e9mu ve spr\u00e1vn\u00e9m form\u00e1tu. Samoz\u0159ejm\u011b, m\u011blo by b\u00fdt jasn\u00e9, \u017ee uchov\u00e1vat v DataLake ohromn\u00fd csv-soubor, kter\u00fd je t\u0159eba nejprve p\u0159e\u010d\u00edst \u0159\u00e1dek po \u0159\u00e1dku, aby se extrahovaly sloupce \u2013 nen\u00ed p\u0159\u00edli\u0161 rozumn\u00e9. Zva\u017ete je\u0161t\u011b jednou dv\u011b v\u00fd\u0161e uveden\u00e9 body, pokud zat\u00edm nen\u00ed jasn\u00e9, pro\u010d je to v\u0161echno d\u016fle\u017eit\u00e9.<\/p>\n<h3>AWS Athena \u2013 \"d\u00e9mon\" z krabi\u010dky<\/h3>\n<p>\nA zde, p\u0159i vytv\u00e1\u0159en\u00ed jezera, jsme jaksi mimochodem narazili na Amazon Athena. Najednou se uk\u00e1zalo, \u017ee pokud pe\u010dliv\u011b rozd\u011bl\u00edme na\u0161e obrovsk\u00e9 logov\u00e9 soubory do shard\u016f-slo\u017eek ve spr\u00e1vn\u00e9m (parquet) sloupcov\u00e9m form\u00e1tu \u2013 m\u016f\u017eeme na nich velmi rychle prov\u00e1d\u011bt mimo\u0159\u00e1dn\u011b informativn\u00ed dotazy a vytv\u00e1\u0159et zpr\u00e1vy BEZ Apache Spark\/Glue clusteru.<\/p>\n<p>Engin Athena, kter\u00fd pracuje s daty v s3, je zalo\u017een na legend\u00e1rn\u00edm <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/big-data\/what-is-presto\/\">Presto<\/a><\/noindex> \u2014 z\u00e1stupce rodiny MPP (masivn\u011b paraleln\u00ed zpracov\u00e1n\u00ed) p\u0159\u00edstup\u016f k zpracov\u00e1n\u00ed dat, kter\u00fd \u010derp\u00e1 data tam, kde le\u017e\u00ed, od s3 a Hadoop a\u017e po Cassandru a b\u011b\u017en\u00e9 textov\u00e9 soubory. Sta\u010d\u00ed po\u017e\u00e1dat Athenu, aby provedla SQL dotaz, a pak u\u017e \u201ev\u0161echno funguje rychle a samo\u201c. Je d\u016fle\u017eit\u00e9 poznamenat, \u017ee Athena je \u201einteligentn\u00ed\u201c, chod\u00ed jen do pot\u0159ebn\u00fdch shardovan\u00fdch slo\u017eek a \u010dte pouze pot\u0159ebn\u00e9 sloupce v dotazu.<\/p>\n<p>Takt\u00e9\u017e tariface dotaz\u016f na Athen\u011b je zaj\u00edmav\u00e1. Plat\u00edme za <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/athena\/pricing\/\">objem prosklen\u00fdch dat<\/a><\/noindex>. Tzn. ne za po\u010det stroj\u016f v clusteru za minutu, ale\u2026 za skute\u010dn\u011b prosklen\u00e1 na 100-500 stroj\u00edch pouze nezbytn\u00e1 data pro vykon\u00e1n\u00ed dotazu.<\/p>\n<p>A kdy\u017e po\u017eadujeme pouze pot\u0159ebn\u00e9 sloupce z dob\u0159e shardovan\u00fdch slo\u017eek, uk\u00e1zalo se, \u017ee slu\u017eba Athena n\u00e1s stoj\u00ed des\u00edtky dolar\u016f za m\u011bs\u00edc. No, je to skv\u011bl\u00e9, t\u00e9m\u011b\u0159 zdarma ve srovn\u00e1n\u00ed s anal\u00fdzou na clusterech!<\/p>\n<p>Tady je, jak shardujeme sv\u00e1 data v s3:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/00bd9ae48c1cd13f3c4f7d32692c9209.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV d\u016fsledku toho za\u010daly r\u016fzn\u00e9 odd\u011blen\u00ed ve spole\u010dnosti, od informa\u010dn\u00ed bezpe\u010dnosti po analytiku, aktivn\u011b vytv\u00e1\u0159et dotazy na Athen\u011b a rychle, b\u011bhem sekund, dost\u00e1vat u\u017eite\u010dn\u00e9 odpov\u011bdi z \u201evelk\u00fdch\u201c dat za pom\u011brn\u011b dlouh\u00e1 obdob\u00ed: m\u011bs\u00edce, pololet\u00ed atd.<\/p>\n<p>Ale \u0161li jsme d\u00e1l a za\u010dali hledat odpov\u011bdi v cloudu <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/connect-with-odbc.html\">prost\u0159ednictv\u00edm ODBC ovlada\u010de<\/a><\/noindex>: analytik v obvykl\u00e9 konzoli p\u00ed\u0161e SQL dotaz, kter\u00fd na 100-500 stroj\u00edch \u201eza babku\u201c prohled\u00e1v\u00e1 data v s3 a vrac\u00ed odpov\u011b\u010f obvykle za jednotky sekund. Pohodln\u00e9. A rychl\u00e9. Po\u0159\u00e1d tomu nemohu uv\u011b\u0159it.<\/p>\n<p>Nakonec, kdy\u017e jsme se rozhodli ukl\u00e1dat data v s3, v efektivn\u00edm sloupcov\u00e9m form\u00e1tu a s rozumn\u00fdm shardov\u00e1n\u00edm dat podle slo\u017eek\u2026 z\u00edskali jsme DataLake a rychl\u00fd a levn\u00fd analytick\u00fd engine \u2014 zdarma. A stal se velmi popul\u00e1rn\u00ed ve spole\u010dnosti, proto\u017ee rozum\u00ed SQL a pracuje mnohem rychleji ne\u017e p\u0159i spu\u0161t\u011bn\u00ed\/zastaven\u00ed\/nastaven\u00ed clusterech. \u201eA pokud je v\u00fdsledek stejn\u00fd, pro\u010d platit v\u00edc?\u201c<\/p>\n<p>Dotaz na Athen\u011b vypad\u00e1 p\u0159ibli\u017en\u011b takto. Pokud si p\u0159ejete, samoz\u0159ejm\u011b, m\u016f\u017eete formulovat pom\u011brn\u011b <noindex><a rel=\"nofollow\" href=\"https:\/\/prestodb.io\/docs\/0.172\/index.html\">slo\u017eit\u00fd a v\u00edcestr\u00e1nkov\u00fd SQL dotaz<\/a><\/noindex>, ale omez\u00edme se na jednoduch\u00e9 seskupen\u00ed. Pod\u00edv\u00e1me se, jak\u00e9 k\u00f3dy odpov\u011bd\u00ed m\u011bl klient p\u0159ed n\u011bkolika t\u00fddny v logech pr\u00e1ce webov\u00e9ho serveru a ujist\u00edme se, \u017ee nebyly \u017e\u00e1dn\u00e9 chyby:<\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme zorganizovali vysoce efektivn\u00ed a cenov\u011b dostupn\u00fd DataLake a pro\u010d pr\u00e1v\u011b tak\" src=\"\/wp-content\/uploads\/2020\/08\/30028991467b9e52f597faa617d374b5.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Z\u00e1v\u011bry<\/h3>\n<p>\nPro\u0161li jsme, ne\u0159ekl bych, \u017ee dlouhou, ale bolestnou cestu, kdy jsme neust\u00e1le adekv\u00e1tn\u011b hodnotili rizika, \u00farove\u0148 slo\u017eitosti a n\u00e1klady na \u00fadr\u017ebu, a na\u0161li jsme \u0159e\u0161en\u00ed pro DataLake a anal\u00fdzu, kter\u00e9 n\u00e1s nep\u0159est\u00e1v\u00e1 t\u011b\u0161it jak rychlost\u00ed, tak n\u00e1klady na vlastnictv\u00ed.<\/p>\n<p>Uk\u00e1zalo se, \u017ee vybudovat efektivn\u00ed, rychl\u00e9 a levn\u00e9 DataLake pro pot\u0159eby zcela r\u016fzn\u00fdch odd\u011blen\u00ed spole\u010dnosti je \u00fapln\u011b v sil\u00e1ch i zku\u0161en\u00fdch v\u00fdvoj\u00e1\u0159\u016f, kte\u0159\u00ed nikdy nebyli architekty, neum\u00ed kreslit \u010dtverce na \u010dtverc\u00edch se \u0161ipkami a znaj\u00ed 50 term\u00edn\u016f z ekosyst\u00e9mu Hadoop.<\/p>\n<p>Na za\u010d\u00e1tku cesty m\u011b bolela hlava z mnoha divok\u00fdch zoologick\u00fdch zahrad otev\u0159en\u00e9ho a uzav\u0159en\u00e9ho softwaru a z pocitu odpov\u011bdnosti v\u016f\u010di potomk\u016fm. Prost\u011b za\u010dn\u011bte budovat sv\u016fj DataLake od jednoduch\u00fdch n\u00e1stroj\u016f: nagios\/munin -&gt; elastic\/kibana -&gt; Hadoop\/Spark\/s3 ..., sb\u00edrejte zp\u011btnou vazbu a hluboce pochopte fyziku prob\u00edhaj\u00edc\u00edch proces\u016f. V\u0161e slo\u017eit\u00e9 a nejasn\u00e9 d\u00e1vejte nep\u0159\u00e1tel\u016fm a konkurent\u016fm.<\/p>\n<p>Pokud nechcete do cloudu a r\u00e1di udr\u017eujete, aktualizujete a patchujete otev\u0159en\u00e9 projekty, lze vybudovat podobnou na\u0161i sch\u00e9ma lok\u00e1ln\u011b, na kancel\u00e1\u0159sk\u00fdch levn\u00fdch stroj\u00edch s Hadoop a Presto navrch. Hlavn\u00ed je nezastavovat se, j\u00edt d\u00e1l, po\u010d\u00edtat, hledat jednoduch\u00e1 a jasn\u00e1 \u0159e\u0161en\u00ed a ur\u010dit\u011b to vyjde! Hodn\u011b \u0161t\u011bst\u00ed v\u0161em a na shledanou!<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/bitrix\/blog\/516374\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432 \u043a\u043e\u043c\u043c\u0435\u0440\u0447\u0435\u0441\u043a\u0443\u044e \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044e. \u0410 \u043a\u043e\u0433\u0434\u0430 \u043d\u0443\u0436\u043d\u043e \u0431\u0443\u0434\u0435\u0442 \u043e\u0431\u043d\u043e\u0432\u043b\u044f\u0442\u044c\u0441\u044f\/\u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c\u0441\u044f \u0438\u043b\u0438 \u043a\u0442\u043e-\u0442\u043e \u0441\u043b\u0443\u0447\u0430\u0439\u043d\u043e \u043f\u0435\u0440\u0435\u0437\u0430\u0433\u0440\u0443\u0437\u0438\u0442 \u043f\u0430\u0440\u0443 \u043c\u0430\u0448\u0438\u043d \u2014 \u043e\u0441\u043e\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043d\u0430\u0447\u0430\u043b\u0441\u044f \u043a\u0430\u043a\u043e\u0439-\u0442\u043e \u043d\u0430\u0432\u044f\u0437\u0447\u0438\u0432\u044b\u0439 \u0434\u0443\u0440\u043d\u043e\u0439 \u0441\u043e\u043d \u043d\u0430\u044f\u0432\u0443, \u0432\u0441\u0435 \u0440\u0435\u0437\u043a\u043e \u0443\u0441\u043b\u043e\u0436\u043d\u0438\u043b\u043e\u0441\u044c \u0434\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":92509,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-92508","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Jak jsme organizovali vysoce efektivn\u00ed a levn\u00e9 DataLake a pro\u010d pr\u00e1v\u011b tak | ProHoster","description":"\u017dijeme v \u00fa\u017easn\u00e9 dob\u011b, kdy je mo\u017en\u00e9 rychle a jednodu\u0161e propojit n\u011bkolik hotov\u00fdch otev\u0159en\u00fdch n\u00e1stroj\u016f, nastavit je s \"vypnut\u00fdm v\u011bdom\u00edm\" podle rad stackoverflow, ani\u017e bychom se pono\u0159ili do \"mnoha p\u00edsmen\", a spustit je.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster","og:description":"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-28T05:42:10+00:00","article:modified_time":"2020-08-28T05:42:10+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"92508","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:07:39","updated":"2022-10-01 09:50:53","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/92508","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=92508"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/92508\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/92509"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=92508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=92508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=92508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}