{"id":55701,"date":"2020-01-26T00:00:00","date_gmt":"2020-01-25T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh"},"modified":"2020-02-18T14:03:50","modified_gmt":"2020-02-18T11:03:50","slug":"nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","title":{"rendered":"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Toto je p\u0159eklad m\u00e9ho \u010dl\u00e1nku na Medium \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/rock-your-data\/getting-started-with-data-lake-4bb13643f9\">\u00davod do datov\u00e9ho jezera<\/a><\/noindex>, kter\u00fd se uk\u00e1zal jako docela popul\u00e1rn\u00ed, pravd\u011bpodobn\u011b d\u00edky sv\u00e9 jednoduchosti. Proto jsem se rozhodl napsat ho v ru\u0161tin\u011b a trochu doplnit, aby oby\u010dejn\u00e9mu \u010dlov\u011bku, kter\u00fd nen\u00ed specialistou na data, bylo jasn\u00e9, co je to datov\u00e9 skladi\u0161t\u011b (DW), co je to datov\u00e9 jezero (Data Lake) a jak spolu funguj\u00ed. <\/p>\n<p>Pro\u010d jsem cht\u011bl ps\u00e1t o datov\u00e9m jeze\u0159e? Pracuji s daty a analytikou v\u00edce ne\u017e 10 let, a nyn\u00ed pracuji s velk\u00fdmi daty v Amazon Alexa AI v Cambridge, kter\u00fd je v Bostonu, p\u0159i\u010dem\u017e s\u00e1m \u017eiji ve Victorii na Vancouveru a \u010dasto jsem jak v Bostonu, tak v Seattlu a Vancouveru, a n\u011bkdy dokonce i v Moskv\u011b vystupuji na konferenc\u00edch. Tak\u00e9 ob\u010das p\u00ed\u0161u, ale v\u011bt\u0161inou p\u00ed\u0161u v angli\u010dtin\u011b a napsal jsem u\u017e <noindex><a rel=\"nofollow\" href=\"https:\/\/www.amazon.com\/Dmitry-Anoshin\/e\/B01A5PVT2M\">n\u011bkolik knih<\/a><\/noindex>, tak\u00e9 m\u00e1m pot\u0159ebu sd\u00edlet trendyd analytiky ze Severn\u00ed Ameriky, a ob\u010das p\u00ed\u0161u na <noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/rockyourdata\">Telegram<\/a><\/noindex>.<\/p>\n<p>V\u017edy jsem pracoval s datov\u00fdmi skladi\u0161ti, a od roku 2015 jsem se intenzivn\u011b v\u011bnoval Amazon Web Services, a v\u016fbec jsem p\u0159e\u0161el na cloudovou analytiku (AWS, Azure, GCP). Sleduji v\u00fdvoj analytick\u00fdch \u0159e\u0161en\u00ed od roku 2007 a s\u00e1m jsem dokonce pracoval pro dodavatele datov\u00fdch skladi\u0161\u0165 Teradata a implementoval je ve Sberbank, tehdy se objevila Big Data s Hadoop. V\u0161ichni za\u010dali \u0159\u00edkat, \u017ee \u00e9ra datov\u00fdch skladi\u0161\u0165 skon\u010dila a te\u010f je v\u0161echno na Hadoopu, a pak se za\u010dalo mluvit o datov\u00e9m jeze\u0159e, op\u011bt, \u017ee nyn\u00ed datov\u00e9mu skladi\u0161ti ur\u010dit\u011b p\u0159i\u0161el konec. Ale na\u0161t\u011bst\u00ed (mo\u017en\u00e1 pro n\u011bkoho bohu\u017eel, kdo vyd\u011bl\u00e1val spoustu pen\u011bz na nastaven\u00ed Hadoopu) datov\u00e9 skladi\u0161t\u011b nezmizelo. <br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nV tomto \u010dl\u00e1nku se pod\u00edv\u00e1me na to, co je datov\u00e9 jezero. \u010cl\u00e1nek je ur\u010den lidem, kte\u0159\u00ed maj\u00ed m\u00e1lo zku\u0161enost\u00ed s datov\u00fdmi skladi\u0161ti nebo je nemaj\u00ed v\u016fbec.<\/p>\n<p><img decoding=\"async\" alt=\"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?\" src=\"\/wp-content\/uploads\/2020\/01\/85b3eda809ce19fc33efb2fad4e93a41.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa obr\u00e1zku je jezero Bled, co\u017e je jedno z m\u00fdch obl\u00edben\u00fdch jezer, i kdy\u017e jsem tam byl jen jednou, ale pamatuji si ho na cel\u00fd \u017eivot. Ale budeme hovo\u0159it o jin\u00e9m typu jezera \u2014 datov\u00e9m jeze\u0159e. Mo\u017en\u00e1 u\u017e jste mnohokr\u00e1t sly\u0161eli tento term\u00edn, ale dal\u0161\u00ed definice nikdy neu\u0161kod\u00ed.<\/p>\n<p>Nejprve zde jsou nejpopul\u00e1rn\u011bj\u0161\u00ed definice datov\u00e9ho jezera:<\/p>\n<blockquote><p>\"souborov\u00e9 \u00falo\u017ei\u0161t\u011b v\u0161ech typ\u016f surov\u00fdch dat, kter\u00e1 jsou dostupn\u00e1 k anal\u00fdze k\u00fdmkoli v organizaci\" \u2014 Martin Fowler.<\/p><\/blockquote>\n<blockquote><p>\u201ePokud si mysl\u00edte, \u017ee datov\u00e1 vitr\u00edna je jako l\u00e1hev vody \u2013 vy\u010di\u0161t\u011bn\u00e9, zabaleno a p\u0159ipraven\u00e9 k pohodln\u00e9 konzumaci, pak na\u0161e datov\u00e9 jezero je obrovsk\u00e1 n\u00e1dr\u017e s vodou v jej\u00edm p\u0159irozen\u00e9m stavu. U\u017eivatel\u00e9 si mohou nabrat vodu pro sebe, pono\u0159it se do hloubky, zkoumat\u201c \u2013 James Dixon. <\/p><\/blockquote>\n<p> Nyn\u00ed u\u017e v\u00edme, \u017ee datov\u00e9 jezero se t\u00fdk\u00e1 analytiky, umo\u017e\u0148uje n\u00e1m uchov\u00e1vat velk\u00e9 objemy dat v jejich p\u016fvodn\u00ed podob\u011b a m\u00e1me pot\u0159ebn\u00fd a pohodln\u00fd p\u0159\u00edstup k t\u011bmto dat\u016fm.<\/p>\n<p>\u010casto r\u00e1d zjednodu\u0161uji v\u011bci, pokud mohu slo\u017eit\u00fd term\u00edn vysv\u011btlit jednoduch\u00fdmi slovy, znamen\u00e1 to, \u017ee jsem pochopil, jak to funguje a k \u010demu je to pot\u0159eba. Nap\u0159\u00edklad, jak jsem si prohl\u00ed\u017eel fotografie na iPhonu, do\u0161lo mi, \u017ee to je opravdov\u00e9 datov\u00e9 jezero, dokonce jsem vytvo\u0159il sn\u00edmek pro konference:<\/p>\n<p><img decoding=\"async\" alt=\"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?\" src=\"\/wp-content\/uploads\/2020\/01\/1956ddf4091ca00f6d86c33a6780d495.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u0161echno je velmi jednoduch\u00e9. Ud\u011bl\u00e1me fotografii telefonem, fotografie se ulo\u017e\u00ed do telefonu a m\u016f\u017ee b\u00fdt ulo\u017eena v iCloudu (\u00falo\u017en\u00e9 m\u00edsto v cloudu). Tak\u00e9 telefon sb\u00edr\u00e1 metadata fotografie: co je na obr\u00e1zku, geolokace, \u010das. V\u00fdsledkem je, \u017ee m\u016f\u017eeme pou\u017e\u00edt pohodln\u00e9 rozhran\u00ed iPhonu k nalezen\u00ed na\u0161\u00ed fotografie a dokonce vid\u00edme ukazatele, nap\u0159\u00edklad, kdy\u017e hled\u00e1m fotografie se slovem ohe\u0148 (fire), najdu 3 fotografie s obr\u00e1zkem ohni\u0161t\u011b. Pro m\u011b je to jako n\u00e1stroj Business Intelligence, kter\u00fd funguje velmi rychle a p\u0159esn\u011b. <\/p>\n<p>A samoz\u0159ejm\u011b, nesm\u00edme zapom\u00ednat na bezpe\u010dnost (autentizaci a autorizaci), jinak se na\u0161e data mohou snadno dostat do ve\u0159ejn\u00e9ho p\u0159\u00edstupu. Existuje mnoho zpr\u00e1v o velk\u00fdch korporac\u00edch a startupech, jejich\u017e data se dostala do ve\u0159ejn\u00e9ho p\u0159\u00edstupu kv\u016fli nedbalosti v\u00fdvoj\u00e1\u0159\u016f a nedodr\u017eov\u00e1n\u00ed jednoduch\u00fdch pravidel.<\/p>\n<p>Dokonce i takov\u00fd jednoduch\u00fd obr\u00e1zek n\u00e1m pom\u00e1h\u00e1 p\u0159edstavit si, co je to datov\u00e9 jezero, jeho rozd\u00edly od tradi\u010dn\u00edho datov\u00e9ho \u00falo\u017ei\u0161t\u011b a jeho hlavn\u00ed prvky:<\/p>\n<ol>\n<li><b>Na\u010d\u00edt\u00e1n\u00ed dat <\/b>(Ingestion) \u2014 kl\u00ed\u010dov\u00e1 slo\u017eka datov\u00e9ho jezera. Data mohou vstupovat do datov\u00e9ho \u00falo\u017ei\u0161t\u011b dv\u011bma zp\u016fsoby \u2014 batch (nahr\u00e1v\u00e1n\u00ed s intervaly) a streaming (p\u0159enos dat).<\/li>\n<li><b>\u00dalo\u017en\u00fd prostor pro soubory<\/b> (Storage) \u2014 hlavn\u00ed slo\u017eka Datov\u00e9ho Jezera. Mus\u00edme m\u00edt \u00falo\u017ei\u0161t\u011b, kter\u00e9 je snadno \u0161k\u00e1lovateln\u00e9, mimo\u0159\u00e1dn\u011b spolehliv\u00e9 a m\u00e1 n\u00edzk\u00e9 n\u00e1klady. Nap\u0159\u00edklad v AWS je to S3.<\/li>\n<li><b>Katalog a Vyhled\u00e1v\u00e1n\u00ed<\/b> (Katalog a vyhled\u00e1v\u00e1n\u00ed) \u2014 abychom se vyhnuli Datov\u00fdm ba\u017ein\u00e1m (co\u017e je situace, kdy shrom\u00e1\u017ed\u00edme v\u0161echna data do jedn\u00e9 hromady a n\u00e1sledn\u011b je nelze efektivn\u011b zpracov\u00e1vat), je nezbytn\u00e9 vytvo\u0159it vrstvu metadat pro klasifikaci dat, aby u\u017eivatel\u00e9 snadno na\u0161li data, kter\u00e1 pot\u0159ebuj\u00ed pro anal\u00fdzu. D\u00e1le je mo\u017en\u00e9 pou\u017e\u00edt dal\u0161\u00ed hledac\u00ed \u0159e\u0161en\u00ed, nap\u0159\u00edklad ElasticSearch. Hled\u00e1n\u00ed pom\u00e1h\u00e1 u\u017eivateli vyhled\u00e1vat pot\u0159ebn\u00e1 data prost\u0159ednictv\u00edm u\u017eivatelsky p\u0159\u00edv\u011btiv\u00e9ho rozhran\u00ed.<\/li>\n<li><b>Zpracov\u00e1n\u00ed<\/b> (Proces) \u2014 tento krok se zab\u00fdv\u00e1 zpracov\u00e1n\u00edm a transformac\u00ed dat. M\u016f\u017eeme transformovat data, m\u011bnit jejich struktury, \u010distit a mnoho dal\u0161\u00edho. <\/li>\n<li><b>Bezpe\u010dnost<\/b> (Bezpe\u010dnost) \u2014 je d\u016fle\u017eit\u00e9 investovat \u010das do n\u00e1vrhu bezpe\u010dnosti \u0159e\u0161en\u00ed. Nap\u0159\u00edklad \u0161ifrov\u00e1n\u00ed dat b\u011bhem jejich uchov\u00e1v\u00e1n\u00ed, zpracov\u00e1n\u00ed a p\u0159enosu. Je d\u016fle\u017eit\u00e9 pou\u017e\u00edvat metody autentizace a autorizace. Na z\u00e1v\u011br pot\u0159ebujeme n\u00e1stroj pro audit.<\/li>\n<\/ol>\n<p>\nZ praktick\u00e9ho hlediska m\u016f\u017eeme charakterizovat datov\u00e9 jezero t\u0159emi atributy:<\/p>\n<ol>\n<li><b>Shroma\u017e\u010fujte a ukl\u00e1dejte cokoliv<\/b> \u2014 datov\u00e9 jezero obsahuje v\u0161echna data, jak surov\u00e1 nepropracovan\u00e1 data za jak\u00e9koliv \u010dasov\u00e9 obdob\u00ed, tak i zpracovan\u00e1\/\u010dist\u00e1 data.<\/li>\n<li><b>Hlubok\u00e1 anal\u00fdza<\/b> \u2014 datov\u00e9 jezero umo\u017e\u0148uje u\u017eivatel\u016fm zkoumat a analyzovat data.<\/li>\n<li><b>Flexibiln\u00ed p\u0159\u00edstup<\/b> \u2014 datov\u00e9 jezero poskytuje flexibiln\u00ed p\u0159\u00edstup k r\u016fzn\u00fdm dat\u016fm a r\u016fzn\u00fdm sc\u00e9n\u00e1\u0159\u016fm.<\/li>\n<\/ol>\n<p>\nNyn\u00ed se m\u016f\u017eeme pod\u00edvat na rozd\u00edl mezi \u00falo\u017ei\u0161t\u011bm dat a datov\u00fdm jezerem. Lid\u00e9 se obvykle ptaj\u00ed:<\/p>\n<ul>\n<li>A co \u00falo\u017ei\u0161t\u011b dat?<\/li>\n<li>Nahrazujeme \u00falo\u017ei\u0161t\u011b dat datov\u00fdm jezerem, nebo ho roz\u0161i\u0159ujeme?<\/li>\n<li>M\u016f\u017eeme se vlastn\u011b obej\u00edt bez datov\u00e9ho jezera?<\/li>\n<\/ul>\n<p>\nStru\u010dn\u011b \u0159e\u010deno, jasn\u00e1 odpov\u011b\u010f neexistuje. V\u0161e z\u00e1vis\u00ed na konkr\u00e9tn\u00ed situaci, dovednostech v t\u00fdmu a rozpo\u010dtu. Nap\u0159\u00edklad migrace \u00falo\u017ei\u0161t\u011b dat na Oracle do AWS a vytvo\u0159en\u00ed datov\u00e9ho jezera dce\u0159inou spole\u010dnost\u00ed Amazon \u2014 Woot \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/blogs\/big-data\/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws\/\">N\u00e1\u0161 p\u0159\u00edb\u011bh o datov\u00e9m jeze\u0159e: Jak Woot.com postavil serverless datov\u00e9 jezero na AWS<\/a><\/noindex>. <\/p>\n<p>Na druhou stranu dodavatel Snowflake tvrd\u00ed, \u017ee u\u017e nemus\u00edte p\u0159em\u00fd\u0161let o datov\u00e9m jeze\u0159e, proto\u017ee jejich datov\u00e1 platforma (do roku 2020 to bylo \u00falo\u017ei\u0161t\u011b dat) v\u00e1m umo\u017e\u0148uje kombinovat datov\u00e9 jezero a \u00falo\u017ei\u0161t\u011b dat. Pracoval jsem trochu se Snowflake, a to je skute\u010dn\u011b jedine\u010dn\u00fd produkt, kter\u00fd to dok\u00e1\u017ee. Cena je jin\u00e1 ot\u00e1zka.<\/p>\n<p>Na z\u00e1v\u011br, moje osobn\u00ed n\u00e1zory jsou, \u017ee st\u00e1le pot\u0159ebujeme datov\u00e9 sklad jako hlavn\u00ed zdroj dat pro na\u0161e reportov\u00e1n\u00ed, a v\u0161e, co se nevejde, uchov\u00e1v\u00e1me v datov\u00e9m jeze\u0159e. Cel\u00e1 role analytiky spo\u010d\u00edv\u00e1 v poskytov\u00e1n\u00ed snadn\u00e9ho p\u0159\u00edstupu podnik\u016fm pro rozhodov\u00e1n\u00ed. A\u0165 se n\u00e1m to l\u00edb\u00ed nebo ne, podnikov\u00ed u\u017eivatel\u00e9 pracuj\u00ed efektivn\u011bji s datov\u00fdm skladem ne\u017e s datov\u00fdm jezerem, nap\u0159\u00edklad v Amazonu \u2014 je tu Redshift (analytick\u00e9 datov\u00e9 sklad) a je tu Redshift Spectrum\/Athena (SQL rozhran\u00ed pro datov\u00e9 jezero v S3 na b\u00e1zi Hive\/Presto). Tot\u00e9\u017e plat\u00ed pro dal\u0161\u00ed modern\u00ed analytick\u00e9 datov\u00e9 sklady.<\/p>\n<p>Pod\u00edvejme se na typickou architekturu datov\u00e9ho skladu:<\/p>\n<p><img decoding=\"async\" alt=\"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?\" src=\"\/wp-content\/uploads\/2020\/01\/a2015f7f5e28e8a671699682105e011e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTo je klasick\u00e9 \u0159e\u0161en\u00ed. M\u00e1me zdrojov\u00e9 syst\u00e9my, pomoc\u00ed ETL\/ELT kop\u00edrujeme data do analytick\u00e9ho datov\u00e9ho skladu a p\u0159ipojujeme k \u0159e\u0161en\u00ed Business Intelligence (moje obl\u00edben\u00e9 Tableau, a va\u0161e?). <\/p>\n<p>Takov\u00e9 \u0159e\u0161en\u00ed m\u00e1 n\u00e1sleduj\u00edc\u00ed nev\u00fdhody:<\/p>\n<ul>\n<li>ETL\/ELT operace vy\u017eaduj\u00ed \u010das a zdroje.<\/li>\n<li>Obvykle je pam\u011b\u0165 pro uchov\u00e1v\u00e1n\u00ed dat v analytick\u00e9m datov\u00e9m skladu drah\u00e1 (nap\u0159\u00edklad Redshift, BigQuery, Teradata), proto\u017ee mus\u00edme zakoupit cel\u00e9 klastr.<\/li>\n<li>Podnikov\u00ed u\u017eivatel\u00e9 maj\u00ed p\u0159\u00edstup k vy\u010di\u0161t\u011bn\u00fdm a \u010dasto agregovan\u00fdm dat\u016fm a nemaj\u00ed mo\u017enost z\u00edskat syrov\u00e1 data.<\/li>\n<\/ul>\n<p>\nSamoz\u0159ejm\u011b, v\u0161e z\u00e1vis\u00ed na va\u0161em p\u0159\u00edpad\u011b. Pokud nem\u00e1te probl\u00e9my s va\u0161\u00edm datov\u00fdm skladem, nen\u00ed v\u00e1m datov\u00e9 jezero v\u016fbec pot\u0159eba. Ale kdy\u017e se objev\u00ed probl\u00e9my s nedostatkem m\u00edsta, v\u00fdkonem nebo cena hraje kl\u00ed\u010dovou roli, m\u016f\u017eete zv\u00e1\u017eit variantu datov\u00e9ho jezera. Pr\u00e1v\u011b proto je datov\u00e9 jezero velmi popul\u00e1rn\u00ed. Zde je p\u0159\u00edklad architektury datov\u00e9ho jezera:<br \/>\n<img decoding=\"async\" alt=\"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?\" src=\"\/wp-content\/uploads\/2020\/01\/c952e2202a6ce2c8d7d91215aa5390cc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nPou\u017eit\u00edm p\u0159\u00edstupu datov\u00e9ho jezera nahr\u00e1v\u00e1me syrov\u00e1 data do na\u0161eho datov\u00e9ho jezera (batch nebo streaming), d\u00e1le zpracov\u00e1v\u00e1me data podle pot\u0159eby. Datov\u00e9 jezero umo\u017e\u0148uje podnikov\u00fdm u\u017eivatel\u016fm vytv\u00e1\u0159et vlastn\u00ed datov\u00e9 transformace (ETL\/ELT) nebo analyzovat data v \u0159e\u0161en\u00edch Business Intelligence (pokud je k dispozici po\u017eadovan\u00fd ovlada\u010d).<\/p>\n<blockquote><p>C\u00edlem jak\u00e9hokoli analytick\u00e9ho \u0159e\u0161en\u00ed je slou\u017eit podnikov\u00fdm u\u017eivatel\u016fm. Proto bychom m\u011bli v\u017edy pracovat na z\u00e1klad\u011b po\u017eadavk\u016f podniku. (V Amazonu to je jeden z princip\u016f \u2014 working backwards).<\/p><\/blockquote>\n<p> Prac\u00ed jak s datov\u00fdm skladem, tak s datov\u00fdm jezerem m\u016f\u017eeme porovnat ob\u011b \u0159e\u0161en\u00ed:<\/p>\n<p><img decoding=\"async\" alt=\"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?\" src=\"\/wp-content\/uploads\/2020\/01\/07b62ac9838438a0b28caf1b22b2dccd.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nHlavn\u00ed z\u00e1v\u011br, kter\u00fd lze u\u010dinit, je, \u017ee datov\u00e9 sklady nesout\u011b\u017e\u00ed s datov\u00fdmi jezery, ale sp\u00ed\u0161e je dopl\u0148uj\u00ed. Je to v\u0161ak na v\u00e1s, co se hod\u00ed pro va\u0161i situaci. V\u017edy je zaj\u00edmav\u00e9 to zkusit sami a u\u010dinit spr\u00e1vn\u00e9 z\u00e1v\u011bry.<\/p>\n<p>Cht\u011bl bych tak\u00e9 vypr\u00e1v\u011bt o jednom z p\u0159\u00edpad\u016f, kdy jsem za\u010dal pou\u017e\u00edvat p\u0159\u00edstup datov\u00e9ho jezera. V\u0161echno je to docela ban\u00e1ln\u00ed, sna\u017eil jsem se pou\u017e\u00edt n\u00e1stroj ELT (m\u011bli jsme Matillion ETL) a Amazon Redshift, m\u00e9 \u0159e\u0161en\u00ed fungovalo, ale neodpov\u00eddalo po\u017eadavk\u016fm.<\/p>\n<p>Pot\u0159eboval jsem vz\u00edt webov\u00e9 protokoly, transformovat je a agregovat, abych poskytl data pro dva p\u0159\u00edpady:<\/p>\n<ol>\n<li>Marketingov\u00fd t\u00fdm cht\u011bl analyzovat aktivitu bot\u016f pro SEO<\/li>\n<li>IT cht\u011blo sledovat metriky provozu webov\u00fdch str\u00e1nek<\/li>\n<\/ol>\n<p>\nVelmi jednoduch\u00e9, opravdu jednoduch\u00e9 protokoly. Zde je p\u0159\u00edklad:<\/p>\n<pre><code class=\"plaintext\">https 2018-07-02T22:23:00.186641Z app\/my-loadbalancer\/50dc6c495c0c9188 \n192.168.131.39:2817 10.0.0.1:80 0.086 0.048 0.037 200 200 0 57 \n\"GET https:\/\/www.example.com:443\/ HTTP\/1.1\" \"curl\/7.46.0\" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 \narn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup\/my-targets\/73e2d6bc24d8a067\n\"Root=1-58337281-1d84f3d73c47ec4e58577259\" \"www.example.com\" \"arn:aws:acm:us-east-2:123456789012:certificate\/12345678-1234-1234-1234-123456789012\"\n1 2018-07-02T22:22:48.364000Z \"authenticate,forward\" \"-\" \"-\"<\/code><\/pre>\n<p>\nJeden soubor v\u00e1\u017eil 1-4 megabajty.<\/p>\n<p>Ale byla tu jedna obt\u00ed\u017enost. M\u011bli jsme 7 dom\u00e9n po cel\u00e9m sv\u011bt\u011b a za jeden den se vytv\u00e1\u0159elo 7000 soubor\u016f. To nen\u00ed a\u017e tak velk\u00fd objem, celkem 50 gigabajt\u016f. Ale velikost na\u0161eho klastru Redshift byla tak\u00e9 mal\u00e1 (4 uzly). Na\u010d\u00edt\u00e1n\u00ed tradi\u010dn\u00edm zp\u016fsobem jednoho souboru trvalo p\u0159ibli\u017en\u011b minutu. To znamen\u00e1, \u017ee probl\u00e9m nebylo mo\u017en\u00e9 vy\u0159e\u0161it p\u0159\u00edmo. A to byl ten p\u0159\u00edpad, kdy jsem se rozhodl pou\u017e\u00edt p\u0159\u00edstup datov\u00e9ho jezera. \u0158e\u0161en\u00ed vypadalo p\u0159ibli\u017en\u011b takto:<\/p>\n<p><img decoding=\"async\" alt=\"Pot\u0159ebujeme datov\u00e9 jezero? A co s datov\u00fdm skladi\u0161t\u011bm?\" src=\"\/wp-content\/uploads\/2020\/01\/21ca33e82da56f83b3deab4c32eb2345.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe to dost jednoduch\u00e9 (chci poznamenat, \u017ee v\u00fdhodou pr\u00e1ce v cloudu je jednoduchost). Pou\u017eil jsem:<\/p>\n<ul>\n<li>AWS Elastic Map Reduce (Hadoop) jako v\u00fdpo\u010detn\u00ed v\u00fdkon <\/li>\n<li>AWS S3 jako \u00falo\u017ei\u0161t\u011b soubor\u016f s mo\u017enost\u00ed \u0161ifrov\u00e1n\u00ed dat a rozli\u0161ov\u00e1n\u00ed p\u0159\u00edstupu<\/li>\n<li>Spark jako InMemory v\u00fdpo\u010detn\u00ed v\u00fdkon a PySpark pro logiku a transformaci dat<\/li>\n<li>Parquet jako v\u00fdsledek pr\u00e1ce Spark<\/li>\n<li>AWS Glue Crawler jako sb\u011bra\u010d metadat o nov\u00fdch datech a parti\u00edch<\/li>\n<li>Redshift Spectrum jako SQL rozhran\u00ed k datov\u00e9mu jezeru pro st\u00e1vaj\u00edc\u00ed u\u017eivatele Redshift<\/li>\n<\/ul>\n<p>\nNejmen\u0161\u00ed klastr EMR+Spark zpracoval v\u0161echny d\u00e1vky soubor\u016f za 30 minut. Existuj\u00ed i dal\u0161\u00ed p\u0159\u00edpady pro AWS, obzvl\u00e1\u0161t\u011b hodn\u011b souvisej\u00edc\u00edch s Alexou, kde je dat velmi mnoho. <\/p>\n<p>Ned\u00e1vno jsem se dozv\u011bd\u011bl o jedn\u00e9 nev\u00fdhod\u011b datov\u00e9ho jezera \u2013 t\u00fdk\u00e1 se to GDPR. Probl\u00e9m nast\u00e1v\u00e1, kdy\u017e klient po\u017e\u00e1d\u00e1 o jejich odstran\u011bn\u00ed a data se nach\u00e1z\u00ed v jednom z soubor\u016f; nem\u016f\u017eeme pou\u017e\u00edt jazyk pro manipulaci s daty a operaci DELETE, jako to d\u011bl\u00e1me v datab\u00e1zi.<\/p>\n<p>Douf\u00e1m, \u017ee \u010dl\u00e1nek objasnil rozd\u00edl mezi datov\u00fdm skladem a datov\u00fdm jezerem. Pokud v\u00e1s to zaujalo, mohu p\u0159elo\u017eit dal\u0161\u00ed sv\u00e9 \u010dl\u00e1nky nebo \u010dl\u00e1nky profesion\u00e1l\u016f, kter\u00e9 \u010dtu. Tak\u00e9 mohu p\u0159ibl\u00ed\u017eit \u0159e\u0161en\u00ed, se kter\u00fdmi pracuji, a jejich architekturu.<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/485180\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043e\u043a\u0430\u0437\u0430\u043b\u0430\u0441\u044c \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u043e\u0439, \u043d\u0430\u0432\u0435\u0440\u043d\u043e\u0435 \u0438\u0437-\u0437\u0430 \u0441\u0432\u043e\u0435\u0439 \u043f\u0440\u043e\u0441\u0442\u043e\u0442\u044b. \u041f\u043e\u044d\u0442\u043e\u043c\u0443 \u044f \u0440\u0435\u0448\u0438\u043b \u043d\u0430\u043f\u0438\u0441\u0430\u0442\u044c \u0435\u0435 \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u043e\u043c \u044f\u0437\u044b\u043a\u0435 \u0438 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u044c, \u0447\u0442\u043e\u0431\u044b \u043f\u0440\u043e\u0441\u0442\u043e\u043c\u0443 \u0447\u0435\u043b\u043e\u0432\u0435\u043a\u0443, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043d\u0435 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u0438\u0441\u0442\u043e\u043c \u043f\u043e \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0441\u0442\u0430\u043b\u043e \u043f\u043e\u043d\u044f\u0442\u043d\u043e, \u0447\u0442\u043e \u0442\u0430\u043a\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 (DW), \u0430 \u0447\u0442\u043e \u0442\u0430\u043a\u043e\u0435 \u043e\u0437\u0435\u0440\u043e \u0434\u0430\u043d\u043d\u044b\u0445 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-55701","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041d\u0443\u0436\u043d\u043e \u043b\u0438 \u043d\u0430\u043c \u043e\u0437\u0435\u0440\u043e \u0434\u0430\u043d\u043d\u044b\u0445? \u0410 \u0447\u0442\u043e \u0434\u0435\u043b\u0430\u0442\u044c \u0441 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435\u043c \u0434\u0430\u043d\u043d\u044b\u0445? | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-01-25T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:03:50+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Pot\u0159ebujeme datov\u00e9 jezero? A co d\u011blat se datov\u00fdm skladem? | ProHoster","description":"Tento \u010dl\u00e1nek je p\u0159eklad m\u00e9ho \u010dl\u00e1nku na medium \u2013 Jak za\u010d\u00edt s datov\u00fdm jezerem.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041d\u0443\u0436\u043d\u043e \u043b\u0438 \u043d\u0430\u043c \u043e\u0437\u0435\u0440\u043e \u0434\u0430\u043d\u043d\u044b\u0445? \u0410 \u0447\u0442\u043e \u0434\u0435\u043b\u0430\u0442\u044c \u0441 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435\u043c \u0434\u0430\u043d\u043d\u044b\u0445? | ProHoster","og:description":"\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-01-25T21:00:00+00:00","article:modified_time":"2020-02-18T11:03:50+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"55701","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:38:32","updated":"2022-09-27 20:27:01","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/55701","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=55701"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/55701\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=55701"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=55701"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=55701"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}