{"id":92508,"date":"2020-08-28T07:42:10","date_gmt":"2020-08-28T05:42:10","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak"},"modified":"2020-08-28T07:42:10","modified_gmt":"2020-08-28T05:42:10","slug":"kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","title":{"rendered":"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Tr\u0103im \u00eentr-o vreme uimitoare, c\u00e2nd putem conecta rapid \u0219i simplu mai multe instrumente deschise, s\u0103 le configur\u0103m cu un \u201econ\u0219tiin\u021b\u0103 dezactivat\u0103\u201d dup\u0103 sfaturile de pe stackoverflow, f\u0103r\u0103 a ne implica \u00een \u201emulte litere\u201d, \u0219i s\u0103 le lans\u0103m \u00een exploatare comercial\u0103. Iar c\u00e2nd va fi nevoie s\u0103 ne actualiz\u0103m\/extindem sau c\u00e2nd cineva va reporni din gre\u0219eal\u0103 c\u00e2teva ma\u0219ini \u2013 ne vom da seama c\u0103 a \u00eenceput un vis ur\u00e2t obsesiv \u00een realitate, totul s-a complicat brusc p\u00e2n\u0103 la nerecunoa\u0219tere, nu mai exist\u0103 cale de \u00eentoarcere, viitorul este neclar \u0219i mai sigur, \u00een loc de programare, se recomand\u0103 cre\u0219terea albinelor \u0219i fabricarea br\u00e2nzei.<\/p>\n<p>Nu degeaba, colegii mai experimenta\u021bi, cu capetele albe pres\u0103rate de bug-uri, contempl\u00e2nd desf\u0103\u0219urarea incredibil de rapid\u0103 a pachetelor de \u201econtainere\u201d \u00een \u201ecuburi\u201d pe zeci de servere folosind \u201elimbi la mod\u0103\u201d cu suport \u00eencorporat pentru input\/output asincron-neblocant \u2013 z\u00e2mbesc modest. \u0218i continu\u0103 \u00een t\u0103cere s\u0103 r\u0103sfoiasc\u0103 \u201eman ps\u201d, cu ochii s\u00e2ngeri\u021bi din cauza surselor \u201enginx\u201d \u0219i scriu-scriu-scriu teste unitare. Colegii \u0219tiu c\u0103 cele mai interesante lucruri sunt \u00eenainte, c\u00e2nd \u201etoate acestea\u201d vor deveni \u00eentr-o noapte un morman sub bradul de Cr\u0103ciun. \u0218i \u00eei va ajuta doar o \u00een\u021belegere profund\u0103 a naturii unix, tabela st\u0103rii TCP\/IP \u00eenv\u0103\u021bat\u0103 pe de rost \u0219i algoritmii de baz\u0103 pentru sortare\/c\u0103utare. Pentru a readuce sistemul la via\u021b\u0103 sub b\u0103taia clopotelor.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nAh da, m-am cam ab\u0103tut, dar sper c\u0103 am reu\u0219it s\u0103 transmit starea de anticipare.<br \/>\nAst\u0103zi vreau s\u0103 \u00eemp\u0103rt\u0103\u0219esc experien\u021ba noastr\u0103 de desf\u0103\u0219urare a unui stack convenabil \u0219i ieftin pentru DataLake, care rezolv\u0103 majoritatea problemelor analitice din companie pentru diferite structuri organiza\u021bionale.<\/p>\n<p>Cu ceva timp \u00een urm\u0103, am ajuns la concluzia c\u0103 companiile au nevoie din ce \u00een ce mai mult de fructele at\u00e2t ale analizei de produs, c\u00e2t \u0219i ale celei tehnice (s\u0103 nu mai vorbim de cire\u0219ele de pe tort \u00een form\u0103 de machine learning) \u0219i pentru a \u00een\u021belege tendin\u021bele \u0219i riscurile \u2013 este necesar s\u0103 colect\u0103m \u0219i s\u0103 analiz\u0103m din ce \u00een ce mai multe metrici.<\/p>\n<h3>Analiza tehnic\u0103 de baz\u0103 \u00een \u201eBitrix24\u201d<\/h3>\n<p>\nCu c\u00e2\u021biva ani \u00een urm\u0103, odat\u0103 cu lansarea serviciului \u201eBitrix24\u201d, am investit activ timp \u0219i resurse \u00een crearea unei platforme analitice simple \u0219i fiabile, care s\u0103 ne ajute s\u0103 vedem rapid problemele din infrastructur\u0103 \u0219i s\u0103 planific\u0103m urm\u0103torul pas. Desigur, am dorit s\u0103 folosim instrumente gata f\u0103cute, c\u00e2t mai simple \u0219i clare. Ca urmare, am ales Nagios pentru monitorizare \u0219i Munin pentru analiz\u0103 \u0219i vizualizare. Acum avem mii de verific\u0103ri \u00een Nagios, sute de grafice \u00een Munin, iar colegii le folosesc zilnic \u0219i cu succes. Metricile sunt clare, graficele sunt informative, sistemul func\u021bioneaz\u0103 fiabil de c\u00e2\u021biva ani \u0219i se adaug\u0103 regulat noi teste \u0219i grafice: introducem un nou serviciu \u00een exploatare \u2014 ad\u0103ug\u0103m c\u00e2teva teste \u0219i grafice. Drum bun.<\/p>\n<h3>M\u00e2n\u0103 pe puls \u2014 analiz\u0103 tehnic\u0103 extins\u0103<\/h3>\n<p>\nDorin\u021ba de a ob\u021bine informa\u021bii despre probleme \u201ec\u00e2t mai repede posibil\u201d ne-a dus la experimente active cu instrumente simple \u0219i clare \u2014 Pinba \u0219i Xhprof.<\/p>\n<p>Pinba ne trimitea statistici despre viteza de func\u021bionare a p\u0103r\u021bilor paginilor web pe PHP \u00een pachete UDP, iar noi puteam vedea online \u00een baza de date MySQL (Pinba are propriul motor MySQL pentru analiz\u0103 rapid\u0103 a evenimentelor) o list\u0103 scurt\u0103 de probleme \u0219i putea reac\u021biona la acestea. Xhprof permitea, \u00een mod automat, s\u0103 colect\u0103m graficele de execu\u021bie ale celor mai lente pagini PHP ale clien\u021bilor \u0219i s\u0103 analiz\u0103m ce ar fi putut duce la aceasta \u2014 lini\u0219ti\u021bi, savur\u00e2nd ceai sau ceva mai puternic.<\/p>\n<p>Cu ceva timp \u00een urm\u0103, instrumentele au fost completate cu un alt motor destul de simplu \u0219i clar, bazat pe un algoritm de indexare invers\u0103, excelent implementat \u00een celebra bibliotec\u0103 Lucene \u2014 Elastic\/Kibana. Ideea simpl\u0103 de a scrie documente \u00een mod multiprocesat \u00een indexul invers Lucene pe baza evenimentelor din loguri \u0219i c\u0103utarea rapid\u0103 \u00een acestea folosind divizarea pe fa\u021bete s-a dovedit, \u00eentr-adev\u0103r, util\u0103.<\/p>\n<p>\u00cen ciuda aspectului tehnic al vizualiz\u0103rilor din Kibana cu concepte la scar\u0103 mic\u0103 de tip \u201ebucket\u201d \u0219i un limbaj re-inventat care nu a fost deloc uitat din algebra rela\u021bional\u0103 \u2014 instrumentul ne ajut\u0103 bine \u00een urm\u0103toarele sarcini:<\/p>\n<ul>\n<li>C\u00e2te erori PHP a avut clientul Bitrix24 pe portalul p1 \u00een ultima or\u0103 \u0219i de care? S\u0103 \u00een\u021belegem, s\u0103 iert\u0103m \u0219i s\u0103 corect\u0103m rapid.<\/li>\n<li>C\u00e2te apeluri video au avut loc pe portalurile din Germania \u00een ultimele 24 de ore, cu ce calitate \u0219i au existat probleme cu canalul\/re\u021beaua?<\/li>\n<li>C\u00e2t de bine func\u021bioneaz\u0103 func\u021bionalitatea sistemului (extensia noastr\u0103 \u00een C pentru PHP), compilat\u0103 din surse \u00een ultima actualizare a serviciului \u0219i distribuit\u0103 clien\u021bilor? Exist\u0103 segfaults?<\/li>\n<li>Datele clien\u021bilor sunt stocate \u00een memoria PHP? Exist\u0103 erori de dep\u0103\u0219ire a memoriei alocate proceselor: \u201eout of memory\u201d? G\u0103si\u021bi \u0219i remedia\u021bi.<\/li>\n<\/ul>\n<p>\nIat\u0103 un exemplu concret. \u00cen ciuda test\u0103rii riguroase \u0219i multilaterale, clientul a \u00eent\u00e2mpinat o eroare jenant\u0103 \u0219i nea\u0219teptat\u0103 \u00eentr-un caz foarte neobi\u0219nuit, cu date de intrare corupte, alarma a sunat \u0219i a \u00eenceput procesul de corectare rapid\u0103:<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/8a802dba41b5d1a85c0dc41dfbf8b84e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u00cen plus, Kibana permite organizarea de notific\u0103ri pe baza evenimentelor specificate, iar \u00een scurt timp, instrumentul a \u00eenceput s\u0103 fie utilizat de zeci de angaja\u021bi din diferite departamente \u2013 de la suport tehnic \u0219i dezvoltare la QA.<\/p>\n<p>Activitatea oric\u0103rui departament din cadrul companiei a devenit u\u0219or de monitorizat \u0219i m\u0103surat \u2013 \u00een loc de analiza manual\u0103 a jurnalelor pe servere, este suficient s\u0103 configurezi o dat\u0103 parserul de jurnale \u0219i trimiterea acestora \u00een clusterul Elastic, pentru a te bucura, de exemplu, de vizualizarea \u00een dashboard-ul Kibana a num\u0103rului de pisici cu dou\u0103 capete v\u00e2ndute, tip\u0103rite pe imprimanta 3D \u00een ultima lun\u0103 lunar\u0103.<\/p>\n<h3>Analiza de afaceri de baz\u0103<\/h3>\n<p>\nToat\u0103 lumea \u0219tie c\u0103 adesea analiza de afaceri \u00een companii \u00eencepe cu o utilizare extrem de activ\u0103, da, da, Excel. Dar, cel mai important, este s\u0103 nu se \u00eencheie acolo. Se mai adaug\u0103 pu\u021bin combustibil focului cu Google Analytics \u00een cloud \u2013 la bine te obi\u0219nuie\u0219ti rapid.<\/p>\n<p>\u00cen compania noastr\u0103, \u00een continu\u0103 dezvoltare armonioas\u0103, au \u00eenceput s\u0103 apar\u0103 ici \u0219i colo \u201eprofe\u021bi\u201d ai muncii mai intensive cu date mai mari. S-au f\u0103cut frecvent cereri pentru rapoarte mai profunde \u0219i mai complexe, iar cu eforturile colegilor din diferite departamente a fost organizat\u0103 cu ceva timp \u00een urm\u0103 o solu\u021bie simpl\u0103 \u0219i practic\u0103 \u2013 combina\u021bia ClickHouse \u0219i PowerBI.<\/p>\n<p>O perioad\u0103 destul de lung\u0103, aceast\u0103 solu\u021bie flexibil\u0103 a fost de mare ajutor, dar treptat a ap\u0103rut con\u0219tientizarea c\u0103 ClickHouse nu este elastic \u0219i nu poate fi tratat at\u00e2t de brutal.<\/p>\n<p>Este important s\u0103 \u00een\u021belegem bine c\u0103 ClickHouse, la fel ca Druid, Vertica \u0219i Amazon RedShift (care se bazeaz\u0103 pe Postgres), sunt motoare analitice optimizate pentru analize destul de accessible (sumariz\u0103ri, agreg\u0103ri, minim-maxim pe coloane \u0219i c\u00e2teva join-uri), deoarece sunt organizate pentru stocarea eficient\u0103 a coloanelor \u00een tabele rela\u021bionale, spre deosebire de binecunoscutul MySQL \u0219i alte baze de date (row-oriented).<\/p>\n<p>Practic, ClickHouse este doar o \"baz\u0103\" de date mai capacitar\u0103, cu o inser\u021bie punctual\u0103 nu foarte convenabil\u0103 (a\u0219a a fost g\u00e2ndit\u0103, totul e \u00een regul\u0103), dar cu o experien\u021b\u0103 analitic\u0103 pl\u0103cut\u0103 \u0219i un set interesant de func\u021bii puternice pentru manipularea datelor. Da, se poate chiar crea un cluster \u2014 dar, \u00een\u021belege\u021bi, s\u0103 dai cu un ciocan nu este chiar corect, a\u0219a c\u0103 am \u00eenceput s\u0103 c\u0103ut\u0103m alte solu\u021bii.<\/p>\n<h3>Cererea pentru Python \u0219i anali\u0219ti<\/h3>\n<p>\n\u00cen compania noastr\u0103 sunt mul\u021bi dezvoltatori care scriu cod aproape \u00een fiecare zi timp de 10-20 de ani \u00een PHP, JavaScript, C#, C\/C++, Java, Go, Rust, Python, Bash. De asemenea, avem mul\u021bi administratori de sistem experimenta\u021bi, care au trecut printr-o adev\u0103rat\u0103 catastrof\u0103 incredibil\u0103, care nu se conforma legilor statistice (de exemplu, c\u00e2nd majoritatea discurilor dintr-un raid-10 sunt distruse de un fulger puternic). \u00cen aceste condi\u021bii, mult timp nu a fost clar ce \u00eenseamn\u0103 \u201eanalist pe Python\u201d. Python este ca PHP, doar c\u0103 numele e pu\u021bin mai lung \u0219i urmele substan\u021belor care altereaz\u0103 con\u0219tiin\u021ba \u00een codul surs\u0103 al interpretului sunt pu\u021bin mai mici. Totu\u0219i, pe m\u0103sur\u0103 ce au fost create tot mai multe rapoarte analitice, dezvoltatorii experimenta\u021bi au \u00eenceput s\u0103 \u00een\u021beleag\u0103 din ce \u00een ce mai profund importan\u021ba specializ\u0103rii \u00eenguste \u00een instrumente precum numpy, pandas, matplotlib, seaborn.<br \/>\nProbabil, rolul decisiv l-au jucat le\u0219inurile bru\u0219te ale angaja\u021bilor la combina\u021bia de cuvinte \u201eregresie logistic\u0103\u201d \u0219i demonstrarea construirii eficiente de rapoarte cu date voluminoase folosind, da, pyspark.<\/p>\n<p>Apache Spark, paradigma sa func\u021bional\u0103, care se potrive\u0219te perfect cu algebra rela\u021bional\u0103 \u0219i capacit\u0103\u021bile sale, au impresionat at\u00e2t de mult dezvoltatorii obi\u0219nui\u021bi cu MySQL, \u00eenc\u00e2t necesitatea \u00eent\u0103ririi r\u00e2ndurilor cu anali\u0219ti experimenta\u021bi a devenit clar\u0103 ca ziua.<\/p>\n<h3>\u00cencerc\u0103rile ulterioare ale Apache Spark\/Hadoop de a decola \u0219i ce nu a decurs conform scenariului<\/h3>\n<p>\nCu toate acestea, a devenit evident c\u0103, \u00een mod clar, cu Spark, ceva nu este \u00een regul\u0103 sau poate ar trebui doar s\u0103 ne sp\u0103l\u0103m mai bine pe m\u00e2ini. Dac\u0103 stiva Hadoop\/MapReduce\/Lucene a fost realizat\u0103 de programatori destul de experimenta\u021bi, ceea ce devine evident dac\u0103 privind cu aten\u021bie codul surs\u0103 Java sau ideile lui Doug Cutting \u00een Lucene, atunci Spark, dintr-o dat\u0103, este scris \u00eentr-un limbaj exotic, foarte contestat din perspectiva practicabilit\u0103\u021bii \u0219i care acum nu se dezvolt\u0103, Scala. Iar c\u0103derile regulate ale calculelor pe clusterul Spark din cauza gestion\u0103rii ilogice \u0219i destul de opace a memoriei pentru opera\u021biile reduce (sunt trimise multe chei simultan) au creat \u00een jurul s\u0103u o aureol\u0103 a ceva ce are unde s\u0103 evolueze. \u00cen plus, situa\u021bia era agravat\u0103 de num\u0103rul mare de porturi deschise ciudate, fi\u0219iere temporare cresc\u00e2nde \u00een cele mai neclare locuri \u0219i miriade de dependen\u021be jar - ceea ce provoca administratorilor de sistem acel sentiment familiar: o ur\u0103 puternic\u0103 (poate c\u0103 trebuia s\u0103 ne sp\u0103l\u0103m pe m\u00e2ini cu s\u0103pun).<\/p>\n<p>\u00cen urma acestui proces, am \u201esuportat\u201d c\u00e2teva proiecte analitice interne, care foloseau activ Apache Spark (inclusiv Spark Streaming, Spark SQL) \u0219i ecosistemul Hadoop (\u0219i multe altele). \u00cen ciuda faptului c\u0103, \u00een timp, am \u00eenv\u0103\u021bat s\u0103 \u201epreg\u0103tim\u201d destul de bine \u201eaceasta\u201d \u0219i s\u0103 monitoriz\u0103m, iar \u201eaceasta\u201d a \u00eencetat practic s\u0103 cad\u0103 din cauza schimb\u0103rii naturii datelor \u0219i dezechilibr\u0103rii hashing-ului uniform RDD, dorin\u021ba de a lua ceva deja gata, actualizat \u0219i administrat undeva \u00een cloud s-a intensificat din ce \u00een ce mai mult. \u00cen acel moment, am \u00eencercat s\u0103 folosim o construc\u021bie cloud gata f\u0103cut\u0103 de Amazon Web Services - <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/emr\/\">EMR<\/a><\/noindex> \u0219i, ulterior, am \u00eencercat s\u0103 rezolv\u0103m problemele pe aceast\u0103 platform\u0103. EMR este varianta Apache Spark preg\u0103tit\u0103 de Amazon cu software suplimentar din ecosistem, asem\u0103n\u0103tor cu construc\u021biile Cloudera\/Hortonworks.<\/p>\n<h3>Un stocare de fi\u0219iere \u201eelastic\u0103\u201d pentru analize - o nevoie urgent\u0103<\/h3>\n<p>\nExperien\u021ba \u201eprepar\u0103rii\u201d Hadoop\/Spark cu arsuri pe diverse p\u0103r\u021bi ale corpului nu a trecut f\u0103r\u0103 urm\u0103ri. A devenit din ce \u00een ce mai evident\u0103 necesitatea de a crea un stocare de fi\u0219iere unic\u0103, ieftin\u0103 \u0219i fiabil\u0103, care s\u0103 fie rezistent\u0103 la defec\u021biuni hardware \u0219i \u00een care s\u0103 poat\u0103 fi stocate fi\u0219iere \u00een diferite formate din diferite sisteme \u0219i s\u0103 se fac\u0103 selec\u021bii eficiente \u0219i realizabile \u00eentr-un timp rezonabil pentru rapoarte.<\/p>\n<p>De asemenea, a\u0219 dori ca actualizarea software-ului acestei platforme s\u0103 nu devin\u0103 un co\u0219mar de noapte de Anul Nou, cu citirea unor stack-uri Java de 20 de pagini \u0219i analiza unor log-uri kilometrice ale func\u021bion\u0103rii clustelui folosind Spark History Server \u0219i o lup\u0103 cu iluminare. A\u0219 fi dorit un instrument simplu \u0219i transparent, care s\u0103 nu necesite o verificare regulat\u0103 sub capot\u0103, \u00een cazul \u00een care o interogare standard MapReduce a dezvoltatorului nu mai func\u021bioneaz\u0103 din cauza unei alegeri nefericite a algoritmului de parti\u021bionare a datelor ini\u021biale.<\/p>\n<h3>Amazon S3 \u2014 candidatul pentru DataLake?<\/h3>\n<p>\nExperien\u021ba cu Hadoop\/MapReduce m-a \u00eenv\u0103\u021bat c\u0103 este nevoie de un sistem de fi\u0219iere fiabil \u0219i scalabil, precum \u0219i de lucr\u0103tori scalabili care \u201evin\u201d mai aproape de date, pentru a nu transfera datele prin re\u021bea. Lucr\u0103torii trebuie s\u0103 fie capabili s\u0103 citeasc\u0103 date \u00een diferite formate, dar, de preferat, s\u0103 nu citeasc\u0103 informa\u021bii suplimentare \u0219i s\u0103 se poat\u0103 stoca datele \u00een formate convenabile pentru lucr\u0103tori.<\/p>\n<p><b>\u00cenc\u0103 o dat\u0103 \u2014 ideea principal\u0103.<\/b> Nu exist\u0103 dorin\u021ba de a \u201e\u00eenc\u0103rca\u201d mari cantit\u0103\u021bi de date \u00eentr-un singur motor analitic de cluster, care oricum va colapsa la un moment dat \u0219i va trebui s\u0103-l \u00eemp\u0103r\u021bim nepl\u0103cut. A\u0219 dori s\u0103 stochez fi\u0219iere, pur \u0219i simplu fi\u0219iere, \u00eentr-un format clar \u0219i s\u0103 efectuez pe ele interog\u0103ri analitice eficiente cu instrumente diferite, dar u\u0219or de \u00een\u021beles. Iar fi\u0219ierele \u00een diferite formate vor continua s\u0103 creasc\u0103. Este mai bine s\u0103 \u00eemp\u0103r\u021bim datele originale, dec\u00e2t motorul. Ne trebuie un DataLake extins \u0219i universal, am decis noi...<\/p>\n<p>Ce-ar fi s\u0103 stoc\u0103m fi\u0219iere \u00een binecunoscuta \u0219i utilizat\u0103 de mul\u021bi solu\u021bie de stocare Cloud scalabil\u0103 Amazon S3, f\u0103r\u0103 s\u0103 ne ocup\u0103m de propriile noastre preparate din Hadoop?<\/p>\n<p>Este clar, datele personale sunt \u201einterzise\u201d, dar ce se \u00eent\u00e2mpl\u0103 cu alte date dac\u0103 le ducem acolo \u0219i le \u201efugim eficient\u201d?<\/p>\n<h3>Ecosistemul analitic de big data clusterizat Amazon Web Services \u2014 foarte pe scurt<\/h3>\n<p>\nJudec\u00e2nd dup\u0103 experien\u021ba noastr\u0103 de lucru cu AWS, Apache Hadoop\/MapReduce este folosit de mult timp \u0219i activ sub diferite forme \u00een serviciul DataPipeline (invidiez colegii, au \u00eenv\u0103\u021bat cum s\u0103-l preg\u0103teasc\u0103 corect). Aici am configurat backup-uri din diferite servicii din tabelele DynamoDB:<br \/>\n<img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/058dc54ed032a7bf3e9e129646202440.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u0218i acestea se efectueaz\u0103 regulat pe clusterele Hadoop\/MapReduce ca un ceas deja de c\u00e2\u021biva ani. \u201eAm configurat \u0219i am uitat:\u201d<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/a6569da8cafdb96c63250bb32bf51704.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDe asemenea, po\u021bi s\u0103 te dedici eficient datascience-ului, ridic\u00e2nd pentru anali\u0219ti c\u0103r\u021bi de lucru Jupiter \u00een cloud \u0219i folosind AWS SageMaker pentru antrenarea \u0219i implementarea modelelor AI. Iat\u0103 cum arat\u0103 la noi:<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/c825d979c9278a8edf8e1e747ef6def8.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u0218i da, po\u021bi s\u0103 ridici o carte de lucru \u00een cloud pentru tine sau pentru analist \u0219i s\u0103 o ata\u0219ezi la un cluster Hadoop\/Spark, s\u0103 efectuezi calcule \u0219i apoi s\u0103 le \"finalizezi\":<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/73cea18c54d2a9ce8d0441463991808b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAdev\u0103rat, este convenabil pentru proiecte analitice individuale \u0219i pentru unele dintre acestea am folosit cu succes serviciul EMR pentru calcule \u0219i analize pe scar\u0103 mare. Dar ce zici de o solu\u021bie sistemic\u0103 pentru DataLake? Va fi posibil? \u00cen acel moment eram la limita speran\u021bei \u0219i disper\u0103rii \u0219i continu\u0103m c\u0103utarea.<\/p>\n<h3>AWS Glue \u2014 un Apache Spark \"la superlativ\"<\/h3>\n<p>\nS-a dovedit c\u0103 AWS are o versiune \u201eproprie\u201d a stivei \u201eHive\/Pig\/Spark\u201d. Rolul lui Hive, adic\u0103 catalogul fi\u0219ierelor \u0219i tipurilor acestora \u00een DataLake, este \u00eendeplinit de serviciul \u201eData catalog\u201d, care nu ascunde compatibilitatea sa cu formatul Apache Hive. La acest serviciu trebuie s\u0103 adaugi informa\u021bii despre unde se afl\u0103 fi\u0219ierele tale \u0219i \u00een ce format sunt. Datele pot fi nu doar \u00een s3, ci \u0219i \u00een bazele de date, dar despre asta nu este acest post. Iat\u0103 cum este organizat catalogul de date DataLake la noi:<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/adb45d09698fdacbf41c86bbadde8bb2.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFi\u0219ierele sunt \u00eenregistrate, excelent. Dac\u0103 fi\u0219ierele s-au actualizat \u2014 pornim manual sau conform unui program crawlers care vor actualiza informa\u021biile din lac \u0219i le vor salva. Mai departe, datele din lac pot fi procesate, iar rezultatele exportate undeva. \u00cen cel mai simplu caz \u2014 export\u0103m de asemenea \u00een s3. Procesarea datelor poate fi realizat\u0103 oriunde, dar se propune s\u0103 configur\u0103m procesul de procesare pe un cluster Apache Spark folosind capabilit\u0103\u021bile avansate prin API AWS Glue. Practic, po\u021bi lua vechiul \u0219i familiarul cod pe python folosind bibliotec\u0103 pyspark \u0219i s\u0103 \u00eel configurezi s\u0103 ruleze pe N noduri ale unui cluster de anumit\u0103 putere cu monitorizare, f\u0103r\u0103 a te ocupa de detaliile Hadoop \u0219i f\u0103r\u0103 a transpune containere docker \u0219i a rezolva conflicte de dependen\u021be.<\/p>\n<p><b>\u00cenc\u0103 o dat\u0103 \u2014 o idee simpl\u0103.<\/b> Nu trebuie s\u0103 configurezi Apache Spark, trebuie doar s\u0103 scrii cod pe python pentru pyspark, s\u0103-l testezi local pe desktop \u0219i apoi s\u0103-l lansezi pe un cluster mare \u00een cloud, indic\u00e2nd unde se afl\u0103 datele de baz\u0103 \u0219i unde s\u0103 pui rezultatul. Uneori este necesar \u0219i util, iar iat\u0103 cum este configurat la noi:<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/dc03181573bb3f5cfcc3a8760bc7e07b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAstfel, dac\u0103 trebuie s\u0103 efectuezi calcule pe un cluster Spark cu date \u00een s3 \u2014 scriem cod pe python\/pyspark, test\u0103m \u0219i plec\u0103m \u00een cloud.<\/p>\n<p>Ce este cu orchestration? \u0218i dac\u0103 o sarcin\u0103 a c\u0103zut \u0219i s-a pierdut? Da, se propune s\u0103 facem un pipeline frumos \u00een stilul Apache Pig \u0219i chiar le-am \u00eencercat, dar am decis s\u0103 folosim, pentru moment, orchestration-ul nostru profund personalizat pe PHP \u0219i JavaScript (\u00een\u021beleg c\u0103 apare un disconfort cognitiv, dar func\u021bioneaz\u0103, de ani de zile \u0219i f\u0103r\u0103 erori).<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/e00ed2047c5c6492e36fccc82e7278a3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Formatul fi\u0219ierelor stocate \u00een lac este cheia performan\u021bei<\/h3>\n<p>\nEste foarte, foarte important s\u0103 \u00een\u021belegem \u00eenc\u0103 dou\u0103 aspecte esen\u021biale. Pentru ca cererile de date din fi\u0219ierele stocate \u00een lac s\u0103 fie executate c\u00e2t mai rapid \u0219i performan\u021ba s\u0103 nu degradeze \u00een urma ad\u0103ug\u0103rii de informa\u021bii noi, este nevoie de:<\/p>\n<ul>\n<li>Coloanele fi\u0219ierelor trebuie p\u0103strate separat (pentru a nu fi necesar s\u0103 se citeasc\u0103 toate liniile pentru a \u00een\u021belege ce este \u00een coloane). Pentru aceasta, am adoptat formatul parquet cu compresie.<\/li>\n<li>Este foarte important s\u0103 shard-uim fi\u0219ierele \u00een foldere precum: limb\u0103, an, lun\u0103, zi, s\u0103pt\u0103m\u00e2n\u0103. Motoarele care \u00een\u021beleg acest tip de shard-uire vor c\u0103uta doar \u00een folderele necesare, f\u0103r\u0103 a analiza toate datele.<\/li>\n<\/ul>\n<p>\nPractic, \u00een acest fel, pune\u021bi datele originale \u00een cea mai eficient\u0103 form\u0103 pentru motoarele analitice de deasupra, care pot accesa selective folderele shard-uite \u0219i citi doar coloanele necesare din fi\u0219iere. Nu trebuie s\u0103 \u201e\u00eenc\u0103rca\u021bi\u201d datele nic\u0103ieri (stocarea se va sparge) \u2014 pur \u0219i simplu pune\u021bi-le imediat \u00een sistemul de fi\u0219iere \u00een formatul corect. Desigur, trebuie s\u0103 fie clar c\u0103 p\u0103strarea unui fi\u0219ier CSV uria\u0219 \u00een DataLake, care trebuie citit r\u00e2nd cu r\u00e2nd de un cluster pentru a extrage coloanele \u2014 nu este foarte eficient. Reflecta\u021bi asupra celor dou\u0103 puncte de mai sus din nou, dac\u0103 \u00eenc\u0103 nu este clar de ce toate acestea.<\/p>\n<h3>AWS Athena \u2014 \u201edemonul\u201d din cutie<\/h3>\n<p>\n\u0218i aici, cre\u00e2nd un lac, am dat, oarecum \u00eent\u00e2mpl\u0103tor, peste Amazon Athena. A devenit brusc evident c\u0103, aranj\u00e2nd cu aten\u021bie fi\u0219ierele noastre uria\u0219e de jurnale dup\u0103 folderele corecte (parquet) \u00een formatul coloanal \u2014 putem face selec\u021bii extrem de informative foarte repede \u0219i s\u0103 gener\u0103m rapoarte F\u0102R\u0102, f\u0103r\u0103 un cluster Apache Spark\/Glue.<\/p>\n<p>Motorul Athena, care lucreaz\u0103 cu datele din s3, se bazeaz\u0103 pe legendarul <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/big-data\/what-is-presto\/\">Presto<\/a><\/noindex> \u2014 reprezentant al familiei de abord\u0103ri MPP (procesare paralelel masiv) pentru prelucrarea datelor, care ia datele exact de unde sunt, de la S3 \u0219i Hadoop p\u00e2n\u0103 la Cassandra \u0219i fi\u0219ierele text obi\u0219nuite. Trebuie doar s\u0103 cerem lui Athena s\u0103 execute o interogare SQL, iar restul \u201efunc\u021bioneaz\u0103 rapid \u0219i de la sine\u201d. Este important de men\u021bionat c\u0103 Athena este \u201einteligent\u0103\u201d, acceseaz\u0103 doar folderele fragmentate necesare \u0219i cite\u0219te doar coloanele necesare din interogare.<\/p>\n<p>Tarifarea interog\u0103rilor c\u0103tre Athena este, de asemenea, interesant\u0103. Pl\u0103tim pentru <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/ru\/athena\/pricing\/\">volumul de date scanate<\/a><\/noindex>. Adic\u0103, nu pentru num\u0103rul de ma\u0219ini \u00een cluster pe minut, ci... pentru datele realmente scanate pe 100-500 de ma\u0219ini, doar cele necesare pentru a executa interogarea.<\/p>\n<p>\u0218i, solicit\u00e2nd doar coloanele necesare din folderele corect fragmentate, s-a dovedit c\u0103 serviciul Athena ne cost\u0103 zeci de dolari pe lun\u0103. Ei bine, este minunat, aproape gratuit, comparativ cu analizele pe clustere!<\/p>\n<p>Iat\u0103 cum ne fragment\u0103m datele \u00een S3:<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/00bd9ae48c1cd13f3c4f7d32692c9209.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDrept urmare, \u00eentr-un timp scurt, diferite departamente din companie, de la securitatea informa\u021biilor la analiz\u0103, au \u00eenceput s\u0103 fac\u0103 interog\u0103ri active c\u0103tre Athena \u0219i s\u0103 ob\u021bin\u0103 rapid, \u00een c\u00e2teva secunde, r\u0103spunsuri utile din \u201edatele mari\u201d pe perioade relativ extinse: luni, semestre etc.<\/p>\n<p>Dar am mers mai departe \u0219i am \u00eenceput s\u0103 c\u0103ut\u0103m r\u0103spunsuri \u00een cloud <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.aws.amazon.com\/athena\/latest\/ug\/connect-with-odbc.html\">prin intermediul driverului ODBC<\/a><\/noindex>: analistul, \u00een consola obi\u0219nuit\u0103, scrie o interogare SQL, care \u201epe 100-500 de ma\u0219ini, pentru un cost mic\u201d scaneaz\u0103 datele \u00een S3 \u0219i returneaz\u0103 r\u0103spunsul de obicei \u00een c\u00e2teva secunde. Este convenabil. \u0218i rapid. Nici acum nu-mi vine s\u0103 cred.<\/p>\n<p>\u00cen cele din urm\u0103, av\u00e2nd \u00een vedere c\u0103 am decis s\u0103 stoc\u0103m datele \u00een S3, \u00eentr-un format columnar eficient \u0219i cu fragmentarea rezonabil\u0103 a datelor pe foldere... am ob\u021binut un DataLake \u0219i un motor analitic rapid \u0219i ieftin \u2014 gratuit. \u0218i a devenit foarte popular \u00een companie, deoarece \u00een\u021belege SQL \u0219i func\u021bioneaz\u0103 de ordinul magnitudinii mai rapid dec\u00e2t prin lans\u0103ri\/opriri\/set\u0103ri ale clusterelor. \u201e\u0218i dac\u0103 rezultatul este acela\u0219i, de ce s\u0103 pl\u0103te\u0219ti mai mult?\u201d<\/p>\n<p>O interogare c\u0103tre Athena arat\u0103 aproximativ a\u0219a. Dac\u0103 este necesar, desigur, se poate formula o interogare SQL suficient de <noindex><a rel=\"nofollow\" href=\"https:\/\/prestodb.io\/docs\/0.172\/index.html\">complex\u0103 \u0219i de mai multe pagini<\/a><\/noindex>, dar ne vom limita la o simpl\u0103 grupare. S\u0103 vedem ce coduri de r\u0103spuns a avut clientul acum c\u00e2teva s\u0103pt\u0103m\u00e2ni \u00een jurnalele de activitate ale serverului web \u0219i s\u0103 ne asigur\u0103m c\u0103 nu sunt erori:<\/p>\n<p><img decoding=\"async\" alt=\"Cum am organizat un DataLake foarte eficient \u0219i ieftin \u0219i de ce exact a\u0219a\" src=\"\/wp-content\/uploads\/2020\/08\/30028991467b9e52f597faa617d374b5.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Conclusions<\/h3>\n<p>\nDup\u0103 un parcurs care nu a fost lung, dar a fost dureros, evalu\u00e2nd constant riscurile, nivelul de dificultate \u0219i costurile de suport, am g\u0103sit o solu\u021bie pentru DataLake \u0219i analitic\u0103, care ne bucur\u0103 at\u00e2t prin vitez\u0103, c\u00e2t \u0219i prin costurile de proprietate.<\/p>\n<p>A devenit clar c\u0103 este complet realizabil s\u0103 construie\u0219ti un DataLake eficient, rapid \u0219i cu costuri reduse de operare pentru nevoile diverselor departamente ale companiei, chiar \u0219i pentru dezvoltatori experimenta\u021bi care nu au fost niciodat\u0103 arhitec\u021bi \u0219i care nu \u0219tiu s\u0103 deseneze p\u0103tr\u0103\u021bele pe p\u0103tr\u0103\u021bele cu s\u0103ge\u021bi, cunoa\u0219terea a 50 de termeni din ecosystema Hadoop.<\/p>\n<p>La \u00eenceputul c\u0103l\u0103toriei, capul \u00eemi exploda din cauza multitudinii de zoologii s\u0103lbatice de software deschis \u0219i \u00eenchis \u0219i a con\u0219tiin\u021bei poverii responsabilit\u0103\u021bii fa\u021b\u0103 de urma\u0219i. \u00cencepe\u021bi s\u0103 construi\u021bi DataLake-ul vostru cu instrumente simple: nagios\/munin -&gt; elastic\/kibana -&gt; Hadoop\/Spark\/s3 ..., adun\u00e2nd feedback \u0219i \u00een\u021beleg\u00e2nd profund fizica proceselor care se desf\u0103\u0219oar\u0103. Toate lucrurile complicate \u0219i neclare - l\u0103sa\u021bi-le du\u0219manilor \u0219i concuren\u021bilor.<\/p>\n<p>Dac\u0103 nu dori\u021bi s\u0103 utiliza\u021bi cloudul \u0219i v\u0103 place s\u0103 sus\u021bine\u021bi, s\u0103 actualiza\u021bi \u0219i s\u0103 aplica\u021bi patch-uri proiectelor open-source, pute\u021bi construi o schem\u0103 similar\u0103 cu a noastr\u0103 local, pe ma\u0219ini de birou ieftine, folosind Hadoop \u0219i Presto deasupra. Principalul este s\u0103 nu v\u0103 opri\u021bi \u0219i s\u0103 merge\u021bi \u00eenainte, s\u0103 calcula\u021bi, s\u0103 c\u0103uta\u021bi solu\u021bii simple \u0219i clare, \u0219i totul va func\u021biona! Buna \u0219ans\u0103 tuturor \u0219i pe cur\u00e2nd!<br \/>\n<br \/>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/bitrix\/blog\/516374\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432 \u043a\u043e\u043c\u043c\u0435\u0440\u0447\u0435\u0441\u043a\u0443\u044e \u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0430\u0446\u0438\u044e. \u0410 \u043a\u043e\u0433\u0434\u0430 \u043d\u0443\u0436\u043d\u043e \u0431\u0443\u0434\u0435\u0442 \u043e\u0431\u043d\u043e\u0432\u043b\u044f\u0442\u044c\u0441\u044f\/\u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c\u0441\u044f \u0438\u043b\u0438 \u043a\u0442\u043e-\u0442\u043e \u0441\u043b\u0443\u0447\u0430\u0439\u043d\u043e \u043f\u0435\u0440\u0435\u0437\u0430\u0433\u0440\u0443\u0437\u0438\u0442 \u043f\u0430\u0440\u0443 \u043c\u0430\u0448\u0438\u043d \u2014 \u043e\u0441\u043e\u0437\u043d\u0430\u0442\u044c, \u0447\u0442\u043e \u043d\u0430\u0447\u0430\u043b\u0441\u044f \u043a\u0430\u043a\u043e\u0439-\u0442\u043e \u043d\u0430\u0432\u044f\u0437\u0447\u0438\u0432\u044b\u0439 \u0434\u0443\u0440\u043d\u043e\u0439 \u0441\u043e\u043d \u043d\u0430\u044f\u0432\u0443, \u0432\u0441\u0435 \u0440\u0435\u0437\u043a\u043e \u0443\u0441\u043b\u043e\u0436\u043d\u0438\u043b\u043e\u0441\u044c \u0434\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":92509,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-92508","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-28T05:42:10+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Cum am organizat un DataLake de \u00eenalt\u0103 eficien\u021b\u0103 \u0219i cu costuri reduse \u0219i de ce exact a\u0219a | ProHoster","description":"Tr\u0103im \u00eentr-o vreme fascinant\u0103, c\u00e2nd putem rapid \u0219i simplu s\u0103 conect\u0103m mai multe instrumente open-source gata f\u0103cute, s\u0103 le configur\u0103m cu \u00abmintea \u00eenchis\u0103\u00bb conform sfaturilor de pe stackoverflow, f\u0103r\u0103 a ne ad\u00e2nci \u00een \u00abmult liter\u0103\u00bb, \u0219i s\u0103 le lans\u0103m.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043e\u0432\u0430\u043b\u0438 \u0432\u044b\u0441\u043e\u043a\u043e\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438 \u043d\u0435\u0434\u043e\u0440\u043e\u0433\u043e\u0435 DataLake \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u0438\u043c\u0435\u043d\u043d\u043e \u0442\u0430\u043a | ProHoster","og:description":"\u041c\u044b \u0436\u0438\u0432\u0435\u043c \u0432 \u0443\u0434\u0438\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u043a\u043e\u0433\u0434\u0430 \u043c\u043e\u0436\u043d\u043e \u0431\u044b\u0441\u0442\u0440\u043e \u0438 \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043e\u0441\u0442\u044b\u043a\u043e\u0432\u0430\u0442\u044c \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0433\u043e\u0442\u043e\u0432\u044b\u0445 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u0432, \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c \u0438\u0445 \u0441 \u00ab\u043e\u0442\u043a\u043b\u044e\u0447\u0435\u043d\u043d\u044b\u043c \u0441\u043e\u0437\u043d\u0430\u043d\u0438\u0435\u043c\u00bb \u043f\u043e \u0441\u043e\u0432\u0435\u0442\u0430\u043c stackoverflow, \u043d\u0435 \u0432\u043d\u0438\u043a\u0430\u044f \u0432 \u00ab\u043c\u043d\u043e\u0433\u043e\u0431\u0443\u043a\u0432\u00bb, \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u0432.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-organizovali-vysokoeffektivnoe-i-nedorogoe-datalake-i-pochemu-imenno-tak","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-28T05:42:10+00:00","article:modified_time":"2020-08-28T05:42:10+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"92508","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:07:39","updated":"2022-10-01 09:50:53","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/92508","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=92508"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/92508\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/92509"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=92508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=92508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=92508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}