{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"Uroniti u Delta Lake: obvezna primjena i evolucija sheme","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Bok, Habr! Predstavljam vam prijevod \u010dlanka <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\u00abZaronite u Delta Lake: Zakon i Evolucija \u0160ema\u00bb<\/a><\/noindex> autora Burak Yavuz, Brenner Heintz i Denny Lee, koja je pripremljena uo\u010di po\u010detka te\u010daja <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\u201eData Engineer\u201c<\/a><\/noindex> od OTUS-a.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Uroniti u Delta Lake: obvezna primjena i evolucija sheme\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Podaci, kao i na\u0161e iskustvo, neprestano se akumuliraju i razvijaju. Da bismo ostali u korak, na\u0161i mentalni modeli svijeta moraju se prilagoditi novim podacima, od kojih neki donose nove dimenzije \u2014 nove na\u010dine posmatranja stvari o kojima ranije nismo imali pojma. Ovi mentalni modeli malo se razlikuju od shema tablica koje definiraju kako klasificiramo i obra\u0111ujemo nove informacije.<\/p>\n<p>To nas dovodi do pitanja upravljanja shemama. Kako se poslovni zadaci i zahtjevi mijenjaju tokom vremena, struktura va\u0161ih podataka se tako\u0111er mijenja. Delta Lake omogu\u0107ava lako uvo\u0111enje novih dimenzija kada se podaci mijenjaju. Korisnici imaju pristup jednostavnoj semantici za upravljanje shemama svojih tablica. Ovi alati uklju\u010duju prisilnu primjenu sheme (Schema Enforcement), koja \u0161titi korisnike od ne\u017eeljenog zaga\u0111enja svojih tablica gre\u0161kama ili suvi\u0161nim podacima, kao i evoluciju sheme (Schema Evolution), koja automatski omogu\u0107ava dodavanje novih stupaca s vrijednim podacima na odgovaraju\u0107a mjesta. U ovom \u010dlanku \u0107emo se dublje posvetiti kori\u0161tenju ovih alata.<\/p>\n<h2>Razumijevanje shema tablica<\/h2>\n<p>\nSvaki DataFrame u Apache Spark sadr\u017ei shemu koja odre\u0111uje oblik podataka, kao \u0161to su tipovi podataka, stupci i metapodaci. Pomo\u0107u Delta Lake, shema tablice se \u010duva u JSON formatu unutar dnevnika transakcija.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>\u0160to je prisilna primjena sheme?<\/h2>\n<p>\nPrisilna primjena sheme (Schema Enforcement), tako\u0111er poznata kao provjera sheme (Schema Validation), je za\u0161titni mehanizam u Delta Lake koji osigurava kvalitetu podataka odbacuju\u0107i zapise koji ne odgovaraju shemi tablice. Kao hostess na recepciji popularnog restorana koji prima samo po prethodnoj rezervaciji, ona provjerava jesu li svi stupci podataka koji se unose u tablicu na odgovaraju\u0107em popisu o\u010dekivanih stupaca (drugim rije\u010dima, imaju li svi \u201erezervacije\u201d), i odbacuje sve zapise sa stupcima koji nisu na tom popisu.<\/p>\n<h2>Kako funkcioni\u0161e prisilna primjena sheme?<\/h2>\n<p>\nDelta Lake koristi provjeru sheme prilikom zapisivanja, \u0161to zna\u010di da se svi novi unosi u tablicu provjeravaju na uskla\u0111enost sa shemom ciljne tablice tijekom zapisivanja. Ako shema nije uskla\u0111ena, Delta Lake potpuno poni\u0161tava transakciju (podatci se ne zapisuju) i generira iznimku kako bi obavijestio korisnika o nesukladnosti.<br \/>\nZa odre\u0111ivanje uskla\u0111enosti unosa Delta Lake koristi sljede\u0107a pravila. Zapisani DataFrame:<\/p>\n<ul>\n<li>ne mo\u017ee sadr\u017eavati dodatne stupce koji nisu u shemi ciljne tablice. I obrnuto, sve je u redu ako ulazni podaci ne sadr\u017ee sve stupce iz tablice \u2014 tim \u0107e stupcima jednostavno biti dodijeljene null vrijednosti.<\/li>\n<li>ne mo\u017ee imati tipove podataka stupaca koji se razlikuju od tipova podataka stupaca u ciljnoj tablici. Ako stupac ciljne tablice sadr\u017ei podatke tipa StringType, ali odgovaraju\u0107i stupac u DataFrame sadr\u017ei podatke tipa IntegerType, prisilno primjenjivanje sheme izazvat \u0107e iznimku i sprije\u010diti izvr\u0161enje operacije zapisivanja.<\/li>\n<li>ne mogu sadr\u017eavati imena stupaca koja se razlikuju samo po velikim i malim slovima. To zna\u010di da ne mo\u017eete imati stupce s imenima 'Foo' i 'foo' definirane u istoj tablici. Iako se Spark mo\u017ee koristiti u osjetljivom ili neosjetljivom na\u010dinu rada (po zadanom), Delta Lake \u010duva jezik, ali nije osjetljiv unutar pohrane sheme. Parquet je osjetljiv na jezik prilikom pohrane i vra\u0107anja informacija o stupcima. Kako bismo izbjegli mogu\u0107e gre\u0161ke, o\u0161te\u0107enje podataka ili njihovu izgubljenost (s \u010dime smo se osobno susreli u Databricksu), odlu\u010dili smo dodati ovo ograni\u010denje.<\/li>\n<\/ul>\n<p>\nDa bismo to ilustrirali, pogledajmo \u0161to se doga\u0111a u donjem kodu prilikom poku\u0161aja dodavanja nekih nedavno generiranih stupaca u Delta Lake tablicu koja jo\u0161 nije postavljena za njihovo prihva\u0107anje.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\nUmjesto automatskog dodavanja novih stupaca, Delta Lake name\u0107e shemu i zaustavlja zapisivanje. Kako bi pomogao odrediti koji stupac (ili njihova mno\u0161tvo) uzrokuje nesukladnost, Spark ispisuje obje sheme iz stack tra\u010deva za usporedbu.<\/p>\n<h2>Koja je korist od prisilnog primjenjivanja sheme?<\/h2>\n<p>\nBudu\u0107i da je prisilna primjena sheme stroga provjera, ona je izvrstan alat za kori\u0161tenje kao \u010duvar \u010distog, potpuno transformiranog skupa podataka spremnog za proizvodnju ili potro\u0161nju. Obi\u010dno se primjenjuje na tablice koje izravno pru\u017eaju podatke:<\/p>\n<ul>\n<li>Algoritmi stroj\u043d\u043e\u0433 u\u010denja<\/li>\n<li>BI nadzornim plo\u010dama<\/li>\n<li>Analitici podataka i alatima za vizualizaciju<\/li>\n<li>Svakom proizvodnom sustavu koji zahtijeva strogo strukturirane, strogo tipizirane semanti\u010dke sheme.<\/li>\n<\/ul>\n<p>\nKako biste pripremili svoje podatke za ovu kona\u010dnu barijeru, mnogi korisnici koriste jednostavnu 'multi-hop' arhitekturu koja postupno unosi strukturu u njihove tablice. Da biste saznali vi\u0161e o tome, mo\u017eete pro\u010ditati \u010dlanak <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Strojno u\u010denje proizvodne razine s Delta Lake.<\/a><\/noindex><\/p>\n<p>Naravno, prisilnu primjenu sheme mo\u017eete koristiti bilo gdje u svom pipeline-u, ali imajte na umu da bi zapisivanje podataka u tablicu u tom slu\u010daju moglo biti frustriraju\u0107e, zbog toga \u0161to ste, na primjer, zaboravili dodati jo\u0161 jedan stupac u ulazne podatke.<\/p>\n<h2>Sprje\u010davanje razrje\u0111ivanja podataka<\/h2>\n<p>\nU ovom trenutku se mo\u017eda pitate, za\u0161to tolika pompa? Na kraju krajeva, ponekad neo\u010dekivana pogre\u0161ka 'neslaganja sheme' mo\u017ee vas saplesti u va\u0161em radnom procesu, posebno ako ste po\u010detnik u Delta Lake. Za\u0161to jednostavno ne dopustiti shemi da se promijeni kako treba da mogu zapisati svoj DataFrame, bez obzira na sve?<\/p>\n<p>Kao \u0161to stara poslovica ka\u017ee, 'unca prevencije vrijedi kilograma lije\u010denja'. U nekom trenutku, ako se ne pobrinete za primjenu svoje sheme, problemi s kompatibilno\u0161\u0107u tipova podataka izbiti \u0107e na vidjelo - naizgled homogeni izvori sirovih podataka mogu sadr\u017eavati rubne slu\u010dajeve, o\u0161te\u0107ene stupce, pogre\u0161no oblikovane prikaze ili druge zastra\u0161uju\u0107e stvari koje vas proganjaju u no\u0107nim morama. Najbolji pristup je zaustaviti te neprijatelje na vratima - pomo\u0107u prisilne primjene sheme - i suo\u010diti se s njima na svjetlu, a ne kasnije, kada po\u010dnu premetati u mra\u010dnim dubinama va\u0161eg radnog koda.<\/p>\n<p>Obavezna primjena sheme osigurava da se shema va\u0161e tablice ne mijenja, osim ako sami ne potvrdite da \u017eelite promjenu. Ovo sprje\u010dava 'razvodnjavanje' podataka koje se mo\u017ee dogoditi kada se novi stupci dodaju tako \u010desto da ranije vrijedne, sa\u017eete tablice gube svoju va\u017enost i korisnost uslijed preplavljivanja podacima. Poticanjem vas na namjernost, postavljanje visoki standarda i o\u010dekivanje visoke kvalitete, obavezna primjena sheme radi ono \u010demu je namijenjena \u2014 poma\u017ee vam ostati savjestan, a va\u0161im tablicama zna\u010di \u010diste.<\/p>\n<p>Ako nakon daljnjeg razmatranja odlu\u010dite da zapravo <i>treba<\/i> dodate novi stupac \u2014 nema problema, u nastavku je jedinstveni ispravak. Rje\u0161enje \u2014 evolucija sheme!<\/p>\n<h2>\u0160to je evolucija sheme?<\/h2>\n<p>\nEvolucija sheme je funkcija koja omogu\u0107uje korisnicima da lako mijenjaju trenutnu shemu tablice u skladu s podacima koji se mijenjaju tijekom vremena. Naj\u010de\u0161\u0107e se koristi prilikom obavljanja operacije dodavanja ili prepisivanja kako bi se automatski prilagodila shema za uklju\u010divanje jednog ili vi\u0161e novih stupaca.<\/p>\n<h2>Kako funkcioni\u0161e evolucija sheme?<\/h2>\n<p>\nSlijede\u0107i primjer iz prethodnog odjeljka, programeri mogu lako koristiti evoluciju sheme za dodavanje novih stupaca koji su prethodno odbijeni zbog nesukladnosti sa shemom. Evolucija sheme se aktivira dodavanjem <code>.option('mergeSchema', 'true')<\/code> u va\u0161em Spark timu <code>.write ili .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nZa pregled grafikona izvr\u0161ite sljede\u0107i Spark SQL upit<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Uroniti u Delta Lake: obvezna primjena i evolucija sheme\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nKao alternativu, ovu opciju mo\u017eete postaviti za cijelu Spark sesiju dodavanjem <code>spark.databricks.delta.schema.autoMerge = True<\/code> u konfiguraciju Sparka. Ali budite oprezni \u0161to se ti\u010de toga, jer obavezna primjena sheme ne\u0107e vas vi\u0161e upozoravati na nepredvi\u0111ene nesukladnosti sa shemom.<\/p>\n<p>Uklju\u010divanjem u upit parametra <code>mergeSchema<\/code>, svi stupci koji su prisutni u DataFrame-u, ali odsutni u ciljnoj tablici, automatski se dodaju na kraj sheme u okviru transakcije zapisivanja. Tako\u0111er se mogu dodati ugnije\u017edene podatke, a oni \u0107e tako\u0111er biti dodani na kraj odgovaraju\u0107ih stupaca strukture.<\/p>\n<p>In\u017eenjeri i znanstvenici mogu koristiti ovu opciju za dodavanje novih stupaca (mogu\u0107e nedavno pra\u0107ene metrike ili stupca prodajnih pokazatelja za ovaj mjesec) u svoje postoje\u0107e tablice za strojarstvo bez naru\u0161avanja postoje\u0107ih modela temeljenih na starim stupcima. <\/p>\n<p>Sljede\u0107e vrste promjena sheme su dopu\u0161tene u okviru evolucije sheme tijekom dodavanja ili prepisivanja tablice:<\/p>\n<ul>\n<li>Dodavanje novih stupaca (ovo je naj\u010de\u0161\u0107i scenarij)<\/li>\n<li>Promjena tipova podataka iz NullType -&gt; bilo koju drugu vrstu ili unapre\u0111enje iz ByteType -&gt; ShortType -&gt; IntegerType<\/li>\n<\/ul>\n<p>\nOstale promjene, koje nisu dopu\u0161tene u okviru evolucije sheme, zahtijevaju da shema i podaci budu ponovno napisani dodavanjem <code>.option(\"overwriteSchema\", \"true\")<\/code>. Na primjer, u slu\u010daju da je stupac \"Foo\" izvorno bio integer, a nova shema bi bila tipa string, tada bi svi Parquet datoteke (podaci) morali biti ponovno zapisani. Takve promjene uklju\u010duju:<\/p>\n<ul>\n<li>uklanjanje stupca<\/li>\n<li>promjenu tipa podataka postoje\u0107eg stupca (na mjestu)<\/li>\n<li>preimenovanje stupaca koji se razlikuju samo po velikim i malim slovima (primjerice, \"Foo\" i \"foo\")<\/li>\n<\/ul>\n<p>\nNa kraju, s sljede\u0107im izdanjem Spark 3.0 bit \u0107e potpuno podr\u017ean eksplicitan DDL (koriste\u0107i ALTER TABLE), \u0161to \u0107e korisnicima omogu\u0107iti sljede\u0107e radnje nad shemama tablica:<\/p>\n<ul>\n<li>dodavanje stupaca<\/li>\n<li>promjena komentara za stupce<\/li>\n<li>pode\u0161avanje svojstava tablice koja definiraju pona\u0161anje tablice, na primjer, postavljanje trajanja skladi\u0161tenja dnevnika transakcija.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Koja je korist evolucije sheme?<\/h2>\n<p>\nEvoluciju sheme mo\u017eete koristiti uvijek kada <i>namjeravate<\/i> promijeniti shemu svoje tablice (za razliku od situacija kada ste slu\u010dajno dodali stupce u svoj DataFrame koji ne bi trebali biti tamo). Ovo je najlak\u0161i na\u010din za migraciju va\u0161e sheme, jer automatski dodaje ispravna imena stupaca i tipove podataka bez potrebe za njihovim eksplicitnim izjavama.<\/p>\n<h2>Zaklju\u010dak<\/h2>\n<p>\nObvezno primjenjivanje sheme odbacuje sve nove stupce ili druge promjene sheme koje nisu kompatibilne s va\u0161om tablicom. Postavljaju\u0107i i odr\u017eavaju\u0107i ove visoke standarde, analiti\u010dari i in\u017eenjeri mogu se osloniti na to da njihovi podaci imaju najvi\u0161i nivo integriteta, raspravljaju\u0107i o tome jasno i precizno, \u0161to im omogu\u0107uje dono\u0161enje u\u010dinkovitijih poslovnih odluka.<\/p>\n<p>S druge strane, evolucija sheme dopunjuje obvezno primjenjivanje, pojednostavljuju\u0107i <i>pretpostavljene<\/i> automatske promjene sheme. Na kraju, to ne bi trebao biti slo\u017een proces \u2014 dodati stupac.<\/p>\n<p>Obvezno primjenjivanje sheme je yang, dok je evolucija sheme yin. U kombinaciji, ove funkcije nikada nisu bile jednostavnije za suzbijanje \u0161uma i pode\u0161avanje signala.<\/p>\n<p><i>Tako\u0111er bismo \u017eeljeli zahvaliti Mukulu Murti i Pranavu Anandu na njihovom doprinosu ovom \u010dlanku.<\/i><\/p>\n<p>Drugi \u010dlanci iz ove serije:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Uronjavanje u Delta Lake: razotkrivanje dnevnika transakcija<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"Pogledajte video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>\u010clanci na temu<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Strojno u\u010denje na razini proizvodnje s Delta Lake<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">\u0160to je jezero podataka?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">Saznajte vi\u0161e o te\u010daju<\/a><\/noindex><\/p>\n<p>Izvor: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"hr_HR\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Uronjavanje u Delta Lake: obvezno primjenjivanje i evolucija sheme | ProHoster","description":"Bok, Habr!","canonical_url":"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"hr_HR","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!","og:url":"https:\/\/prohoster.info\/hr\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/hr\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}