{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"Ponorenie do Delta Lake: n\u00faten\u00e9 aplikovanie a evol\u00facia sch\u00e9my","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Ahoj, Habr! Predstavujem v\u00e1m preklad \u010dl\u00e1nku <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\u201eVstup do Delta Lake: Vynucovanie a v\u00fdvoj sch\u00e9my\u201c<\/a><\/noindex> autorov Burak Yavuz, Brenner Heintz a Denny Lee, ktor\u00fd bol pripraven\u00fd pred za\u010diatkom kurzu <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\u201eData Engineer\u201c<\/a><\/noindex> od OTUS.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Ponorenie do Delta Lake: n\u00faten\u00e9 aplikovanie a evol\u00facia sch\u00e9my\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>\u00dadaje, rovnako ako na\u0161e sk\u00fasenosti, sa neust\u00e1le zhroma\u017e\u010fuj\u00fa a vyv\u00edjaj\u00fa. Aby sme nezaost\u00e1vali, musia sa na\u0161e ment\u00e1lne modely sveta prisp\u00f4sobi\u0165 nov\u00fdm \u00fadajom, z ktor\u00fdch niektor\u00e9 obsahuj\u00fa nov\u00e9 dimenzie \u2014 nov\u00e9 sp\u00f4soby, ako pozorova\u0165 veci, o ktor\u00fdch sme predt\u00fdm nemali predstavu. Tieto ment\u00e1lne modely sa m\u00e1lo l\u00ed\u0161ia od sch\u00e9m tabuliek, ktor\u00e9 ur\u010duj\u00fa, ako klasifikujeme a spracov\u00e1vame nov\u00e9 inform\u00e1cie.<\/p>\n<p>To n\u00e1s priv\u00e1dza k ot\u00e1zke spr\u00e1vy sch\u00e9m. Ke\u010f sa obchodn\u00e9 \u00falohy a po\u017eiadavky v priebehu \u010dasu menia, men\u00ed sa aj \u0161trukt\u00fara va\u0161ich \u00fadajov. Delta Lake umo\u017e\u0148uje jednoducho zav\u00e1dza\u0165 nov\u00e9 dimenzie, ke\u010f sa maj\u00fa \u00fadaje meni\u0165. Pou\u017e\u00edvatelia maj\u00fa pr\u00edstup k jednoduchej semantike na spr\u00e1vu sch\u00e9m svojich tabuliek. Tieto n\u00e1stroje zah\u0155\u0148aj\u00fa vynucovanie sch\u00e9my (Schema Enforcement), ktor\u00e9 chr\u00e1ni pou\u017e\u00edvate\u013eov pred ne\u00famyseln\u00fdm zne\u010disten\u00edm svojich tabuliek chybami alebo nepotrebn\u00fdmi \u00fadajmi, ako aj evol\u00faciu sch\u00e9my (Schema Evolution), ktor\u00e1 umo\u017e\u0148uje automaticky prid\u00e1va\u0165 nov\u00e9 st\u013apce s hodnotn\u00fdmi \u00fadajmi na pr\u00edslu\u0161n\u00e9 miesta. V tomto \u010dl\u00e1nku sa hlb\u0161ie zaober\u00e1me pou\u017e\u00edvan\u00edm t\u00fdchto n\u00e1strojov.<\/p>\n<h2>Pochopenie sch\u00e9m tabuliek<\/h2>\n<p>\nKa\u017ed\u00fd DataFrame v Apache Spark obsahuje sch\u00e9mu, ktor\u00e1 ur\u010duje formu \u00fadajov, ako s\u00fa typy \u00fadajov, st\u013apce a metad\u00e1ta. So syst\u00e9mom Delta Lake sa sch\u00e9ma tabu\u013eky uchov\u00e1va vo form\u00e1te JSON vo vn\u00fatri transak\u010dn\u00e9ho denn\u00edka.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>\u010co je vynucovanie sch\u00e9my?<\/h2>\n<p>\nVynucovanie sch\u00e9my (Schema Enforcement), zn\u00e1me tie\u017e ako overovanie sch\u00e9my (Schema Validation), je ochrann\u00fd mechanizmus v Delta Lake, ktor\u00fd zabezpe\u010duje kvalitu \u00fadajov t\u00fdm, \u017ee odmieta z\u00e1znamy, ktor\u00e9 nezodpovedaj\u00fa sch\u00e9me tabu\u013eky. Podobne ako hosteska na recepcii v popul\u00e1rnej re\u0161taur\u00e1cii, ktor\u00e1 prij\u00edma len s rezerv\u00e1ciou, overuje, \u010di je ka\u017ed\u00fd st\u013apec \u00fadajov, ktor\u00fd sa vklad\u00e1 do tabu\u013eky, v zodpovedaj\u00facom zozname o\u010dak\u00e1van\u00fdch st\u013apcov (in\u00fdmi slovami, \u010di m\u00e1 pre ka\u017ed\u00fd z nich \u201erezerv\u00e1ciu\u201c) a odmieta ak\u00e9ko\u013evek z\u00e1znamy so st\u013apcami, ktor\u00e9 nie s\u00fa v zozname.<\/p>\n<h2>Ako funguje vynucovanie sch\u00e9my?<\/h2>\n<p>\nDelta Lake pri z\u00e1pise vyu\u017e\u00edva kontrolu sch\u00e9my, \u010do znamen\u00e1, \u017ee v\u0161etky nov\u00e9 z\u00e1znamy v tabu\u013eke sa kontroluj\u00fa na zlu\u010dite\u013enos\u0165 so sch\u00e9mou cie\u013eovej tabu\u013eky po\u010das z\u00e1pisu. Ak je sch\u00e9ma nezlu\u010dite\u013en\u00e1, Delta Lake \u00faplne zru\u0161\u00ed transakciu (\u00fadaje sa nezap\u00ed\u0161u) a vytvor\u00ed v\u00fdnimku, aby informoval pou\u017e\u00edvate\u013ea o nezrovnalosti.<br \/>\nAby ur\u010dil zlu\u010dite\u013enos\u0165 z\u00e1znamu, Delta Lake pou\u017e\u00edva nasleduj\u00face pravidl\u00e1. Z\u00e1pisov\u00fd DataFrame:<\/p>\n<ul>\n<li>nem\u00f4\u017ee obsahova\u0165 dodato\u010dn\u00e9 st\u013apce, ktor\u00e9 nie s\u00fa v sch\u00e9me cie\u013eovej tabu\u013eky. Naopak, ak prich\u00e1dzaj\u00face \u00fadaje neobsahuj\u00fa v\u0161etky st\u013apce z tabu\u013eky, je to v poriadku \u2014 t\u00fdmto st\u013apcom bud\u00fa jednoducho priraden\u00e9 nulov\u00e9 hodnoty.<\/li>\n<li>nesmie ma\u0165 typy \u00fadajov st\u013apcov, ktor\u00e9 sa l\u00ed\u0161ia od typov \u00fadajov st\u013apcov v cie\u013eovej tabu\u013eke. Ak st\u013apec cie\u013eovej tabu\u013eky obsahuje \u00fadaje StringType, ale pr\u00edslu\u0161n\u00fd st\u013apec v DataFrame obsahuje \u00fadaje IntegerType, vyn\u00faten\u00e9 uplatnenie sch\u00e9my vyvol\u00e1 v\u00fdnimku a zabr\u00e1ni vykonaniu oper\u00e1cie z\u00e1pisu.<\/li>\n<li>nem\u00f4\u017ee obsahova\u0165 n\u00e1zvy st\u013apcov, ktor\u00e9 sa l\u00ed\u0161ia iba ve\u013ekos\u0165ou p\u00edsmen. To znamen\u00e1, \u017ee nem\u00f4\u017eete ma\u0165 st\u013apce s n\u00e1zvami 'Foo' a 'foo', definovan\u00e9 v jednej tabu\u013eke. Hoci sa Spark m\u00f4\u017ee pou\u017e\u00edva\u0165 v citlivom alebo necitlivom (predvolene) na ve\u013ekos\u0165 p\u00edsmen, Delta Lake uchov\u00e1va ve\u013ekos\u0165 p\u00edsmen, ale je necitliv\u00fd v r\u00e1mci ukladania sch\u00e9my. Parquet je citliv\u00fd na ve\u013ekos\u0165 p\u00edsmen pri ukladan\u00ed a vr\u00e1ten\u00ed inform\u00e1ci\u00ed o st\u013apcoch. Aby sa predi\u0161lo mo\u017en\u00fdm chyb\u00e1m, po\u0161kodeniu \u00fadajov alebo ich strate (s \u010d\u00edm sme sa osobne stretli v Databricks), rozhodli sme sa prida\u0165 toto obmedzenie.<\/li>\n<\/ul>\n<p>\nAby sme to ilustrovali, po\u010fme sa pozrie\u0165 na to, \u010do sa deje v ni\u017e\u0161ie uvedenom k\u00f3de, ke\u010f sa pok\u00fasime prida\u0165 niektor\u00e9 ned\u00e1vno vygenerovan\u00e9 st\u013apce do tabu\u013eky Delta Lake, ktor\u00e1 e\u0161te nie je nastaven\u00e1 na ich prijatie.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\nNamiesto automatick\u00e9ho pridania nov\u00fdch st\u013apcov Delta Lake uplat\u0148uje sch\u00e9mu a zastavuje z\u00e1pis. Aby sme pomohli ur\u010di\u0165, ktor\u00fd st\u013apec (alebo ich mno\u017estvo) je pr\u00ed\u010dinou nezlu\u010dite\u013enosti, Spark vypisuje obe sch\u00e9my zo z\u00e1sobn\u00edka pre porovnanie.<\/p>\n<h2>Ak\u00fd je pr\u00ednos vyn\u00faten\u00e9ho uplatnenia sch\u00e9my?<\/h2>\n<p>\nPreto\u017ee vyn\u00faten\u00e9 pou\u017eitie sch\u00e9my predstavuje dos\u0165 pr\u00edsnu kontrolu, je vynikaj\u00facim n\u00e1strojom na vyu\u017eitie ako br\u00e1na \u010distej, \u00faplne transformovanej sady d\u00e1t, ktor\u00e1 je pripraven\u00e1 na produkciu alebo spotrebu. Zvy\u010dajne sa aplikuje na tabu\u013eky, ktor\u00e9 priamo pod\u00e1vaj\u00fa d\u00e1ta:<\/p>\n<ul>\n<li>Algoritmy strojov\u00e9ho u\u010denia<\/li>\n<li>BI dashboardy<\/li>\n<li>Analytika d\u00e1t a vizualiza\u010dn\u00e9 n\u00e1stroje<\/li>\n<li>Ak\u00fdko\u013evek produk\u010dn\u00fd syst\u00e9m, ktor\u00fd vy\u017eaduje pr\u00edsne \u0161trukt\u00farovan\u00e9, pr\u00edsne typovan\u00e9 s\u00e9mantick\u00e9 sch\u00e9my.<\/li>\n<\/ul>\n<p>\nAby ste pripravili svoje d\u00e1ta na t\u00fato kone\u010dn\u00fa prek\u00e1\u017eku, mnoh\u00ed u\u017e\u00edvatelia pou\u017e\u00edvaj\u00fa jednoduch\u00fa architekt\u00faru \u201emulti-hop\u201c, ktor\u00e1 postupne vn\u00e1\u0161a \u0161trukt\u00faru do ich tabuliek. Ak sa chcete o tom dozvedie\u0165 viac, m\u00f4\u017eete si pre\u010d\u00edta\u0165 \u010dl\u00e1nok <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Strojov\u00e9 u\u010denie na \u00farovni produkcie s Delta Lake.<\/a><\/noindex><\/p>\n<p>Samozrejme, vyn\u00faten\u00e9 pou\u017eitie sch\u00e9my sa m\u00f4\u017ee pou\u017ei\u0165 kdeko\u013evek vo va\u0161om pipeline, ale pam\u00e4tajte, \u017ee streamovanie z\u00e1pisu do tabu\u013eky v takom pr\u00edpade m\u00f4\u017ee by\u0165 frustruj\u00face, preto\u017ee ste napr\u00edklad zabudli, \u017ee ste pridali \u010fal\u0161\u00ed st\u013apec do prich\u00e1dzaj\u00facich d\u00e1t.<\/p>\n<h2>Predch\u00e1dzanie rednutiu d\u00e1t<\/h2>\n<p>\nV tomto bode sa mo\u017eno p\u00fdtate, \u010do sp\u00f4sobilo tak\u00fd rozruch? Nakoniec, ob\u010das v\u00e1s m\u00f4\u017ee neo\u010dak\u00e1van\u00e1 chyba \u201enes\u00falad sch\u00e9my\u201c potiahnu\u0165 dolu vo va\u0161om pracovnom procese, najm\u00e4 ak ste nov\u00e1\u010dik v Delta Lake. Pre\u010do jednoducho nema\u0165 povolen\u00e9, aby sa sch\u00e9ma menila tak, ako to potrebujem, aby som mohol zap\u00edsa\u0165 svoj DataFrame, nech sa deje \u010doko\u013evek?<\/p>\n<p>Ako vrav\u00ed star\u00e9 pr\u00edslovie, \u201euncia prevencie stoj\u00ed libru lie\u010dby\u201c. V nejakom momente, ak sa nestar\u00e1te o uplatnenie svojej sch\u00e9my, objavia sa probl\u00e9my s kompatibilitou typov d\u00e1t \u2014 na prv\u00fd poh\u013ead homog\u00e9nne zdroje surov\u00fdch d\u00e1t m\u00f4\u017eu obsahova\u0165 okrajov\u00e9 pr\u00edpady, po\u0161koden\u00e9 st\u013apce, nespr\u00e1vne vytvoren\u00e9 zobrazenia alebo in\u00e9 stra\u0161n\u00e9 veci, ktor\u00e9 sa sn\u00edvaj\u00fa v no\u010dn\u00fdch mor\u00e1ch. Najlep\u0161\u00edm pr\u00edstupom je zastavi\u0165 t\u00fdchto nepriate\u013eov pri br\u00e1ne \u2014 pomocou vyn\u00faten\u00e9ho pou\u017eitia sch\u00e9my \u2014 a zaobera\u0165 sa nimi na svetle, nie nesk\u00f4r, ke\u010f za\u010dn\u00fa \u010d\u00edha\u0165 v temn\u00fdch h\u013abkach v\u00e1\u0161ho pracovn\u00e9ho k\u00f3du.<\/p>\n<p>Povinn\u00e9 uplatnenie sch\u00e9my zaist\u00ed, \u017ee sa va\u0161a tabu\u013ekov\u00e1 sch\u00e9ma nezmen\u00ed, pokia\u013e to sami nepotvrd\u00edte. To zabra\u0148uje \"zrieden\u00e9mu\" (dilution) obsahu, ktor\u00e9 m\u00f4\u017ee nasta\u0165, ke\u010f sa nov\u00e9 st\u013apce prid\u00e1vaj\u00fa tak \u010dasto, \u017ee predt\u00fdm cenn\u00e9, zhutnen\u00e9 tabu\u013eky str\u00e1caj\u00fa svoj v\u00fdznam a u\u017eito\u010dnos\u0165 v d\u00f4sledku zaplavenia \u00fadajmi. Povzbudzuj\u00fac v\u00e1s, aby ste boli z\u00e1mern\u00ed, nastavovali vysok\u00e9 \u0161tandardy a o\u010dak\u00e1vali vysok\u00fa kvalitu, povinn\u00e9 uplatnenie sch\u00e9my rob\u00ed presne to, na \u010do bolo ur\u010den\u00e9 \u2014 pom\u00e1ha v\u00e1m zosta\u0165 poctiv\u00fdmi, a va\u0161im tabu\u013ek\u00e1m \u2014 \u010dist\u00fdmi.<\/p>\n<p>Ak pri \u010fal\u0161om presk\u00faman\u00ed zist\u00edte, \u017ee naozaj <i>je potrebn\u00e9<\/i> prida\u0165 nov\u00fd st\u013apec \u2014 nie je probl\u00e9m, ni\u017e\u0161ie je uveden\u00e9 jednoradkov\u00e9 rie\u0161enie. Rie\u0161enie \u2014 evol\u00facia sch\u00e9my!<\/p>\n<h2>\u010co je to evol\u00facia sch\u00e9my?<\/h2>\n<p>\nEvol\u00facia sch\u00e9my je funkcia, ktor\u00e1 pou\u017e\u00edvate\u013eom umo\u017e\u0148uje \u013eahko meni\u0165 aktu\u00e1lnu sch\u00e9mu tabu\u013eky pod\u013ea \u00fadajov, ktor\u00e9 sa menia s \u010dasom. Naj\u010dastej\u0161ie sa pou\u017e\u00edva pri vykon\u00e1van\u00ed oper\u00e1cie pridania alebo prep\u00edsania, aby sa automaticky prisp\u00f4sobila sch\u00e9ma na zahrnutie jedn\u00e9ho alebo viacer\u00fdch nov\u00fdch st\u013apcov.<\/p>\n<h2>Ako funguje evol\u00facia sch\u00e9my?<\/h2>\n<p>\nNasleduj\u00fac pr\u00edklad z predch\u00e1dzaj\u00facej \u010dasti, v\u00fdvoj\u00e1ri m\u00f4\u017eu \u013eahko vyu\u017ei\u0165 evol\u00faciu sch\u00e9my na pridanie nov\u00fdch st\u013apcov, ktor\u00e9 boli predt\u00fdm odmietnut\u00e9 z d\u00f4vodu nes\u00fahlasu so sch\u00e9mou. Evol\u00facia sch\u00e9my sa aktivuje pridan\u00edm <code>.option('mergeSchema', 'true')<\/code> k va\u0161ej Spark pr\u00edkazu <code>.write alebo .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nAby ste si prezreli graf, vykonajte nasleduj\u00faci Spark SQL dopyt<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Ponorenie do Delta Lake: n\u00faten\u00e9 aplikovanie a evol\u00facia sch\u00e9my\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAlternat\u00edvne m\u00f4\u017eete nastavi\u0165 t\u00fato vo\u013ebu pre cel\u00fa Spark rel\u00e1ciu pridan\u00edm <code>spark.databricks.delta.schema.autoMerge = True<\/code> do konfigur\u00e1cie Spark. Ale bu\u010fte opatrn\u00ed, preto\u017ee povinn\u00e9 uplatnenie sch\u00e9my u\u017e v\u00e1s nebude varova\u0165 pred ne\u00famyseln\u00fdmi nes\u00faladmi so sch\u00e9mou.<\/p>\n<p>Pri zapnut\u00ed parametru v dopyte <code>mergeSchema<\/code>, v\u0161etky st\u013apce, ktor\u00e9 s\u00fa pr\u00edtomn\u00e9 v DataFrame, ale ch\u00fdbaj\u00fa v cie\u013eovej tabu\u013eke, sa automaticky prid\u00e1vaj\u00fa na koniec sch\u00e9my v r\u00e1mci transakcie z\u00e1pisu. M\u00f4\u017eu sa tie\u017e prida\u0165 vnoren\u00e9 polia, a tie sa tie\u017e pridaj\u00fa na koniec pr\u00edslu\u0161n\u00fdch st\u013apcov \u0161trukt\u00fary.<\/p>\n<p>D\u00e1tum in\u017einieri a vedci m\u00f4\u017eu vyu\u017ei\u0165 t\u00fato mo\u017enos\u0165 na pridanie nov\u00fdch st\u013apcov (mo\u017eno ned\u00e1vno sledovanej metriky alebo st\u013apca predajov za tento mesiac) do svojich existuj\u00facich produktov\u00fdch tabuliek strojov\u00e9ho u\u010denia, bez toho, aby naru\u0161ili existuj\u00face modely zalo\u017een\u00e9 na star\u00fdch st\u013apcoch. <\/p>\n<p>Nasleduj\u00face typy zmien sch\u00e9my s\u00fa prijate\u013en\u00e9 v r\u00e1mci evol\u00facie sch\u00e9my po\u010das prid\u00e1vania alebo prepisovania tabuliek:<\/p>\n<ul>\n<li>Pridanie nov\u00fdch st\u013apcov (to je najbe\u017enej\u0161\u00ed scen\u00e1r)<\/li>\n<li>Zmena typov \u00fadajov z NullType -&gt; ak\u00fdko\u013evek in\u00fd typ alebo zvy\u0161ovanie z ByteType -&gt; ShortType -&gt; IntegerType<\/li>\n<\/ul>\n<p>\nIn\u00e9 zmeny, ktor\u00e9 nie s\u00fa povolen\u00e9 v r\u00e1mci evol\u00facie sch\u00e9my, si vy\u017eaduj\u00fa, aby sch\u00e9ma a d\u00e1ta boli prep\u00edsan\u00e9 pridan\u00edm <code>.option(\"overwriteSchema\", \"true\")<\/code>. Napr\u00edklad, ak bol st\u013apec \u201eFoo\u201c p\u00f4vodne integer a nov\u00e1 sch\u00e9ma by bola typu string, potom by sa v\u0161etky s\u00fabory Parquet (d\u00e1ta) museli prep\u00edsa\u0165. Medzi tak\u00e9to zmeny patr\u00ed:<\/p>\n<ul>\n<li>odstr\u00e1nenie st\u013apca<\/li>\n<li>zmena typu \u00fadajov existuj\u00faceho st\u013apca (na mieste)<\/li>\n<li>premenovanie st\u013apcov, ktor\u00e9 sa l\u00ed\u0161ia len ve\u013ekos\u0165ou p\u00edsmen (napr\u00edklad \u201eFoo\u201c a \u201efoo\u201c)<\/li>\n<\/ul>\n<p>\nNakoniec, s nasleduj\u00facim vydan\u00edm Spark 3.0 bude \u00faplne podporovan\u00e1 explicitn\u00e1 DDL (s pou\u017eit\u00edm ALTER TABLE), \u010do umo\u017en\u00ed pou\u017e\u00edvate\u013eom vykon\u00e1va\u0165 nasleduj\u00face oper\u00e1cie s tabu\u013ekov\u00fdmi sch\u00e9mami:<\/p>\n<ul>\n<li>prid\u00e1vanie st\u013apcov<\/li>\n<li>zmena koment\u00e1rov k st\u013apcom<\/li>\n<li>nastavenie vlastnost\u00ed tabu\u013eky, ktor\u00e9 definuj\u00fa spr\u00e1vanie tabu\u013eky, ako je napr\u00edklad nastavenie trvania uchov\u00e1vania registra transakci\u00ed.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Ak\u00fd je pr\u00ednos evol\u00facie sch\u00e9my?<\/h2>\n<p>\nEvol\u00facia sch\u00e9my sa m\u00f4\u017ee vyu\u017ei\u0165 v\u017edy, ke\u010f <i>zam\u00fd\u0161\u013eate<\/i> zmeni\u0165 sch\u00e9mu svojej tabu\u013eky (na rozdiel od t\u00fdch pr\u00edpadov, ke\u010f ste n\u00e1hodou pridali do svojho DataFrame st\u013apce, ktor\u00e9 tam by\u0165 nemali). Je to najjednoduch\u0161\u00ed sp\u00f4sob, ako migrova\u0165 svoju sch\u00e9mu, preto\u017ee automaticky prid\u00e1va spr\u00e1vne n\u00e1zvy st\u013apcov a typy \u00fadajov bez nutnosti ich explicitnej deklar\u00e1cie.<\/p>\n<h2>Z\u00e1ver<\/h2>\n<p>\nPovinn\u00e9 uplatnenie sch\u00e9my odmieta ak\u00e9ko\u013evek nov\u00e9 st\u013apce alebo in\u00e9 zmeny sch\u00e9my, ktor\u00e9 nie s\u00fa kompatibiln\u00e9 s va\u0161ou tabu\u013ekou. Udr\u017eiavan\u00edm t\u00fdchto vysok\u00fdch \u0161tandardov sa analytici a in\u017einieri m\u00f4\u017eu spolieha\u0165 na to, \u017ee ich \u00fadaje maj\u00fa najvy\u0161\u0161iu \u00farove\u0148 integrity, \u010do im umo\u017e\u0148uje prij\u00edma\u0165 efekt\u00edvnej\u0161ie obchodn\u00e9 rozhodnutia.<\/p>\n<p>Na druhej strane, evol\u00facia sch\u00e9my dop\u013a\u0148a povinn\u00e9 uplatnenie, \u010do zjednodu\u0161uje <i>predpokladan\u00e9<\/i> automatick\u00e9 zmeny sch\u00e9my. Nakoniec to nemus\u00ed by\u0165 zlo\u017eit\u00e1 \u00faloha \u2014 prida\u0165 st\u013apec.<\/p>\n<p>Povinn\u00e9 uplatnenie sch\u00e9my je jang, zatia\u013e \u010do evol\u00facia sch\u00e9my je jin. Spolu tieto funkcie zjednodu\u0161uj\u00fa potla\u010denie \u0161umu a ladenie sign\u00e1lu viac ne\u017e kedyko\u013evek predt\u00fdm.<\/p>\n<p><i>R\u00e1di by sme tie\u017e po\u010fakovali Mukulovi Murti a Pranavovi Anandovi za ich pr\u00ednos k tomuto \u010dl\u00e1nku.<\/i><\/p>\n<p>\u010eal\u0161ie \u010dl\u00e1nky z tejto s\u00e9rie:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Zabudovanie do Delta Lake: rozbalenie transak\u010dn\u00e9ho denn\u00edka<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"Prehrajte video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>S\u00favisiace \u010dl\u00e1nky<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Strojov\u00e9 u\u010denie na \u00farovni v\u00fdrobn\u00e9ho prostredia s Delta Lake<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">\u010co je jazero d\u00e1t?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">Zistite viac o kurze<\/a><\/noindex><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Zabudovanie do Delta Lake: povinn\u00e9 uplatnenie a evol\u00facia sch\u00e9my | ProHoster","description":"Ahoj, Habr!","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}