{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"Pohled na Delta Lake: nucen\u00e9 aplikace a evoluce sch\u00e9matu","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Ahoj, Habr! P\u0159edstavuji v\u00e1m p\u0159eklad \u010dl\u00e1nku <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\u201ePono\u0159en\u00ed do Delta Lake: Vynucen\u00ed a evoluce sch\u00e9matu\u201c<\/a><\/noindex> autor\u016f Burak Yavuz, Brenner Heintz a Denny Lee, kter\u00fd byl p\u0159ipraven na zah\u00e1jen\u00ed kurzu <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\u201eDatov\u00fd in\u017een\u00fdr\u201c<\/a><\/noindex> od OTUS.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Pohled na Delta Lake: nucen\u00e9 aplikace a evoluce sch\u00e9matu\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Data, stejn\u011b jako na\u0161e zku\u0161enosti, se neust\u00e1le shroma\u017e\u010fuj\u00ed a vyv\u00edjej\u00ed. Abychom nez\u016fstali pozadu, na\u0161e ment\u00e1ln\u00ed modely sv\u011bta se mus\u00ed p\u0159izp\u016fsobit nov\u00fdm dat\u016fm, z nich\u017e n\u011bkter\u00e1 obsahuj\u00ed nov\u00e9 dimenze \u2014 nov\u00e9 zp\u016fsoby, jak pozorovat v\u011bci, o kter\u00fdch jsme d\u0159\u00edve nem\u011bli pon\u011bt\u00ed. Tyto ment\u00e1ln\u00ed modely se od sebe m\u00e1lo li\u0161\u00ed od sch\u00e9mat tabulek, kter\u00e9 ur\u010duj\u00ed, jak klasifikujeme a zpracov\u00e1v\u00e1me nov\u00e9 informace.<\/p>\n<p>To n\u00e1s p\u0159iv\u00e1d\u00ed k ot\u00e1zce spr\u00e1vy sch\u00e9mat. Jak se obchodn\u00ed \u00fakoly a po\u017eadavky v pr\u016fb\u011bhu \u010dasu m\u011bn\u00ed, m\u011bn\u00ed se tak\u00e9 struktura va\u0161ich dat. Delta Lake umo\u017e\u0148uje snadno implementovat nov\u00e9 dimenze p\u0159i zm\u011bn\u011b dat. U\u017eivatel\u00e9 maj\u00ed p\u0159\u00edstup k jednoduch\u00e9 s\u00e9mantice pro spr\u00e1vu sch\u00e9mat sv\u00fdch tabulek. Tyto n\u00e1stroje zahrnuj\u00ed vynucen\u00ed sch\u00e9matu (Schema Enforcement), kter\u00e9 chr\u00e1n\u00ed u\u017eivatele p\u0159ed ne\u00famysln\u00fdm zne\u010di\u0161t\u011bn\u00edm jejich tabulek chybami nebo ne\u017e\u00e1douc\u00edmi daty, a tak\u00e9 evoluci sch\u00e9matu (Schema Evolution), kter\u00e1 umo\u017e\u0148uje automaticky p\u0159id\u00e1vat nov\u00e9 sloupce s cenn\u00fdmi daty na odpov\u00eddaj\u00edc\u00ed m\u00edsta. V tomto \u010dl\u00e1nku se pod\u00edv\u00e1me hloub\u011bji na pou\u017eit\u00ed t\u011bchto n\u00e1stroj\u016f.<\/p>\n<h2>Porozum\u011bn\u00ed sch\u00e9mat\u016fm tabulek<\/h2>\n<p>\nKa\u017ed\u00fd DataFrame v Apache Spark obsahuje sch\u00e9ma, kter\u00e9 definuje formu dat, jako jsou datov\u00e9 typy, sloupce a metadata. Pomoc\u00ed Delta Lake se sch\u00e9ma tabulky zachov\u00e1v\u00e1 ve form\u00e1tu JSON uvnit\u0159 transak\u010dn\u00edho protokolu.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Co je to vynucen\u00ed sch\u00e9matu?<\/h2>\n<p>\nVynucen\u00ed sch\u00e9matu (Schema Enforcement), tak\u00e9 zn\u00e1m\u00e9 jako ov\u011b\u0159ov\u00e1n\u00ed sch\u00e9matu (Schema Validation), je ochrann\u00fd mechanismus v Delta Lake, kter\u00fd zaru\u010duje kvalitu dat t\u00edm, \u017ee odm\u00edt\u00e1 z\u00e1znamy, kter\u00e9 neodpov\u00eddaj\u00ed sch\u00e9matu tabulky. Podobn\u011b jako hosteska na recepci v obl\u00edben\u00e9 restauraci, kter\u00e1 p\u0159ij\u00edm\u00e1 pouze s p\u0159edchoz\u00ed rezervac\u00ed, kontroluje, zda ka\u017ed\u00fd sloupec dat, kter\u00fd je vkl\u00e1d\u00e1n do tabulky, pat\u0159\u00ed do p\u0159\u00edslu\u0161n\u00e9ho seznamu o\u010dek\u00e1van\u00fdch sloupc\u016f (jin\u00fdmi slovy, zda pro ka\u017ed\u00fd z nich existuje \u201erezervace\u201c) a odm\u00edt\u00e1 jak\u00e9koli z\u00e1znamy se sloupci, kter\u00e9 nejsou na seznamu.<\/p>\n<h2>Jak funguje vynucen\u00ed sch\u00e9matu?<\/h2>\n<p>\nDelta Lake pou\u017e\u00edv\u00e1 kontrolu sch\u00e9matu p\u0159i z\u00e1pisu, co\u017e znamen\u00e1, \u017ee v\u0161echny nov\u00e9 z\u00e1znamy do tabulky jsou b\u011bhem z\u00e1pisu kontrolov\u00e1ny na shodu se sch\u00e9matem c\u00edlov\u00e9 tabulky. Pokud nen\u00ed sch\u00e9ma kompatibiln\u00ed, Delta Lake transakci zcela zru\u0161\u00ed (data nejsou zaps\u00e1na) a vyvol\u00e1 v\u00fdjimku, aby u\u017eivatele informoval o nesouladu.<br \/>\nPro ur\u010den\u00ed kompatibility z\u00e1znamu s tabulkou Delta Lake pou\u017e\u00edv\u00e1 n\u00e1sleduj\u00edc\u00ed pravidla. Zapisovan\u00fd DataFrame:<\/p>\n<ul>\n<li>nesm\u00ed obsahovat dal\u0161\u00ed sloupce, kter\u00e9 nejsou ve sch\u00e9matu c\u00edlov\u00e9 tabulky. Naopak, pokud vstupn\u00ed data neobsahuj\u00ed naprosto v\u0161echny sloupce z tabulky, je to v po\u0159\u00e1dku \u2014 t\u011bmto sloupc\u016fm budou jednodu\u0161e p\u0159i\u0159azeny nulov\u00e9 hodnoty.<\/li>\n<li>nesm\u00ed m\u00edt datov\u00e9 typy sloupc\u016f, kter\u00e9 se li\u0161\u00ed od typ\u016f dat sloupc\u016f v c\u00edlov\u00e9 tabulce. Pokud sloupec c\u00edlov\u00e9 tabulky obsahuje data typu StringType, ale odpov\u00eddaj\u00edc\u00ed sloupec v DataFrame obsahuje data typu IntegerType, vynucen\u00ed sch\u00e9matu vyvol\u00e1 v\u00fdjimku a zabr\u00e1n\u00ed proveden\u00ed operace z\u00e1pisu.<\/li>\n<li>ne m\u016f\u017ee obsahovat n\u00e1zvy sloupc\u016f, kter\u00e9 se li\u0161\u00ed pouze v velikosti p\u00edsmen. To znamen\u00e1, \u017ee nem\u016f\u017eete m\u00edt sloupce s n\u00e1zvy 'Foo' a 'foo' definovan\u00e9 v jedn\u00e9 tabulce. A\u010dkoliv lze Spark pou\u017e\u00edt v re\u017eimu citliv\u00e9m nebo necitliv\u00e9m na velikost p\u00edsmen (v\u00fdchoz\u00ed), Delta Lake zachov\u00e1v\u00e1 velikost p\u00edsmen, ale je necitliv\u00fd v r\u00e1mci ulo\u017een\u00ed sch\u00e9matu. Parquet je citliv\u00fd na velikost p\u00edsmen p\u0159i ukl\u00e1d\u00e1n\u00ed a vracen\u00ed informac\u00ed o sloupc\u00edch. Abychom se vyhnuli mo\u017en\u00fdm chyb\u00e1m, po\u0161kozen\u00ed dat nebo jejich ztr\u00e1t\u011b (s \u010d\u00edm\u017e jsme se osobn\u011b setkali v Databricks), rozhodli jsme se p\u0159idat toto omezen\u00ed.<\/li>\n<\/ul>\n<p>\nAbychom to ilustrovali, poj\u010fme se pod\u00edvat na to, co se d\u011bje v n\u00ed\u017ee uveden\u00e9m k\u00f3du, kdy\u017e se pokus\u00edme p\u0159idat n\u011bkter\u00e9 ned\u00e1vno vygenerovan\u00e9 sloupce do tabulky Delta Lake, kter\u00e1 je\u0161t\u011b nen\u00ed nastavena na jejich p\u0159ijet\u00ed.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\nM\u00edsto automatick\u00e9ho p\u0159id\u00e1v\u00e1n\u00ed nov\u00fdch sloupc\u016f Delta Lake prosazuje sch\u00e9ma a zastavuje z\u00e1pis. Abychom pomohli ur\u010dit, kter\u00fd sloupec (nebo jich v\u00edce) je p\u0159\u00ed\u010dinou nesouladu, Spark vyp\u00ed\u0161e ob\u011b sch\u00e9mata z trasov\u00e1n\u00ed z\u00e1sobn\u00edku pro porovn\u00e1n\u00ed.<\/p>\n<h2>Jak\u00fd je p\u0159\u00ednos vynucen\u00ed sch\u00e9matu?<\/h2>\n<p>\nProto\u017ee vynucen\u00e9 pou\u017eit\u00ed sch\u00e9matu p\u0159edstavuje pom\u011brn\u011b p\u0159\u00edsnou kontrolu, je skv\u011bl\u00fdm n\u00e1strojem pro zaji\u0161t\u011bn\u00ed \u010dist\u00e9ho, pln\u011b transformovan\u00e9ho souboru dat, kter\u00fd je p\u0159ipraven pro v\u00fdrobu nebo spot\u0159ebu. Obvykle se aplikuje na tabulky, kter\u00e9 p\u0159\u00edmo poskytuj\u00ed data:<\/p>\n<ul>\n<li>Algoritmy strojov\u00e9ho u\u010den\u00ed<\/li>\n<li>BI dashboardy<\/li>\n<li>Analytiku dat a vizualiza\u010dn\u00ed n\u00e1stroje<\/li>\n<li>Jak\u00fdkoli v\u00fdrobn\u00ed syst\u00e9m vy\u017eaduj\u00edc\u00ed p\u0159\u00edsn\u011b strukturovan\u00e9, p\u0159\u00edsn\u011b typizovan\u00e9 s\u00e9mantick\u00e9 sch\u00e9mata.<\/li>\n<\/ul>\n<p>\nAby p\u0159ipravili sv\u00e1 data na tuto z\u00e1v\u011bre\u010dnou p\u0159ek\u00e1\u017eku, mnoho u\u017eivatel\u016f pou\u017e\u00edv\u00e1 jednoduchou architekturu \u201emulti-hop\u201c, kter\u00e1 postupn\u011b vkl\u00e1d\u00e1 strukturu do jejich tabulek. Chcete-li se o tom dozv\u011bd\u011bt v\u00edce, m\u016f\u017eete si p\u0159e\u010d\u00edst \u010dl\u00e1nek <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Strojov\u00e9 u\u010den\u00ed na \u00farovni v\u00fdroby s Delta Lake.<\/a><\/noindex><\/p>\n<p>Samoz\u0159ejm\u011b, \u017ee vynucen\u00e9 pou\u017eit\u00ed sch\u00e9matu lze pou\u017e\u00edt kdekoli ve va\u0161em pipeline, ale m\u011bjte na pam\u011bti, \u017ee streamov\u00e1n\u00ed z\u00e1pisu do tabulky m\u016f\u017ee b\u00fdt frustruj\u00edc\u00ed, pokud jste nap\u0159\u00edklad zapomn\u011bli, \u017ee jste p\u0159idali dal\u0161\u00ed sloupec do p\u0159\u00edchoz\u00edch dat.<\/p>\n<h2>Prevence z\u0159ed\u011bn\u00ed dat<\/h2>\n<p>\nV tuto chv\u00edli se m\u016f\u017eete pt\u00e1t, pro\u010d je kolem toho takov\u00fd rozruch? Nakonec, ob\u010das m\u016f\u017ee nep\u0159edv\u00eddateln\u00e1 chyba \u201eneshodnosti sch\u00e9matu\u201c podrazit nohy va\u0161\u00ed pracovn\u00edmu procesu, zejm\u00e9na pokud jste nov\u00e1\u010dkem v Delta Lake. Pro\u010d prost\u011b nedovolit sch\u00e9matu zm\u011bnit se, jak pot\u0159ebuje, abych mohl svou datovou sadu zapsat, bez ohledu na okolnosti?<\/p>\n<p>Jak \u0159\u00edk\u00e1 star\u00e9 p\u0159\u00edslov\u00ed, \u201eunce prevence za kilogram l\u00e9\u010dby\u201c. V ur\u010dit\u00e9m okam\u017eiku, pokud se nestar\u00e1te o aplikaci sv\u00e9ho sch\u00e9matu, se mohou na povrch dostat probl\u00e9my s kompatibilitou typ\u016f dat \u2014 na prvn\u00ed pohled homogenn\u00ed zdroje surov\u00fdch dat mohou obsahovat okrajov\u00e9 p\u0159\u00edpady, po\u0161kozen\u00e9 sloupce, nespr\u00e1vn\u011b utvo\u0159en\u00e9 mapov\u00e1n\u00ed nebo jin\u00e9 hr\u016fzy, kter\u00e9 se v\u00e1m sn\u00ed o no\u010dn\u00edch m\u016fr\u00e1ch. Nejlep\u0161\u00ed p\u0159\u00edstup je zastavit tyto nep\u0159\u00e1tele u bran \u2014 pomoc\u00ed vynucen\u00e9ho pou\u017eit\u00ed sch\u00e9matu \u2014 a vy\u0159e\u0161it je na sv\u011btle, nam\u00edsto pozd\u011bji, kdy\u017e za\u010dnou prohl\u00ed\u017eet ve tm\u011b va\u0161eho pracovn\u00edho k\u00f3du.<\/p>\n<p>Povinn\u00e9 prosazen\u00ed sch\u00e9matu zaji\u0161\u0165uje, \u017ee sch\u00e9ma va\u0161\u00ed tabulky se nezm\u011bn\u00ed, pokud sami nepotvrd\u00edte variantu zm\u011bny. Zabra\u0148uje \u201ez\u0159ed\u011bn\u00ed\u201c (dilution) dat, kter\u00e9 m\u016f\u017ee nastat, kdy\u017e jsou nov\u00e9 sloupce p\u0159id\u00e1v\u00e1ny tak \u010dasto, \u017ee d\u0159\u00edve hodnotn\u00e9, komprimovan\u00e9 tabulky ztr\u00e1cej\u00ed svou hodnotu a u\u017eite\u010dnost v d\u016fsledku zaplaven\u00ed daty. T\u00edm, \u017ee v\u00e1s povzbud\u00ed k tomu, abyste byli z\u00e1m\u011brn\u00ed, nastavili vysok\u00e9 standardy a o\u010dek\u00e1vali vysokou kvalitu, povinn\u00e9 prosazen\u00ed sch\u00e9matu d\u011bl\u00e1 p\u0159esn\u011b to, k \u010demu bylo ur\u010deno \u2013 pom\u00e1h\u00e1 v\u00e1m z\u016fstat sv\u011bdomit\u00ed a va\u0161im tabulk\u00e1m z\u016fst\u00e1vat \u010dist\u00e9.<\/p>\n<p>Pokud p\u0159i dal\u0161\u00edm zva\u017eov\u00e1n\u00ed zjist\u00edte, \u017ee vlastn\u011b <i>je t\u0159eba<\/i> chcete p\u0159idat nov\u00fd sloupec \u2013 \u017e\u00e1dn\u00fd probl\u00e9m, n\u00ed\u017ee je jednor\u00e1zov\u00e9 \u0159e\u0161en\u00ed. \u0158e\u0161en\u00ed \u2013 evoluce sch\u00e9matu!<\/p>\n<h2>Co je evoluce sch\u00e9matu?<\/h2>\n<p>\nEvoluce sch\u00e9matu je funkce, kter\u00e1 u\u017eivatel\u016fm umo\u017e\u0148uje snadno m\u011bnit aktu\u00e1ln\u00ed sch\u00e9ma tabulky podle dat, kter\u00e1 se v pr\u016fb\u011bhu \u010dasu m\u011bn\u00ed. Nej\u010dast\u011bji se pou\u017e\u00edv\u00e1 p\u0159i prov\u00e1d\u011bn\u00ed operace p\u0159id\u00e1n\u00ed nebo p\u0159eps\u00e1n\u00ed, aby automaticky p\u0159izp\u016fsobila sch\u00e9ma tak, aby zahrnovalo jeden nebo v\u00edce nov\u00fdch sloupc\u016f.<\/p>\n<h2>Jak funguje evoluce sch\u00e9matu?<\/h2>\n<p>\nN\u00e1sleduj\u00edc\u00edm p\u0159\u00edkladem z p\u0159edchoz\u00edho odd\u00edlu mohou v\u00fdvoj\u00e1\u0159i snadno pou\u017e\u00edt evoluci sch\u00e9matu k p\u0159id\u00e1n\u00ed nov\u00fdch sloupc\u016f, kter\u00e9 byly d\u0159\u00edve zam\u00edtnuty kv\u016fli nesouladu se sch\u00e9matem. Evoluce sch\u00e9matu se aktivuje p\u0159id\u00e1n\u00edm <code>.option('mergeSchema', 'true')<\/code> k va\u0161emu Spark p\u0159\u00edkazu <code>.write nebo .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nChcete-li zobrazit graf, prove\u010fte n\u00e1sleduj\u00edc\u00ed Spark SQL dotaz<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Pohled na Delta Lake: nucen\u00e9 aplikace a evoluce sch\u00e9matu\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAlternativn\u011b m\u016f\u017eete tento parametr nastavit pro celou Spark relaci p\u0159id\u00e1n\u00edm <code>spark.databricks.delta.schema.autoMerge = True<\/code> do konfigurace Spark. Ale bu\u010fte opatrn\u00ed, proto\u017ee povinn\u00e9 prosazen\u00ed sch\u00e9matu v\u00e1s ji\u017e nebude varovat p\u0159ed ne\u00famysln\u00fdmi nesoulady se sch\u00e9matem.<\/p>\n<p>P\u0159id\u00e1n\u00edm parametru <code>mergeSchema<\/code>, v\u0161echny sloupce, kter\u00e9 jsou p\u0159\u00edtomny v DataFrame, ale chyb\u00ed v c\u00edlov\u00e9 tabulce, se automaticky p\u0159id\u00e1vaj\u00ed na konec sch\u00e9matu v r\u00e1mci transakce z\u00e1pisu. Tak\u00e9 mohou b\u00fdt p\u0159id\u00e1na vno\u0159en\u00e1 pole, a tak\u00e9 budou p\u0159id\u00e1na na konec odpov\u00eddaj\u00edc\u00edch sloupc\u016f struktury.<\/p>\n<p>In\u017een\u00fd\u0159i a v\u011bdci mohou tuto mo\u017enost vyu\u017e\u00edt k p\u0159id\u00e1n\u00ed nov\u00fdch sloupc\u016f (nap\u0159. ned\u00e1vno sledovan\u00e9 metriky nebo sloupec pro prodeje za tento m\u011bs\u00edc) do sv\u00fdch st\u00e1vaj\u00edc\u00edch v\u00fdrobn\u00edch tabulek strojov\u00e9ho u\u010den\u00ed, ani\u017e by do\u0161lo k naru\u0161en\u00ed st\u00e1vaj\u00edc\u00edch model\u016f zalo\u017een\u00fdch na star\u00fdch sloupc\u00edch. <\/p>\n<p>N\u00e1sleduj\u00edc\u00ed typy zm\u011bn sch\u00e9matu jsou povoleny v r\u00e1mci evoluce sch\u00e9matu p\u0159i p\u0159id\u00e1v\u00e1n\u00ed nebo p\u0159episov\u00e1n\u00ed tabulky:<\/p>\n<ul>\n<li>P\u0159id\u00e1n\u00ed nov\u00fdch sloupc\u016f (to je nejb\u011b\u017en\u011bj\u0161\u00ed sc\u00e9n\u00e1\u0159)<\/li>\n<li>Zm\u011bna datov\u00fdch typ\u016f z NullType -&gt; jak\u00fdkoli jin\u00fd typ nebo zv\u00fd\u0161en\u00ed z ByteType -&gt; ShortType -&gt; IntegerType<\/li>\n<\/ul>\n<p>\nDal\u0161\u00ed zm\u011bny, kter\u00e9 nejsou povoleny v r\u00e1mci evoluce sch\u00e9matu, vy\u017eaduj\u00ed, aby bylo sch\u00e9ma a data p\u0159eps\u00e1na p\u0159id\u00e1n\u00edm <code>.option(&quot;overwriteSchema&quot;, &quot;true&quot;)<\/code>. Nap\u0159\u00edklad pokud byl sloupec \"Foo\" p\u016fvodn\u011b integer, a nov\u00e9 sch\u00e9ma by bylo typu String, pak by v\u0161echny soubory Parquet (data) musely b\u00fdt p\u0159eps\u00e1ny. Mezi tyto zm\u011bny pat\u0159\u00ed:<\/p>\n<ul>\n<li>odstran\u011bn\u00ed sloupce<\/li>\n<li>zm\u011bna datov\u00e9ho typu existuj\u00edc\u00edho sloupce (na m\u00edst\u011b)<\/li>\n<li>p\u0159ejmenov\u00e1n\u00ed sloupc\u016f, kter\u00e9 se li\u0161\u00ed pouze velikost\u00ed p\u00edsmen (nap\u0159. \"Foo\" a \"foo\")<\/li>\n<\/ul>\n<p>\nNakonec, s n\u00e1sleduj\u00edc\u00ed verz\u00ed Spark 3.0 bude pln\u011b podporov\u00e1na explicitn\u00ed DDL (s vyu\u017eit\u00edm ALTER TABLE), co\u017e umo\u017en\u00ed u\u017eivatel\u016fm prov\u00e1d\u011bt n\u00e1sleduj\u00edc\u00ed akce nad sch\u00e9maty tabulek:<\/p>\n<ul>\n<li>p\u0159id\u00e1n\u00ed sloupc\u016f<\/li>\n<li>zm\u011bna koment\u00e1\u0159\u016f k sloupc\u016fm<\/li>\n<li>nastaven\u00ed vlastnost\u00ed tabulky, kter\u00e9 ur\u010duj\u00ed chov\u00e1n\u00ed tabulky, nap\u0159\u00edklad nastaven\u00ed doby uchov\u00e1v\u00e1n\u00ed transak\u010dn\u00edho protokolu.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Jak\u00e9 jsou v\u00fdhody evoluce sch\u00e9matu?<\/h2>\n<p>\nEvoluci sch\u00e9matu m\u016f\u017eete vyu\u017e\u00edt v\u017edy, kdy\u017e <i>pl\u00e1nujete<\/i> zm\u011bnit sch\u00e9ma sv\u00e9 tabulky (na rozd\u00edl od p\u0159\u00edpad\u016f, kdy jste omylem p\u0159idali do sv\u00e9ho DataFrame sloupce, kter\u00e9 by tam nem\u011bly b\u00fdt). To je nejjednodu\u0161\u0161\u00ed zp\u016fsob, jak migrovat va\u0161e sch\u00e9ma, proto\u017ee automaticky p\u0159id\u00e1v\u00e1 spr\u00e1vn\u00e9 n\u00e1zvy sloupc\u016f a datov\u00e9 typy bez nutnosti jejich explicitn\u00edho vyhla\u0161ov\u00e1n\u00ed.<\/p>\n<h2>Z\u00e1v\u011br<\/h2>\n<p>\nPovinn\u00e9 pou\u017eit\u00ed sch\u00e9matu odm\u00edt\u00e1 jak\u00e9koli nov\u00e9 sloupce nebo jin\u00e9 zm\u011bny sch\u00e9matu, kter\u00e9 nejsou kompatibiln\u00ed s va\u0161\u00ed tabulkou. Stanoven\u00edm a dodr\u017eov\u00e1n\u00edm t\u011bchto vysok\u00fdch standard\u016f se analytici a in\u017een\u00fd\u0159i mohou spolehnout na to, \u017ee jejich data maj\u00ed nejvy\u0161\u0161\u00ed \u00farove\u0148 integrity, p\u0159i\u010dem\u017e jasn\u011b formuluj\u00ed sv\u00e9 my\u0161lenky, co\u017e jim umo\u017e\u0148uje p\u0159ij\u00edmat efektivn\u011bj\u0161\u00ed obchodn\u00ed rozhodnut\u00ed.<\/p>\n<p>Na druh\u00e9 stran\u011b evoluce sch\u00e9matu dopl\u0148uje povinn\u00e9 pou\u017eit\u00ed, zjednodu\u0161uj\u00edc\u00ed <i>p\u0159edpokl\u00e1dan\u00e9<\/i> automatick\u00e9 zm\u011bny sch\u00e9matu. Nakonec by to nem\u011blo b\u00fdt komplikovan\u00e9 \u2013 p\u0159idat sloupec.<\/p>\n<p>Povinn\u00e9 pou\u017eit\u00ed sch\u00e9matu je jang, zat\u00edmco evoluce sch\u00e9matu je jin. Spole\u010dn\u011b tyto funkce usnad\u0148uj\u00ed potla\u010den\u00ed \u0161umu a lad\u011bn\u00ed sign\u00e1lu jako nikdy p\u0159edt\u00edm.<\/p>\n<p><i>R\u00e1di bychom tak\u00e9 pod\u011bkovali Mukulovi Murti a Pranavovi Anandovi za jejich p\u0159\u00edsp\u011bvek k tomuto \u010dl\u00e1nku.<\/i><\/p>\n<p>Dal\u0161\u00ed \u010dl\u00e1nky v t\u00e9to s\u00e9rii:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Pono\u0159en\u00ed do Delta Lake: rozbalen\u00ed transak\u010dn\u00edho den\u00edku<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"P\u0159ehr\u00e1t video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>\u010cl\u00e1nky na t\u00e9ma<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Strojov\u00e9 u\u010den\u00ed na podnikov\u00e9 \u00farovni s Delta Lake<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">Co je datov\u00e9 jezero?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">Zjistit v\u00edce o kurzu<\/a><\/noindex><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Pono\u0159en\u00ed do Delta Lake: povinn\u00e9 pou\u017eit\u00ed a evoluce sch\u00e9matu | ProHoster","description":"Ahoj, Habr!","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}