{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"Sissejuhatus Delta Lake'i: sundrakendamine ja skeemi evolutsioon","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Tere, Habr! Esitan teile artikli t\u00f5lke <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\u00abSukeldumine Delta Lake'i: skeemi j\u00f5ud ja areng\u00bb<\/a><\/noindex> autorid Burak Yavuz, Brenner Heintz ja Denny Lee, mis koostati kursuse alguse eel <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\u201eAndmeinsener\u201c<\/a><\/noindex> OTUSelt.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Sissejuhatus Delta Lake&#039;i: sundrakendamine ja skeemi evolutsioon\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Andmed, nagu ka meie kogemus, kogunevad ja arenevad pidevalt. Et mitte mahaj\u00e4\u00e4da, peavad meie vaimsed mudelid maailmast kohanduma uute andmetega, millest m\u00f5ned h\u00f5lmavad uusi m\u00f5\u00f5tmeid - uusi viise t\u00e4helepanekute tegemiseks asjadest, millest varem polnud meil aimugi. Need vaimsed mudelid ei erine eriti tabeliskeemidest, mis m\u00e4\u00e4ratlevad, kuidas me klassifitseerime ja t\u00f6\u00f6tleme uut teavet.<\/p>\n<p>See viib meid skeemihalduse k\u00fcsimuse juurde. Aja m\u00f6\u00f6dudes muutuvad \u00e4ri\u00fclesanded ja n\u00f5uded, millega muutub ka teie andmete struktuur. Delta Lake v\u00f5imaldab uute m\u00f5\u00f5tmete h\u00f5lpsat rakendamist andmete muutmisel. Kasutajatel on lihtne semantika oma tabelite skeemide haldamiseks. Need t\u00f6\u00f6riistad h\u00f5lmavad skeemi sundrakendamist (Schema Enforcement), mis kaitseb kasutajaid nende tabelite ekslikest v\u00f5i mittevajalikest andmetest p\u00f5hjustatud saastumise eest, ning skeemi arengut (Schema Evolution), mis v\u00f5imaldab automaatselt lisada uusi veerge v\u00e4\u00e4rtuslike andmetega asjakohastesse kohtadesse. Selles artiklis s\u00fcveneme nende t\u00f6\u00f6riistade kasutamisse.<\/p>\n<h2>Tabeli skeemide m\u00f5istmine<\/h2>\n<p>\nIga DataFrame Apache Sparkis sisaldab skeemi, mis m\u00e4\u00e4ratleb andmete kuju, n\u00e4iteks andmet\u00fc\u00fcbid, veerud ja metaandmed. Delta Lake'i puhul salvestatakse tabeli skeem JSON-formaadis tehingute ajaloos.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Mis on skeemi sundrakendamine?<\/h2>\n<p>\nSchema Enforcement, tuntud ka kui skeemi valideerimine, on kaitsemehhanism Delta Lake'is, mis tagab andmete kvaliteedi, t\u00f5rjudes salvestusi, mis ei vasta tabeli skeemile. Nagu hostes populaarses restoranis, kes v\u00f5tab vastu ainult eelnevalt broneeritud, kontrollib see, kas iga tabelisse sisestatava andmeveeru puhul on olemas vastav oodatud veergude nimekiri (teisis\u00f5nu, kas sellel on iga\u00fche jaoks \u00abbroneering\u00bb), ja t\u00f5rjub k\u00f5ik salvestused, kus veerge, mis nimekirjas ei ole.<\/p>\n<h2>Kuidas schema enforcement t\u00f6\u00f6tab?<\/h2>\n<p>\nDelta Lake kasutab skeemi kontrollimist kirje kirjutamisel, mis t\u00e4hendab, et k\u00f5ik uued salvestused tabelisse kontrollitakse sihttabeli skeemiga koosk\u00f5las olemise osas kirjutamise ajal. Kui skeem ei ole \u00fchilduv, t\u00fchistab Delta Lake t\u00e4ielikult tehingu (andmeid ei kirjutata) ja tekitab erandi, et teavitada kasutajat vastuolust.<br \/>\nDelta Lake kasutab kirje \u00fchilduvuse m\u00e4\u00e4ramiseks j\u00e4rgmisi reegleid. Salvestatav DataFrame:<\/p>\n<ul>\n<li>ei saa sisaldada t\u00e4iendavaid veerge, mida sihttabeli skeem ei h\u00f5lma. Ja vastupidi, k\u00f5ik on korras, kui sisendandmed ei sisalda absoluutselt k\u00f5iki tabeli veerge \u2014 need veerud saavad lihtsalt nullv\u00e4\u00e4rtused.<\/li>\n<li>ei saa omada veergude andmet\u00fc\u00fcpe, mis erinevad sihttabeli veergude andmet\u00fc\u00fcpidest. Kui sihttabeli veerg sisaldab StringType andmeid, kuid vastav veerg DataFrame'is sisaldab IntegerType andmeid, p\u00f5hjustab skeemi sundrakendamine erandi ja takistab kirjutamisoperatsiooni t\u00e4itmist.<\/li>\n<li>\u043d\u0435 \u043c\u043e\u0436\u0435\u0442 \u0441\u043e\u0434\u0435\u0440\u0436\u0430\u0442\u044c \u0438\u043c\u0435\u043d\u0430 \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u0432, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043e\u0442\u043b\u0438\u0447\u0430\u044e\u0442\u0441\u044f \u0442\u043e\u043b\u044c\u043a\u043e \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u043e\u043c. \u042d\u0442\u043e \u0437\u043d\u0430\u0447\u0438\u0442, \u0447\u0442\u043e \u0432\u044b \u043d\u0435 \u043c\u043e\u0436\u0435\u0442\u0435 \u0438\u043c\u0435\u0442\u044c \u0441\u0442\u043e\u043b\u0431\u0446\u044b \u0441 \u0438\u043c\u0435\u043d\u0430\u043c\u0438 &#8216;Foo&#8217; \u0438 &#8216;foo&#8217;, \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u043d\u044b\u0435 \u0432 \u043e\u0434\u043d\u043e\u0439 \u0442\u0430\u0431\u043b\u0438\u0446\u0435. \u0425\u043e\u0442\u044f Spark \u043c\u043e\u0436\u043d\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0432 \u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u043c \u0438\u043b\u0438 \u043d\u0435\u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u043c (\u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e) \u043a \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0443 \u0440\u0435\u0436\u0438\u043c\u0435, Delta Lake \u0441\u043e\u0445\u0440\u0430\u043d\u044f\u0435\u0442 \u0440\u0435\u0433\u0438\u0441\u0442\u0440, \u043d\u043e \u043d\u0435\u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u0435\u043d \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0438 \u0441\u0445\u0435\u043c\u044b. Parquet \u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u0435\u043d \u043a \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0443 \u043f\u0440\u0438 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0438 \u0438 \u0432\u043e\u0437\u0432\u0440\u0430\u0442\u0435 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u0438 \u0441\u0442\u043e\u043b\u0431\u0446\u0430. \u0427\u0442\u043e\u0431\u044b \u0438\u0437\u0431\u0435\u0436\u0430\u0442\u044c \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u044b\u0445 \u043e\u0448\u0438\u0431\u043e\u043a, \u043f\u043e\u0432\u0440\u0435\u0436\u0434\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u043b\u0438 \u0438\u0445 \u043f\u043e\u0442\u0435\u0440\u0438 (\u0441 \u0447\u0435\u043c \u043c\u044b \u043b\u0438\u0447\u043d\u043e \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u043b\u0438\u0441\u044c \u0432 Databricks), \u043c\u044b \u0440\u0435\u0448\u0438\u043b\u0438 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u044d\u0442\u043e \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u0438\u0435.<\/li>\n<\/ul>\n<p>\nKuna illustreerida, vaatame, mis toimub allolevas koodis, kui proovime lisada m\u00f5ned hiljuti genereeritud veerud Delta Lake tabelisse, mis pole nende vastuv\u00f5tmiseks veel seadistatud.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\nUute veergude automaatse lisamise asemel kehtestab Delta Lake skeemi ja peatab kirje. Et aidata v\u00e4lja selgitada, milline veerg (v\u00f5i mitu) p\u00f5hjustab ebak\u00f5la, v\u00e4ljastab Spark v\u00f5rreldamiseks m\u00f5lema skeemi stack trace'i.<\/p>\n<h2>Milline on skeemi sundrakendamise kasu?<\/h2>\n<p>\nKuna skeemi sundrakendamine on piisavalt ranged kontroll, on see suurep\u00e4rane t\u00f6\u00f6riist, mida kasutada puhta, t\u00e4ielikult muudetud andmestiku v\u00e4ravas, mis on valmis tootmiseks v\u00f5i tarbimiseks. Tavaliselt rakendatakse seda tabelitele, mis otse t\u00f5id andmeid:<\/p>\n<ul>\n<li>Masin\u00f5ppe algoritmid<\/li>\n<li>BI juhtpaneelid<\/li>\n<li>Andmeanal\u00fc\u00fcs ja visualiseerimist\u00f6\u00f6riistad<\/li>\n<li>Mis tahes tootmiss\u00fcsteem, mis vajab rangelt struktureeritud, rangelt t\u00fcpiseeritud semantilisi skeeme.<\/li>\n<\/ul>\n<p>\nEttevalmistamiseks oma andmeid sellele viimasele barj\u00e4\u00e4ri paljud kasutajad rakendavad lihtsat \"multi-hop\" arhitektuuri, mis j\u00e4rk-j\u00e4rgult lisab struktuuri nende tabelitesse. Lisainformatsiooni saamiseks v\u00f5ite tutvuda artikliga <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Tootmiselt masin\u00f5pe Delta Lake'iga.<\/a><\/noindex><\/p>\n<p>Muidugi, sunnitud skeemi rakendamist saab kasutada igal kohal teie andmestikus, kuid pidage meeles, et voogesitus tabelisse sellisel juhul v\u00f5ib olla t\u00fclikas, kuna n\u00e4iteks te unustasite, et lisasite veel \u00fche veeru sisendandmetesse.<\/p>\n<h2>Andmete lahjendamise ennetamine<\/h2>\n<p>\nSelle hetke seisuga v\u00f5ite k\u00fcsida, miks selline elevus? L\u00f5ppude l\u00f5puks v\u00f5ib m\u00f5nikord ootamatu \"skeemi mittevastavuse\" viga segada teie t\u00f6\u00f6protsessi, eriti kui olete Delta Lake'iga uus. Miks mitte lihtsalt lubada skeemil muutuda nii, nagu on vajalik, et ma saaksin m\u00f5lemad salvestada oma DataFrame\u2019i, hoolimata k\u00f5igest?<\/p>\n<p>Vanalt \u00f6eldakse, et \u00abtass ennetust maksab naela ravi\u00bb. \u00dchel hetkel, kui te ei hooli oma skeemi rakendamisest, t\u00f5usevad andmet\u00fc\u00fcpidega seotud probleemid \u2014 esmapilgul \u00fchtlaselt n\u00e4ivad toorandmeallikad v\u00f5ivad sisaldada \u00e4\u00e4rmuslikke juhtumeid, kahjustatud veerge, valesti vormindatud seoseid v\u00f5i muid kohutavaid asju, mis k\u00fclastavad \u00f5udusunen\u00e4gusid. Parim l\u00e4henemine on peatada need vaenlased v\u00e4ravas \u2014 sundides skeemi rakendama \u2014 ja tegeleda nendega p\u00e4evavalgele, mitte hiljem, kui nad hakkavad hiilima teie t\u00f6\u00f6koodeksi pimedatesse s\u00fcgavustesse.<\/p>\n<p>Skeemi sundrakendamine tagab, et teie tabeli struktuur ei muutu, kui te ise ei kinnita muudatust. See takistab andmete 'lahjendamist', mis v\u00f5ib toimuda, kui uusi veerge lisatakse liiga tihti, mist\u00f5ttu varem v\u00e4\u00e4rtuslikud, kokkusurutud tabelid kaotavad oma v\u00e4\u00e4rtuse ja kasulikkuse andmemere t\u00f5ttu. Sundrakendamine julgustab teid olema teadlik, seadma k\u00f5rgeid standardeid ja ootama k\u00f5rge kvaliteediga, tehes seda t\u00e4pselt selleks, milleks see oli m\u00f5eldud \u2014 aitama teil j\u00e4\u00e4da ausaks ja teie tabelitel puhtaks.<\/p>\n<p>Kui edasise arutelu k\u00e4igus otsustate, et soovite tegelikult <i>vaja<\/i> lisada uue veeru \u2014 pole probleemi, allpool on \u00fche rea lahendus. Lahendus on skeemi evolutsioon!<\/p>\n<h2>Mis on skeemi evolutsioon?<\/h2>\n<p>\nSch\u00e9made evolutsioon on funktsioon, mis v\u00f5imaldab kasutajatel kergesti muuta tabeli praegust skeemi vastavalt andmetele, mis aja jooksul muutuvad. Seda kasutatakse k\u00f5ige sagedamini lisamise v\u00f5i \u00fcle kirjutamise operatsiooni k\u00e4igus, et skeemi automaatselt kohandada, et lisada \u00fcks v\u00f5i mitu uut veergu.<\/p>\n<h2>Kuidas sch\u00e9made evolutsioon t\u00f6\u00f6tab?<\/h2>\n<p>\nJ\u00e4tkates eelnevas jaotises toodud n\u00e4idet, saavad arendajad kergesti kasutada sch\u00e9made evolutsiooni uute veergude lisamiseks, mis olid varem skeemiga kokkusobimatuse t\u00f5ttu tagasi l\u00fckatud. Sch\u00e9made evolutsioon aktiveeritakse, lisades <code>.option('mergeSchema', 'true')<\/code> teie Spark komandole <code>.write v\u00f5i .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nGraafiku vaatamiseks tehke j\u00e4rgmine Spark SQL p\u00e4ring<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Sissejuhatus Delta Lake&#039;i: sundrakendamine ja skeemi evolutsioon\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAlternatiivselt saate selle valiku seada kogu Spark seansi jaoks, lisades <code>spark.databricks.delta.schema.autoMerge = True<\/code> Spark konfiguratsioonile. Kuid kasutage seda ettevaatlikult, kuna skeemi sundrakendamine ei hoiatada teid enam etten\u00e4gematute skeemide vastuolude eest.<\/p>\n<p>Kuna p\u00e4ringus on parameeter <code>mergeSchema<\/code>, k\u00f5ik DataFrame'is olemasolevad, kuid sihttabelis puuduvad veerud lisatakse automaatselt skeemi l\u00f5ppu kirjutamise tehingu k\u00e4igus. Samuti v\u00f5ivad olla lisatud sisemised v\u00e4ljad, ja need lisatakse ka vastavate veergude struktuuri l\u00f5ppu.<\/p>\n<p>Kuup\u00e4evainsenerid ja teadlased saavad kasutada seda valikut, et lisada uusi veerge (v\u00f5imalik uudse j\u00e4lgitav m\u00f5\u00f5dik v\u00f5i m\u00fc\u00fcgim\u00e4\u00e4rad selle kuu veerg) oma olemasolevatesse masin\u00f5ppe tootmistabelitesse, ilma et varasemate veergude p\u00f5hjal p\u00f5hinevad mudelid oleksid katki. <\/p>\n<p>Allj\u00e4rgnevad skeemi muudatuste t\u00fc\u00fcbid on lubatud skeemi evolutsiooni k\u00e4igus tabeli lisamisel v\u00f5i kirjutamisel:<\/p>\n<ul>\n<li>Uute veergude lisamine (see on k\u00f5ige levinum stsenaarium)<\/li>\n<li>Andmet\u00fc\u00fcbi muutmine NullType'ist -&gt; mis tahes muusse tippu v\u00f5i t\u00f5stmine ByteType'ist -&gt; ShortType'ist -&gt; IntegerType'ist<\/li>\n<\/ul>\n<p>\nMuud muudatused, mis ei ole skeemi evolutsiooni raames lubatud, n\u00f5uavad, et skeem ja andmed tuleb \u00fcmber kirjutada, lisades <code>.option(&quot;overwriteSchema&quot;, &quot;true&quot;)<\/code>. N\u00e4iteks juhul, kui veerg \u201eFoo\u201d oli algselt t\u00e4isarv ja uus skeem oleks olnud stringi andmet\u00fc\u00fcp, oleks k\u00f5iki Parquet-faile (andmed) pidanud \u00fcle kirjutama. Selliste muutuste hulka kuuluvad:<\/p>\n<ul>\n<li>veeru eemaldamine<\/li>\n<li>olemasoleva veeru andmet\u00fc\u00fcbi muutmine (kohapeal)<\/li>\n<li>veergude \u00fcmbernimetamine, mis erinevad vaid suur- ja v\u00e4iket\u00e4htede poolest (n\u00e4iteks \u201eFoo\u201d ja \u201efoo\u201d)<\/li>\n<\/ul>\n<p>\nL\u00f5puks toetab j\u00e4rgmise Spark 3.0 v\u00e4ljaandega t\u00e4ielikult selges DDL-i (kasutades ALTER TABLE), mis v\u00f5imaldab kasutajatel teostada j\u00e4rgmisi toiminguid tabelite skeemidega:<\/p>\n<ul>\n<li>veergude lisamine<\/li>\n<li>veergude kommentaaride muutmine<\/li>\n<li>tabeli omaduste kohandamine, mis m\u00e4\u00e4ravad tabeli k\u00e4itumise, n\u00e4iteks tehinguajakava s\u00e4ilivuse m\u00e4\u00e4ramine.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Milline on skeemi arengu eelised?<\/h2>\n<p>\nSkeemi arendamist saab kasutada alati, kui te <i>kavatsete<\/i> muuda oma tabeli skeemi (vastupidiselt juhtumitele, kus olete kogemata lisanud oma DataFrame'i veerge, mida seal ei tohiks olla). See on k\u00f5ige lihtsam viis oma skeemi migreerimiseks, kuna see lisab automaatselt \u00f5iged veerunimed ja andmet\u00fc\u00fcbid ilma nende selges\u00f5nalise kuulutamiseta.<\/p>\n<h2>Kokkuv\u00f5te<\/h2>\n<p>\nSkeemi sundrakendamine l\u00fckkab tagasi k\u00f5ik uued veerud v\u00f5i muud skeemi muudatused, mis ei \u00fchti teie tabeliga. Seades ja s\u00e4ilitades need k\u00f5rged standardid, saavad anal\u00fc\u00fctikud ja insenerid tugineda oma andmete k\u00f5rgeimale kvaliteedile, m\u00f5eldes sellele selgelt ja arusaadavalt, mis v\u00f5imaldab neil teha efektiivsemaid \u00e4riotsuseid.<\/p>\n<p>Teisest k\u00fcljest t\u00e4iendab skeemi evolutsioon sundrakendamist, lihtsustades <i>eeldatavad<\/i> automaatseid skeemi muudatusi. L\u00f5ppude l\u00f5puks ei tohiks see olla keeruline \u2014 lisada veerg.<\/p>\n<p>Skeemi sundrakendamine on j\u00e4nes, kus skeemi evolutsioon on yin. Koos kasutamisel lihtsustavad need funktsioonid nagu kunagi varem m\u00fcra allasurumist ja signaali seadmist.<\/p>\n<p><i>Soovime samuti t\u00e4nada Mukula Murti ja Pranava Ananda nende panuse eest sellesse artiklisse.<\/i><\/p>\n<p>Teised artiklid sellest sarjast:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Sukeldumine Delta Lake'i: tehingu ajalugu avamine<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"Vaata videot\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>Seotud artiklid<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Tootmisastme masin\u00f5pe Delta Lake'i abil<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">Mis on andmej\u00e4rv?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">Tutvu kursusega l\u00e4hemalt<\/a><\/noindex><\/p>\n<p>Allikas: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c,\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"et_EE\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c,\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47 Sukeldumine Delta Lake'i: skeemi kohustuslik rakendamine ja evolutsioon | ProHoster","description":"Tere, Habr! Esitan teile t\u00f5lgitud artikli \"Diving Into Delta Lake: Schema Enforcement &amp; Evolution\" autoritelt Burak Yavuz, Brenner Heintz ja Denny Lee, mis on koostatud OTUS-i \"Data Engineer\" kursuse alguse eel. Andmed, nagu ka meie kogemused, pidevalt kogunevad ja arenevad. Et mitte maha j\u00e4\u00e4da, peavad meie vaimsed mudelid maailmast kohanema uute andmetega.","canonical_url":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"et_EE","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c,","og:url":"https:\/\/prohoster.info\/et\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45"},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}