{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"\u00cencercare \u00een Delta Lake: aplicarea for\u021bat\u0103 \u0219i evolu\u021bia schemei","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Salut, Habr! V\u0103 prezint traducerea articolului <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\u201eExplor\u00e2nd Delta Lake: Impunerea \u0219i Evolu\u021bia Schemei\u201c<\/a><\/noindex> autorii Burak Yavuz, Brenner Heintz \u0219i Denny Lee, care a fost preg\u0103tit \u00eenainte de \u00eenceperea cursului <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\u201eInginer de date\u201d<\/a><\/noindex> de la OTUS.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"\u00cencercare \u00een Delta Lake: aplicarea for\u021bat\u0103 \u0219i evolu\u021bia schemei\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Datele, la fel ca experien\u021ba noastr\u0103, se acumuleaz\u0103 \u0219i evolueaz\u0103 constant. Pentru a nu r\u0103m\u00e2ne \u00een urm\u0103, modelele noastre mentale despre lume trebuie s\u0103 se adapteze la noi date, unele dintre ele con\u021bin\u00e2nd noi dimensiuni - noi modalit\u0103\u021bi de a observa lucruri despre care anterior nu aveam cuno\u0219tin\u021b\u0103. Aceste modele mentale nu sunt foarte diferite de schemele de tabele, care determin\u0103 modul \u00een care clasific\u0103m \u0219i proces\u0103m noua informa\u021bie.<\/p>\n<p>Aceasta ne conduce la problema gestion\u0103rii schemelor. Pe m\u0103sur\u0103 ce cerin\u021bele \u0219i sarcinile de afaceri se schimb\u0103 \u00een timp, structura datelor tale se schimb\u0103. Delta Lake faciliteaz\u0103 implementarea u\u0219oar\u0103 a noilor dimensiuni atunci c\u00e2nd datele se schimb\u0103. Utilizatorii au acces la o semantica simpl\u0103 pentru gestionarea schemelor tabelelor lor. Aceste instrumente includ impunerea schemei (Schema Enforcement), care protejeaz\u0103 utilizatorii de contaminarea neinten\u021bionat\u0103 a tabelelor lor cu erori sau date nedorite, precum \u0219i evolu\u021bia schemei (Schema Evolution), care permite ad\u0103ugarea automat\u0103 de coloane noi cu date valoroase \u00een locuri corespunz\u0103toare. \u00cen acest articol, ne vom aprofunda \u00een utilizarea acestor instrumente.<\/p>\n<h2>\u00cen\u021belegerea schemelor de tabele<\/h2>\n<p>\nFiecare DataFrame din Apache Spark con\u021bine o schem\u0103 care define\u0219te forma datelor, cum ar fi tipurile de date, coloanele \u0219i metadatele. Cu Delta Lake, schema tabelei este p\u0103strat\u0103 \u00een format JSON \u00een cadrul jurnalului de tranzac\u021bii.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Ce este impunerea schemei?<\/h2>\n<p>\nImpunerea schemei (Schema Enforcement), cunoscut\u0103 \u0219i sub denumirea de validarea schemei (Schema Validation), este un mecanism de protec\u021bie \u00een Delta Lake care garanteaz\u0103 calitatea datelor, resping\u00e2nd \u00eenregistr\u0103rile care nu se conformeaz\u0103 schemei tabelei. La fel ca o gazd\u0103 la recep\u021bia unui restaurant popular, care accept\u0103 doar rezerv\u0103ri, aceasta verific\u0103 dac\u0103 fiecare coloan\u0103 de date introdus\u0103 \u00een tabel se afl\u0103 \u00een lista corespunz\u0103toare de coloane a\u0219teptate (cu alte cuvinte, dac\u0103 fiecare are \u201erezervare\u201d), \u0219i respinge orice \u00eenregistr\u0103ri cu coloane care nu sunt \u00een list\u0103.<\/p>\n<h2>Cum func\u021bioneaz\u0103 impunerea schemei?<\/h2>\n<p>\nDelta Lake utilizeaz\u0103 validarea schemelor la scriere, ceea ce \u00eenseamn\u0103 c\u0103 toate noile \u00eenregistr\u0103ri \u00een tabel sunt verificate pentru compatibilitate cu schema tabelului \u021bint\u0103 \u00een timpul scrierii. Dac\u0103 schema nu este compatibil\u0103, Delta Lake anuleaz\u0103 complet tranzac\u021bia (datele nu sunt scrise) \u0219i genereaz\u0103 o excep\u021bie pentru a informa utilizatorul despre neconcordan\u021b\u0103.<br \/>\nPentru a determina compatibilitatea \u00eenregistr\u0103rii, Delta Lake utilizeaz\u0103 urm\u0103toarele reguli. DataFrame-ul scris:<\/p>\n<ul>\n<li>nu poate con\u021bine coloane suplimentare care nu sunt \u00een schema tabelului \u021bint\u0103. \u0218i invers, totul este \u00een regul\u0103 dac\u0103 datele de intrare nu con\u021bin toate coloanele din tabel \u2014 acestor coloane li se vor atribui valori nule.<\/li>\n<li>nu poate avea tipuri de date ale coloanelor care difer\u0103 de tipurile de date ale coloanelor din tabelul \u021bint\u0103. Dac\u0103 o coloan\u0103 din tabelul \u021bint\u0103 con\u021bine date de tip StringType, dar coloana corespunz\u0103toare din DataFrame con\u021bine date de tip IntegerType, aplicarea for\u021bat\u0103 a schemei va genera o excep\u021bie \u0219i va \u00eempiedica executarea opera\u021biei de scriere.<\/li>\n<li>nu poate con\u021bine nume de coloane care difer\u0103 doar prin registru. Acest lucru \u00eenseamn\u0103 c\u0103 nu pute\u021bi avea coloane cu numele 'Foo' \u0219i 'foo', definite \u00een aceea\u0219i tabel\u0103. De\u0219i Spark poate fi utilizat \u00een modul sensibil sau nesensibil (implicit) la registru, Delta Lake p\u0103streaz\u0103 registrul, dar este nesensibil \u00een cadrul stoc\u0103rii schemei. Parquet este sensibil la registru la stocarea \u0219i returnarea informa\u021biilor coloanei. Pentru a evita posibilele erori, coruperea datelor sau pierderea acestora (cu care ne-am confruntat personal \u00een Databricks), am decis s\u0103 ad\u0103ug\u0103m aceast\u0103 restric\u021bie.<\/li>\n<\/ul>\n<p>\nPentru a ilustra acest lucru, s\u0103 ne uit\u0103m la ceea ce se \u00eent\u00e2mpl\u0103 \u00een codul de mai jos atunci c\u00e2nd \u00eencerc\u0103m s\u0103 ad\u0103ug\u0103m unele coloane recent generate \u00een tabelul Delta Lake, care nu este \u00eenc\u0103 configurat pentru a le accepta.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\n\u00cen loc s\u0103 adauge automat coloanele noi, Delta Lake impune schema \u0219i opre\u0219te scrierea. Pentru a ajuta la identificarea coloanei (sau coloanelor) care cauzeaz\u0103 neconcordan\u021ba, Spark afi\u0219eaz\u0103 ambele scheme din stiva de erori pentru compara\u021bie.<\/p>\n<h2>Care este beneficiul aplic\u0103rii for\u021bate a schemei?<\/h2>\n<p>\nDeoarece aplicarea for\u021bat\u0103 a schemei reprezint\u0103 un control destul de strict, aceasta este un instrument excelent pentru a fi utilizat ca un gardian al unui set de date curat, complet transformat, gata pentru produc\u021bie sau consum. De obicei, este aplicat tabelor care ofer\u0103 date \u00een mod direct:<\/p>\n<ul>\n<li>Algoritmilor de \u00eenv\u0103\u021bare automat\u0103<\/li>\n<li>Dashboard-urilor BI<\/li>\n<li>Analizei datelor \u0219i instrumentelor de vizualizare<\/li>\n<li>Oric\u0103rei sisteme de produc\u021bie care necesit\u0103 scheme semantice strict structurate \u0219i tipizate.<\/li>\n<\/ul>\n<p>\nPentru a-\u0219i preg\u0103ti datele pentru aceast\u0103 barier\u0103 final\u0103, mul\u021bi utilizatori folosesc o arhitectur\u0103 simpl\u0103 \u201emulti-hop\u201d care aduce treptat structur\u0103 \u00een tabelele lor. Pentru a afla mai multe despre aceasta, pute\u021bi consulta articolul <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">\u00cenv\u0103\u021barea automat\u0103 la nivel de produc\u021bie cu Delta Lake.<\/a><\/noindex><\/p>\n<p>Desigur, aplicarea for\u021bat\u0103 a schemei poate fi utilizat\u0103 oriunde \u00een pipeline-ul dumneavoastr\u0103, dar \u021bine\u021bi cont c\u0103 scrierea \u00een flux \u00eentr-un tabel \u00een acest caz poate fi frustrant\u0103, deoarece, de exemplu, a\u021bi uitat c\u0103 a\u021bi ad\u0103ugat o coloan\u0103 suplimentar\u0103 \u00een datele de intrare.<\/p>\n<h2>Prevenirea dilu\u0103rii datelor<\/h2>\n<p>\nP\u00e2n\u0103 \u00een acest punct, v\u0103 pute\u021bi \u00eentreba de ce at\u00e2ta agita\u021bie? La urma urmei, uneori, o eroare nea\u0219teptat\u0103 de \u201enecorespundere a schemei\u201d v\u0103 poate \u00eempiedica \u00een fluxul de lucru, mai ales dac\u0103 sunte\u021bi nou \u00een Delta Lake. De ce s\u0103 nu permit doar schemei s\u0103 se schimbe astfel \u00eenc\u00e2t s\u0103 pot salva DataFrame-ul meu, indiferent de situa\u021bie?<\/p>\n<p>A\u0219a cum spune vechea zical\u0103, \u201eo uncie de preven\u021bie valoreaz\u0103 o livr\u0103 de tratament\u201d. La un moment dat, dac\u0103 nu v\u0103 ocupa\u021bi de aplicarea schemei, problemele de compatibilitate a tipurilor de date \u00ee\u0219i vor ridica capul ur\u00e2t \u2014 surse aparent omogene de date brute pot con\u021bine cazuri limit\u0103, coloane deteriorate, map\u0103ri prost formate sau alte lucruri terifiante care b\u00e2ntuie co\u0219marurile. Cea mai bun\u0103 abordare este s\u0103 opri\u021bi ace\u0219ti du\u0219mani la poart\u0103 \u2014 prin aplicarea for\u021bat\u0103 a schemei \u2014 \u0219i s\u0103 v\u0103 ocupa\u021bi de ei la lumin\u0103, nu mai t\u00e2rziu, c\u00e2nd \u00eencep s\u0103 hoin\u0103reasc\u0103 \u00een ad\u00e2ncurile \u00eentunecate ale codului dumneavoastr\u0103 de lucru.<\/p>\n<p>Aplicarea for\u021bat\u0103 a schemei asigur\u0103 c\u0103 schema tabelului nu se va schimba, dec\u00e2t dac\u0103 confirma\u021bi personal varia\u021bia modific\u0103rii. Aceasta previne \"dilua\u021bia\" datelor, care poate ap\u0103rea atunci c\u00e2nd coloanele noi sunt ad\u0103ugate at\u00e2t de des \u00eenc\u00e2t tabelele valoroase, compacte \u00ee\u0219i pierd semnifica\u021bia \u0219i utilitatea din cauza inunda\u021biei de date. \u00cencuraj\u00e2ndu-v\u0103 s\u0103 fi\u021bi inten\u021bionat, s\u0103 stabili\u021bi standarde ridicate \u0219i s\u0103 a\u0219tepta\u021bi calitate \u00eenalt\u0103, aplicarea for\u021bat\u0103 a schemei face exact ceea ce a fost destinat \u2014 ajut\u00e2ndu-v\u0103 s\u0103 r\u0103m\u00e2ne\u021bi de bun\u0103-credin\u021b\u0103, iar tabelele dumneavoastr\u0103 s\u0103 r\u0103m\u00e2n\u0103 curate.<\/p>\n<p>Dac\u0103, \u00een urma unei examin\u0103ri ulterioare, decide\u021bi c\u0103 de fapt <i>nevoie<\/i> dori\u021bi s\u0103 ad\u0103uga\u021bi o coloan\u0103 nou\u0103 \u2014 nicio problem\u0103, mai jos este o solu\u021bie pe o linie. Solu\u021bia este evolu\u021bia schemei!<\/p>\n<h2>Ce este evolu\u021bia schemei?<\/h2>\n<p>\nEvolu\u021bia schemei este o func\u021bie care le permite utilizatorilor s\u0103 modifice cu u\u0219urin\u021b\u0103 schema actual\u0103 a tabelului \u00een conformitate cu datele care se schimb\u0103 \u00een timp. Este cel mai des utilizat\u0103 \u00een opera\u021biuni de ad\u0103ugare sau scriere, pentru a adapta automat schema pentru a include una sau mai multe coloane noi.<\/p>\n<h2>Cum func\u021bioneaz\u0103 evolu\u021bia schemei?<\/h2>\n<p>\nUrm\u00e2nd exemplul din sec\u021biunea anterioar\u0103, dezvoltatorii pot utiliza cu u\u0219urin\u021b\u0103 evolu\u021bia schemei pentru a ad\u0103uga coloane noi care au fost anterior respinse din cauza neconcordan\u021bei cu schema. Evolu\u021bia schemei este activat\u0103 prin ad\u0103ugarea <code>.option('mergeSchema', 'true')<\/code> la comanda dumneavoastr\u0103 Spark <code>.write sau .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nPentru a vizualiza schema, rula\u021bi urm\u0103toarea interogare Spark SQL<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"\u00cencercare \u00een Delta Lake: aplicarea for\u021bat\u0103 \u0219i evolu\u021bia schemei\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nCa alternativ\u0103, pute\u021bi seta aceast\u0103 op\u021biune pentru \u00eentreaga sesiune Spark, ad\u0103ug\u00e2nd <code>spark.databricks.delta.schema.autoMerge = True<\/code> \u00een configura\u021bia Spark. Dar folosi\u021bi aceast\u0103 op\u021biune cu precau\u021bie, deoarece aplicarea for\u021bat\u0103 a schemei nu v\u0103 va mai avertiza despre neconcordan\u021bele neinten\u021bionate cu schema.<\/p>\n<p>Prin ad\u0103ugarea parametrului <code>mergeSchema<\/code>, toate coloanele care sunt prezente \u00een DataFrame, dar lipsesc \u00een tabelul \u021bint\u0103, sunt ad\u0103ugate automat la sf\u00e2r\u0219itul schemei \u00een cadrul tranzac\u021biei de scriere. De asemenea, pot fi ad\u0103ugate c\u00e2mpuri \u00eencorporate, iar acestea vor fi ad\u0103ugate \u0219i la sf\u00e2r\u0219itul coloanelor respective ale structurii.<\/p>\n<p>Inginerii \u0219i cercet\u0103torii pot folosi aceast\u0103 op\u021biune pentru a ad\u0103uga coloane noi (posibil o metric\u0103 recent urm\u0103rit\u0103 sau o coloan\u0103 cu indicatorii v\u00e2nz\u0103rilor din aceast\u0103 lun\u0103) \u00een tabelele existente de produc\u021bie pentru \u00eenv\u0103\u021barea automat\u0103, f\u0103r\u0103 a afecta modelele existente, bazate pe coloanele mai vechi. <\/p>\n<p>Urm\u0103toarele tipuri de modific\u0103ri ale schemei sunt acceptabile \u00een cadrul evolu\u021biei schemei \u00een timpul ad\u0103ug\u0103rii sau rescrierii unei tabele:<\/p>\n<ul>\n<li>Ad\u0103ugarea de coloane noi (aceasta este cea mai comun\u0103 situa\u021bie)<\/li>\n<li>Modificarea tipurilor de date din NullType -&gt; orice alt tip sau promovarea din ByteType -&gt; ShortType -&gt; IntegerType<\/li>\n<\/ul>\n<p>\nAlte modific\u0103ri, neacceptate \u00een cadrul evolu\u021biei schemei, necesit\u0103 ca schema \u0219i datele s\u0103 fie rescrise prin ad\u0103ugare <code>.option(\"overwriteSchema\", \"true\")<\/code>. De exemplu, \u00een cazul \u00een care coloana \u201eFoo\u201d a fost ini\u021bial de tip integer, iar noua schem\u0103 ar fi de tip string, atunci toate fi\u0219ierele Parquet (datele) ar trebui s\u0103 fie rescrise. Aceste modific\u0103ri includ:<\/p>\n<ul>\n<li>eliminarea coloanelor<\/li>\n<li>modificarea tipului de date al unei coloane existente (in situ)<\/li>\n<li>renumirea coloanelor care difer\u0103 doar prin caz (de exemplu, \u201eFoo\u201d \u0219i \u201efoo\u201d)<\/li>\n<\/ul>\n<p>\n\u00cen cele din urm\u0103, cu urm\u0103toarea versiune Spark 3.0, va fi complet sus\u021binut DDL explicit (folosind ALTER TABLE), ceea ce va permite utilizatorilor s\u0103 efectueze urm\u0103toarele ac\u021biuni asupra schemelor de tabele:<\/p>\n<ul>\n<li>ad\u0103ugarea de coloane<\/li>\n<li>modificarea comentariilor asupra coloanelor<\/li>\n<li>configurarea propriet\u0103\u021bilor tabelei care definesc comportamentul tabelei, de exemplu, stabilirea duratei de p\u0103strare a jurnalului de tranzac\u021bii.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Care este beneficiul evolu\u021biei schemei?<\/h2>\n<p>\nEvolu\u021bia schemei poate fi utilizat\u0103 \u00eentotdeauna c\u00e2nd <i>inten\u021biona\u021bi<\/i> s\u0103 modifica\u021bi schema tabelei dvs. (spre deosebire de cazurile \u00een care a\u021bi ad\u0103ugat accidental coloane \u00een DataFrame-ul dvs. care nu ar fi trebuit s\u0103 fie acolo). Aceasta este cea mai simpl\u0103 metod\u0103 de a migra schema dvs., deoarece adaug\u0103 automat numele corecte ale coloanelor \u0219i tipurile de date f\u0103r\u0103 a necesita declara\u021bia lor explicit\u0103.<\/p>\n<h2>Concluzie<\/h2>\n<p>\nAplicarea for\u021bat\u0103 a schemei respinge orice coloane noi sau alte modific\u0103ri ale schemei care nu sunt compatibile cu tabelul dumneavoastr\u0103. Stabilind \u0219i men\u021bin\u00e2nd aceste standarde ridicate, anali\u0219tii \u0219i inginerii pot avea \u00eencredere c\u0103 datele lor au cea mai \u00eenalt\u0103 integritate, permi\u021b\u00e2ndu-le s\u0103 ia decizii de afaceri mai eficiente.<\/p>\n<p>Pe de alt\u0103 parte, evolu\u021bia schemei completeaz\u0103 aplicarea for\u021bat\u0103, simplific\u00e2nd <i>modific\u0103rile presupuse<\/i> automate ale schemei. La urma urmei, nu ar trebui s\u0103 fie o dificultate \u2014 ad\u0103ugarea unei coloane.<\/p>\n<p>Aplicarea for\u021bat\u0103 a schemei este yang, \u00een timp ce evolu\u021bia schemei este yin. Folosite \u00eempreun\u0103, aceste func\u021bii faciliteaz\u0103 supresia zgomotului \u0219i ajustarea semnalului.<\/p>\n<p><i>De asemenea, am dori s\u0103 mul\u021bumim lui Mukul Murti \u0219i Pranav Anand pentru contribu\u021bia lor la acest articol.<\/i><\/p>\n<p>Alte articole din aceast\u0103 serie:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Explor\u00e2nd Delta Lake: desf\u0103\u0219urarea jurnalului de tranzac\u021bii<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"Reda\u021bi video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>Articole conexe<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">\u00cenv\u0103\u021bare automat\u0103 la nivel de produc\u021bie cu Delta Lake<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">Ce este un lac de date?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">Afl\u0103 mai multe despre curs<\/a><\/noindex><\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Explor\u00e2nd Delta Lake: aplicarea for\u021bat\u0103 \u0219i evolu\u021bia schemei | ProHoster","description":"Salut, Habr!","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}