{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"P\u00ebrkufizimi i Delta Lake: zbatimi i detyruesh\u00ebm dhe evolucioni i skem\u00ebs","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>P\u00ebrsh\u00ebndetje, Habr! Ju prezantoj p\u00ebrkthimin e artikullit <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\u00abP\u00ebrkufizimi i Delta Lake: Zbatimi i Skem\u00ebs &amp; Evolucioni\u00bb<\/a><\/noindex> nga autor\u00ebt Burak Yavuz, Brenner Heintz dhe Denny Lee, i p\u00ebrgatitur para fillimit t\u00eb kursit <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\u00abInxhiner i T\u00eb Dh\u00ebnave\u00bb<\/a><\/noindex> nga OTUS.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrkufizimi i Delta Lake: zbatimi i detyruesh\u00ebm dhe evolucioni i skem\u00ebs\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>T\u00eb dh\u00ebnat, si dhe p\u00ebrvoja jon\u00eb, vazhdimisht akumulohen dhe zhvillohen. P\u00ebr t\u00eb mos mbetur prapa, modelet tona mendore t\u00eb bot\u00ebs duhet t\u00eb adaptohen me t\u00eb dh\u00ebna t\u00eb reja, disa prej t\u00eb cilave p\u00ebrmbajn\u00eb dimensione t\u00eb reja \u2014 m\u00ebnyra t\u00eb reja p\u00ebr t\u00eb v\u00ebzhguar gj\u00ebra p\u00ebr t\u00eb cilat m\u00eb par\u00eb nuk kishim ide. K\u00ebto modele mendore nuk dallohen shum\u00eb nga skemat e tabelave, t\u00eb cilat p\u00ebrcaktojn\u00eb se si ne klasifikojm\u00eb dhe p\u00ebrpunojm\u00eb informacionin e ri.<\/p>\n<p>Kjo na \u00e7on n\u00eb \u00e7\u00ebshtjen e menaxhimit t\u00eb skemave. Nd\u00ebrsa detyrat dhe k\u00ebrkesat e biznesit ndryshojn\u00eb me kalimin e koh\u00ebs, ashtu ndryshon edhe struktura e t\u00eb dh\u00ebnave tuaja. Delta Lake lejon integrimin e leht\u00eb t\u00eb dimensioneve t\u00eb reja kur ndodhin ndryshime n\u00eb t\u00eb dh\u00ebna. P\u00ebrdoruesit kan\u00eb qasje n\u00eb nj\u00eb semantik\u00eb t\u00eb thjesht\u00eb p\u00ebr t\u00eb menaxhuar skemat e tabelave t\u00eb tyre. K\u00ebto mjete p\u00ebrfshijn\u00eb forcimin e skem\u00ebs (Schema Enforcement), i cili mbron p\u00ebrdoruesit nga ndotja e paq\u00ebllimt\u00eb t\u00eb tabelave t\u00eb tyre me gabime ose t\u00eb dh\u00ebna t\u00eb panevojshme, si dhe evolucionin e skem\u00ebs (Schema Evolution), i cili lejon shtimin automatik t\u00eb kolonave t\u00eb reja me t\u00eb dh\u00ebna t\u00eb vlefshme n\u00eb vende t\u00eb p\u00ebrshtatshme. N\u00eb k\u00ebt\u00eb artikull, ne do t\u00eb thellohemi n\u00eb p\u00ebrdorimin e k\u00ebtyre mjeteve.<\/p>\n<h2>Kuptimi i skemave t\u00eb tabelave<\/h2>\n<p>\n\u00c7do DataFrame n\u00eb Apache Spark p\u00ebrmban nj\u00eb skem\u00eb q\u00eb p\u00ebrcakton formatin e t\u00eb dh\u00ebnave, si\u00e7 jan\u00eb tipet e t\u00eb dh\u00ebnave, kolonat dhe metadatat. Me Delta Lake, skema e tabel\u00ebs ruhet n\u00eb formatin JSON brenda regjistrit t\u00eb transaksioneve.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>\u00c7far\u00eb \u00ebsht\u00eb forcimi i skem\u00ebs?<\/h2>\n<p>\n\u041f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0441\u0445\u0435\u043c\u044b (Schema Enforcement), \u0442\u0430\u043a\u0436\u0435 \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u043e\u0435 \u043a\u0430\u043a \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u0441\u0445\u0435\u043c\u044b (Schema Validation), \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0437\u0430\u0449\u0438\u0442\u043d\u044b\u043c \u043c\u0435\u0445\u0430\u043d\u0438\u0437\u043c\u043e\u043c \u0432 Delta Lake, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0433\u0430\u0440\u0430\u043d\u0442\u0438\u0440\u0443\u0435\u0442 \u043a\u0430\u0447\u0435\u0441\u0442\u0432\u043e \u0434\u0430\u043d\u043d\u044b\u0445, \u043e\u0442\u043a\u043b\u043e\u043d\u044f\u044f \u0437\u0430\u043f\u0438\u0441\u0438, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0435 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0442 \u0441\u0445\u0435\u043c\u0435 \u0442\u0430\u0431\u043b\u0438\u0446\u044b. \u041a\u0430\u043a \u0438 \u0445\u043e\u0441\u0442\u0435\u0441 \u043d\u0430 \u0441\u0442\u043e\u0439\u043a\u0435 \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0430\u0446\u0438\u0438 \u0432 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u043e\u043c \u0440\u0435\u0441\u0442\u043e\u0440\u0430\u043d\u0435, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043f\u0440\u0438\u043d\u0438\u043c\u0430\u0435\u0442 \u0442\u043e\u043b\u044c\u043a\u043e \u043f\u043e \u043f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0439 \u0431\u0440\u043e\u043d\u0438, \u043e\u043d \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0435\u0442, \u0435\u0441\u0442\u044c \u043b\u0438 \u043a\u0430\u0436\u0434\u044b\u0439 \u0441\u0442\u043e\u043b\u0431\u0435\u0446 \u0434\u0430\u043d\u043d\u044b\u0445, \u0432\u0432\u043e\u0434\u0438\u043c\u044b\u0445 \u0432 \u0442\u0430\u0431\u043b\u0438\u0446\u0443, \u0432 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0435\u043c \u0441\u043f\u0438\u0441\u043a\u0435 \u043e\u0436\u0438\u0434\u0430\u0435\u043c\u044b\u0445 \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u0432 (\u0434\u0440\u0443\u0433\u0438\u043c\u0438 \u0441\u043b\u043e\u0432\u0430\u043c\u0438, \u0435\u0441\u0442\u044c \u043b\u0438 \u0434\u043b\u044f \u043a\u0430\u0436\u0434\u043e\u0433\u043e \u0438\u0437 \u043d\u0438\u0445 \u00ab\u0431\u0440\u043e\u043d\u044c\u00bb), \u0438 \u043e\u0442\u043a\u043b\u043e\u043d\u044f\u0435\u0442 \u043b\u044e\u0431\u044b\u0435 \u0437\u0430\u043f\u0438\u0441\u0438 \u0441\u043e \u0441\u0442\u043e\u043b\u0431\u0446\u0430\u043c\u0438, \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u043d\u0435\u0442 \u0432 \u0441\u043f\u0438\u0441\u043a\u0435.<\/p>\n<h2>\u041a\u0430\u043a \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0441\u0445\u0435\u043c\u044b?<\/h2>\n<p>\nDelta Lake \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0443 \u0441\u0445\u0435\u043c\u044b \u043f\u0440\u0438 \u0437\u0430\u043f\u0438\u0441\u0438, \u0447\u0442\u043e \u043e\u0437\u043d\u0430\u0447\u0430\u0435\u0442, \u0447\u0442\u043e \u0432\u0441\u0435 \u043d\u043e\u0432\u044b\u0435 \u0437\u0430\u043f\u0438\u0441\u0438 \u0432 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u044e\u0442\u0441\u044f \u043d\u0430 \u0441\u043e\u0432\u043c\u0435\u0441\u0442\u0438\u043c\u043e\u0441\u0442\u044c \u0441\u043e \u0441\u0445\u0435\u043c\u043e\u0439 \u0446\u0435\u043b\u0435\u0432\u043e\u0439 \u0442\u0430\u0431\u043b\u0438\u0446\u044b \u0432\u043e \u0432\u0440\u0435\u043c\u044f \u0437\u0430\u043f\u0438\u0441\u0438. \u0415\u0441\u043b\u0438 \u0441\u0445\u0435\u043c\u0430 \u043d\u0435\u0441\u043e\u0432\u043c\u0435\u0441\u0442\u0438\u043c\u0430, Delta Lake \u043f\u043e\u043b\u043d\u043e\u0441\u0442\u044c\u044e \u043e\u0442\u043c\u0435\u043d\u044f\u0435\u0442 \u0442\u0440\u0430\u043d\u0437\u0430\u043a\u0446\u0438\u044e (\u0434\u0430\u043d\u043d\u044b\u0435 \u043d\u0435 \u0437\u0430\u043f\u0438\u0441\u044b\u0432\u0430\u044e\u0442\u0441\u044f) \u0438 \u0441\u043e\u0437\u0434\u0430\u0435\u0442 \u0438\u0441\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435, \u0447\u0442\u043e\u0431\u044b \u0441\u043e\u043e\u0431\u0449\u0438\u0442\u044c \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044e \u043e \u043d\u0435\u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0438\u0438.<br \/>\n\u0414\u043b\u044f \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f \u0441\u043e\u0432\u043c\u0435\u0441\u0442\u0438\u043c\u043e\u0441\u0442\u0438 \u0437\u0430\u043f\u0438\u0441\u0438 \u0441 \u0442\u0430\u0431\u043b\u0438\u0446\u0435\u0439 Delta Lake \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0435 \u043f\u0440\u0430\u0432\u0438\u043b\u0430. \u0417\u0430\u043f\u0438\u0441\u044b\u0432\u0430\u0435\u043c\u044b\u0439 DataFrame:<\/p>\n<ul>\n<li>nuk mund t\u00eb p\u00ebrmbaj\u00eb kolona shtes\u00eb q\u00eb nuk jan\u00eb n\u00eb skem\u00ebn e tabel\u00ebs s\u00eb synuar. Dhe p\u00ebrkundrazi, gjith\u00e7ka \u00ebsht\u00eb n\u00eb rregull n\u00ebse t\u00eb dh\u00ebnat hyr\u00ebse nuk p\u00ebrmbajn\u00eb t\u00eb gjitha kolonat nga tabela \u2014 k\u00ebtyre kolonave thjesht do t'u jepen vlera zero.<\/li>\n<li>nuk mund t\u00eb ket\u00eb tipe t\u00eb dh\u00ebnash t\u00eb kolonave q\u00eb ndryshojn\u00eb nga tipet e dh\u00ebnash t\u00eb kolonave n\u00eb tabel\u00ebn e synuar. N\u00ebse nj\u00eb kolone e tabel\u00ebs s\u00eb synuar p\u00ebrmban t\u00eb dh\u00ebna StringType, por kolona p\u00ebrkat\u00ebse n\u00eb DataFrame p\u00ebrmban t\u00eb dh\u00ebna IntegerType, forcimi i aplikimit t\u00eb skem\u00ebs do t\u00eb shkaktoj\u00eb nj\u00eb p\u00ebrjashtim dhe do t\u00eb parandaloj\u00eb ekzekutimin e operacionit t\u00eb shkrimit.<\/li>\n<li>\u043d\u0435 \u043c\u043e\u0436\u0435\u0442 \u0441\u043e\u0434\u0435\u0440\u0436\u0430\u0442\u044c \u0438\u043c\u0435\u043d\u0430 \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u0432, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043e\u0442\u043b\u0438\u0447\u0430\u044e\u0442\u0441\u044f \u0442\u043e\u043b\u044c\u043a\u043e \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u043e\u043c. \u042d\u0442\u043e \u0437\u043d\u0430\u0447\u0438\u0442, \u0447\u0442\u043e \u0432\u044b \u043d\u0435 \u043c\u043e\u0436\u0435\u0442\u0435 \u0438\u043c\u0435\u0442\u044c \u0441\u0442\u043e\u043b\u0431\u0446\u044b \u0441 \u0438\u043c\u0435\u043d\u0430\u043c\u0438 &#8216;Foo&#8217; \u0438 &#8216;foo&#8217;, \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u043d\u044b\u0435 \u0432 \u043e\u0434\u043d\u043e\u0439 \u0442\u0430\u0431\u043b\u0438\u0446\u0435. \u0425\u043e\u0442\u044f Spark \u043c\u043e\u0436\u043d\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0432 \u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u043c \u0438\u043b\u0438 \u043d\u0435\u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u043c (\u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e) \u043a \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0443 \u0440\u0435\u0436\u0438\u043c\u0435, Delta Lake \u0441\u043e\u0445\u0440\u0430\u043d\u044f\u0435\u0442 \u0440\u0435\u0433\u0438\u0441\u0442\u0440, \u043d\u043e \u043d\u0435\u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u0435\u043d \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0438 \u0441\u0445\u0435\u043c\u044b. Parquet \u0447\u0443\u0432\u0441\u0442\u0432\u0438\u0442\u0435\u043b\u0435\u043d \u043a \u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0443 \u043f\u0440\u0438 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0438 \u0438 \u0432\u043e\u0437\u0432\u0440\u0430\u0442\u0435 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u0438 \u0441\u0442\u043e\u043b\u0431\u0446\u0430. \u0427\u0442\u043e\u0431\u044b \u0438\u0437\u0431\u0435\u0436\u0430\u0442\u044c \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u044b\u0445 \u043e\u0448\u0438\u0431\u043e\u043a, \u043f\u043e\u0432\u0440\u0435\u0436\u0434\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u043b\u0438 \u0438\u0445 \u043f\u043e\u0442\u0435\u0440\u0438 (\u0441 \u0447\u0435\u043c \u043c\u044b \u043b\u0438\u0447\u043d\u043e \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u043b\u0438\u0441\u044c \u0432 Databricks), \u043c\u044b \u0440\u0435\u0448\u0438\u043b\u0438 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u044d\u0442\u043e \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u0438\u0435.<\/li>\n<\/ul>\n<p>\nP\u00ebr t\u00eb ilustruar k\u00ebt\u00eb, le t\u00eb shohim se \u00e7far\u00eb ndodh n\u00eb kodin e dh\u00ebn\u00eb m\u00eb posht\u00eb kur p\u00ebrpiqemi t\u00eb shtojm\u00eb disa kolona t\u00eb reja t\u00eb gjeneruara n\u00eb tabel\u00ebn Delta Lake, e cila ende nuk \u00ebsht\u00eb e konfigurueshme p\u00ebr t'i pranuar ato.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\nN\u00eb vend q\u00eb t\u00eb shtoj\u00eb automatikisht kolona t\u00eb reja, Delta Lake imponon nj\u00eb skem\u00eb dhe ndalon shkrimin. P\u00ebr t\u00eb ndihmuar n\u00eb identifikimin e cil\u00ebs do kolon\u00eb (apo disa) \u00ebsht\u00eb shkaku i mosmas\u00eb, Spark shfaq t\u00eb dyja skemat nga stack trace p\u00ebr krahasim.<\/p>\n<h2>Cila \u00ebsht\u00eb p\u00ebrfitimi i forcimit t\u00eb skem\u00ebs?<\/h2>\n<p>\nDuke qen\u00eb se forcimi i skem\u00ebs p\u00ebrb\u00ebn nj\u00eb kontroll mjaft strik, ai \u00ebsht\u00eb nj\u00eb mjet i shk\u00eblqyer p\u00ebr t'u p\u00ebrdorur si portier i nj\u00eb grumbulli t\u00eb past\u00ebr dhe t\u00eb plot\u00ebsisht t\u00eb transformuar t\u00eb dh\u00ebnash, i cili \u00ebsht\u00eb gati p\u00ebr prodhim ose konsum. N\u00eb p\u00ebrgjith\u00ebsi, aplikohet n\u00eb tabelat q\u00eb sh\u00ebrbejn\u00eb drejtp\u00ebrdrejt t\u00eb dh\u00ebnat:<\/p>\n<ul>\n<li>Algoritmet e m\u00ebsimit t\u00eb makinave<\/li>\n<li>Dashboard-e BI<\/li>\n<li>Analiza e t\u00eb dh\u00ebnave dhe mjetet e vizualizimit<\/li>\n<li>Cil\u00ebsdo sistem prodhimi q\u00eb k\u00ebrkon skema semantike strikt t\u00eb strukturuara dhe tipizuara.<\/li>\n<\/ul>\n<p>\nP\u00ebr t\u00eb p\u00ebrgatitur t\u00eb dh\u00ebnat e tua p\u00ebr k\u00ebt\u00eb barrier\u00eb p\u00ebrfundimtare, shum\u00eb p\u00ebrdorues p\u00ebrdorin nj\u00eb arkitektur\u00eb t\u00eb thjesht\u00eb \u201cmulti-hop\u201d q\u00eb gradualisht i jep struktur\u00eb tabela t\u00eb tyre. P\u00ebr t\u00eb m\u00ebsuar m\u00eb shum\u00eb rreth k\u00ebsaj, mund t\u00eb shqyrtosh artikullin <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">M\u00ebsoni makinat e nivelit prodhues me Delta Lake.<\/a><\/noindex><\/p>\n<p>Sigurisht, aplikimi i detyruar i skem\u00ebs mund t\u00eb p\u00ebrdoret n\u00eb \u00e7do vend t\u00eb pipeline tuaj, por mbani mend se shkruajta n\u00eb tabel\u00eb n\u00eb k\u00ebt\u00eb rast mund t\u00eb jet\u00eb e frustrueshme, pasi p\u00ebr shembull, mund t\u00eb keni harruar se keni shtuar nj\u00eb kolon\u00eb t\u00eb re n\u00eb t\u00eb dh\u00ebnat hyr\u00ebse.<\/p>\n<h2>Parandalimi i hollimit t\u00eb t\u00eb dh\u00ebnave<\/h2>\n<p>\nN\u00eb k\u00ebt\u00eb pik\u00eb, mund t\u00eb pyesni vet\u00ebn pse ka kaq shum\u00eb zhurm\u00eb? N\u00eb fund t\u00eb fundit, ndonj\u00ebher\u00eb nj\u00eb gabim i papritur \u201cmos p\u00ebrputhje skeme\u201d mund t\u2019ju v\u00ebr\u00eb n\u00eb nj\u00eb telashe n\u00eb procesin tuaj t\u00eb pun\u00ebs, ve\u00e7an\u00ebrisht n\u00ebse jeni fillestar me Delta Lake. Pse thjesht t\u00eb mos lejoj\u00eb skem\u00ebn t\u00eb ndryshoj\u00eb si\u00e7 \u00ebsht\u00eb e nevojshme q\u00eb un\u00eb t\u00eb mund t\u00eb shkruaj DataFrame tim, pavar\u00ebsisht nga gjith\u00e7ka?<\/p>\n<p>Si\u00e7 thot\u00eb nj\u00eb proverb i vjet\u00ebr, \"nj\u00eb on\u00e7\u00eb parandalim kushton nj\u00eb paund trajtim\". N\u00eb nj\u00eb moment, n\u00ebse nuk kujdeseni p\u00ebr t\u00eb zbatuar skem\u00ebn tuaj, problemet me p\u00ebrputhshm\u00ebrin\u00eb e llojeve t\u00eb t\u00eb dh\u00ebnave do t\u00eb ngrihen n\u00eb m\u00ebnyr\u00eb t\u00eb pak\u00ebndshme \u2014 burimet e dukshme homogjene t\u00eb t\u00eb dh\u00ebnave t\u00eb pap\u00ebrpunuara mund t\u00eb p\u00ebrmbajn\u00eb raste t\u00eb skajshme, kolona t\u00eb d\u00ebmtuara, p\u00ebrshkrime t\u00eb formuara gabimisht ose gj\u00ebra t\u00eb tjera t\u00eb frikshme q\u00eb shfaqen n\u00eb makthet. Qasja m\u00eb e mir\u00eb \u00ebsht\u00eb t\u00eb ndaloni k\u00ebta armiq n\u00eb port\u00eb \u2014 duke imponuar zbatimin e skem\u00ebs \u2014 dhe t'i p\u00ebrballeni atyre n\u00eb drit\u00eb, jo m\u00eb von\u00eb, kur ata fillojn\u00eb t\u00eb rregullohen n\u00eb thell\u00ebsit\u00eb e err\u00ebta t\u00eb kodit tuaj t\u00eb pun\u00ebs.<\/p>\n<p>Zbatja e detyrueshme e skem\u00ebs jep siguri se skema e tabel\u00ebs suaj nuk do t\u00eb ndryshoj\u00eb, p\u00ebrve\u00e7 n\u00ebse ju vet\u00eb e konfirmoni opsionin e ndryshimit. Kjo parandalon \"t\u00eb hollimin\" e t\u00eb dh\u00ebnave, q\u00eb mund t\u00eb ndodh\u00eb kur kolonat e reja shtohen me kaq shum\u00eb shpejt\u00ebsi saq\u00eb tabelat e vjetra, t\u00eb vlefshme dhe t\u00eb p\u00ebrmbledhura, humbin vler\u00ebn dhe p\u00ebrdorshm\u00ebrin\u00eb e tyre p\u00ebr shkak t\u00eb ndotjes me t\u00eb dh\u00ebna. Duke inkurajuar q\u00eb t\u00eb jeni t\u00eb q\u00ebllimsh\u00ebm, t\u00eb vendosni standarde t\u00eb larta dhe t\u00eb prisni cil\u00ebsi t\u00eb lart\u00eb, zbatimi i detyruesh\u00ebm i skem\u00ebs b\u00ebn pik\u00ebrisht at\u00eb p\u00ebr t\u00eb cil\u00ebn ishte menduar - t\u00eb ndihmoj\u00eb q\u00eb t\u00eb mbeteni t\u00eb ndersh\u00ebm dhe tabelat tuaja t\u00eb mbeten t\u00eb pastra.<\/p>\n<p>N\u00ebse, pas shqyrtimit t\u00eb m\u00ebtejsh\u00ebm, vendosni se n\u00eb t\u00eb v\u00ebrtet\u00eb <i>nevojitet<\/i> t\u00eb shtoni nj\u00eb kolon\u00eb t\u00eb re - asnj\u00eb problem, m\u00eb posht\u00eb \u00ebsht\u00eb nj\u00eb zgjidhje nj\u00eb-linj\u00eb. Zgjidhja \u00ebsht\u00eb evolucioni i skem\u00ebs!<\/p>\n<h2>\u00c7far\u00eb \u00ebsht\u00eb evolucioni i skem\u00ebs?<\/h2>\n<p>\nEvolucioni i skem\u00ebs \u00ebsht\u00eb nj\u00eb funksion q\u00eb lejon p\u00ebrdoruesit t\u00eb ndryshojn\u00eb leht\u00ebsisht skem\u00ebn aktuale t\u00eb tabel\u00ebs n\u00eb p\u00ebrputhje me t\u00eb dh\u00ebnat q\u00eb ndryshojn\u00eb me kalimin e koh\u00ebs. P\u00ebrdoret m\u00eb s\u00eb shpeshti gjat\u00eb operacioneve t\u00eb shtimit ose p\u00ebrshkrimit, p\u00ebr t\u00eb adaptuar automatikisht skem\u00ebn p\u00ebr t\u00eb p\u00ebrfshir\u00eb nj\u00eb ose disa kolona t\u00eb reja.<\/p>\n<h2>Si funksionon evolucioni i skem\u00ebs?<\/h2>\n<p>\nDuke ndjekur shembullin nga seksioni i m\u00ebparsh\u00ebm, zhvilluesit mund t\u00eb p\u00ebrdorin leht\u00ebsisht evolucionin e skem\u00ebs p\u00ebr t\u00eb shtuar kolona t\u00eb reja, t\u00eb cilat m\u00eb par\u00eb ishin refuzuar p\u00ebr shkak t\u00eb mos p\u00ebrputhshm\u00ebris\u00eb me skem\u00ebn. Evolucioni i skem\u00ebs aktivizohet duke shtuar <code>.option('mergeSchema', 'true')<\/code> n\u00eb komand\u00ebn tuaj t\u00eb Spark <code>.write ose .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nP\u00ebr t\u00eb par\u00eb grafikun, kryeni k\u00ebt\u00eb k\u00ebrkes\u00eb t\u00eb Spark SQL<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"P\u00ebrkufizimi i Delta Lake: zbatimi i detyruesh\u00ebm dhe evolucioni i skem\u00ebs\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAlternativisht, mund t\u00eb vendosni k\u00ebt\u00eb opsion p\u00ebr t\u00eb gjith\u00eb seanc\u00ebn e Spark, duke shtuar <code>spark.databricks.delta.schema.autoMerge = True<\/code> n\u00eb konfigurimin e Spark. Por, p\u00ebrdoreni k\u00ebt\u00eb me kujdes, pasi zbatimi i detyruesh\u00ebm i skem\u00ebs nuk do t'ju paralajm\u00ebroj\u00eb m\u00eb p\u00ebr mos p\u00ebrputhshm\u00ebrit\u00eb e papara me skem\u00ebn.<\/p>\n<p>Duke p\u00ebrfshir\u00eb n\u00eb k\u00ebrkes\u00eb parametrin <code>mergeSchema<\/code>, gjith\u00eb kolumnat q\u00eb jan\u00eb n\u00eb DataFrame, por mungojn\u00eb n\u00eb tabel\u00ebn e destinacionit, automatikisht shtohen n\u00eb fund t\u00eb skem\u00ebs n\u00eb kuad\u00ebr t\u00eb transaksionit t\u00eb shkruar. Po ashtu, mund t\u00eb shtohen fusha t\u00eb ngulitura, dhe ato gjithashtu do t\u00eb shtohen n\u00eb fund t\u00eb kolumnave p\u00ebrkat\u00ebse t\u00eb struktur\u00ebs.<\/p>\n<p>Inxhinier\u00ebt dhe shkenc\u00ebtar\u00ebt e t\u00eb dh\u00ebnave mund t\u00eb p\u00ebrdorin k\u00ebt\u00eb opsion p\u00ebr t\u00eb shtuar kolona t\u00eb reja (ndoshta nj\u00eb metrik\u00eb q\u00eb \u00ebsht\u00eb monitoruar s\u00eb fundmi ose nj\u00eb kolona treguesish shitjesh p\u00ebr k\u00ebt\u00eb muaj) n\u00eb tabelat e tyre ekzistuese t\u00eb prodhimit p\u00ebr m\u00ebsimin e makineris\u00eb, pa prishur modelet ekzistuese t\u00eb bazuara n\u00eb kolona t\u00eb vjetra. <\/p>\n<p>Llojet e ardhshme t\u00eb ndryshimeve t\u00eb skem\u00ebs jan\u00eb t\u00eb pranuara brenda evolucionit t\u00eb skem\u00ebs gjat\u00eb shtimit ose rinovimit t\u00eb tabel\u00ebs:<\/p>\n<ul>\n<li>Shtimi i kolumnave t\u00eb reja (kjo \u00ebsht\u00eb skenari m\u00eb i zakonsh\u00ebm)<\/li>\n<li>Ndryshimi i llojeve t\u00eb t\u00eb dh\u00ebnave nga NullType -&gt; \u00e7do lloj tjet\u00ebr ose rritja nga ByteType -&gt; ShortType -&gt; IntegerType<\/li>\n<\/ul>\n<p>\nNdryshime t\u00eb tjera, q\u00eb jan\u00eb t\u00eb papranueshme brenda evolucionit t\u00eb skem\u00ebs, k\u00ebrkojn\u00eb q\u00eb skema dhe t\u00eb dh\u00ebnat t\u00eb rivendosen duke shtuar <code>.option(&quot;overwriteSchema&quot;, &quot;true&quot;)<\/code>. P\u00ebr shembull, n\u00eb rastin kur kolona \u00abFoo\u00bb fillimisht ishte integer, dhe skema e re do t\u00eb ishte e tipit string, at\u00ebher\u00eb t\u00eb gjith\u00eb skedar\u00ebt Parquet (t\u00eb dh\u00ebnat) do t\u00eb duhej t\u00eb shkruheshin p\u00ebrs\u00ebri. Ndryshime t\u00eb tilla p\u00ebrfshijn\u00eb:<\/p>\n<ul>\n<li>heqjen e kolon\u00ebs<\/li>\n<li>ndryshimin e tipit t\u00eb t\u00eb dh\u00ebnave t\u00eb kolon\u00ebs ekzistuese (n\u00eb vend)<\/li>\n<li>rilindjen e kolonave q\u00eb ndryshojn\u00eb vet\u00ebm n\u00eb regjist\u00ebr (p.sh., \u00abFoo\u00bb dhe \u00abfoo\u00bb)<\/li>\n<\/ul>\n<p>\nM\u00eb n\u00eb fund, me rilindjen e ardhshme Spark 3.0 do t\u00eb mb\u00ebshtetet plot\u00ebsisht DDL e qart\u00eb (duke p\u00ebrdorur ALTER TABLE), e cila do t'u lejoj\u00eb p\u00ebrdoruesve t\u00eb kryejn\u00eb veprimet e m\u00ebposhtme mbi skemat e tabelave:<\/p>\n<ul>\n<li>shtimin e kolonave<\/li>\n<li>ndryshimin e komenteve t\u00eb kolonave<\/li>\n<li>konfigurimin e pronave t\u00eb tabel\u00ebs q\u00eb p\u00ebrcaktojn\u00eb sjelljen e tabel\u00ebs, p\u00ebr shembull, caktimin e koh\u00ebzgjatjes s\u00eb ruajtjes s\u00eb regjistrit t\u00eb transaksioneve.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Cila \u00ebsht\u00eb p\u00ebrfitimi i evolucionit t\u00eb skem\u00ebs?<\/h2>\n<p>\nEvolucioni i skem\u00ebs mund t\u00eb p\u00ebrdoret gjithmon\u00eb kur ju <i>keni nd\u00ebrmend<\/i> ndryshoni skem\u00ebn e tabel\u00ebs suaj (ndryshe nga rastet kur aksidentalisht keni shtuar n\u00eb DataFrame tuaj kolona q\u00eb nuk duhet t\u00eb ishin aty). Kjo \u00ebsht\u00eb m\u00ebnyra m\u00eb e leht\u00eb p\u00ebr t\u00eb migruar skem\u00ebn tuaj, sepse automatikisht shton emrat e duhur t\u00eb kolonave dhe llojet e t\u00eb dh\u00ebnave pa pasur nevoj\u00eb p\u00ebr shpalljen e tyre t\u00eb qart\u00eb.<\/p>\n<h2>P\u00ebrfundimi<\/h2>\n<p>\nZbatimi i detyruesh\u00ebm i skem\u00ebs refuzon \u00e7do kolon\u00eb t\u00eb re ose ndryshime t\u00eb tjera t\u00eb skem\u00ebs q\u00eb nuk jan\u00eb t\u00eb p\u00ebrshtatshme me tabel\u00ebn tuaj. Duke vendosur dhe ruajtur k\u00ebto standarde t\u00eb larta, analist\u00ebt dhe inxhinier\u00ebt mund t\u00eb besojn\u00eb se t\u00eb dh\u00ebnat e tyre kan\u00eb nj\u00eb nivel t\u00eb lart\u00eb integriteti, duke argumentuar p\u00ebr k\u00ebt\u00eb qart\u00eb dhe ndryshe, duke u lejuar atyre t\u00eb marrin vendime m\u00eb efektive n\u00eb biznes.<\/p>\n<p>Nga ana tjet\u00ebr, evolucioni i skem\u00ebs plot\u00ebson zbatimin e detyruesh\u00ebm, duke thjeshtuar <i>ndryshimet e supozuara<\/i> automatik. N\u00eb fund t\u00eb fundit, kjo nuk duhet t\u00eb jet\u00eb komplekse \u2014 t\u00eb shtosh nj\u00eb kolon\u00eb.<\/p>\n<p>Zbatimi i detyruesh\u00ebm i skem\u00ebs \u00ebsht\u00eb yan, nd\u00ebrsa evolucioni i skem\u00ebs \u00ebsht\u00eb yin. Kur p\u00ebrdoren s\u00eb bashku, k\u00ebto funksione e b\u00ebjn\u00eb m\u00eb t\u00eb leht\u00eb se kurr\u00eb minimizimin e zhurm\u00ebs dhe optimizimin e sinjalit.<\/p>\n<p><i>Po ashtu d\u00ebshirojm\u00eb t\u00eb fal\u00ebnderojm\u00eb Mukul Murti dhe Pranav Anand p\u00ebr kontributin e tyre n\u00eb k\u00ebt\u00eb artikull.<\/i><\/p>\n<p>Artikujt e tjer\u00eb n\u00eb k\u00ebt\u00eb seri:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Zhytesh n\u00eb Delta Lake: Zhbllokimi i regjistrit t\u00eb transaksioneve<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"Luaj videon\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>Artikujt n\u00eb lidhje me tem\u00ebn<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">M\u00ebsimi i makinerive n\u00eb nivel prodhimi me Delta Lake<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">\u00c7far\u00eb \u00ebsht\u00eb nj\u00eb liqen t\u00eb dh\u00ebnash?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">M\u00ebso m\u00eb shum\u00eb rreth kursit<\/a><\/noindex><\/p>\n<p>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c,\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c,\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Zhytesh n\u00eb Delta Lake: Zbatimi dhe evolucioni i detyr\u00ebs | ProHoster","description":"P\u00ebrsh\u00ebndetje, Habr! Po ju paraqes p\u00ebrkthimin e artikullit \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb nga autor\u00ebt Burak Yavuz, Brenner Heintz dhe Denny Lee, i cili u p\u00ebrgatit n\u00eb prag t\u00eb nisjes s\u00eb kursit \u00abData Engineer\u00bb nga OTUS. T\u00eb dh\u00ebnat, ashtu si dhe p\u00ebrvoja jon\u00eb, vazhdimisht grumbullohen dhe zhvillohen. P\u00ebr t\u00eb mos mbetur prapa, modelet tona mendore t\u00eb bot\u00ebs duhet t\u00eb adaptohen ndaj t\u00eb dh\u00ebnave t\u00eb reja,","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c,","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45"},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}