{"id":82038,"date":"2020-05-19T01:42:26","date_gmt":"2020-05-18T23:42:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy"},"modified":"2020-05-19T01:42:26","modified_gmt":"2020-05-18T23:42:26","slug":"pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","title":{"rendered":"Diving into Delta Lake: Schema Enforcement and Evolution","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><i><b>Hello, Habr! I present to your attention the translation of the article. <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/09\/24\/diving-into-delta-lake-schema-enforcement-evolution.html\">\"Diving Into Delta Lake: Schema Enforcement &amp; Evolution\"<\/a><\/noindex> by Burak Yavuz, Brenner Heintz, and Denny Lee, prepared ahead of the course launch <noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">\"Data Engineer\"<\/a><\/noindex> from OTUS.<\/b><\/i><\/p>\n<p><img decoding=\"async\" alt=\"Diving into Delta Lake: Schema Enforcement and Evolution\" src=\"\/wp-content\/uploads\/2020\/05\/1d20921d8d4a1c1e0f8afee4fc019024.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Data, much like our experience, continuously accumulates and evolves. To keep pace, our mental models of the world must adapt to new data, some of which introduces new dimensions\u2014new ways to observe things previously unknown to us. These mental models are little different from table schemas that define how we classify and process new information.<\/p>\n<p>This brings us to the topic of schema management. As business tasks and requirements change over time, so does the structure of your data. Delta Lake makes it easy to implement new dimensions as data changes. Users have access to a straightforward semantics for managing their table schemas. These tools include Schema Enforcement, which protects users from inadvertently cluttering their tables with errors or unnecessary data, and Schema Evolution, which allows for the automatic addition of new columns with valuable data in the appropriate places. In this article, we will delve into the use of these tools.<\/p>\n<h2>Understanding Table Schemas<\/h2>\n<p>\nEvery DataFrame in Apache Spark contains a schema that defines the shape of the data, such as data types, columns, and metadata. With Delta Lake, the table schema is stored in JSON format within the transaction log.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>What is Schema Enforcement?<\/h2>\n<p>\nSchema Enforcement, also known as Schema Validation, is a protective mechanism in Delta Lake that ensures data quality by rejecting records that do not conform to the table schema. Like a hostess at the registration desk of a popular restaurant who only accepts reservations, it checks whether each column of data being entered into the table is on the corresponding list of expected columns (in other words, whether each has a 'reservation'), and rejects any records with columns not on the list.<\/p>\n<h2>How does Schema Enforcement work?<\/h2>\n<p>\nDelta Lake uses schema enforcement upon writing, meaning that all new records being added to the table are checked for compatibility with the schema of the target table at write time. If the schema is incompatible, Delta Lake fully rolls back the transaction (data is not written) and raises an exception to alert the user about the mismatch.<br \/>\nTo determine the compatibility of a record, Delta Lake uses the following rules. The DataFrame being written:<\/p>\n<ul>\n<li>must not contain additional columns that are not present in the target table schema. Conversely, it is acceptable if the incoming data does not contain all the columns from the table\u2014those columns will simply be assigned null values.<\/li>\n<li>must not have data types for its columns that differ from those of the columns in the target table. For instance, if a column in the target table contains data of StringType, but the corresponding column in the DataFrame contains data of IntegerType, forcing schema enforcement will raise an exception and prevent the write operation from proceeding.<\/li>\n<li>cannot contain column names that differ only by case. This means you cannot have columns named 'Foo' and 'foo' defined within the same table. While Spark can be used in case-sensitive or case-insensitive mode (the latter being the default), Delta Lake preserves case but is insensitive within schema storage. Parquet is case-sensitive when storing and retrieving column information. To avoid potential errors, data corruption, or loss (which we have personally encountered at Databricks), we have decided to implement this restriction.<\/li>\n<\/ul>\n<p>\nTo illustrate this, let\u2019s take a look at what happens in the code example below when attempting to add some newly generated columns to a Delta Lake table that is not yet configured to accept them.<\/p>\n<pre><code class=\"apache\"># \u0421\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u043c DataFrame \u0441\u0441\u0443\u0434, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043c\u044b \u0434\u043e\u0431\u0430\u0432\u0438\u043c \u0432 \u043d\u0430\u0448\u0443 \u0442\u0430\u0431\u043b\u0438\u0446\u0443 Delta Lake\nloans = sql(&quot;&quot;&quot;\n            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,\n            CAST(rand(10) * 10000 * count AS double) AS amount\n            FROM loan_by_state_delta\n            &quot;&quot;&quot;)\n\n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u0438\u0441\u0445\u043e\u0434\u043d\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\noriginal_loans.printSchema()\n\nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n \n# \u0412\u044b\u0432\u0435\u0441\u0442\u0438 \u043d\u043e\u0432\u0443\u044e \u0441\u0445\u0435\u043c\u0443 DataFrame\nloans.printSchema()\n \nroot\n  |-- addr_state: string (nullable = true)\n  |-- count: integer (nullable = true)\n  |-- amount: double (nullable = true) # new column\n \n# \u041f\u043e\u043f\u044b\u0442\u043a\u0430 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u043d\u043e\u0432\u044b\u0439 DataFrame (\u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c) \u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0443\u044e \u0442\u0430\u0431\u043b\u0438\u0446\u0443\nloans.write.format(&quot;delta&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_PATH)\n\nReturns:\n\nA schema mismatch detected when writing to the Delta table.\n \nTo enable schema migration, please set:\n'.option(&quot;mergeSchema&quot;, &quot;true&quot;)'\n \nTable schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n \nData schema:\nroot\n-- addr_state: string (nullable = true)\n-- count: long (nullable = true)\n-- amount: double (nullable = true)\n \nIf Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.<\/code><\/pre>\n<p>\nInstead of automatically adding new columns, Delta Lake enforces the schema and halts the write operation. To assist in identifying which column (or columns) is causing the mismatch, Spark outputs both schemas from the stack trace for comparison.<\/p>\n<h2>What is the benefit of enforcing schema?<\/h2>\n<p>\nSince schema enforcement represents a fairly strict check, it serves as an excellent tool for acting as a gatekeeper of clean, fully transformed datasets ready for production or consumption. It is typically applied to tables that directly serve data:<\/p>\n<ul>\n<li>Machine learning algorithms<\/li>\n<li>BI dashboards<\/li>\n<li>Data analytics and visualization tools<\/li>\n<li>Any production system requiring strictly structured, strongly typed semantic schemas.<\/li>\n<\/ul>\n<p>\nTo prepare your data for this final barrier, many users employ a simple multi-hop architecture that gradually adds structure to their tables. To learn more about this, you can check out the article <noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Production-grade machine learning with Delta Lake.<\/a><\/noindex><\/p>\n<p>Of course, schema enforcement can be used anywhere in your pipeline, but be aware that streaming writes to a table can be frustrating if, for example, you've forgotten that you've added another column to the incoming data.<\/p>\n<h2>Preventing data dilution<\/h2>\n<p>\nAt this point, you might be wondering what all the fuss is about. After all, an unexpected 'schema mismatch' error can trip you up in your workflow, especially if you are new to Delta Lake. Why not just allow the schema to change as needed so I can write my DataFrame, come what may?<\/p>\n<p>As the old saying goes, 'an ounce of prevention is worth a pound of cure.' At some point, if you don't take care to enforce your schema, data type compatibility issues will raise their ugly heads\u2014seemingly homogeneous sources of raw data may contain edge cases, corrupted columns, poorly formed mappings, or other scary things that haunt your nightmares. The best approach is to stop these foes at the gate\u2014with schema enforcement\u2014and deal with them in the light, rather than later, when they start prowling in the dark depths of your working code.<\/p>\n<p>Enforcing a schema ensures that your table's schema won't change unless you explicitly allow it. This prevents data 'dilution' that can occur when new columns are added so frequently that previously valuable, compact tables lose their significance and usability due to data flooding. By encouraging you to be deliberate, set high standards, and expect quality, schema enforcement does exactly what it's intended to do \u2014 helps you remain diligent while keeping your tables clean.<\/p>\n<p>If upon further consideration you decide that you actually <i>we need<\/i> need to add a new column \u2014 no problem, here\u2019s a one-liner fix. The solution is schema evolution!<\/p>\n<h2>What is schema evolution?<\/h2>\n<p>\nSchema evolution is a feature that allows users to easily modify the current schema of a table according to data that changes over time. It is most often used when performing an add or overwrite operation to automatically adapt the schema to include one or more new columns.<\/p>\n<h2>How does schema evolution work?<\/h2>\n<p>\nFollowing the example in the previous section, developers can easily use schema evolution to add new columns that were previously rejected due to schema mismatches. Schema evolution is activated by adding <code>.option('mergeSchema', 'true')<\/code> to your Spark command <code>.write or .writeStream.<\/code><\/p>\n<pre><code class=\"apache\"># \u0414\u043e\u0431\u0430\u0432\u044c\u0442\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440 mergeSchema\nloans.write.format(&quot;delta&quot;) \n           .option(&quot;mergeSchema&quot;, &quot;true&quot;) \n           .mode(&quot;append&quot;) \n           .save(DELTALAKE_SILVER_PATH)<\/code><\/pre>\n<p>\nTo view the graph, execute the following Spark SQL query<\/p>\n<pre><code class=\"apache\"># \u0421\u043e\u0437\u0434\u0430\u0439\u0442\u0435 \u0433\u0440\u0430\u0444\u0438\u043a \u0441 \u043d\u043e\u0432\u044b\u043c \u0441\u0442\u043e\u043b\u0431\u0446\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u0434\u0442\u0432\u0435\u0440\u0434\u0438\u0442\u044c, \u0447\u0442\u043e \u0437\u0430\u043f\u0438\u0441\u044c \u043f\u0440\u043e\u0448\u043b\u0430 \u0443\u0441\u043f\u0435\u0448\u043d\u043e\n%sql\nSELECT addr_state, sum(`amount`) AS amount\nFROM loan_by_state_delta\nGROUP BY addr_state\nORDER BY sum(`amount`)\nDESC LIMIT 10<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Diving into Delta Lake: Schema Enforcement and Evolution\" src=\"\/wp-content\/uploads\/2020\/05\/873ce67da00f69696d5b9328bd90793a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nAlternatively, you can set this option for the entire Spark session by adding <code>spark.databricks.delta.schema.autoMerge = True<\/code> to the Spark configuration. But use this cautiously, as schema enforcement will no longer warn you about unintended schema mismatches.<\/p>\n<p>By including the parameter <code>mergeSchema<\/code>, all columns that are present in the DataFrame but absent in the target table are automatically added to the end of the schema within the write transaction. Nested fields may also be added and will be included at the end of the corresponding column structures.<\/p>\n<p>Data engineers and scientists can use this option to add new columns (possibly a newly tracked metric or this month's sales metric column) to their existing machine learning production tables without disrupting existing models based on old columns. <\/p>\n<p>The following types of schema changes are permissible as part of the schema evolution when adding or overwriting a table:<\/p>\n<ul>\n<li>Adding new columns (this is the most common scenario)<\/li>\n<li>Changing data types from NullType -&gt; any other type or promotion from ByteType -&gt; ShortType -&gt; IntegerType<\/li>\n<\/ul>\n<p>\nOther changes not permitted within schema evolution require the schema and data to be overwritten by additions <code>.option(\"overwriteSchema\", \"true\")<\/code>. For example, if the column \"Foo\" was originally an integer and the new schema was to be a string data type, then all Parquet files (data) would need to be rewritten. Such changes include:<\/p>\n<ul>\n<li>removing a column<\/li>\n<li>changing the data type of an existing column (in place)<\/li>\n<li>renaming columns that differ only in case (for example, \"Foo\" and \"foo\")<\/li>\n<\/ul>\n<p>\nFinally, with the upcoming release of Spark 3.0, explicit DDL (using ALTER TABLE) will be fully supported, allowing users to perform the following operations on table schemas:<\/p>\n<ul>\n<li>adding columns<\/li>\n<li>changing column comments<\/li>\n<li>setting table properties that define table behavior, such as setting transaction log retention duration.<\/li>\n<\/ul>\n<p><\/p>\n<h2>What are the benefits of schema evolution?<\/h2>\n<p>\nSchema evolution can be utilized whenever you <i>intend to<\/i> change the schema of your table (as opposed to cases where you accidentally added columns to your DataFrame that should not be there). This is the easiest way to migrate your schema because it automatically adds the correct column names and data types without needing to explicitly declare them.<\/p>\n<h2>Conclusion<\/h2>\n<p>\nEnforcement of schema applies to any new columns or other schema changes that are not compatible with your table. By setting and maintaining these high standards, analysts and engineers can rely on their data having the highest level of integrity, reasoning about it clearly and explicitly, which enables them to make more effective business decisions.<\/p>\n<p>On the other hand, schema evolution complements schema enforcement, simplifying <i>the implied<\/i> automatic changes to the schema. Ultimately, it shouldn't be complicated \u2014 just adding a column.<\/p>\n<p>Schema enforcement is the yin, while schema evolution is the yang. Together, these features effectively simplify noise suppression and signal tuning.<\/p>\n<p><i>We would also like to thank Mukul Murti and Pranav Anand for their contributions to this article.<\/i><\/p>\n<p>Other articles in this series:<\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/21\/diving-into-delta-lake-unpacking-the-transaction-log.html\">Diving into Delta Lake: unpacking the transaction log<\/a><\/noindex><\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"tjb10n5wVs8\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/tjb10n5wVs8\/hqdefault.jpg\" alt=\"Play video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<h2>Related articles<\/h2>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/blog\/2019\/08\/14\/productionizing-machine-learning-with-delta-lake.html\">Production-level machine learning with Delta Lake<\/a><\/noindex><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/databricks.com\/discover\/data-lakes\/introduction\">What is a data lake?<\/a><\/noindex><\/p>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/otus.pw\/J1P5\/\">Learn more about the course<\/a><\/noindex><\/p>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/otus\/blog\/502324\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u041f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u044e \u0432\u0430\u0448\u0435\u043c\u0443 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u044e \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u00abDiving Into Delta Lake: Schema Enforcement &amp; Evolution\u00bb \u0430\u0432\u0442\u043e\u0440\u043e\u0432 Burak Yavuz, Brenner Heintz and Denny Lee, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u044b\u043b \u043f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043b\u0435\u043d \u0432 \u043f\u0440\u0435\u0434\u0434\u0432\u0435\u0440\u0438\u0438 \u0441\u0442\u0430\u0440\u0442\u0430 \u043a\u0443\u0440\u0441\u0430 \u00abData Engineer\u00bb \u043e\u0442 OTUS. \u0414\u0430\u043d\u043d\u044b\u0435, \u043a\u0430\u043a \u0438 \u043d\u0430\u0448 \u043e\u043f\u044b\u0442, \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u043d\u0430\u043a\u0430\u043f\u043b\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u0438 \u0440\u0430\u0437\u0432\u0438\u0432\u0430\u044e\u0442\u0441\u044f. \u0427\u0442\u043e\u0431\u044b \u043d\u0435 \u043e\u0442\u0441\u0442\u0430\u0432\u0430\u0442\u044c, \u043d\u0430\u0448\u0438 \u043c\u0435\u043d\u0442\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u043e\u0434\u0435\u043b\u0438 \u043c\u0438\u0440\u0430 \u0434\u043e\u043b\u0436\u043d\u044b \u0430\u0434\u0430\u043f\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u043a \u043d\u043e\u0432\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u043c, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":82039,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-82038","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-18T23:42:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Diving into Delta Lake: Schema Enforcement and Evolution | ProHoster","description":"Hello, Habr!","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 Delta Lake: \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438 \u044d\u0432\u043e\u043b\u044e\u0446\u0438\u044f \u0441\u0445\u0435\u043c\u044b | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440!","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/pogruzhenie-v-delta-lake-prinuditelnoe-primenenie-i-evolyucziya-shemy","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-18T23:42:26+00:00","article:modified_time":"2020-05-18T23:42:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"82038","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 15:45:28","updated":"2022-09-29 13:43:45","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/82038","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=82038"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/82038\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/82039"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=82038"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=82038"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=82038"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}