{"id":30268,"date":"2019-10-31T21:34:35","date_gmt":"2019-10-31T18:34:35","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-peremeshhat-vygruzhat-i-integrirovat-ochen-bolshie-dannye-dyoshevo-i-bystro-chto-takoe-pushdown-optimizatsiya\/"},"modified":"2019-10-31T21:34:35","modified_gmt":"2019-10-31T18:34:35","slug":"kak-peremeshhat-vygruzhat-i-integrirovat-ochen-bolshie-dannye-dyoshevo-i-bystro-chto-takoe-pushdown-optimizatsiya","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/kak-peremeshhat-vygruzhat-i-integrirovat-ochen-bolshie-dannye-dyoshevo-i-bystro-chto-takoe-pushdown-optimizatsiya","title":{"rendered":"How to move, unload, and integrate very large data cheaply and quickly? What is pushdown optimization?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Any operation involving large data sets requires significant computing power. Moving data from a database to Hadoop can take weeks or cost as much as an airplane wing. Don't want to wait and waste money? Balance the load across different platforms. One way to do this is through pushdown optimization.<\/p>\n

<\/p>\n

I asked leading trainer in Russia for the development and administration of Informatica products, Alexey Ananyev, to explain the pushdown optimization feature in Informatica Big Data Management (BDM). Did you ever learn to work with Informatica products? Chances are, it was Alexey who taught you the basics of PowerCenter and explained how to build mappings. <\/p>\n

<\/p>\n

Alexey Ananyev, Head of the Training Division at DIS Group<\/strong><\/p>\n

<\/p>\n

What is pushdown?<\/strong><\/p>\n

<\/p>\n

Many of you are already familiar with Informatica Big Data Management (BDM). The product can integrate large data from various sources, move it between different systems, provide easy access to it, allow profiling, and much more.
\nIn skilled hands, BDM can work wonders: tasks will be performed quickly and with minimal computational resources. <\/p>\n

<\/p>\n

Do you want this too? Learn to use the pushdown feature in BDM to distribute computational load between different platforms. The pushdown technology allows you to turn a mapping into a script and select the environment in which this script will run. The ability to choose allows you to combine the strengths of different platforms and achieve their maximum performance.<\/p>\n

<\/p>\n

To configure the script execution environment, you need to select the type of pushdown. The script can be fully executed on Hadoop or partially distributed between the source and the sink. There are 4 possible types of pushdown. A mapping can remain as a native mapping without being turned into a script. The mapping can be executed as much as possible on the source (source) or fully on the source (full). The mapping can also be turned into a Hadoop script (none).<\/p>\n

<\/a><\/noindex><\/p>\n

Pushdown optimization<\/strong><\/p>\n

<\/p>\n

These 4 types can be combined in various ways to optimize pushdown for the specific needs of the system. For example, it is often more effective to extract data from the database using its own capabilities and process the data using Hadoop to avoid overloading the database.<\/p>\n

<\/p>\n

Let\u2019s consider a case where both the source and the receiver are in the database, and the execution platform for transformations can be chosen: depending on the settings, this could be Informatica, a database server, or Hadoop. This example will best illustrate the technical side of this mechanism. Naturally, in real life, such a situation does not occur, but it is best suited for demonstrating functionality. <\/p>\n

<\/p>\n

Let\u2019s take a mapping for reading two tables in a single Oracle database. And let the results of the reading be written to a table in the same database. The mapping scheme will look like this:<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

In the form of mapping in Informatica BDM 10.2.1, it looks like this:<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

Pushdown type \u2013 native<\/strong><\/p>\n

<\/p>\n

If we choose the native pushdown type, the mapping will be executed on server<\/a> Informatica. Data will be read from the Oracle server, transferred to the Informatica server, transformed there, and sent to Hadoop. In other words, we will get a typical ETL process.<\/p>\n

<\/p>\n

Pushdown type \u2013 source <\/strong><\/p>\n

<\/p>\n

By selecting the source type, we are able to distribute our process between proxy server<\/a> the databases (DB) and Hadoop. When executing the process with this setting, queries will be sent to the database to retrieve data from the tables. The rest will be executed in steps on Hadoop.
\nThe execution scheme will look like this:<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

Below is an example of setting up the execution environment.<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

In this case, mapping will be executed in two steps. In its settings, we will see that it has turned into a script that will be sent to the source. Moreover, joining the tables and transforming the data will be done as a rewritten query at the source.
\nIn the image below, we see the optimized mapping in BDM, and at the source \u2013 the rewritten query.<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

The role of Hadoop in this configuration will be reduced to managing the data flow \u2013 conducting it. The result of the query will be directed to Hadoop. After the reading is complete, the file from Hadoop will be written to the receiver.<\/p>\n

<\/p>\n

Pushdown type \u2013 full <\/strong><\/p>\n

<\/p>\n

When selecting the full type, the mapping will completely turn into a query on the database. The result of the query will be directed to Hadoop. The scheme of such a process is presented below.<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

An example of the setup is presented below.<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

As a result, we will get an optimized mapping similar to the previous one. The only difference is that all the logic is transferred to the receiver in the form of overriding its insertion. An example of the optimized mapping is presented below.<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

Here, as in the previous case, Hadoop acts as the conductor. However, here the entire source is read, and then at the receiver level, the logic for data processing is executed.<\/p>\n

<\/p>\n

Pushdown type \u2013 null<\/strong><\/p>\n

<\/p>\n

And the last variant is the pushdown type, within which our mapping will turn into a script on Hadoop.<\/p>\n

<\/p>\n

The optimized mapping will now look like this:<\/p>\n

<\/p>\n

\"How<\/p>\n

<\/p>\n

Here, the data from the source files will first be read on Hadoop. Then, using its tools, these two files will be merged. After this, the data will be transformed and exported to the database.<\/p>\n

<\/p>\n

By understanding the principles of pushdown optimization, many processes for handling big data can be organized very efficiently. For instance, just recently, a large company exported massive data from storage to Hadoop in just a few weeks, data that had been collected over several years.<\/p>\n

Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041b\u044e\u0431\u0430\u044f \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u044f \u0441 \u0431\u043e\u043b\u044c\u0448\u0438\u043c\u0438 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0432\u044b\u0447\u0438\u0441\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043c\u043e\u0449\u043d\u043e\u0441\u0442\u0435\u0439. \u041e\u0431\u044b\u0447\u043d\u043e\u0435 \u043f\u0435\u0440\u0435\u043c\u0435\u0449\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u0437 \u0431\u0430\u0437\u044b \u043d\u0430 Hadoop \u043c\u043e\u0436\u0435\u0442 \u0434\u043b\u0438\u0442\u044c\u0441\u044f \u043d\u0435\u0434\u0435\u043b\u044f\u043c\u0438 \u0438\u043b\u0438 \u0441\u0442\u043e\u0438\u0442\u044c, \u043a\u0430\u043a \u043a\u0440\u044b\u043b\u043e \u0441\u0430\u043c\u043e\u043b\u0451\u0442\u0430. \u041d\u0435 \u0445\u043e\u0442\u0438\u0442\u0435 \u0436\u0434\u0430\u0442\u044c \u0438 \u0442\u0440\u0430\u0442\u0438\u0442\u044c\u0441\u044f? \u0421\u0431\u0430\u043b\u0430\u043d\u0441\u0438\u0440\u0443\u0439\u0442\u0435 \u043d\u0430\u0433\u0440\u0443\u0437\u043a\u0443 \u043d\u0430 \u0440\u0430\u0437\u043d\u044b\u0435 \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u044b. \u041e\u0434\u0438\u043d \u0438\u0437 \u0441\u043f\u043e\u0441\u043e\u0431\u043e\u0432 \u2013 pushdown-\u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u044f. \u042f \u043f\u043e\u043f\u0440\u043e\u0441\u0438\u043b \u0432\u0435\u0434\u0443\u0449\u0435\u0433\u043e \u0432 \u0420\u043e\u0441\u0441\u0438\u0438 \u0442\u0440\u0435\u043d\u0435\u0440\u0430 \u043f\u043e \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0435 \u0438 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432 Informatica \u0410\u043b\u0435\u043a\u0441\u0435\u044f \u0410\u043d\u0430\u043d\u044c\u0435\u0432\u0430 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-30268","post","type-post","status-publish","format-standard","hentry","category-blog"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47How to move, export, and integrate very large data cheaply and quickly? What is pushdown optimization? | ProHoster","description":"Any operation with big data requires significant computing power. Transferring data from a database to Hadoop can take weeks or cost as much as an airplane wing.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/kak-peremeshhat-vygruzhat-i-integrirovat-ochen-bolshie-dannye-dyoshevo-i-bystro-chto-takoe-pushdown-optimizatsiya","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043f\u0435\u0440\u0435\u043c\u0435\u0449\u0430\u0442\u044c, \u0432\u044b\u0433\u0440\u0443\u0436\u0430\u0442\u044c \u0438 \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043e\u0447\u0435\u043d\u044c \u0431\u043e\u043b\u044c\u0448\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u0451\u0448\u0435\u0432\u043e \u0438 \u0431\u044b\u0441\u0442\u0440\u043e? \u0427\u0442\u043e \u0442\u0430\u043a\u043e\u0435 pushdown-\u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u044f? | ProHoster","og:description":"\u041b\u044e\u0431\u0430\u044f \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u044f \u0441 \u0431\u043e\u043b\u044c\u0448\u0438\u043c\u0438 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0432\u044b\u0447\u0438\u0441\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043c\u043e\u0449\u043d\u043e\u0441\u0442\u0435\u0439. \u041e\u0431\u044b\u0447\u043d\u043e\u0435 \u043f\u0435\u0440\u0435\u043c\u0435\u0449\u0435\u043d\u0438\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u0437 \u0431\u0430\u0437\u044b \u043d\u0430 Hadoop \u043c\u043e\u0436\u0435\u0442 \u0434\u043b\u0438\u0442\u044c\u0441\u044f \u043d\u0435\u0434\u0435\u043b\u044f\u043c\u0438 \u0438\u043b\u0438 \u0441\u0442\u043e\u0438\u0442\u044c, \u043a\u0430\u043a \u043a\u0440\u044b\u043b\u043e \u0441\u0430\u043c\u043e\u043b\u0451\u0442\u0430.","og:url":"https:\/\/prohoster.info\/en\/blog\/kak-peremeshhat-vygruzhat-i-integrirovat-ochen-bolshie-dannye-dyoshevo-i-bystro-chto-takoe-pushdown-optimizatsiya","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:34:35+00:00","article:modified_time":"2019-10-31T18:34:35+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30268","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"Article","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-09 21:49:15","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:38:39","updated":"2026-02-09 21:49:15","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/30268","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=30268"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/30268\/revisions"}],"predecessor-version":[{"id":160240,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/30268\/revisions\/160240"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=30268"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=30268"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=30268"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}