{"id":37567,"date":"2019-10-31T22:18:22","date_gmt":"2019-10-31T19:18:22","guid":{"rendered":"https:\/\/prohoster.info\/blog\/apache-nifi-kratkij-obzor-vozmozhnostej-na-praktike\/"},"modified":"2019-10-31T22:18:22","modified_gmt":"2019-10-31T19:18:22","slug":"apache-nifi-kratkij-obzor-vozmozhnostej-na-praktike","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/apache-nifi-kratkij-obzor-vozmozhnostej-na-praktike","title":{"rendered":"Apache NIFI \u2014 A Brief Overview of Practical Capabilities","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Introduction<\/h3>\n

<\/p>\n

It so happened that at my current job, I had to get acquainted with this technology. Let me start with a brief backstory. During a recent meeting, our team was told that we needed to create an integration with a well-known system<\/em>. The integration meant that this well-known system would send us requests via HTTP to a specific endpoint, and we, strangely enough, would send back responses in the form of SOAP messages. It seems all simple and trivial. This implies that we need\u2026<\/a><\/noindex><\/p>\n

<\/p>\n

Task<\/h3>\n

<\/p>\n

Create 3 services. The first of these is the Database Update Service. This service updates the data in the database and generates a file in CSV format when new data arrives from an external system, to be sent to the next system. The endpoint of the second service \u2013 the FTP Transfer Service \u2013 is called, which receives the transmitted file, validates it, and stores it in file storage via FTP. The third service \u2013 the Data Delivery Service \u2013 works asynchronously with the first two. It accepts a request from an external system for the file mentioned above, takes the prepared response file, modifies it (updates the fields id, description, linkToFile), and sends the response as a SOAP message. So, the overall picture is as follows: the first two services start their work only when data for updating arrives. The third service works constantly since there are many information consumers, about 1000 requests for data retrieval per minute. The services are available constantly, and their instances are located in different environments such as testing, demo, pre-production, and production. Below is a diagram illustrating the operation of these services. I should clarify that some details are simplified to avoid unnecessary complexity.<\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

Technical deep dive<\/h3>\n

<\/p>\n

When planning a solution to the task, we initially decided to create applications in Java using the Spring framework, with Nginx as a load balancer, a Postgres database, and other technical and non-technical components. Since the time allocated for developing the technical solution allowed us to consider other approaches to this problem, we took a look at the trendy technology Apache NIFI, which is popular in certain circles. I must say that this technology enabled us to notice these 3 services. This article will describe the development of the file transport service and the data transfer service to the consumer; however, if the article resonates well, I will write about the data updating service in the database.<\/p>\n

<\/p>\n

What is it?<\/h3>\n

<\/p>\n

NIFI represents a distributed architecture for fast parallel loading and processing of data, a large number of plugins for sources and transformations, configuration versioning, and much more. A nice bonus is that it is very easy to use. Trivial processes, such as getFile, sendHttpRequest, and others, can be represented as squares. Each square represents a certain process, the interaction of which can be seen in the illustration below. More detailed documentation on how to interact with and configure processes is provided here<\/a><\/noindex> , for those who prefer Russian \u2014 here<\/a><\/noindex>. The documentation is well-written, detailing how to unpack and run NIFI, as well as how to create processes, the so-called squares.
\nThe idea to write this article arose after extensive searching and structuring of the gathered information into something coherent, as well as the desire to make life a bit easier for future developers.<\/p>\n

<\/p>\n

Example<\/h3>\n

<\/p>\n

This example examines how squares interact with each other. The overall scheme is quite simple: We receive an HTTP request (theoretically with a file in the body of the request. For demonstration purposes in NIFI, in this example the request starts the process of retrieving a file from the local storage), then we send back a response that the request has been received, while simultaneously initiating the process of retrieving the file from the storage and then the process of moving it via FTP to the storage. It is worth explaining that the processes interact with each other through what is called a flowFile. This is a basic entity in NIFI, which stores attributes and content. The content is the data presented in the stream file. So, roughly speaking, if you received a file from one square and are transferring it to another, the content will be your file.<\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

As you can see \u2014 this diagram depicts the overall process. HandleHttpRequest \u2014 receives requests, ReplaceText \u2014 generates the response body, HandleHttpResponse \u2014 sends back the response. FetchFile \u2014 retrieves the file from the storage and passes it to the PutSftp square \u2014 which places the file on FTP at the specified address. Now let's take a closer look at this process. <\/p>\n

<\/p>\n

In this case \u2014 request is the start of everything. Let's take a look at its configuration parameters. <\/p>\n

<\/p>\n

\"Apache<\/p>\n

\nEverything here is quite trivial except for StandardHttpContextMap \u2014 this is a service that allows sending and receiving requests. More details and even examples can be found \u2014 here<\/a><\/noindex><\/p>\n

Next, let's look at the configuration parameters of the ReplaceText square. Pay attention to ReplacementValue \u2014 this is what will be returned to the user as a response. In settings, you can adjust the logging level; the logs can be viewed at {where you unpacked nifi}\/nifi-1.9.2\/logs where there are also parameters for failure\/success \u2014 based on these parameters, you can regulate the process as a whole. So, in case of successful text processing \u2014 the process of sending a response to the user will be triggered, while in another case we will simply log the unsuccessful process. <\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

In the properties of HandleHttpResponse, there is nothing particularly interesting except for the status upon successful creation of the response. <\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

Having dealt with the request and response \u2014 let\u2019s move on to retrieving the file and placing it on the FTP server. FetchFile \u2014 retrieves the file from the path specified in the settings and transfers it to the next process. <\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

The PutSftp block places the file in the file storage. The configuration parameters can be seen below.<\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

It is important to note that each block is a separate process that must be initiated. We examined the simplest example, which does not require any complex customization. Next, we will discuss a slightly more complicated process, where we will write a bit in Groovy. <\/p>\n

<\/p>\n

A more complex example<\/h3>\n

<\/p>\n

The data transfer service to the consumer has become somewhat more complex due to the process of modifying the SOAP message. The overall process is depicted in the diagram below.<\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

Here, the idea is also not particularly complicated: we received a request from the consumer for data, sent a response confirming the message receipt, initiated the process of obtaining the response file, then edited it with certain logic, after which we transmitted the file to the consumer as a SOAP message to the server. <\/p>\n

<\/p>\n

I don't think it's necessary to describe the blocks we've seen above again \u2014 let's move directly to new ones. If you need to edit any file and standard blocks like ReplaceText are not suitable, you will have to write your own script. This can be done using the ExecuteGroovyScript block. Its settings are presented below.<\/p>\n

<\/p>\n

\"Apache<\/p>\n

<\/p>\n

There are two ways to load a script into this square. The first is by uploading a file with the script. The second is by inserting the script into scriptBody. As far as I know, the executeScript square supports several programming languages \u2014 one of them is Groovy. I must disappoint Java developers \u2014 you cannot write scripts in Java in such squares. For those who really want to \u2014 you need to create your own custom square and add it to the NIFI system. This entire operation involves quite lengthy rituals, which we won't delve into in this article. I chose the Groovy language. Below is a test script that simply incrementally updates the id in the SOAP message. It's important to note that you take a file from flowFile, update it, and don\u2019t forget to put the updated file back. It\u2019s also worth mentioning that not all libraries are connected. It may turn out that you will still have to import one of the libraries. Another downside is that debugging the script in this square is quite difficult. There is a way to connect to the NIFI JVM and start the debugging process. Personally, I launched a local application and simulated receiving a file from the session. I also performed debugging locally. The errors that arise when loading the script can be easily Googled and are logged by NIFI itself.<\/p>\n

<\/p>\n

import org.apache.commons.io.IOUtils\nimport groovy.xml.XmlUtil\nimport java.nio.charset.*\nimport groovy.xml.StreamingMarkupBuilder\n\ndef flowFile = session.get()\nif (!flowFile) return\ntry {\n    flowFile = session.write(flowFile, { inputStream, outputStream ->\n        String result = IOUtils.toString(inputStream, \"UTF-8\");\n        def recordIn = new XmlSlurper().parseText(result)\n        def element = recordIn.depthFirst().find {\n            it.name() == 'id'\n        }\n\n        def newId = Integer.parseInt(element.toString()) + 1\n        def recordOut = new XmlSlurper().parseText(result)\n        recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId\n\n        def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()\n        outputStream.write(res.getBytes(StandardCharsets.UTF_8))\n} as StreamCallback)\n     session.transfer(flowFile, REL_SUCCESS)\n}\ncatch(Exception e) {\n    log.error(\"Error during processing of validate.groovy\", e)\n    session.transfer(flowFile, REL_FAILURE)\n}<\/code><\/pre>\n
<\/p>\n
This is where the customization of the square ends. The updated file is then passed to the square that sends the file to the server. Below are the settings for this square.<\/p>\n
<\/p>\n
\"Apache<\/p>\n
<\/p>\n
We describe the method by which the SOAP message will be sent. We specify where to send it. Next, we need to indicate that this is specifically SOAP.<\/p>\n
<\/p>\n
\"Apache<\/p>\n
<\/p>\n
We are adding several properties such as host and action (soapAction). We save and check. For more details on how to send SOAP requests, you can take a look. here<\/a><\/noindex><\/p>\n
<\/p>\n
We explored several use cases of NIFI processes. How they interact and what real benefits they provide. The examples discussed are test cases and slightly differ from what is actually in production. I hope this article will be somewhat useful for developers. Thank you for your attention. If you have any questions, feel free to reach out. I'll do my best to respond.<\/p>\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0412\u0432\u0435\u0434\u0435\u043d\u0438\u0435 \u0422\u0430\u043a \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c, \u0447\u0442\u043e \u043d\u0430 \u043c\u043e\u0435\u043c \u0442\u0435\u043a\u0443\u0449\u0435\u043c \u043c\u0435\u0441\u0442\u0435 \u0440\u0430\u0431\u043e\u0442\u044b \u043c\u043d\u0435 \u043f\u0440\u0438\u0448\u043b\u043e\u0441\u044c \u043f\u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0434\u0430\u043d\u043d\u043e\u0439 \u0442\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0435\u0439. \u041d\u0430\u0447\u043d\u0443 \u0441 \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u043f\u0440\u0435\u0434\u044b\u0441\u0442\u043e\u0440\u0438\u0438. \u041d\u0430 \u043e\u0447\u0435\u0440\u0435\u0434\u043d\u043e\u043c \u043c\u0438\u0442\u0438\u043d\u0433\u0435, \u043d\u0430\u0448\u0435\u0439 \u043a\u043e\u043c\u0430\u043d\u0434\u0435 \u0441\u043a\u0430\u0437\u0430\u043b\u0438, \u0447\u0442\u043e \u043d\u0443\u0436\u043d\u043e \u0441\u043e\u0437\u0434\u0430\u0442\u044c \u0438\u043d\u0442\u0435\u0433\u0440\u0430\u0446\u0438\u044e \u0441 \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u043e\u0439. \u041f\u043e\u0434 \u0438\u043d\u0442\u0435\u0433\u0440\u0430\u0446\u0438\u0435\u0439 \u043f\u043e\u0434\u0440\u0430\u0437\u0443\u043c\u0435\u0432\u0430\u043b\u043e\u0441\u044c, \u0447\u0442\u043e \u044d\u0442\u0430 \u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u0441\u0438\u0441\u0442\u0435\u043c\u0430 \u0431\u0443\u0434\u0435\u0442 \u043d\u0430\u043c \u0441\u043b\u0430\u0442\u044c \u0437\u0430\u043f\u0440\u043e\u0441\u044b \u0447\u0435\u0440\u0435\u0437 HTTP \u043d\u0430 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u043d\u044b\u0439 \u0435\u043d\u0434\u043f\u043e\u0438\u043d\u0442, \u0430 \u043c\u044b, \u043a\u0430\u043a \u044d\u0442\u043e \u043d\u0438 \u0441\u0442\u0440\u0430\u043d\u043d\u043e, \u0441\u043b\u0430\u0442\u044c […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":28196,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-37567","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Apache NIFI \u2014 A Brief Overview of Practical Capabilities | ProHoster","description":"Introduction It so happened that at my current job, I had to become familiar with this technology. Let me start with a brief backstory.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/news\/apache-nifi-kratkij-obzor-vozmozhnostej-na-praktike","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Apache NIFI \u2014 \u041a\u0440\u0430\u0442\u043a\u0438\u0439 \u043e\u0431\u0437\u043e\u0440 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0435\u0439 \u043d\u0430 \u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0435 | ProHoster","og:description":"\u0412\u0432\u0435\u0434\u0435\u043d\u0438\u0435 \u0422\u0430\u043a \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c, \u0447\u0442\u043e \u043d\u0430 \u043c\u043e\u0435\u043c \u0442\u0435\u043a\u0443\u0449\u0435\u043c \u043c\u0435\u0441\u0442\u0435 \u0440\u0430\u0431\u043e\u0442\u044b \u043c\u043d\u0435 \u043f\u0440\u0438\u0448\u043b\u043e\u0441\u044c \u043f\u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0434\u0430\u043d\u043d\u043e\u0439 \u0442\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0435\u0439. \u041d\u0430\u0447\u043d\u0443 \u0441 \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u043f\u0440\u0435\u0434\u044b\u0441\u0442\u043e\u0440\u0438\u0438.","og:url":"https:\/\/prohoster.info\/en\/blog\/news\/apache-nifi-kratkij-obzor-vozmozhnostej-na-praktike","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:18:22+00:00","article:modified_time":"2019-10-31T19:18:22+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"37567","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-23 18:24:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:24:24","updated":"2026-01-23 18:24:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/37567","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=37567"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/37567\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/28196"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=37567"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=37567"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=37567"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}