Apache NIFI - Pfupi Pfupi yeZvinhu muKudzidzira

Nhanganyaya

Zvakazoitika kuti pabasa pangu pari zvino ndaifanira kujairana neruzivo urwu. Ndichatanga nekumashure zvishoma. Pamusangano wakatevera, timu yedu yakaudzwa kuti taifanira kugadzira kubatanidzwa inozivikanwa system. Nekubatanidza zvaireva kuti iyi inozivikanwa system yaizotitumira zvikumbiro kuburikidza neHTTP kune imwe nzvimbo yekupedzisira, uye isu, zvisingaite, taizotumira mhinduro nenzira yeSOAP meseji. Zvose zvinoratidzika zviri nyore uye zvisingakoshi. Kubva pane izvi zvinotevera kuti unoda ...

Basa

Gadzira masevhisi matatu. Yekutanga yavo ndeye Database Update Service. Iyi sevhisi, kana data nyowani yasvika kubva kune yechitatu-bato system, inovandudza iyo data mudhatabhesi uye inogadzira faira muCSV fomati kuti iendese kune inotevera system. Mhedziso yebasa rechipiri inonzi - FTP Transport Service, iyo inogamuchira faira yakatamiswa, inoisimbisa, uye inoiisa mukuchengetedza mafaira kuburikidza neFTP. Iyo yechitatu sevhisi, iyo yevatengi data data sevhisi, inoshanda asynchronously nevaviri vekutanga. Iyo inogamuchira chikumbiro kubva kune yechitatu-bato rekunze system kuti igamuchire faira yakurukurwa pamusoro, inotora yakagadzirira mhinduro faira, inoishandura (inogadzirisa id, tsananguro, linkToFile minda) uye inotumira mhinduro nenzira yeSOAP meseji. Ndiko kuti, mufananidzo wakazara ndewekutevera: masevhisi maviri ekutanga anotanga basa rawo chete kana data rekuvandudza rasvika. Iyo yechitatu sevhisi inoshanda nguva dzose nekuti kune vakawanda vatengi veruzivo, nezve 3 zvikumbiro zve data paminiti. Masevhisi anowanikwa nguva dzose uye zviitiko zvavo zviri munzvimbo dzakasiyana, senge bvunzo, demo, pre-kugadzirwa uye prod. Pazasi pane mufananidzo wekuti masevhisi aya anoshanda sei. Rega ndijekese ipapo kuti zvimwe zvinhu zvakarerutswa kudzivirira kuoma kusingakoshi.

Technical Deepening

Pakuronga mhinduro kudambudziko, takatanga tafunga kuita zvikumbiro muJava tichishandisa iyo Spring framework, Nginx balancer, Postgres dhatabhesi uye humwe hunyanzvi uye kwete zvinhu zvehunyanzvi. Sezvo nguva yekugadzira mhinduro yehunyanzvi yakatibvumira kufunga dzimwe nzira dzekugadzirisa dambudziko iri, maziso edu akawira paApache NIFI tekinoroji, iri fashoni mune mamwe madenderedzwa. Ini ndichataura ipapo kuti tekinoroji iyi yakatibvumira kuona aya masevhisi matatu. Ichi chinyorwa chichatsanangura kuvandudzwa kwesevhisi yekufambisa faira uye sevhisi yekufambisa data kumutengi, asi kana chinyorwa chinobatsira, ini ndichanyora nezve sevhisi yekuvandudza data mudhatabhesi.

Chii ichocho?

NIFI chivakwa chakagoverwa chekukurumidza kufanana kurodha uye kugadzirisa data, nhamba huru yemapulagi kune masosi uye shanduko, shanduro yezvigadziriso uye zvimwe zvakawanda. Bhonasi yakanaka ndeyekuti iri nyore kushandisa. Trivial maitiro akadai getFile, sendHttpRequest uye mamwe anogona kumiririrwa semakwere. Chikwere chimwe nechimwe chinomiririra maitiro, kupindirana kwacho kunogona kuonekwa mumufananidzo uri pasi apa. Mamwe magwaro akadzama nezve process setup interactions akanyorwa pano , kune avo vanotaura chiRussian - pano. Zvinyorwa zvinonyatso tsanangura nzira yekuburitsa nekumhanyisa NIFI, pamwe nemagadzirirwo ekugadzira maitiro, anozivikanwawo semakwere.
Pfungwa yekunyora chinyorwa yakazvarwa mushure mekutsvaga kwenguva refu nekugadzirisa ruzivo rwakagamuchirwa mune chimwe chinhu chinoziva, pamwe nechishuwo chekuita kuti hupenyu huve nyore kune vanozogadzira ramangwana.

Muenzaniso:

Muenzaniso wekuti masikweya anodyidzana sei anotariswa. Iyo general scheme iri nyore: Isu tinogashira chikumbiro cheHTTP (Muchirevo, ine faira mumuviri wechikumbiro. Kuti uratidze kugona kweNIFI, mumuenzaniso uyu chikumbiro chinotanga nzira yekugamuchira faira kubva kunzvimbo yekuchengetedza faira. ), tobva tatumira zvakare mhinduro yekuti chikumbiro chagamuchirwa, nenzira yakafanana nzira yekugamuchira faira kubva kuFH uyezve nzira yekuifambisa kuburikidza neFTP kuenda kuFH. Izvo zvakakodzera kujekesa kuti maitiro anodyidzana nemumwe kuburikidza neanonzi flowFile. Ichi ndicho chikamu chekutanga muNIFI chinochengeta hunhu uye zvirimo. Zvemukati ndiyo data inomiririrwa neiyo rwizi faira. Ndokureva, kutaura, kana iwe ukagashira faira kubva kune imwe mativi uye woiendesa kune imwe, zvirimo zvichave faira rako.

Sezvauri kuona, mufananidzo uyu unoratidza maitiro ese. HandleHttpRequest - inogamuchira zvikumbiro, ReplaceText - inogadzira muviri wekupindura, HandleHttpResponse - inotumira mhinduro. FetchFile - inogamuchira faira kubva kune faira chengetedzo, inoiendesa kune yakaenzana PutSftp - inoisa iyi faira paFTP, pakero yakatarwa. Zvino zvakawanda pamusoro pemaitiro aya.

Muchiitiko ichi, kukumbira ndiko kutanga kwezvose. Ngatitarisei magadzirirwo ayo paramita.

Zvese zviri pano zvidiki kunze kweStandardHttpContextMap - iyi imhando yesevhisi inobvumidza iwe kutumira uye kugamuchira zvikumbiro. Mune zvimwe zvakadzama uye kunyangwe nemienzaniso, unogona kuona - pano

Tevere, ngatitarisei kuReplaceText kumisikidza maparamendi echikwere. Zvakakodzera kuteerera ReplacementValue - izvi ndizvo zvichadzoserwa kumushandisi nenzira yemhinduro. Muzvirongwa iwe unogona kugadzirisa chiyero chekutema, iwe unogona kuona matanda {pawakasunungura nifi}/nifi-1.9.2/logs, panewo kukundikana/kubudirira parameters - zvichibva pane izvi paramita unogona kudzora maitiro ese. . Ndiko kuti, munyaya yekubudirira kwekunyora kwemavara, nzira yekutumira mhinduro kumushandisi ichadanwa, uye mune imwe nyaya isu tichangonyora maitiro asina kubudirira.

Hapana chinhu chinonyanya kufadza muHandleHttpResponse zvivakwa kunze kwechimiro kana mhinduro yagadzirwa zvinobudirira.

Isu takagadzirisa chikumbiro nemhinduro - ngatienderere mberi nekugamuchira faira uye kuriisa paFTP server. FetchFile - inogamuchira faira munzira inotsanangurwa muzvirongwa uye inopfuudza kune inotevera maitiro.

Uye ipapo iyo PutSftp sikweya - inoisa iyo faira mukuchengetedza faira. Tinogona kuona zvigadziriso parameters pazasi.

Izvo zvakakodzera kutarisisa kune chokwadi chekuti imwe neimwe square inzira yakaparadzana inofanirwa kutangwa. Takatarisa muenzaniso wakapusa usingade chero yakaoma kugadzirisa. Zvadaro, tichatarisa nzira yacho zvishoma zvakaoma, apo tichanyora zvishoma pane grooves.

Muenzaniso wakaoma

Iyo data yekufambisa sevhisi kumutengi yakave yakanyanya kuomarara nekuda kwemaitiro ekugadzirisa iyo SOAP meseji. Iyo general process inoratidzwa mumufananidzo uri pasi apa.

Pano pfungwa yacho haina kunyanya kuoma: takagamuchira chikumbiro kubva kumutengi kuti aida data, akatumira mhinduro kuti akanga agamuchira meseji, akatanga nzira yekugamuchira faira remhinduro, ndokuzorigadzirisa neimwe pfungwa, uye ipapo. yakaendesa faira kumutengi nenzira yeSOAP meseji kune server.

Ini ndinofunga hapana chikonzero chekutsanangura zvakare iwo masikweya atawona pamusoro - ngatifambei takananga kune matsva. Kana iwe uchida kugadzirisa chero faira uye akajairwa ReplaceText mhando masikweya haana kukodzera, uchafanirwa kunyora yako script. Izvi zvinogona kuitwa uchishandisa iyo ExecuteGroogyScript square. Zvirongwa zvayo zvinoratidzwa pazasi.

Pane nzira mbiri dzekuisa script muchikwere ichi. Yekutanga ndeyekudhawunirodha faira rine script. Yechipiri ndeyekuisa script muscriptBody. Sekuziva kwangu, iyo executeScript square inotsigira akati wandei mitauro - imwe yacho ndeye groovy. Ini ndichaodza mwoyo vanogadzira java - haugone kunyora zvinyorwa mujava munzvimbo dzakadai. Kune avo vanonyatsoda, iwe unofanirwa kugadzira yako tsika sikweya uye woiwedzera kune iyo NIFI system. Uku kuvhiya kwese kunoperekedzwa nekutamba kwakareba netamborini, izvo zvatisingazobata nazvo munyaya ino. Ndakasarudza mutauro wegroovy. Pazasi pane bvunzo script iyo inongowedzera kuwedzera id mune SOAP meseji. Zvakakosha kucherechedza. Iwe unotora iyo faira kubva flowFile uye kuigadzirisa, usakanganwa kuti iwe unofanirwa kuidzosera ipapo, yakagadziridzwa. Zvakakoshawo kuziva kuti haasi mabhuku emabhuku ose anobatanidzwa. Zvinogona kuitika kuti iwe uchiri kufanira kuunza imwe ye libs. Chimwe chinokanganisa ndechekuti script mune iyi sikweya yakaoma kugadzirisa. Pane nzira yekubatanidza kuNIFI JVM uye tanga iyo debugging maitiro. Ini pachangu, ndakatangisa application yemuno uye ndakatevedzera kugamuchira faira kubva muchikamu. Ndakaitawo debugging munharaunda. Zvikanganiso zvinoonekwa pakurodha script zviri nyore kuGoogle uye zvinonyorwa neNIFI pachayo kulogi.

import org.apache.commons.io.IOUtils
import groovy.xml.XmlUtil
import java.nio.charset.*
import groovy.xml.StreamingMarkupBuilder

def flowFile = session.get()
if (!flowFile) return
try {
    flowFile = session.write(flowFile, { inputStream, outputStream ->
        String result = IOUtils.toString(inputStream, "UTF-8");
        def recordIn = new XmlSlurper().parseText(result)
        def element = recordIn.depthFirst().find {
            it.name() == 'id'
        }

        def newId = Integer.parseInt(element.toString()) + 1
        def recordOut = new XmlSlurper().parseText(result)
        recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId

        def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()
        outputStream.write(res.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
     session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
    log.error("Error during processing of validate.groovy", e)
    session.transfer(flowFile, REL_FAILURE)
}

Chaizvoizvo, apa ndipo panoperera kugadzirisa kweskweya. Tevere, iyo yakagadziridzwa faira inoendeswa kune sikweya, iyo ine basa rekutumira iyo faira kune server. Pazasi pane marongero echikwere ichi.

Isu tinotsanangura nzira iyo meseji yeSOAP ichafambiswa. Tinonyora kupi. Tevere unofanira kuratidza kuti iyi iSOAP.

Wedzera akati wandei zvivakwa zvakaita semugamuchiri uye chiito (soapAction). Tinochengetedza uye tarisa. Iwe unogona kuona rumwe ruzivo rwekutumira zvikumbiro zveSOAP pano

Takatarisa nzira dzinoverengeka dzekushandisa maNIFI maitiro. Vanopindirana sei uye chii chavanobatsirika chaicho? Mienzaniso inotariswa ndeyeyedzo uye yakati siyanei nezvinoitika muhondo. Ndinovimba chinyorwa ichi chichava chinobatsira zvishoma kune vanogadzira. Ndinokutendai nekuteerera kwenyu. Kana uine mibvunzo, nyora. Ndichaedza kupindura.

Source: www.habr.com