I-Apache NIFI-Ujongo olufutshane lweeMpawu zokuSebenza

Intshayelelo

Kwenzeka ukuba kwindawo endisebenza kuyo ngoku kufuneka ndiqhelane nobu bugcisa. Ndiza kuqala ngemvelaphi encinci. Kwintlanganiso elandelayo, iqela lethu laxelelwa ukuba kufuneka senze udibaniso kunye inkqubo eyaziwayo. Ngokudityaniswa kwakuthetha ukuba le nkqubo eyaziwayo iya kusithumela izicelo nge-HTTP ukuya kwindawo ethile yokuphela, kwaye thina, ngokungaqhelekanga, siya kuthumela iimpendulo zamva ngendlela yomyalezo we-SOAP. Yonke into ibonakala ilula kwaye ingenamsebenzi. Ukusuka koku kulandela ukuba ufuna...

Injongo

Yenza iinkonzo ezi-3. Eyokuqala kubo yiNkonzo yoHlaziyo lweDatabase. Le nkonzo, xa idatha entsha ifika kwi-third party system, ihlaziya idatha kwi-database kwaye ivelise ifayile kwifomathi ye-CSV ukuyidlulisela kwinkqubo elandelayo. Isiphelo senkonzo yesibini sibizwa ngokuba yiNkonzo yezoThutho ye-FTP, efumana ifayile edluliselweyo, iqinisekisa, kwaye ibeka kwindawo yokugcina ifayile ngeFTP. Inkonzo yesithathu, inkonzo yokudlulisa idatha yabathengi, isebenza ngokungafaniyo nezimbini zokuqala. Ifumana isicelo esivela kwinkqubo yangaphandle yangaphandle ukufumana ifayile exoxwe ngasentla, ithatha ifayile yokuphendula ekulungeleyo, iyiguqule (uhlaziywa i-id, inkcazo, i-linkToFile fields) kwaye ithumela impendulo ngendlela yomyalezo we-SOAP. Oko kukuthi, umfanekiso opheleleyo ngolu hlobo lulandelayo: iinkonzo ezimbini zokuqala ziqala umsebenzi wazo kuphela xa idatha yokuhlaziywa ifikile. Inkonzo yesithathu isebenza rhoqo kuba kukho abathengi abaninzi bolwazi, malunga nezicelo ze-1000 zedatha ngomzuzu. Iinkonzo zifumaneka rhoqo kwaye iimeko zabo zibekwe kwiindawo ezahlukeneyo, ezifana novavanyo, idemo, ukuveliswa kwangaphambili kunye nemveliso. Apha ngezantsi ngumzobo wendlela ezi nkonzo zisebenza ngayo. Makhe ndicacise kwangoko ukuba ezinye iinkcukacha zenziwe lula ukuphepha ukuntsonkotha okungeyomfuneko.

Ubunzulu bobugcisa

Xa sicwangcisa isisombululo kwingxaki, saqala sagqiba ekubeni senze izicelo kwiJava usebenzisa i-Spring framework, i-balancer ye-Nginx, i-database ye-Postgres kunye nezinye izinto zobugcisa kunye nezobugcisa. Ukusukela ixesha lokuphuhlisa isisombululo sobugcisa wasivumela ukuba siqwalasele ezinye iindlela zokusombulula le ngxaki, amehlo ethu awela kwitekhnoloji ye-Apache NIFI, enemfashini kwizangqa ezithile. Ndiza kuthetha kwangoko ukuba le teknoloji isivumele ukuba siqaphele ezi nkonzo ze-3. Eli nqaku liza kuchaza ukuphuhliswa kwenkonzo yokuthutha ifayile kunye nenkonzo yokudlulisa idatha kumthengi, kodwa ukuba inqaku lincedo, ndiya kubhala malunga nenkonzo yokuhlaziya idatha kwisiseko sedatha.

Yintoni na?

I-NIFI yi-architecture esasazwayo yokulayishwa ngokukhawuleza okufanayo kunye nokucubungula idatha, inani elikhulu leeplagi zemithombo kunye nokuguqulwa, ukuguqulwa koqwalaselo kunye nokunye okuninzi. Ibhonasi entle kukuba kulula kakhulu ukuyisebenzisa. Iinkqubo ezincinci ezifana ne-getFile, sendHttpRequest kunye nezinye zinokumelwa njengezikwere. Isikwere ngasinye simele inkqubo, intsebenziswano enokubonwa kulo mfanekiso ungezantsi. Amaxwebhu aneenkcukacha ezithe vetshe malunga nonxibelelwano lwenkqubo yokuseta abhaliwe apha , kwabo bathetha isiRashiya - apha. Amaxwebhu achaza ngokugqibeleleyo indlela yokukhupha kunye nokuqhuba i-NIFI, kunye nendlela yokwenza iinkqubo, ezaziwa ngokuba zizikwere.
Umbono wokubhala inqaku wazalwa emva kokukhangela ixesha elide kunye nokucwangciswa kolwazi olufunyenweyo kwinto eyaziwayo, kunye nomnqweno wokwenza ubomi bube lula kubaphuhlisi bexesha elizayo.

Umzekelo:

Umzekelo wendlela izikwere ezidibana ngayo ziyaqwalaselwa. Iskimu esiqhelekileyo silula kakhulu: Sifumana isicelo se-HTTP (Kwithiyori, kunye nefayile emzimbeni wesicelo. Ukubonisa amandla e-NIFI, kulo mzekelo isicelo siqala inkqubo yokufumana ifayile kwindawo yokugcina ifayile yendawo. ), emva koko sithumela impendulo ukuba isicelo sifunyenwe, ngokuhambelana nenkqubo yokufumana ifayile kwi-FH kwaye emva koko inkqubo yokuyihambisa nge-FTP ukuya kwi-FH. Kuyafaneleka ukucacisa ukuba iinkqubo zisebenzisana enye kwenye ngokubizwa ngokuba yi-flowFile. Eli liqumrhu elisisiseko kwi-NIFI eligcina iimpawu kunye nomxholo. Umxholo yidatha emelwe yifayile yomlambo. Oko kukuthi, ukuthetha ngokuthe ngqo, ukuba ufumana ifayile kwisikwere esinye kwaye uyigqithisele kwesinye, umxholo uya kuba yifayile yakho.

Njengoko ubona, lo mfanekiso ubonisa inkqubo jikelele. HandleHttpRequest - yamkela izicelo, iReplaceText - ivelisa umzimba wokuphendula, i-HandleHttpResponse - ithumela impendulo. I-FetchFile - ifumana ifayile kwindawo yokugcina ifayile, idlulisela kwi-square PutSftp - ibeka le fayile kwi-FTP, kwidilesi echaziweyo. Ngoku ngakumbi malunga nale nkqubo.

Kule meko, isicelo sisiqalo sayo yonke into. Makhe sijonge iiparameters zayo zoqwalaselo.

Yonke into elapha incinci ngaphandle kweStandardHttpContextMap - olu luhlobo lwenkonzo ekuvumela ukuba uthumele kwaye ufumane izicelo. Ngeenkcukacha ezingakumbi kunye nemizekelo, ungabona - apha

Okulandelayo, masijonge kwi-ReplaceText yoqwalaselo iiparamitha zesikwere. Kufanelekile ukunikela ingqalelo kwi-ReplacementValue - yile nto iya kubuyiselwa kumsebenzisi ngendlela yokuphendula. Kwiisetingi ungahlengahlengisa inqanaba lokugawulwa kwemithi, ungabona iilogi {apho ukhuphe khona i-nifi}/nifi-1.9.2/logs, kukho kwakhona ukusilela/ukuphumelela iiparamitha-ngokusekwe kwezi parameters ungalawula inkqubo xa iyonke. . Oko kukuthi, kwimeko yokuphumelela kwesicatshulwa, inkqubo yokuthumela impendulo kumsebenzisi iya kubizwa, kwaye kwelinye icala siya kulogisha inkqubo engaphumelelanga.

Akukho nto inomdla ngokukodwa kwiipropati ze-HandleHttpResponse ngaphandle kwemeko xa impendulo yenziwe ngempumelelo.

Siye salungisa isicelo kunye nempendulo - masiqhubele phambili ekufumaneni ifayile kwaye siyibeke kwiseva yeFTP. I-FetchFile - ifumana ifayile kwindlela echazwe kwizicwangciso kwaye idlulisele kwinkqubo elandelayo.

Kwaye ke isikwere sePutSftp - sibeka ifayile kwindawo yokugcina ifayile. Sinokubona iiparamitha zoqwalaselo ngezantsi.

Kufanelekile ukunikela ingqalelo kwinto yokuba isikwere ngasinye yinkqubo eyahlukileyo ekufuneka iqaliswe. Sijonge owona mzekelo ulula ongadingi naluphi na uhlengahlengiso oluntsonkothileyo. Emva koko, siza kujonga inkqubo encinci ngakumbi, apho siya kubhala kancinci kwi-grooves.

Umzekelo onzima ngakumbi

Inkonzo yokudlulisa idatha kumthengi yajika yaba nzima ngakumbi ngenxa yenkqubo yokuguqula umyalezo we-SOAP. Inkqubo ngokubanzi iboniswe kumzobo ongezantsi.

Apha ingcamango nayo ayinzima kakhulu: sifumene isicelo kumthengi ukuba ufuna idatha, wathumela impendulo efumene umyalezo, waqala inkqubo yokufumana ifayile yokuphendula, emva koko wayihlela ngengqiqo ethile, emva koko. idluliselwe ifayile kumthengi ngohlobo lomyalezo weSEPHA kumncedisi.

Ndicinga ukuba akukho mfuneko yokuchaza kwakhona ezi zikwere sizibonileyo ngasentla - masihambe siye ngqo kwezintsha. Ukuba ufuna ukuhlela nayiphi na ifayile kunye nezikwere zohlobo lweReplaceText eziqhelekileyo azilunganga, kuya kufuneka ubhale esakho umbhalo. Oku kunokwenziwa kusetyenziswa isikwere se-ExecuteGroogyScript. Iisetingi zayo zinikwe ngezantsi.

Kukho iinketho ezimbini zokulayisha iscript kwesi sikwere. Eyokuqala kukukhuphela ifayile ngescript. Okwesibini kukufaka umbhalo kwi-scriptBody. Ngokokwazi kwam, isikwere se-executeScript sixhasa iilwimi ezininzi-enye yazo yi-groovy. Ndiza kuphoxa abaphuhlisi be-java - awukwazi ukubhala izikripthi kwi-java kwizikwere ezinjalo. Kwabo bafuna ngokwenene, kufuneka wenze esakho isikwere kwaye uyongeze kwinkqubo ye-NIFI. Lo msebenzi wonke ukhatshwa ngumdaniso omde kakhulu ngentambula, esingayi kujongana nayo kweli nqaku. Ndakhetha ulwimi lwe-groovy. Apha ngezantsi siscript sovavanyo esihlaziya ngokwandayo id kumyalezo weSEPHA. Kubalulekile ukuqaphela. Uthatha ifayile kwi-flowFile kwaye uyihlaziye, ungalibali ukuba kufuneka uyibeke apho, ihlaziywe. Kukwafanelekile ukuqaphela ukuba asingawo onke amathala eencwadi afakiweyo. Kungenzeka ukuba kusafuneka ungenise ngaphandle enye yee-libs. Enye into engezantsi kukuba okushicilelweyo kwesi sikwere kunzima kakhulu ukulungisa ingxaki. Kukho indlela yokuqhagamshela kwi-NIFI JVM kwaye uqale inkqubo yokulungisa ingxaki. Ngokomntu, ndasungula isicelo sendawo kwaye ndalinganisa ukufumana ifayile kwiseshoni. Ndenze ukulungisa ingxaki kwalapha. Iimpazamo ezivelayo xa kulayishwa iskripthi kulula kakhulu kuGoogle kwaye zibhalwe yiNIFI ngokwayo kwilog.

import org.apache.commons.io.IOUtils
import groovy.xml.XmlUtil
import java.nio.charset.*
import groovy.xml.StreamingMarkupBuilder

def flowFile = session.get()
if (!flowFile) return
try {
    flowFile = session.write(flowFile, { inputStream, outputStream ->
        String result = IOUtils.toString(inputStream, "UTF-8");
        def recordIn = new XmlSlurper().parseText(result)
        def element = recordIn.depthFirst().find {
            it.name() == 'id'
        }

        def newId = Integer.parseInt(element.toString()) + 1
        def recordOut = new XmlSlurper().parseText(result)
        recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId

        def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()
        outputStream.write(res.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
     session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
    log.error("Error during processing of validate.groovy", e)
    session.transfer(flowFile, REL_FAILURE)
}

Ngokwenyani, kulapho ulungelelwaniso lwesikwere luphela khona. Emva koko, ifayile ehlaziyiweyo idluliselwa kwisikwere, esinoxanduva lokuthumela ifayile kumncedisi. Ngezantsi kukho izicwangciso zesi sikwere.

Sichaza indlela oza kuhanjiswa ngayo umyalezo we-SOAP. Sibhala phi. Okulandelayo kufuneka ubonise ukuba le yiSEPHA.

Yongeza iipropathi ezininzi ezifana ne-host kunye nesenzo (soapAction). Sigcina kwaye sijonge. Ungabona iinkcukacha ezingaphezulu malunga nendlela yokuthumela izicelo ze-SEAP apha

Sijonge iindlela ezininzi zokusebenzisa iinkqubo ze-NIFI. Basebenzisana njani kwaye yintoni inzuzo yabo yokwenene? Imizekelo eqwalaselwayo yovavanyo kwaye yahluke kancinane kwinto eyenzekayo ekulweni. Ndiyathemba ukuba eli nqaku liya kuba luncedo kancinane kubaphuhlisi. Enkosi ngosinaka kwakho. Ukuba unayo nayiphi na imibuzo, bhala. Ndizakuzama ukuphendula.

umthombo: www.habr.com