I-Apache NIFI - Uhlolojikelele olufushane lwezici Zokuzilolonga

Isingeniso

Kwenzeka ukuthi lapho engisebenza khona kwadingeka ngijwayelane nalobu buchwepheshe. Ngizoqala ngesizinda esincane. Emhlanganweni olandelayo, ithimba lethu latshelwa ukuthi sidinga ukudala ukuhlanganisa isistimu eyaziwayo. Ngokuhlanganisa kwakusho ukuthi lolu hlelo olwaziwayo luzosithumela izicelo nge-HTTP endaweni ethile yokugcina, futhi thina, ngokuxakile, sizothumela izimpendulo emuva ngendlela yomlayezo we-SOAP. Konke kubonakala kulula futhi kuyinto engasho lutho. Kulokhu kulandela ukuthi udinga...

Inhloso

Dala amasevisi angu-3. Eyokuqala yazo yi-Database Update Service. Le sevisi, lapho idatha entsha ifika ivela ohlelweni lwezinkampani zangaphandle, ibuyekeza idatha kusizindalwazi futhi ikhiqize ifayela ngefomethi ye-CSV ukuze idluliselwe ohlelweni olulandelayo. Indawo yokugcina yesevisi yesibili ibizwa ngokuthi - Isevisi Yezokuthutha ye-FTP, ethola ifayela elidlulisiwe, eliqinisekisayo, futhi libeke endaweni yokugcina ifayela nge-FTP. Isevisi yesithathu, isevisi yokudluliswa kwedatha yomthengi, isebenza ngokulinganayo nezimbili zokuqala. Ithola isicelo esivela ohlelweni lwangaphandle lwenkampani yangaphandle ukuze ithole ifayela okukhulunywe ngalo ngenhla, ithatha ifayela lokuphendula eselilungile, ililungise (ibuyekeza i-id, incazelo, izinkambu ze-linkToFile) futhi ithumele impendulo ngendlela yomlayezo we-SOAP. Okusho ukuthi, isithombe esiphelele simi kanje: izinsizakalo ezimbili zokuqala ziqala umsebenzi wazo kuphela lapho idatha yokubuyekezwa isifikile. Isevisi yesithathu isebenza njalo ngoba kukhona abathengi abaningi bolwazi, mayelana nezicelo ze-1000 zedatha ngomzuzu. Izinsizakalo zitholakala njalo futhi izimo zazo zitholakala ezindaweni ezihlukene, njengokuhlola, idemo, ukukhiqizwa kwangaphambili kanye nokukhiqiza. Ngezansi kunomdwebo wokuthi lezi zinsizakalo zisebenza kanjani. Ake ngicacise zisuka nje ukuthi eminye imininingwane yenziwe lula ukuze kugwenywe inkimbinkimbi engenasidingo.

Ukujula Kwezobuchwepheshe

Lapho sihlela isisombululo senkinga, siqale sanquma ukwenza izinhlelo zokusebenza ku-Java sisebenzisa uhlaka lwe-Spring, ibhalansi ye-Nginx, isizindalwazi se-Postgres nezinye izinto zobuchwepheshe hhayi kangako. Njengoba isikhathi sokuthuthukisa isixazululo sezobuchwepheshe sasivumela ukuthi sicabangele ezinye izindlela zokuxazulula le nkinga, amehlo ethu awela kubuchwepheshe be-Apache NIFI, obunemfashini emibuthanweni ethile. Ngizosho ngokushesha ukuthi lobu buchwepheshe busivumele ukuthi siqaphele lezi zinsizakalo ezi-3. Lesi sihloko sizochaza ukuthuthukiswa kwesevisi yokuthutha ifayela kanye nesevisi yokudlulisa idatha kumthengi, kodwa uma isihloko siwusizo, ngizobhala mayelana nesevisi yokubuyekeza idatha ku-database.

Kuyini?

I-NIFI iyisakhiwo esabalalisiwe sokulayisha okusheshayo okuhambisanayo nokucubungula idatha, inani elikhulu lama-plugin wemithombo nokuguqulwa, ukuguqulwa kokucushwa nokunye okuningi. Ibhonasi enhle ukuthi kulula kakhulu ukuyisebenzisa. Izinqubo ezingabalulekile ezifana ne-getFile, sendHttpRequest nezinye zingamelwa njengezikwele. Isikwele ngasinye simele inqubo, ukusebenzelana kwayo okungabonakala esithombeni esingezansi. Imibhalo enemininingwane eyengeziwe mayelana nokusebenzisana kokusetha inqubo isibhaliwe lapha , kulabo abakhuluma isiRashiya - lapha. Amadokhumenti achaza kahle indlela yokuqaqa nokusebenzisa i-NIFI, kanye nendlela yokudala izinqubo, ezaziwa nangokuthi izikwele.
Umqondo wokubhala i-athikili wazalwa ngemva kokusesha isikhathi eside nokuhlela ulwazi olutholiwe lube kokuthile okuqaphelayo, kanye nesifiso sokwenza ukuphila kube lula kubathuthukisi besikhathi esizayo.

Isibonelo:

Isibonelo sokuthi izikwele zihlangana kanjani ziyacatshangelwa. Uhlelo olujwayelekile lulula kakhulu: Sithola isicelo se-HTTP (Ngokwembono, ngefayela emzimbeni wesicelo. Ukuze ubonise amakhono e-NIFI, kulesi sibonelo isicelo siqala inqubo yokuthola ifayela endaweni yokugcina ifayela lendawo. ), bese sibuyisela impendulo ukuthi isicelo sitholiwe, ngokuhambisana nenqubo yokwamukela ifayela elivela ku-FH kanye nenqubo yokulihambisa nge-FTP liye ku-FH. Kuyafaneleka ukucacisa ukuthi izinqubo ziyasebenzisana ngokusebenzisa okuthiwa yi-flowFile. Leli yibhizinisi eliyisisekelo ku-NIFI eligcina izibaluli nokuqukethwe. Okuqukethwe yidatha emelelwa yifayela lokusakaza. Okusho ukuthi, uma nje uthola ifayela kwesinye isikwele bese ulidlulisela kwesinye, okuqukethwe kuzoba ifayela lakho.

Njengoba ubona, lesi sithombe sibonisa inqubo evamile. I-HandleHttpRequest - yamukela izicelo, i-ReplaceText - ikhiqiza indikimba yokuphendula, i-HandleHttpResponse - ithumela impendulo. I-FetchFile - ithola ifayela endaweni yokugcina ifayela, ilidlulisele esigcawini se-PutSftp - ibeka leli fayela ku-FTP, ekhelini elishiwo. Manje okwengeziwe ngale nqubo.

Kulokhu, isicelo siyisiqalo sayo yonke into. Ake sibheke imingcele yayo yokucushwa.

Yonke into lapha incane impela ngaphandle kwe-StandardHttpContextMap - lolu uhlobo lwesevisi olukuvumela ukuthi uthumele futhi wamukele izicelo. Ngemininingwane eyengeziwe ngisho nangezibonelo, ungabona - lapha

Okulandelayo, ake sibheke imingcele yokumisa i-ReplaceText yesikwele. Kuyafaneleka ukunaka i-ReplacementValue - yilokhu okuzobuyiselwa kumsebenzisi ngendlela yempendulo. Kuzilungiselelo ungakwazi ukulungisa izinga lokungena, ungabona amalogi {lapho ukhiphe khona i-nifi}/nifi-1.9.2/logs, akhona namapharamitha wokuhluleka/wempumelelo - ngokusekelwe kulawa mapharamitha ungakwazi ukulawula inqubo iyonke. . Okusho ukuthi, esimweni sokucutshungulwa kombhalo okuphumelelayo, inqubo yokuthumela impendulo kumsebenzisi izobizwa, futhi kwesinye isimo sizomane sibhale inqubo engaphumelelanga.

Akukho okuthakaselayo ngokukhethekile ezicini ze-HandleHttpResponse ngaphandle kwesimo lapho impendulo idalwe ngempumelelo.

Silungise isicelo kanye nempendulo - ake siqhubekele phambili ekwamukeleni ifayela futhi silibeke kuseva ye-FTP. I-FetchFile - ithola ifayela ngendlela eshiwo kuzilungiselelo bese idlulisela inqubo elandelayo.

Bese isikwele se-PutSftp - sibeka ifayela endaweni yokugcina ifayela. Singabona imingcele yokumisa ngezansi.

Kuyafaneleka ukunaka iqiniso lokuthi isikwele ngasinye siyinqubo ehlukile okufanele yethulwe. Sibheke isibonelo esilula esingadingi ukwenza ngokwezifiso okuyinkimbinkimbi. Okulandelayo, sizobheka inqubo kancane inkimbinkimbi, lapho sizobhala khona kancane kuma-grooves.

Isibonelo esiyinkimbinkimbi

Isevisi yokudlulisa idatha kumthengi iphenduke yaba inkimbinkimbi ngenxa yenqubo yokulungisa umlayezo we-SOAP. Inqubo evamile iboniswa esithombeni esingezansi.

Lapha umqondo nawo awuyona inkimbinkimbi ikakhulukazi: sithole isicelo esivela kumthengi ukuthi udinga idatha, wathumela impendulo ukuthi uthole umlayezo, waqala inqubo yokuthola ifayela lokuphendula, wabe esehlela nge-logic ethile, bese idlulisele ifayela kumthengi ngendlela yomlayezo we-SOAP kuseva.

Ngicabanga ukuthi asikho isidingo sokuphinda sichaze lezo zikwele esizibone ngenhla - ake siqonde kwezintsha. Uma udinga ukuhlela noma yiliphi ifayela futhi izikwele zohlobo lwe-ReplaceText ezijwayelekile azifanelekile, kuzodingeka ukuthi ubhale esakho umbhalo. Lokhu kungenziwa kusetshenziswa isikwele se-ExecuteGroogyScript. Izilungiselelo zayo zethulwe ngezansi.

Kunezinketho ezimbili zokulayisha umbhalo kulesi sikwele. Eyokuqala iwukudawuniloda ifayela elineskripthi. Okwesibili ukufaka umbhalo ku-scriptBody. Ngokwazi kwami, isikwele se-executeScript sisekela izilimi ezimbalwa - enye yazo i-groovy. Ngizodumaza abathuthukisi be-java - awukwazi ukubhala imibhalo ku-java ezikweleni ezinjalo. Kulabo abafuna ngempela, udinga ukudala isikwele sakho sangokwezifiso bese usingeza ohlelweni lwe-NIFI. Lo msebenzi wonke uhambisana nomdanso omude onethamborini, esingeke sibhekane nawo kulesi sihloko. Ngakhetha ulimi lwe-groovy. Ngezansi umbhalo wokuhlola ovele ubuyekeze ngokuqhubekayo i-id kumlayezo we-SOAP. Kubalulekile ukuqaphela. Uthatha ifayela ku-flowFile futhi ulibuyekeze, ungakhohlwa ukuthi udinga ukulibuyisela lapho, libuyekeziwe. Kubalulekile futhi ukuqaphela ukuthi akuyona yonke imitapo yolwazi efakiwe. Kungenzeka ukuthi kusafanele ungenise eyodwa yama-libs. Okunye okubi ukuthi iskripthi kulesi sikwele sinzima kakhulu ukusisusa. Kukhona indlela yokuxhuma ku-NIFI JVM bese uqala inqubo yokulungisa iphutha. Ngokwami, ngethule isicelo sendawo futhi ngalingisa ukuthola ifayela kusuka kuseshini. Ngiphinde ngenza ukulungisa iphutha endaweni. Amaphutha avelayo uma kulayishwa umbhalo alula ku-Google futhi abhalwa yi-NIFI ngokwayo kulogi.

import org.apache.commons.io.IOUtils
import groovy.xml.XmlUtil
import java.nio.charset.*
import groovy.xml.StreamingMarkupBuilder

def flowFile = session.get()
if (!flowFile) return
try {
    flowFile = session.write(flowFile, { inputStream, outputStream ->
        String result = IOUtils.toString(inputStream, "UTF-8");
        def recordIn = new XmlSlurper().parseText(result)
        def element = recordIn.depthFirst().find {
            it.name() == 'id'
        }

        def newId = Integer.parseInt(element.toString()) + 1
        def recordOut = new XmlSlurper().parseText(result)
        recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId

        def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()
        outputStream.write(res.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
     session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
    log.error("Error during processing of validate.groovy", e)
    session.transfer(flowFile, REL_FAILURE)
}

Empeleni, kulapho ukwakheka kwesikwele ngokwezifiso kugcina khona. Okulandelayo, ifayela elibuyekeziwe lidluliselwa esigcawini, esinesibopho sokuthumela ifayela kuseva. Ngezansi kukhona izilungiselelo zalesi sikwele.

Sichaza indlela umlayezo we-SOAP ozothunyelwa ngayo. Sibhala kuphi. Okulandelayo udinga ukukhombisa ukuthi lena INSIPHO.

Engeza izakhiwo ezimbalwa ezifana nomsingathi kanye nesenzo (soapAction). Silondoloza futhi sihlole. Ungabona imininingwane eyengeziwe yokuthi uzithumela kanjani izicelo ze-SOAP lapha

Sibheke izinketho ezimbalwa zokusebenzisa izinqubo ze-NIFI. Basebenzelana kanjani futhi iyini inzuzo yabo yangempela? Izibonelo ezicatshangelwayo ezesivivinyo futhi zihluke kancane kulokho okwenzeka ngempela empini. Ngithemba ukuthi lesi sihloko sizoba usizo kancane kubathuthukisi. Ngiyabonga ukulalela kwenu. Uma unemibuzo, bhala. Ngizozama ukuphendula.

Source: www.habr.com