Mau oyamba
Zinachitika kuti kuntchito yanga pano ndiyenera kuzolowera lusoli. Ndiyamba ndi maziko pang'ono. Pamsonkhano wotsatira, gulu lathu linauzidwa kuti tifunika kupanga mgwirizano ndi dongosolo lodziwika. Mwa kuphatikiza zidatanthawuza kuti dongosolo lodziwika bwinoli lingatitumizire zopempha kudzera pa HTTP mpaka kumapeto kwake, ndipo ife, modabwitsa, titha kutumizanso mayankho ngati uthenga wa SOAP. Chilichonse chikuwoneka chophweka komanso chochepa. Izi zikutanthauza kuti muyenera ...
Cholinga
Pangani 3 ntchito. Yoyamba mwa iwo ndi Database Update Service. Utumikiwu, pamene deta yatsopano ifika kuchokera ku dongosolo la chipani chachitatu, imasintha zomwe zili mu database ndikupanga fayilo mumtundu wa CSV kuti usamutsire ku dongosolo lotsatira. Mapeto a utumiki wachiwiri amatchedwa - FTP Transport Service, yomwe imalandira fayilo yotumizidwa, imatsimikizira, ndikuyiyika mu yosungirako mafayilo kudzera pa FTP. Utumiki wachitatu, ntchito yotumizira deta ya ogula, imagwira ntchito mofanana ndi ziwiri zoyambirira. Imalandira pempho kuchokera ku dongosolo lakunja la chipani chachitatu kuti lilandire fayilo yomwe yafotokozedwa pamwambapa, imatenga fayilo yoyankhidwa yokonzeka, kuisintha (kukonzanso id, kufotokozera, minda ya linkToFile) ndikutumiza yankho mu mawonekedwe a uthenga wa SOAP. Ndiko kuti, chithunzi chonse chiri motere: mautumiki awiri oyambirira amayamba ntchito yawo pokhapokha pamene deta yosinthidwa yafika. Utumiki wachitatu umagwira ntchito nthawi zonse chifukwa pali ogula ambiri, pafupifupi zopempha za 1000 pa mphindi imodzi. Ntchito zimapezeka nthawi zonse ndipo zochitika zake zimapezeka m'malo osiyanasiyana, monga kuyesa, kuwonetsetsa, kupanga zisanachitike ndi zopangira. Pansipa pali chithunzi cha momwe mautumikiwa amagwirira ntchito. Ndiloleni ndifotokoze nthawi yomweyo kuti mfundo zina zakhala zosavuta kupewa zovuta zosafunikira.
Kuzama kwaukadaulo
Pokonzekera njira yothetsera vutoli, tidaganiza zoyamba kugwiritsa ntchito Java pogwiritsa ntchito Spring framework, Nginx balancer, Postgres database ndi zina zamakono osati zamakono. Popeza nthawi yopanga yankho laukadaulo idatilola kulingalira njira zina zothetsera vutoli, kuyang'ana kwathu kudagwera paukadaulo wa Apache NIFI, womwe uli wowoneka bwino pamabwalo ena. Ndikunena nthawi yomweyo kuti ukadaulo uwu udatilola kuzindikira mautumiki atatu awa. Nkhaniyi ifotokoza za chitukuko cha ntchito yoyendetsa mafayilo ndi ntchito yotumizira deta kwa ogula, koma ngati nkhaniyi ndi yothandiza, ndilemba za ntchito yokonzanso deta mu database.
Ndi chiyani?
NIFI ndi zomangamanga zomwe zimagawidwa kuti zikhazikitse mofulumira ndi kukonzanso deta, mapulagini ambiri a magwero ndi masinthidwe, kumasulira kwa masanjidwe ndi zina zambiri. Bonasi yabwino ndikuti ndiyosavuta kugwiritsa ntchito. Njira zazing'ono monga getFile, sendHttpRequest ndi zina zitha kuyimiridwa ngati mabwalo. Mzere uliwonse umayimira ndondomeko, kuyanjana komwe kumawoneka mu chithunzi pansipa. Zolemba zambiri zokhudzana ndi kukhazikitsidwa kwa njira zalembedwa , kwa iwo omwe amalankhula Chirasha - . Zolembazo zimalongosola bwino momwe mungatulutsire ndikuyendetsa NIFI, komanso momwe mungapangire njira, zomwe zimadziwikanso kuti mabwalo.
Lingaliro lolemba nkhani lidabadwa pambuyo kusaka kwanthawi yayitali ndikukonza zomwe zalandilidwa kukhala chinthu chodziwika bwino, komanso chikhumbo chofuna kuti moyo ukhale wosavuta kwa opanga mtsogolo.
Chitsanzo:
Chitsanzo cha momwe mabwalo amagwirizanirana wina ndi mzake amaganiziridwa. Chiwembu chonsecho ndi chophweka: Timalandira pempho la HTTP (Mwachidziwitso, ndi fayilo mu thupi la pempho. Kuti muwonetse mphamvu za NIFI, mu chitsanzo ichi pempho likuyamba njira yolandirira fayilo kuchokera kumalo osungirako mafayilo. ), ndiye timatumizanso yankho lomwe pempholo lalandiridwa, mofanana ndi njira yolandirira fayilo kuchokera ku FH ndiyeno njira yosunthira kudzera pa FTP kupita ku FH. Ndikoyenera kufotokozera kuti njira zimagwirizana wina ndi mzake kudzera muzomwe zimatchedwa flowFile. Ili ndiye gawo loyambira mu NIFI lomwe limasunga zomwe zili ndi zomwe zili. Zomwe zili ndi data yomwe imayimiridwa ndi fayilo yamtsinje. Ndiko kunena kuti, ngati mulandira fayilo kuchokera pabwalo limodzi ndikusamutsa kupita ku lina, zomwe zili mufayilo yanu.
Monga mukuwonera, chithunzichi chikuwonetsa zochitika zonse. HandleHttpRequest - imavomereza zopempha, ReplaceText - imapanga gulu loyankha, HandleHttpResponse - imatumiza yankho. FetchFile - imalandira fayilo kuchokera kosungirako mafayilo, ndikuitumiza kumalo apakati PutSftp - imayika fayiloyi pa FTP, pa adilesi yotchulidwa. Tsopano zambiri za njirayi.
Pamenepa, pempho ndilo chiyambi cha chirichonse. Tiyeni tiwone magawo ake kasinthidwe.
Chilichonse pano ndi chochepa kwambiri kupatula StandardHttpContextMap - uwu ndi mtundu wa ntchito zomwe zimakulolani kutumiza ndi kulandira zopempha. Mwatsatanetsatane komanso ngakhale ndi zitsanzo, mutha kuwona -
Kenako, tiyeni tiwone zosintha za ReplaceText za sikweya. Ndikoyenera kumvera ReplacementValue - izi ndizomwe zidzabwezedwe kwa wogwiritsa ntchito ngati yankho. Pazikhazikiko mutha kusintha kuchuluka kwa mitengo, mutha kuwona zipika {kumene mudatulutsa nifi}/nifi-1.9.2/logs, palinso magawo olephera / opambana - potengera magawowa mutha kuwongolera njira yonse. . Ndiko kuti, pankhani yolemba bwino zolemba, njira yotumizira yankho kwa wogwiritsa ntchito idzayitanidwa, ndipo nthawi ina tidzangolemba zomwe sizinachitike.
Palibe chosangalatsa kwambiri muzinthu za HandleHttpResponse kupatula momwe yankho lidapangidwa bwino.
Takonza zopempha ndi kuyankha - tiyeni tipitirire kulandira fayilo ndikuyiyika pa seva ya FTP. FetchFile - imalandira fayilo panjira yomwe yafotokozedwa pazosintha ndikuipereka kunjira ina.
Kenako malo a PutSftp - amayika fayiloyo posungira mafayilo. Titha kuwona magawo osinthira pansipa.
Ndikoyenera kulabadira kuti lalikulu lililonse ndi njira yosiyana yomwe iyenera kukhazikitsidwa. Tinayang'ana chitsanzo chosavuta chomwe sichifuna makonda aliwonse ovuta. Chotsatira, tiwona ndondomekoyi movutikira kwambiri, pomwe tidzalemba pang'ono pa grooves.
Chitsanzo chovuta kwambiri
Ntchito yotumizira deta kwa ogula idakhala yovuta kwambiri chifukwa chakusintha kwa uthenga wa SOAP. Njira zambiri zikuwonetsedwa mu chithunzi pansipa.
Pano lingalirolo silili lovuta kwambiri: tinalandira pempho kuchokera kwa wogula kuti akufunikira deta, adatumiza yankho lomwe adalandira uthenga, anayamba njira yolandira fayilo yoyankhira, kenako ndikuikonza ndi lingaliro linalake, ndiyeno. adasamutsa fayilo kwa ogula ngati uthenga wa SOAP ku seva.
Ndikuganiza kuti palibe chifukwa chofotokozeranso mabwalo omwe tawawona pamwambapa - tiyeni tipite ku zatsopano. Ngati mukufuna kusintha fayilo iliyonse ndipo mabwalo wamba amtundu wa ReplaceText sali oyenera, muyenera kulemba zanu. Izi zitha kuchitika pogwiritsa ntchito lalikulu la ExecuteGroogyScript. Zokonda zake zikuwonetsedwa pansipa.
Pali njira ziwiri zotsitsa zolemba mumzerewu. Choyamba ndikutsitsa fayilo yokhala ndi script. Yachiwiri ndikuyika script mu scriptBody. Monga ndikudziwira, executeScript square imathandizira zilankhulo zingapo - imodzi mwazo ndi groovy. Ndikhumudwitsa opanga java - simungathe kulemba zolemba mu java m'mabwalo otere. Kwa iwo omwe akufunadi, muyenera kupanga malo anu enieni ndikuwonjezera ku NIFI system. Opaleshoni yonseyi imatsagana ndi kuvina kwautali kwambiri ndi maseche, omwe sitikambirana nawo m'nkhaniyi. Ndinasankha chinenero cha groovy. Pansipa pali cholembera choyesera chomwe chimangowonjezera ID mu uthenga wa SOAP. Ndikofunika kuzindikira. Mumatenga fayilo kuchokera ku flowFile ndikuyisintha, musaiwale kuti muyenera kuyiyika pamenepo, kusinthidwa. Ndikoyeneranso kudziwa kuti si malaibulale onse omwe akuphatikizidwa. Zitha kuchitika kuti mukufunikabe kuitanitsa limodzi la libs. Chinanso choyipa ndichakuti script pabwaloli ndizovuta kuyisintha. Pali njira yolumikizira ku NIFI JVM ndikuyamba kukonza zolakwika. Inemwini, ndidayambitsa pulogalamu yakumaloko ndikufanizira kulandira fayilo kuchokera pagawoli. Ndinachitanso debugging kwanuko. Zolakwa zomwe zimawonekera mukatsegula script ndizosavuta kwa Google ndipo zimalembedwa ndi NIFI pa chipikacho.
import org.apache.commons.io.IOUtils
import groovy.xml.XmlUtil
import java.nio.charset.*
import groovy.xml.StreamingMarkupBuilder
def flowFile = session.get()
if (!flowFile) return
try {
flowFile = session.write(flowFile, { inputStream, outputStream ->
String result = IOUtils.toString(inputStream, "UTF-8");
def recordIn = new XmlSlurper().parseText(result)
def element = recordIn.depthFirst().find {
it.name() == 'id'
}
def newId = Integer.parseInt(element.toString()) + 1
def recordOut = new XmlSlurper().parseText(result)
recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId
def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()
outputStream.write(res.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
log.error("Error during processing of validate.groovy", e)
session.transfer(flowFile, REL_FAILURE)
}Kwenikweni, apa ndipamene makonda a square amatha. Kenako, fayilo yosinthidwa imasamutsidwa ku bwalo, yomwe ili ndi udindo wotumiza fayilo ku seva. Pansipa pali zokonda pabwaloli.
Timalongosola njira yomwe uthenga wa SOAP udzatumizidwa. Timalemba kumene. Kenako muyenera kuwonetsa kuti iyi ndi SOAP.
Onjezani zinthu zingapo monga host ndi zochita (soapAction). Timasunga ndi kufufuza. Mutha kuwona zambiri zamomwe mungatumizire zopempha za SOAP
Tinayang'ana njira zingapo zogwiritsira ntchito njira za NIFI. Kodi amalumikizana bwanji ndipo phindu lawo lenileni ndi lotani? Zitsanzo zomwe zimaganiziridwa ndi zoyesa ndipo ndizosiyana pang'ono ndi zomwe zimachitika pankhondo. Ndikukhulupirira kuti nkhaniyi idzakhala yothandiza kwa omanga. Zikomo chifukwa chakumvetsera. Ngati muli ndi mafunso, lembani. Ndiyesera kuyankha.
Source: www.habr.com