Apache NIFI - Takaitaccen Bayani na Dama a Cikin Ayyuka

Gabatarwar

Ya faru ne a wurin aiki na a yanzu dole na san wannan fasaha. Zan fara da ɗan baya. A taro na gaba, an gaya wa ƙungiyarmu cewa muna buƙatar ƙirƙirar haɗin kai tare da tsarin da aka sani. Ta hanyar haɗin kai ana nufin cewa wannan sanannen tsarin zai aiko mana da buƙatun ta hanyar HTTP zuwa takamaiman wurin ƙarshe, kuma mu, abin banƙyama, za mu mayar da martani ta hanyar saƙon SOAP. Komai yana da sauƙi kuma maras muhimmanci. Daga wannan ya biyo bayan kuna buƙatar ...

Manufar

Ƙirƙiri ayyuka 3. Na farko daga cikinsu shine Sabis na Sabunta Database. Wannan sabis ɗin, lokacin da sababbin bayanai suka zo daga tsarin ɓangare na uku, yana sabunta bayanan da ke cikin ma'ajin bayanai kuma yana samar da fayil a tsarin CSV don canja wurin shi zuwa tsarin na gaba. Ana kiran ƙarshen sabis na biyu - Sabis na Sufuri na FTP, wanda ke karɓar fayil ɗin da aka canjawa wuri, ya inganta shi, kuma ya sanya shi cikin ajiyar fayil ta hanyar FTP. Sabis na uku, sabis na canja wurin bayanai na mabukaci, yana aiki a daidaita tare da biyun farko. Yana karɓar buƙatu daga tsarin waje na ɓangare na uku don karɓar fayil ɗin da aka tattauna a sama, ɗaukar fayil ɗin amsa shirye-shiryen, gyara shi (yana sabunta id, bayanin, linkToFile filayen) kuma ya aika da amsa ta hanyar saƙon SOAP. Wato, hoto gaba ɗaya shine kamar haka: sabis biyu na farko suna fara aikinsu ne kawai lokacin da bayanan sabuntawa ya isa. Sabis na uku yana aiki koyaushe saboda akwai masu amfani da bayanai da yawa, game da buƙatun 1000 don bayanai a minti daya. Ana samun ayyuka akai-akai kuma ana samun misalinsu a wurare daban-daban, kamar gwaji, demo, riga-kafi da samfur. A ƙasa akwai zane na yadda waɗannan ayyukan ke aiki. Bari in fayyace nan da nan cewa an sauƙaƙa wasu cikakkun bayanai don guje wa rikitarwa mara amfani.

Zurfafa Fasaha

Lokacin da muke shirin warware matsalar, mun fara yanke shawarar yin aikace-aikace a cikin Java ta amfani da tsarin bazara, Nginx balancer, bayanan Postgres da sauran fasaha kuma ba haka ba. Tun lokacin da za a samar da mafita na fasaha ya ba mu damar yin la'akari da wasu hanyoyin da za a magance wannan matsala, kallonmu ya fadi a kan fasahar Apache NIFI, wanda ke da kyau a wasu da'irori. Zan ce nan da nan cewa wannan fasaha ta ba mu damar lura da waɗannan ayyuka 3. Wannan labarin zai bayyana ci gaban sabis na sufuri na fayil da sabis na canja wurin bayanai ga mabukaci, amma idan labarin yana da amfani, zan rubuta game da sabis don sabunta bayanai a cikin bayanan.

Mene ne?

NIFI tsarin gine-ginen da aka rarraba don sauri daidaitaccen lodi da sarrafa bayanai, adadi mai yawa na plugins don tushe da canje-canje, sigar daidaitawa da ƙari mai yawa. Kyakkyawan kari shine cewa yana da sauƙin amfani. Ana iya wakilta ƙananan matakai kamar getFile, sendHttpRequest da sauransu azaman murabba'ai. Kowane murabba'i yana wakiltar tsari, wanda za'a iya ganin hulɗar sa a cikin hoton da ke ƙasa. An rubuta ƙarin cikakkun bayanai kan hulɗar saitin tsari a nan , ga masu jin Rashanci - a nan. Takaddun sun bayyana daidai yadda ake cire kaya da gudanar da NIFI, da kuma yadda ake ƙirƙirar matakai, wanda kuma aka sani da murabba'ai.
An haifi ra'ayin rubuta labarin bayan dogon bincike da tsara bayanan da aka karɓa a cikin wani abu mai hankali, da kuma sha'awar yin rayuwa ta ɗan sauƙi ga masu haɓakawa na gaba.

Alal misali:

An yi la'akari da misalin yadda murabba'ai ke hulɗa da juna. Tsarin gabaɗaya yana da sauƙi: Muna karɓar buƙatar HTTP (A ka'idar, tare da fayil a jikin buƙatun. Don nuna iyawar NIFI, a cikin wannan misalin buƙatar ta fara aiwatar da karɓar fayil daga ajiyar fayil na gida. ), sa'an nan kuma mu mayar da martani cewa bukatar da aka samu, a cikin layi daya tsarin karban fayil daga FH sa'an nan kuma aiwatar da shi ta hanyar FTP zuwa FH. Yana da kyau a fayyace cewa matakai suna hulɗa da juna ta hanyar abin da ake kira flowFile. Wannan shine tushen tushen a NIFI wanda ke adana halaye da abun ciki. Abun ciki shine bayanan da fayil ɗin rafi ke wakilta. Wato, kusan magana, idan kun karɓi fayil daga murabba'i ɗaya kuma ku canza shi zuwa wani, abun ciki zai zama fayil ɗin ku.

Kamar yadda kake gani, wannan hoton yana nuna tsarin gaba ɗaya. HandleHttpRequest - yana karɓar buƙatun, ReplaceText - yana haifar da amsawar amsawa, HandleHttpResponse - yana aika amsa. FetchFile - yana karɓar fayil daga ajiyar fayil, yana tura shi zuwa fili PutSftp - yana sanya wannan fayil ɗin akan FTP, a ƙayyadadden adireshin. Yanzu ƙarin game da wannan tsari.

A wannan yanayin, buƙatar ita ce farkon komai. Bari mu kalli sigogin tsarin sa.

Duk abin da ke nan ba shi da mahimmanci ban da StandardHttpContextMap - wannan nau'in sabis ne wanda ke ba ku damar aikawa da karɓar buƙatun. A cikin ƙarin daki-daki kuma har ma da misalai, zaku iya gani - a nan

Na gaba, bari mu kalli sigogin daidaitawa na ReplaceText na murabba'in. Yana da kyau a kula da ReplacementValue - wannan shine abin da za a mayar da shi ga mai amfani a cikin hanyar amsawa. A cikin saitunan za ku iya daidaita matakin shiga, za ku iya ganin logs {inda kuka cire nifi}/nifi-1.9.2/logs, akwai kuma gazawar / nasara sigogi - dangane da waɗannan sigogi za ku iya tsara tsarin gaba ɗaya. . Wato idan aka sami nasarar sarrafa rubutun, za a kira tsarin aika martani ga mai amfani, sannan a wani yanayi za mu shiga tsarin da bai yi nasara ba.

Babu wani abu mai ban sha'awa musamman a cikin kaddarorin Amsa na HandleHttp sai matsayi lokacin da aka sami nasarar ƙirƙira martani.

Mun warware buƙatar da amsa - bari mu ci gaba zuwa karɓar fayil ɗin mu sanya shi akan sabar FTP. FetchFile - yana karɓar fayil a hanyar da aka ƙayyade a cikin saitunan kuma ya wuce shi zuwa tsari na gaba.

Sannan filin PutSftp - yana sanya fayil ɗin a cikin ajiyar fayil. Za mu iya ganin sigogin sanyi a ƙasa.

Yana da kyau a kula da gaskiyar cewa kowane murabba'i wani tsari ne daban wanda dole ne a ƙaddamar. Mun kalli misali mafi sauƙi wanda baya buƙatar kowane hadadden gyare-gyare. Na gaba, za mu dubi tsarin dan kadan mafi rikitarwa, inda za mu rubuta kadan a kan tsagi.

Misali mai rikitarwa

Sabis ɗin canja wurin bayanai ga mabukaci ya zama ɗan ƙara rikitarwa saboda tsarin gyaran saƙon SOAP. Ana nuna tsarin gaba ɗaya a cikin adadi a ƙasa.

A nan ra'ayin kuma ba shi da rikitarwa musamman: mun sami buƙatu daga mabukaci cewa yana buƙatar bayanai, ya aika da amsa cewa ya karɓi saƙo, ya fara aiwatar da karɓar fayil ɗin amsa, sannan a gyara shi da wata ma'ana, sannan canja wurin fayil ɗin zuwa mabukaci ta hanyar saƙon SOAP zuwa uwar garken.

Ina tsammanin babu buƙatar sake bayyana wuraren da muka gani a sama - bari mu matsa kai tsaye zuwa sababbi. Idan kana buƙatar gyara kowane fayil kuma nau'in madaukai na ReplaceText bai dace ba, dole ne ka rubuta rubutun naka. Ana iya yin wannan ta amfani da filin ExecuteGroogyScript. Ana gabatar da saitunan sa a ƙasa.

Akwai zaɓuɓɓuka biyu don loda rubutun cikin wannan filin. Na farko shine ta hanyar zazzage fayil tare da rubutun. Na biyu shine ta hanyar shigar da rubutun a cikin scriptBody. Kamar yadda na sani, filin aiwatarwa yana goyan bayan yaruka da yawa - ɗaya daga cikinsu shine groovy. Zan kunyata masu haɓaka java - ba za ku iya rubuta rubutun a java ba a cikin irin waɗannan murabba'ai. Ga waɗanda suke so da gaske, kuna buƙatar ƙirƙirar filin ku na al'ada kuma ku ƙara shi zuwa tsarin NIFI. Wannan duka aiki yana tare da dogon rawa tare da tambourine, wanda ba za mu magance shi ba a cikin wannan labarin. Na zaɓi yaren baƙar fata. A ƙasa akwai rubutun gwaji wanda kawai ke ƙara sabunta id a cikin saƙon SOAP. Yana da mahimmanci a lura. Kuna ɗaukar fayil ɗin daga flowFile kuma sabunta shi, kar ku manta cewa kuna buƙatar mayar da shi can, sabuntawa. Har ila yau, ya kamata a lura cewa ba duk ɗakunan karatu ba ne a ciki. Yana iya faruwa cewa har yanzu dole ne ka shigo da ɗaya daga cikin libs. Wani kasala kuma shine rubutun da ke cikin wannan filin yana da wahalar cirewa. Akwai hanyar haɗi zuwa NIFI JVM kuma fara aiwatar da gyara kuskure. Da kaina, na ƙaddamar da aikace-aikacen gida kuma na kwaikwayi karɓar fayil daga zaman. Na kuma yi kuskure a cikin gida. Kurakurai da ke bayyana lokacin loda rubutun suna da sauƙi ga Google kuma NIFI da kanta ta rubuta zuwa log ɗin.

import org.apache.commons.io.IOUtils
import groovy.xml.XmlUtil
import java.nio.charset.*
import groovy.xml.StreamingMarkupBuilder

def flowFile = session.get()
if (!flowFile) return
try {
    flowFile = session.write(flowFile, { inputStream, outputStream ->
        String result = IOUtils.toString(inputStream, "UTF-8");
        def recordIn = new XmlSlurper().parseText(result)
        def element = recordIn.depthFirst().find {
            it.name() == 'id'
        }

        def newId = Integer.parseInt(element.toString()) + 1
        def recordOut = new XmlSlurper().parseText(result)
        recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId

        def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()
        outputStream.write(res.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
     session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
    log.error("Error during processing of validate.groovy", e)
    session.transfer(flowFile, REL_FAILURE)
}

A gaskiya, wannan shine inda gyaran murabba'in ya ƙare. Na gaba, an canza fayil ɗin da aka sabunta zuwa murabba'in, wanda ke da alhakin aika fayil ɗin zuwa uwar garken. A ƙasa akwai saitunan wannan fili.

Mun bayyana hanyar da za a isar da saƙon SOAP. Muna rubuta inda. Na gaba kana buƙatar nuna cewa wannan SABULU ne.

Ƙara kaddarori da yawa kamar su mai gida da aiki (soapAction). Muna ajiyewa mu duba. Kuna iya ganin ƙarin cikakkun bayanai kan yadda ake aika buƙatun SOAP a nan

Mun duba zaɓuɓɓuka da yawa don amfani da hanyoyin NIFI. Yaya suke mu'amala kuma menene ainihin amfanin su? Misalan da aka yi la'akari da su na gwaji ne kuma sun ɗan bambanta da abin da ke faruwa a cikin yaƙi. Ina fatan wannan labarin zai zama ɗan amfani ga masu haɓakawa. Na gode da kulawar ku. Idan kuna da wasu tambayoyi, rubuta. Zan yi kokarin amsa.

source: www.habr.com