Apache NIFI - Cov ntsiab lus luv luv ntawm cov yam ntxwv hauv kev xyaum

Taw qhia

Nws thiaj li tshwm sim tias ntawm kuv qhov chaw ua haujlwm tam sim no kuv yuav tsum tau paub txog cov cuab yeej no. Kuv mam li pib nrog me ntsis keeb kwm yav dhau. Ntawm lub rooj sib tham tom ntej, peb pab neeg tau hais tias peb yuav tsum tsim kom muaj kev koom ua ke nrog paub qhov system. Los ntawm kev sib koom ua ke nws txhais tau tias qhov kev paub zoo no yuav xa peb cov lus thov ntawm HTTP mus rau qhov kawg ntawm qhov kawg, thiab peb, oddly txaus, yuav xa rov qab cov lus teb hauv daim ntawv SOAP lus. Txhua yam zoo li yooj yim thiab tsis tseem ceeb. Los ntawm no nws ua raws li koj xav tau ...

Hom phiaj

Tsim 3 cov kev pabcuam. Thawj ntawm lawv yog Database Update Service. Qhov kev pabcuam no, thaum cov ntaub ntawv tshiab tuaj txog los ntawm cov txheej txheem thib peb, hloov kho cov ntaub ntawv hauv database thiab tsim cov ntaub ntawv hauv CSV hom kom hloov mus rau lwm qhov system. Qhov kawg ntawm qhov kev pabcuam thib ob yog hu ua - FTP Thauj Kev Pabcuam, uas tau txais cov ntaub ntawv hloov pauv, lees paub nws, thiab muab tso rau hauv cov ntaub ntawv khaws cia ntawm FTP. Qhov kev pabcuam thib peb, cov neeg siv cov ntaub ntawv hloov chaw, ua haujlwm asynchronously nrog thawj ob. Nws tau txais kev thov los ntawm lwm tus neeg sab nrauv kom tau txais cov ntaub ntawv hais txog saum toj no, siv cov ntaub ntawv teb npaj txhij, hloov kho nws (hloov kho tus id, piav qhia, linkToFile teb) thiab xa cov lus teb hauv daim ntawv SOAP lus. Ntawd yog, daim duab tag nrho yog raws li hauv qab no: thawj ob qhov kev pabcuam pib ua haujlwm tsuas yog thaum cov ntaub ntawv hloov kho tau tuaj txog. Qhov kev pabcuam thib peb ua haujlwm tas li vim muaj ntau tus neeg siv khoom ntawm cov ntaub ntawv, txog 1000 thov rau cov ntaub ntawv ib feeb. Cov kev pabcuam muaj nyob tas li thiab lawv qhov xwm txheej nyob rau hauv ib puag ncig sib txawv, xws li kev sim, demo, ua ntej tsim khoom thiab khoom muag. Hauv qab no yog daim duab qhia seb cov kev pabcuam no ua haujlwm li cas. Cia kuv qhia tam sim ntawd tias qee cov ntsiab lus tau yooj yim kom tsis txhob muaj qhov nyuaj.

Technical Deepening

Thaum npaj cov kev daws teeb meem, peb thawj zaug txiav txim siab ua cov ntawv thov hauv Java siv lub caij nplooj ntoos hlav, Nginx balancer, Postgres database thiab lwm yam kev siv thiab tsis yog li ntawd. Txij li thaum lub sijhawm los tsim cov kev daws teeb meem tau tso cai rau peb los txiav txim siab txog lwm txoj hauv kev los daws qhov teeb meem no, peb qhov muag poob rau Apache NIFI thev naus laus zis, uas yog fashionable hauv qee lub voj voog. Kuv mam li hais tam sim ntawd tias lub tshuab no tso cai rau peb pom 3 cov kev pabcuam no. Kab lus no yuav piav qhia txog kev txhim kho cov kev pabcuam kev thauj mus los thiab cov ntaub ntawv xa mus rau cov neeg siv khoom, tab sis yog tias tsab xov xwm muaj txiaj ntsig zoo, kuv yuav sau txog qhov kev pabcuam rau kev hloov kho cov ntaub ntawv hauv database.

Nws yog dab tsi?

NIFI yog ib qho kev faib khoom rau kev thauj mus los sai thiab ua tiav cov ntaub ntawv, ntau tus plugins rau cov peev txheej thiab kev hloov pauv, hloov pauv ntawm kev teeb tsa thiab ntau ntxiv. Ib qho txiaj ntsig zoo yog tias nws yooj yim heev rau siv. Cov txheej txheem tsis tseem ceeb xws li getFile, sendHttpRequest thiab lwm tus tuaj yeem sawv cev ua squares. Txhua square sawv cev rau cov txheej txheem, kev sib cuam tshuam uas tuaj yeem pom hauv daim duab hauv qab no. Cov ntaub ntawv ntxaws ntxiv ntawm kev teeb tsa kev sib cuam tshuam tau raug sau no rau cov neeg uas hais lus Lavxias - no. Cov ntaub ntawv zoo kawg nkaus piav qhia yuav ua li cas txhawm rau ntim thiab khiav NIFI, nrog rau yuav ua li cas los tsim cov txheej txheem, tseem hu ua squares
Lub tswv yim los sau ib tsab xov xwm tau yug los tom qab kev tshawb fawb ntev thiab teeb tsa cov ntaub ntawv tau txais rau hauv ib yam dab tsi nco qab, nrog rau kev xav ua kom lub neej yooj yim me ntsis rau cov neeg tsim khoom yav tom ntej..

Piv Txwv:

Ib qho piv txwv ntawm yuav ua li cas squares cuam tshuam nrog ib leeg yog suav tias yog. Lub tswv yim dav dav yog qhov yooj yim heev: Peb tau txais HTTP thov (hauv txoj kev xav, nrog cov ntaub ntawv hauv lub cev ntawm qhov kev thov. Txhawm rau ua kom pom lub peev xwm ntawm NIFI, hauv qhov piv txwv no qhov kev thov pib txheej txheem tau txais cov ntaub ntawv los ntawm cov ntaub ntawv hauv zos. ), tom qab ntawd peb xa rov qab cov lus teb tias qhov kev thov tau txais, nyob rau tib lub sijhawm ntawm kev txais cov ntaub ntawv los ntawm FH thiab tom qab ntawd tus txheej txheem ntawm kev txav ntawm FTP mus rau FH. Nws tsim nyog qhia meej tias cov txheej txheem cuam tshuam nrog ib leeg los ntawm qhov hu ua flowFile. Qhov no yog lub hauv paus chaw hauv NIFI uas khaws cov cwj pwm thiab cov ntsiab lus. Cov ntsiab lus yog cov ntaub ntawv uas sawv cev los ntawm cov ntaub ntawv ntws. Ntawd yog, hais lus ntxhib, yog tias koj tau txais cov ntaub ntawv los ntawm ib lub xwmfab thiab hloov mus rau lwm qhov, cov ntsiab lus yuav yog koj cov ntaub ntawv.

Raws li koj tuaj yeem pom, daim duab no qhia txog cov txheej txheem dav dav. HandleHttpRequest - lees txais kev thov, ReplaceText - tsim lub cev teb, HandleHttpResponse - xa cov lus teb. FetchFile - tau txais cov ntaub ntawv los ntawm cov ntaub ntawv khaws cia, hloov mus rau lub square PutSftp - muab cov ntaub ntawv no tso rau ntawm FTP, ntawm qhov chaw nyob. Tam sim no ntxiv txog cov txheej txheem no.

Hauv qhov no, kev thov yog qhov pib ntawm txhua yam. Cia peb saib ntawm nws cov configuration parameters.

Txhua yam ntawm no yog qhov tseem ceeb heev nrog rau kev zam ntawm StandardHttpContextMap - qhov no yog ib hom kev pabcuam uas tso cai rau koj xa thiab tau txais kev thov. Hauv kev nthuav dav ntxiv thiab txawm nrog cov piv txwv, koj tuaj yeem pom - no

Tom ntej no, cia peb saib ntawm ReplaceText configuration parameters ntawm lub square. Nws yog tsim nyog them sai sai rau ReplacementValue - qhov no yog dab tsi yuav rov qab mus rau tus neeg siv hauv daim ntawv teb. Hauv kev teeb tsa koj tuaj yeem hloov kho qib ntawm kev nkag mus, koj tuaj yeem pom cov cav {qhov twg koj unpacked nifi}/nifi-1.9.2/logs, kuj tseem muaj kev ua tsis tiav / tsis ua tiav - raws li cov kev txwv no koj tuaj yeem tswj cov txheej txheem tag nrho. . Ntawd yog, nyob rau hauv cov ntaub ntawv ntawm kev ua tiav cov ntawv nyeem, cov txheej txheem ntawm kev xa cov lus teb rau tus neeg siv yuav raug hu, thiab nyob rau hauv lwm qhov teeb meem peb yuav tsuas sau cov txheej txheem tsis ua tiav.

Tsis muaj dab tsi tshwj xeeb tshwj xeeb hauv HandleHttpResponse cov khoom tshwj tsis yog cov xwm txheej thaum cov lus teb tau tsim tiav.

Peb tau txheeb xyuas qhov kev thov thiab cov lus teb - cia peb mus rau qhov tau txais cov ntaub ntawv thiab muab tso rau ntawm FTP server. FetchFile - tau txais cov ntaub ntawv ntawm txoj hauv kev tau teev tseg hauv qhov chaw thiab dhau mus rau cov txheej txheem tom ntej.

Thiab tom qab ntawd PutSftp square - muab cov ntaub ntawv tso rau hauv cov ntaub ntawv cia. Peb tuaj yeem pom cov kev teeb tsa tsis raug hauv qab no.

Nws yog ib qho tsim nyog yuav tsum tau them sai sai rau qhov tseeb tias txhua lub xwmfab yog cov txheej txheem cais uas yuav tsum tau pib. Peb saib ntawm qhov piv txwv yooj yim tshaj plaws uas tsis tas yuav muaj kev hloov kho nyuaj. Tom ntej no, peb yuav saib cov txheej txheem me ntsis nyuab dua, qhov twg peb yuav sau me ntsis ntawm cov grooves.

Ntau yam piv txwv

Cov ntaub ntawv xa mus rau cov neeg siv khoom tau dhau los ua qhov nyuaj me ntsis vim yog cov txheej txheem ntawm kev hloov kho cov lus SOAP. Cov txheej txheem dav dav yog qhia hauv daim duab hauv qab no.

Ntawm no lub tswv yim kuj tsis yog qhov nyuaj tshwj xeeb: peb tau txais kev thov los ntawm cov neeg siv khoom uas nws xav tau cov ntaub ntawv, xa cov lus teb tias nws tau txais cov lus, pib txheej txheem ntawm kev txais cov ntaub ntawv teb, tom qab ntawd kho nws nrog qee qhov laj thawj, thiab tom qab ntawd Hloov cov ntaub ntawv mus rau cov neeg siv khoom hauv daim ntawv SOAP lus rau lub server.

Kuv xav tias tsis tas yuav piav ntxiv txog cov squares uas peb tau pom saum toj no - cia peb txav mus rau qhov tshiab. Yog tias koj xav tau hloov kho cov ntaub ntawv thiab qhov zoo tib yam ReplaceText hom squares tsis haum, koj yuav tau sau koj tus kheej tsab ntawv. Qhov no tuaj yeem ua tiav siv ExecuteGrogyScript square. Nws cov kev teeb tsa tau nthuav tawm hauv qab no.

Muaj ob txoj hauv kev los thauj cov ntawv rau hauv lub xwmfab no. Thawj yog los ntawm rub tawm cov ntaub ntawv nrog ib tsab ntawv. Qhov thib ob yog los ntawm kev ntxig ib tsab ntawv rau hauv scriptBody. Raws li kuv paub, executeScript square txhawb ntau hom lus - ib qho ntawm lawv yog groovy. Kuv yuav poob siab java developers - koj tsis tuaj yeem sau ntawv hauv java hauv cov xwm txheej zoo li no. Rau cov neeg uas xav tau tiag tiag, koj yuav tsum tsim koj tus kheej kev cai square thiab ntxiv rau NIFI system. Qhov kev ua haujlwm tag nrho no yog nrog los ntawm kev ua las voos ntev nrog tambourine, uas peb yuav tsis cuam tshuam nrog hauv kab lus no. Kuv xaiv hom lus groovy. Hauv qab no yog cov ntawv xeem uas tsuas yog hloov kho tus ID hauv SOAP lus. Nws yog ib qho tseem ceeb kom nco ntsoov. Koj nqa cov ntaub ntawv los ntawm flowFile thiab hloov kho nws, tsis txhob hnov ​​​​qab tias koj yuav tsum muab nws rov qab rau ntawd, kho tshiab. Nws tseem tsim nyog sau cia tias tsis yog txhua lub tsev qiv ntawv suav nrog. Tej zaum nws yuav tshwm sim tias koj tseem yuav tsum tau import ib qho ntawm cov libs. Lwm downside yog tias tsab ntawv nyob rau hauv no square yog heev nyuaj rau debug. Muaj ib txoj hauv kev txuas mus rau NIFI JVM thiab pib cov txheej txheem debugging. Tus kheej, kuv tau tsim ib daim ntawv thov hauv zos thiab simulated tau txais cov ntaub ntawv los ntawm kev sib kho. Kuv kuj tau debugging hauv zos. Qhov yuam kev uas tshwm sim thaum thauj cov ntawv yog yooj yim heev rau Google thiab sau los ntawm NIFI nws tus kheej rau lub cav.

import org.apache.commons.io.IOUtils
import groovy.xml.XmlUtil
import java.nio.charset.*
import groovy.xml.StreamingMarkupBuilder

def flowFile = session.get()
if (!flowFile) return
try {
    flowFile = session.write(flowFile, { inputStream, outputStream ->
        String result = IOUtils.toString(inputStream, "UTF-8");
        def recordIn = new XmlSlurper().parseText(result)
        def element = recordIn.depthFirst().find {
            it.name() == 'id'
        }

        def newId = Integer.parseInt(element.toString()) + 1
        def recordOut = new XmlSlurper().parseText(result)
        recordOut.Body.ClientMessage.RequestMessage.RequestContent.content.MessagePrimaryContent.ResponseBody.id = newId

        def res = new StreamingMarkupBuilder().bind { mkp.yield recordOut }.toString()
        outputStream.write(res.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
     session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
    log.error("Error during processing of validate.groovy", e)
    session.transfer(flowFile, REL_FAILURE)
}

Qhov tseeb, qhov no yog qhov kev hloov kho ntawm square xaus. Tom ntej no, cov ntaub ntawv kho tshiab raug xa mus rau lub square, uas yog lub luag haujlwm xa cov ntaub ntawv mus rau lub server. Hauv qab no yog cov chaw rau lub square.

Peb piav qhia txog txoj hauv kev uas cov lus SOAP yuav raug xa mus. Peb sau qhov twg. Tom ntej no koj yuav tsum qhia tias qhov no yog SOAP.

Ntxiv ntau yam khoom xws li tus tswv tsev thiab kev ua haujlwm (soapAction). Peb txuag thiab xyuas. Koj tuaj yeem pom cov ntsiab lus ntxiv txog yuav xa SOAP thov li cas S, SѓS,

Peb tau saib ntau qhov kev xaiv rau kev siv cov txheej txheem NIFI. Lawv cuam tshuam li cas thiab lawv qhov txiaj ntsig tiag tiag yog dab tsi? Cov piv txwv suav tias yog cov kev sim thiab txawv me ntsis ntawm qhov tshwm sim tiag tiag hauv kev sib ntaus sib tua. Kuv vam tias tsab xov xwm no yuav muaj txiaj ntsig me ntsis rau cov neeg tsim khoom. Ua tsaug rau koj mloog. Yog tias koj muaj lus nug, sau. Kuv mam sim teb.

Tau qhov twg los: www.hab.com