Nyob zoo txhua leej txhua tus!

Txoj haujlwm yog raws li hauv qab no - muaj qhov ntws qhia hauv daim duab saum toj no, uas yuav tsum tau muab dov tawm mus rau N servers nrog Apache NiFi. Flow test - cov ntaub ntawv raug tsim tawm thiab xa mus rau lwm qhov piv txwv NiFi. Hloov cov ntaub ntawv tshwm sim siv NiFi Site rau qhov chaw raws tu qauv.

NiFi Site to Site (S2S) yog ib txoj hauv kev ruaj ntseg, kho tau zoo heev los hloov cov ntaub ntawv ntawm NiFi piv txwv. Saib seb S2S ua haujlwm li cas cov ntaub ntawv thiab nws tseem ceeb heev uas yuav tsum nco ntsoov teeb tsa koj li NiFi piv txwv kom tso cai rau S2S pom S, SѓS,.

Thaum nws los txog rau kev hloov ntaub ntawv siv S2S, ib qho piv txwv hu ua tus neeg siv khoom, qhov thib ob yog server. Tus neeg siv khoom xa cov ntaub ntawv, tus neeg rau zaub mov tau txais nws. Ob txoj hauv kev los teeb tsa cov ntaub ntawv hloov chaw ntawm lawv:

laub. Cov ntaub ntawv raug xa los ntawm tus neeg siv khoom siv siv Cov Txheej Txheem Chaw Taws Teeb (RPG). Ntawm lub server piv txwv, cov ntaub ntawv tau txais los ntawm kev siv qhov chaw nres nkoj nkag

Rub. Tus neeg rau zaub mov tau txais cov ntaub ntawv siv RPG, tus neeg siv khoom xa tuaj siv qhov chaw nres nkoj Output.

Flow rau dov yog khaws cia hauv Apache Registry.

Apache NiFi Registry yog ib txoj haujlwm me me ntawm Apache NiFi uas muab cov khoom ntws ntws thiab hloov kho cov cuab yeej. Ib hom GIT. Cov ntaub ntawv hais txog kev txhim kho, teeb tsa thiab ua haujlwm nrog cov npe tuaj yeem pom hauv cov ntaub ntawv raug cai. Flow rau cia yog muab tso rau hauv ib pawg txheej txheem thiab khaws cia rau hauv daim ntawv teev npe hauv daim ntawv no. Peb yuav rov qab mus rau qhov no tom qab hauv tsab xov xwm.

Thaum pib, thaum N yog tus lej me me, qhov ntws tau xa thiab hloov kho los ntawm tes hauv lub sijhawm tsim nyog.

Tab sis thaum N loj hlob, muaj teeb meem ntau ntxiv:

nws yuav siv sij hawm ntau dua los hloov kho qhov ntws. Koj yuav tsum mus rau tag nrho cov servers
muaj qhov yuam kev hloov tshiab templates. Ntawm no lawv hloov kho, tab sis ntawm no lawv tsis nco qab lawm
tib neeg yuam kev thaum ua ntau yam haujlwm zoo sib xws

Tag nrho cov no coj peb mus rau lub fact tias nws yog tsim nyog los automate tus txheej txheem. Kuv tau sim cov hauv qab no los daws qhov teeb meem no:

Siv MiNiFi hloov NiFi
Neeb CLI
NiPyAPI

Siv MiNiFi

ApacheMiNify yog ib qhov project sub ntawm Apache NiFi. MiNiFy yog tus neeg sawv cev sib cog lus uas siv tib cov txheej txheem xws li NiFi, tso cai rau koj los tsim cov dej ntws zoo ib yam li hauv NiFi. Lub lightness ntawm tus neeg sawv cev yog tiav, ntawm lwm yam, vim lub fact tias MiNiFy tsis muaj ib tug graphical interface rau cov flow configuration. MiNiFy qhov tsis muaj graphical interface txhais tau hais tias nws yog qhov tsim nyog los daws qhov teeb meem ntawm kev xa khoom hauv minifi. Txij li thaum MiNiFy nquag siv hauv IOT, muaj ntau yam khoom thiab cov txheej txheem ntawm kev xa dej mus rau qhov kawg minifi yuav tsum tau ua haujlwm. Ib txoj haujlwm paub, txoj cai?

Lwm subproject, MiNiFi C2 Server, yuav pab daws qhov teeb meem no. Cov khoom no yog npaj los ua lub hauv paus ntsiab lus hauv kev xa tawm architecture. Yuav ua li cas rau configure ib puag ncig - piav nyob rau hauv qhov no tsab xov xwm ntawm Habre thiab cov ntaub ntawv txaus los daws qhov teeb meem. MiNiFi ua ke nrog C2 server tau hloov kho nws cov teeb tsa. Qhov tsuas yog qhov tsis zoo ntawm txoj hauv kev no yog tias koj yuav tsum tsim cov qauv ntawm C2 Server, kev cog lus yooj yim rau kev sau npe tsis txaus.

Cov kev xaiv tau piav qhia hauv kab lus saum toj no ua haujlwm thiab tsis yooj yim rau kev siv, tab sis peb yuav tsum tsis txhob hnov qab cov hauv qab no:

minifi tsis muaj tag nrho cov processors los ntawm nifi
CPU versions hauv Minifi lag tom qab CPU versions hauv NiFi.

Thaum lub sijhawm sau ntawv, qhov tseeb version ntawm NiFi yog 1.9.2. Tus processor version ntawm qhov tseeb MiNiFi version yog 1.7.0. Cov txheej txheem tuaj yeem ntxiv rau MiNiFi, tab sis vim yog qhov sib txawv ntawm cov txheej txheem ntawm NiFi thiab MiNiFi processors, qhov no yuav tsis ua haujlwm.

Neeb CLI

Kev txiav txim los ntawm piav qhia cov cuab yeej ntawm lub vev xaib raug cai, qhov no yog ib qho cuab yeej rau kev sib cuam tshuam ntawm NiFI thiab NiFi Registry hauv kev khiav dej num lossis kev tswj xyuas cov txheej txheem. Download tau qhov cuab yeej no los pib. ntawm no.

Khiav cov khoom siv hluav taws xob

./bin/cli.sh
           _     ___  _
 Apache   (_)  .' ..](_)   ,
 _ .--.   __  _| |_  __    )
[ `.-. | [  |'-| |-'[  |  /  
|  | | |  | |  | |   | | '    '
[___||__][___][___] [___]',  ,'
                           `'
          CLI v1.9.2

Type 'help' to see a list of available commands, use tab to auto-complete.

Txhawm rau kom peb thauj cov khoom tsim nyog los ntawm kev sau npe, peb yuav tsum paub cov cim ntawm lub pob tawb (piv txwv li) thiab qhov ntws ntawm nws tus kheej (flow identifier). Cov ntaub ntawv no tuaj yeem tau txais los ntawm cli lossis hauv NiFi kev sau npe web interface. Lub web interface zoo li no:

Siv CLI, koj ua qhov no:

#> registry list-buckets -u http://nifi-registry:18080

#   Name             Id                                     Description
-   --------------   ------------------------------------   -----------
1   test_bucket   709d387a-9ce9-4535-8546-3621efe38e96   (empty)

#> registry list-flows -b 709d387a-9ce9-4535-8546-3621efe38e96 -u http://nifi-registry:18080

#   Name           Id                                     Description
-   ------------   ------------------------------------   -----------
1   test_flow   d27af00a-5b47-4910-89cd-9c664cd91e85

Khiav cov txheej txheem import pawg los ntawm kev sau npe:

#> nifi pg-import -b 709d387a-9ce9-4535-8546-3621efe38e96 -f d27af00a-5b47-4910-89cd-9c664cd91e85 -fv 1 -u http://nifi:8080

7f522a13-016e-1000-e504-d5b15587f2f3

Lub ntsiab lus tseem ceeb yog tias ib qho piv txwv nifi tuaj yeem teev tau raws li tus tswv tsev uas peb dov pawg txheej txheem.

Cov txheej txheem pab pawg ntxiv nrog cov processors nres, lawv yuav tsum tau pib

#> nifi pg-start -pgid 7f522a13-016e-1000-e504-d5b15587f2f3 -u http://nifi:8080

Zoo heev, cov processors tau pib. Txawm li cas los xij, raws li cov xwm txheej ntawm qhov teeb meem, peb xav tau NiFi piv txwv xa cov ntaub ntawv mus rau lwm qhov xwm txheej. Cia peb xav tias Push txoj kev raug xaiv los hloov cov ntaub ntawv mus rau lub server. Txhawm rau npaj cov ntaub ntawv hloov chaw, nws yog ib qho tsim nyog yuav tsum tau ua kom cov ntaub ntawv hloov pauv (Enable transmitting) ntawm qhov ntxiv Remote Process Group (RPG), uas twb tau suav nrog hauv peb cov dej ntws.

Hauv cov ntaub ntawv hauv CLI thiab lwm qhov chaw, kuv tsis pom txoj hauv kev los pab hloov cov ntaub ntawv. Yog tias koj paub yuav ua li cas, thov sau rau hauv cov lus.

Txij li thaum peb muaj bash thiab peb npaj mus rau qhov kawg, peb yuav pom txoj hauv kev tawm! Koj tuaj yeem siv NiFi API los daws qhov teeb meem no. Cia peb siv cov qauv hauv qab no, peb coj tus ID los ntawm cov piv txwv saum toj no (hauv peb cov ntaub ntawv nws yog 7f522a13-016e-1000-e504-d5b15587f2f3). Kev piav qhia ntawm NiFi API Txoj Kev S, SѓS,.

Hauv lub cev, koj yuav tsum dhau JSON, ntawm daim ntawv hauv qab no:

{
    "revision": {
	    "clientId": "value",
	    "version": 0,
	    "lastModifier": "value"
	},
    "state": "value",
    "disconnectedNodeAcknowledged": true
}

Parameters uas yuav tsum tau sau nyob rau hauv thiaj li yuav "ua hauj lwm":
lub xeev - cov ntaub ntawv hloov chaw. Muaj TRANSMITTING los pab kom cov ntaub ntawv hloov chaw, STOPPED kom lov tes taw
version - processor version

version yuav default rau 0 thaum tsim, tab sis cov tsis tuaj yeem tau txais los ntawm txoj kev

Rau cov neeg nyiam ntawm bash scripts, hom no yuav zoo li tsim nyog, tab sis nws nyuaj rau kuv - bash scripts tsis yog kuv nyiam. Txoj kev tom ntej yog nthuav thiab yooj yim dua hauv kuv lub tswv yim.

NiPyAPI

NiPyAPI yog lub tsev qiv ntawv Python rau kev cuam tshuam nrog NiFi piv txwv. Cov ntaub ntawv nplooj ntawv muaj cov ntaub ntawv tsim nyog los ua haujlwm nrog lub tsev qiv ntawv. Kev pib nrawm tau piav qhia hauv qhov project ntawm github.

Peb tsab ntawv rau dov tawm qhov kev teeb tsa yog Python program. Cia peb mus rau coding.
Teem configs rau kev ua haujlwm ntxiv. Peb yuav xav tau cov nram qab no parameter:

nipyapi.config.nifi_config.host = 'http://nifi:8080/nifi-api' #путь до nifi-api инстанса, на котором разворачиваем process group
nipyapi.config.registry_config.host = 'http://nifi-registry:18080/nifi-registry-api' #путь до nifi-registry-api registry
nipyapi.config.registry_name = 'MyBeutifulRegistry' #название registry, как будет называться в инстансе nifi
nipyapi.config.bucket_name = 'BucketName' #название bucket, из которого подтягиваем flow
nipyapi.config.flow_name = 'FlowName' #название flow, которое подтягиваем

Tsis tas li ntawd kuv yuav ntxig cov npe ntawm cov txheej txheem ntawm lub tsev qiv ntawv no, uas tau piav qhia S, SѓS,.

Peb txuas lub npe mus rau nifi piv txwv siv

nipyapi.versioning.create_registry_client

Ntawm cov kauj ruam no, koj tuaj yeem ntxiv ib daim tshev uas sau npe twb tau ntxiv rau qhov piv txwv, rau qhov no koj tuaj yeem siv txoj hauv kev.

nipyapi.versioning.list_registry_clients

Peb nrhiav lub thoob los tshawb nrhiav ntxiv rau qhov ntws hauv pob tawb

nipyapi.versioning.get_registry_bucket

Raws li lub thoob pom, peb tab tom nrhiav kev ntws

nipyapi.versioning.get_flow_in_bucket

Tom ntej no, nws yog ib qho tseem ceeb kom nkag siab yog tias pawg txheej txheem no twb tau ntxiv lawm. Cov txheej txheem pab pawg tau muab tso rau los ntawm kev tswj hwm thiab qhov xwm txheej yuav tshwm sim thaum tus thib ob yog superimposed rau saum ib qho. Kuv tshawb xyuas, nws tuaj yeem yog 🙂 Kom tau txais tag nrho cov txheej txheem ntxiv, siv txoj hauv kev

nipyapi.canvas.list_all_process_groups

thiab tom qab ntawd peb tuaj yeem tshawb nrhiav, piv txwv li, los ntawm lub npe.

Kuv yuav tsis piav qhia txog cov txheej txheem ntawm kev hloov kho cov qauv, Kuv tsuas yog hais tias yog cov txheej txheem ntxiv rau hauv cov qauv tshiab, ces tsis muaj teeb meem nrog cov lus nyob hauv cov kab. Tab sis yog hais tias tus processors raug tshem tawm, ces tej zaum yuav muaj teeb meem (nifi tsis pub lub processor yuav raug tshem tawm yog hais tias ib tug kab lus tau sau nyob rau hauv pem hauv ntej ntawm nws). Yog tias koj xav paub yuav ua li cas kuv daws qhov teeb meem no - sau ntawv rau kuv, thov, peb yuav tham txog qhov no. Hu rau qhov kawg ntawm tsab xov xwm. Cia peb mus rau theem ntawm kev ntxiv ib pawg txheej txheem.

Thaum debugging tsab ntawv, kuv tuaj hla ib tug feature uas qhov tseeb version ntawm ntws yog tsis ib txwm rub, yog li kuv xav kom koj ua ntej kom meej cov version no:

nipyapi.versioning.get_latest_flow_ver

Deploy process group:

nipyapi.versioning.deploy_flow_version

Peb pib cov processors:

nipyapi.canvas.schedule_process_group

Hauv qhov thaiv ntawm CLI, nws tau sau tias kev hloov pauv cov ntaub ntawv tsis tuaj yeem qhib rau hauv pawg txheej txheem chaw taws teeb? Thaum siv tsab ntawv, kuv ntsib qhov teeb meem no thiab. Lub sijhawm ntawd, kuv tsis tuaj yeem pib hloov cov ntaub ntawv siv API thiab kuv txiav txim siab sau ntawv mus rau tus tsim tawm ntawm lub tsev qiv ntawv NiPyAPI thiab thov kev tawm tswv yim / kev pab. Tus tsim tawm teb rau kuv, peb tau tham txog qhov teeb meem thiab nws tau sau tias nws xav tau sijhawm los "kos ib yam dab tsi". Thiab tam sim no, ob peb hnub tom qab, email tuaj txog uas Python muaj nuj nqi tau sau uas daws kuv qhov teeb meem pib !!! Lub sijhawm ntawd, NiPyAPI version yog 0.13.3 thiab, tau kawg, tsis muaj ib yam dab tsi ntawm nws. Tab sis hauv version 0.14.0, uas tau tso tawm tsis ntev los no, cov haujlwm no twb tau suav nrog hauv tsev qiv ntawv. Ntsib

nipyapi.canvas.set_remote_process_group_transmission

Yog li, nrog kev pab los ntawm NiPyAPI lub tsev qiv ntawv, peb txuas nrog lub npe, dov cov ntws, thiab txawm pib cov txheej txheem thiab cov ntaub ntawv hloov chaw. Tom qab ntawd koj tuaj yeem comb cov cai, ntxiv txhua yam kev txheeb xyuas, txiav, thiab qhov ntawd yog nws. Tab sis qhov ntawd yog ib zaj dab neeg sib txawv kiag li.

Ntawm cov kev xaiv automation uas kuv tau txiav txim siab, qhov kawg zoo li kuv qhov ua tau zoo tshaj plaws. Ua ntej, qhov no tseem yog python code, nyob rau hauv uas koj tuaj yeem embed auxiliary program code thiab coj kom zoo dua ntawm cov lus programming. Qhov thib ob, NiPyAPI project tab tom txhim kho thiab yog tias muaj teeb meem koj tuaj yeem sau rau tus tsim tawm. Thib peb, NiPyAPI tseem yog qhov hloov pauv tau yooj yim dua rau kev cuam tshuam nrog NiFi hauv kev daws teeb meem nyuaj. Piv txwv li, hauv kev txiav txim siab seb cov lus queues tam sim no khoob hauv qhov ntws thiab seb nws puas tuaj yeem hloov kho pawg txheej txheem.

Yog tag nrho. Kuv tau piav qhia 3 txoj hauv kev rau automating ntws xa hauv NiFi, qhov pitfalls uas tus tsim tawm yuav ntsib thiab muab cov cai ua haujlwm rau kev xa khoom. Yog tias koj txaus siab rau lub ncauj lus no ib yam li kuv - sau!

Tau qhov twg los: www.hab.com

Flow Delivery Automation hauv Apache NiFi

Siv MiNiFi

Neeb CLI

NiPyAPI

Ntxiv ib saib Отменить ответ