Sawubona wonke umntu!

Umsebenzi umi ngolu hlobo lulandelayo - kukho ukuhamba okuboniswe kumfanekiso ongentla, ekufuneka ukhutshelwe ngaphandle kwiiseva ze-N nge Apache NiFi. Uvavanyo lokuqukuqela - ifayile iyenziwa kwaye ithunyelwe komnye umzekelo weNiFi. Ugqithiso lwedatha lwenzeka kusetyenziswa iSiza seNiFi kwiSayithi protocol.

Indawo ye-NiFi kwiSiza (i-S2S) yindlela ekhuselekileyo, enokwenziwa ngokwezifiso kakhulu yokudlulisa idatha phakathi kweemeko ze-NiFi. Jonga ukuba isebenza njani i-S2S amaxwebhu kwaye kubalulekile ukukhumbula ukuseta umzekelo wakho we-NiFi ukuvumela i-S2S ibone apha.

Xa kuziwa ekudluliselweni kwedatha usebenzisa i-S2S, umzekelo omnye ubizwa ngokuba ngumxhasi, owesibini ngumncedisi. Umxhasi uthumela idatha, umncedisi uyayifumana. Iindlela ezimbini zokuseta ukudluliselwa kwedatha phakathi kwazo:

Dudula. Idatha ithunyelwa kumzekelo womxhasi kusetyenziswa iQela leNkqubo eLawulwa kude (RPG). Kumzekelo weseva, idatha ifunyenwe kusetyenziswa i-Input Port

Tsala. Umncedisi ufumana idata usebenzisa i RPG, umxhasi uthumela usebenzisa Imveliso yezibuko.

Ukuhamba kokuqengqeleka kugcinwa kwiRejistri yeApache.

I-Apache NiFi Registry yi-subproject ye-Apache NiFi ebonelela ngokugcinwa kokuhamba kunye nesixhobo soguqulelo. Uhlobo lwe-GIT. Ulwazi malunga nokufaka, ukuqwalasela kunye nokusebenza kunye nerejista inokufumaneka kwi amaxwebhu asemthethweni. Ukuhamba kokugcinwa kuhlanganiswe kwiqela lenkqubo kwaye kugcinwe kwirejista kule fomu. Siza kubuyela kule kamva kwinqaku.

Ekuqaleni, xa i-N iyinani elincinci, ukuhamba kuhanjiswe kwaye kuhlaziywe ngesandla ngexesha elifanelekileyo.

Kodwa njengoko i-N ikhula, kukho iingxaki ezininzi:

kuthatha ixesha elininzi ukuhlaziya ukuhamba. Kufuneka uye kuzo zonke iiseva
kukho iimpazamo zokuhlaziya itemplates. Apha bahlaziya, kodwa apha balibale
impazamo yomntu xa kusenziwa inani elikhulu lemisebenzi efanayo

Konke oku kusizisa kwinto yokuba kuyimfuneko ukwenza inkqubo ngokuzenzekelayo. Ndizame ezi ndlela zilandelayo ukusombulula le ngxaki:

Sebenzisa iMiNiFi endaweni yeNiFi
NiFi CLI
NiPyAPI

Ukusebenzisa iMiNiFi

ApacheMiNify yiprojekthi ephantsi ye-Apache NiFi. I-MiNiFy yi-arhente edibeneyo esebenzisa iiprosesa ezifanayo njenge-NiFi, ekuvumela ukuba udale ukuhamba okufanayo njenge-NiFi. Ukukhanya kwe-arhente kuphunyeziwe, phakathi kwezinye izinto, ngenxa yokuba i-MiNiFy ayinayo i-graphical interface yoqwalaselo lokuhamba. Ukunqongophala kwe-MiNiFy ye-graphical interface kuthetha ukuba kuyimfuneko ukusombulula ingxaki yokuhanjiswa kwe-flow minifi. Ekubeni i-MiNiFy isetyenziswa ngokusebenzayo kwi-IOT, kukho amacandelo amaninzi kunye nenkqubo yokuhambisa ukuhamba kwiimeko zokugqibela ze-minifi kufuneka zizenzekele. Umsebenzi oqhelekileyo, akunjalo?

Enye i-subproject, i-MiNiFi C2 Server, iya kunceda ukuxazulula le ngxaki. Le mveliso ijoliswe ukuba ibe yindawo ephambili kwi-architecture yokuthunyelwa. Uyiqwalasela njani imeko-bume-ichazwe kwi eli nqaku kuHabré kunye nolwazi lwanele ukusombulula ingxaki. I-MiNiFi ngokubambisana ne-C2 iseva ihlaziya ngokuzenzekelayo ukucwangciswa kwayo. I-drawback kuphela yale ndlela kukuba kufuneka udale iitemplates kwi-C2 Server, ukuzibophelela okulula kwirejista akwanele.

Inketho echazwe kwinqaku elingentla iyasebenza kwaye akunzima ukuyiphumeza, kodwa akufanele silibale oku kulandelayo:

minifi ayinazo zonke iiprosesa ezivela kwi-nifi
Iinguqulelo ze-CPU kwi-Minifi lag ngasemva kweenguqulelo ze-CPU kwi-NiFi.

Ngexesha lokubhala, inguqulo yakamuva ye-NiFi yi-1.9.2. Inguqulo yeprosesa yenguqulo yakutshanje ye-MiNiFi yi-1.7.0. Iiprosesa zinokongezwa kwi-MiNiFi, kodwa ngenxa yokungafani kwenguqulelo phakathi kwe-NiFi kunye ne-MiNiFi processors, oku kungasebenzi.

NiFi CLI

Ukugweba inkcazelo isixhobo kwiwebhusayithi esemthethweni, esi sixhobo sokuzenzekelayo ukusebenzisana phakathi kwe-NiFI kunye ne-NiFi Registry kwintsimi yokuhanjiswa okanye ulawulo lwenkqubo. Khuphela esi sixhobo ukuze uqalise. kusuka apha.

Qhuba usetyenziso

./bin/cli.sh
           _     ___  _
 Apache   (_)  .' ..](_)   ,
 _ .--.   __  _| |_  __    )
[ `.-. | [  |'-| |-'[  |  /  
|  | | |  | |  | |   | | '    '
[___||__][___][___] [___]',  ,'
                           `'
          CLI v1.9.2

Type 'help' to see a list of available commands, use tab to auto-complete.

Ukuze silayishe ukuhamba okuyimfuneko ukusuka kwirejista, kufuneka sazi izichazi zebhaskithi (isazisi sebhakethi) kunye nokuhamba ngokwazo (isazisi sokuhamba). Le datha inokufunyanwa mhlawumbi nge-cli okanye kwi-interface yewebhu ye-NiFi yobhaliso. Ujongano lwewebhu lujongeka ngolu hlobo:

Ukusebenzisa i-CLI, wenza oku:

#> registry list-buckets -u http://nifi-registry:18080

#   Name             Id                                     Description
-   --------------   ------------------------------------   -----------
1   test_bucket   709d387a-9ce9-4535-8546-3621efe38e96   (empty)

#> registry list-flows -b 709d387a-9ce9-4535-8546-3621efe38e96 -u http://nifi-registry:18080

#   Name           Id                                     Description
-   ------------   ------------------------------------   -----------
1   test_flow   d27af00a-5b47-4910-89cd-9c664cd91e85

Qhuba iqela lenkqubo yokuthatha ngaphandle kubhaliso:

#> nifi pg-import -b 709d387a-9ce9-4535-8546-3621efe38e96 -f d27af00a-5b47-4910-89cd-9c664cd91e85 -fv 1 -u http://nifi:8080

7f522a13-016e-1000-e504-d5b15587f2f3

Inqaku elibalulekileyo kukuba nawuphi na umzekelo we-nifi unokuchazwa njengomamkeli apho siqengqela khona iqela lenkqubo.

Iqela lenkqubo elongeziweyo kunye nabaqhubekisi abamisiweyo, kufuneka baqalwe

#> nifi pg-start -pgid 7f522a13-016e-1000-e504-d5b15587f2f3 -u http://nifi:8080

Kulungile, iiprosesa ziqalile. Nangona kunjalo, ngokwemiqathango yengxaki, sifuna iimeko ze-NiFi ukuthumela idatha kwezinye iimeko. Makhe sicinge ukuba indlela yokuTyhala yakhethwa ukudlulisela idatha kumncedisi. Ukuze ulungelelanise ukuhanjiswa kwedatha, kuyimfuneko ukwenza ukuhanjiswa kwedatha (Vumela ukuhanjiswa) kwiQela leNkqubo yeRemote eyongeziweyo (RPG), esele ibandakanyiwe ekuhambeni kwethu.

Kumaxwebhu kwi-CLI nakweminye imithombo, andizange ndifumane indlela yokwenza ukudluliselwa kwedatha. Ukuba uyayazi indlela yokwenza oku, nceda ubhale kwizimvo.

Kuba sine-bash kwaye sikulungele ukuya esiphelweni, siya kufumana indlela yokuphuma! Ungasebenzisa i-NiFi API ukusombulula le ngxaki. Masisebenzise le ndlela ilandelayo, sithatha i-ID kwimizekelo engentla (kwimeko yethu yi-7f522a13-016e-1000-e504-d5b15587f2f3). Inkcazo yeeNdlela ze-NiFi API apha.

Emzimbeni, kufuneka udlule i-JSON, yale fomu ilandelayo:

{
    "revision": {
	    "clientId": "value",
	    "version": 0,
	    "lastModifier": "value"
	},
    "state": "value",
    "disconnectedNodeAcknowledged": true
}

Iiparamitha ekufuneka zizaliswe ukuze "usebenze":
Lumente — imo yogqithiso lwedatha. Ifumaneka TRANSMITTING ukwenza ugqithiso lwedatha, STOPED ukukhubaza
Inguqulelo - inguqulo yeprosesa

uguqulelo lungagqibekanga ukuya ku-0 xa yenziwe, kodwa ezi parameters zinokufunyanwa kusetyenziswa indlela

Kubathandi bezikripthi ze-bash, le ndlela inokubonakala ifanelekile, kodwa inzima kum - izikripthi ze-bash azikho intando yam. Indlela elandelayo inomdla ngakumbi kwaye ilungele ngakumbi kwimbono yam.

NiPyAPI

I-NiPyAPI yilayibrari yePython yokunxibelelana neemeko zeNiFi. Iphepha loxwebhu iqulethe ulwazi oluyimfuneko lokusebenza nethala leencwadi. Ukuqala ngokukhawuleza kuchazwe kwi iprojekthi kwi github.

Isikripthi sethu sokukhupha uqwalaselo yinkqubo yePython. Masiqhubele phambili kwikhowudi.
Cwangcisa uqwalaselo lomsebenzi oqhubekayo. Siya kufuna ezi parameters zilandelayo:

nipyapi.config.nifi_config.host = 'http://nifi:8080/nifi-api' #путь до nifi-api инстанса, на котором разворачиваем process group
nipyapi.config.registry_config.host = 'http://nifi-registry:18080/nifi-registry-api' #путь до nifi-registry-api registry
nipyapi.config.registry_name = 'MyBeutifulRegistry' #название registry, как будет называться в инстансе nifi
nipyapi.config.bucket_name = 'BucketName' #название bucket, из которого подтягиваем flow
nipyapi.config.flow_name = 'FlowName' #название flow, которое подтягиваем

Ngapha koko ndiza kufaka amagama eendlela zeli thala leencwadi, ezichaziweyo apha.

Sidibanisa irejistri kumzekelo we-nifi usebenzisa

nipyapi.versioning.create_registry_client

Kweli nyathelo, unokongeza kwakhona itshekhi ukuba ubhaliso sele longezwe kumzekelo, kuba oku ungasebenzisa indlela.

nipyapi.versioning.list_registry_clients

Sifumana ibhakethi ukukhangela ngakumbi ukuhamba kwebhasikithi

nipyapi.versioning.get_registry_bucket

Ngokwebhakethi efunyenweyo, sijonge ukuhamba

nipyapi.versioning.get_flow_in_bucket

Okulandelayo, kubalulekile ukuqonda ukuba eli qela lenkqubo sele longeziwe. Iqela lenkqubo libekwe yi-coordinates kwaye imeko inokuvela xa isibini sibekwe ngaphezulu kwenye. Ndijongile, inokuba 🙂 Ukufumana lonke iqela lenkqubo eyongeziweyo, sebenzisa indlela

nipyapi.canvas.list_all_process_groups

kwaye ke sinokukhangela, umzekelo, ngegama.

Andiyi kuchaza inkqubo yokuhlaziywa kwethemplate, ndiya kuthetha kuphela ukuba ngaba iiprosesa zongezwa kwinguqu entsha yethemplate, ngoko akukho ngxaki kunye nobukho bemiyalezo kwimigca. Kodwa ukuba iiprosesa zisusiwe, ke iingxaki zinokuvela (i-nifi ayivumeli ukususwa kweprosesa ukuba umgca womyalezo uqokelele phambi kwayo). Ukuba unomdla kwindlela endiyicombulula ngayo le ngxaki - bhala kum, nceda, siza kuxubusha le ngongoma. Abafowunelwa ekupheleni kwenqaku. Masiqhubele phambili kwinyathelo lokongeza iqela lenkqubo.

Xa kulungiswa iskripthi, ndifumene into yokuba inguqulelo yamva nje yokuhamba ayisoloko inyuswa, ke ndincoma ukuba uqale uyicacise le nguqulelo:

nipyapi.versioning.get_latest_flow_ver

Sebenzisa iqela lenkqubo:

nipyapi.versioning.deploy_flow_version

Siqala iiprosesa:

nipyapi.canvas.schedule_process_group

Kwibhloko malunga ne-CLI, kubhaliwe ukuba ukuhanjiswa kwedatha akwenziwanga ngokuzenzekelayo kwiqela lenkqubo ekude? Xa ndiphumeza iskripthi, nam ndidibene nale ngxaki. Ngelo xesha, andizange ndikwazi ukuqalisa ukudluliselwa kwedatha usebenzisa i-API kwaye ndagqiba ekubeni ndibhalele kumthuthukisi welayibrari ye-NiPyAPI kwaye ndicele iingcebiso / uncedo. Umphuhlisi wandiphendula, saxoxa ngengxaki kwaye wabhala ukuba ufuna ixesha "lokukhangela into". Kwaye ngoku, kwiintsuku ezimbalwa kamva, i-imeyile ifika apho kubhalwe umsebenzi wePython osombulula ingxaki yam yokuqalisa !!! Ngelo xesha, inguqulo ye-NiPyAPI yayingu-0.13.3 kwaye, ngokuqinisekileyo, kwakungekho nto yolo hlobo kuyo. Kodwa kwinguqulo 0.14.0, ekhutshwe kutshanje, lo msebenzi sele ufakiwe kwithala leencwadi. Hlangana

nipyapi.canvas.set_remote_process_group_transmission

Ngoko, ngoncedo lwelayibrari ye-NiPyAPI, sadibanisa irejistri, saqengqeleka ukuhamba, saze saqalisa iiprosesa kunye nokudluliselwa kwedatha. Emva koko ungadibanisa ikhowudi, udibanise zonke iintlobo zeetshekhi, ukugawulwa kwemithi, kwaye yiloo nto. Kodwa elo libali elahluke ngokupheleleyo.

Kwiinketho ezizenzekelayo endiziqwalaseleyo, ezi zamva zabonakala kum zisebenza kakhulu. Okokuqala, oku kuseyikhowudi yepython, apho unokubethelela ikhowudi yenkqubo yokuncedisa kwaye ujabulele zonke izibonelelo zolwimi lweprogram. Okwesibini, iprojekthi ye-NiPyAPI iphuhlisa ngokusebenzayo kwaye kwimeko yeengxaki ungabhalela umphuhlisi. Okwesithathu, i-NiPyAPI isesisixhobo esiguquguqukayo ngakumbi sokusebenzisana ne-NiFi ekusombululeni iingxaki ezinzima. Umzekelo, ekumiseleni ukuba ngaba imigca yomyalezo ayinanto ngoku ekuqukuqeleni kunye nokuba kunokwenzeka ukuhlaziya inkqubo yeqela.

Kuko konke. Ndachaza iindlela ze-3 zokuzenzekelayo ukuhanjiswa kokuhamba kwi-NiFi, imigibe enokuthi umphuhlisi ahlangabezane nayo kwaye unikeze ikhowudi yokusebenza yokuhanjiswa ngokuzenzekelayo. Ukuba unomdla kwesi sihloko njengoko ndinjalo - bhala!

umthombo: www.habr.com

Ukuhanjiswa kwe-Automation kwi-Apache NiFi

Ukusebenzisa iMiNiFi

NiFi CLI

NiPyAPI

Yongeza izimvo Phendula impendulo