Sawubona wonke umuntu!

Umsebenzi umi kanje - kukhona ukugeleza, okwethulwa esithombeni esingenhla, okudingeka kudluliselwe kumaseva we-N nge I-Apache NiFi. Ukuhlolwa kokugeleza - ifayela liyakhiqizwa futhi lithunyelwa kwesinye isenzakalo se-NiFi. Ukudluliswa kwedatha kwenzeka kusetshenziswa iphrothokholi ye-NiFi Site to Site.

I-NiFi Site to Site (S2S) iyindlela evikelekile, elungiseka kalula yokudlulisa idatha phakathi kwezimo ze-NiFi. Isebenza kanjani i-S2S, bheka imibhalo futhi kubalulekile ukuthi ungakhohlwa ukulungisa isibonelo se-NiFi ukuze uvumele i-S2S, bheka lapha.

Ezimeni lapho sikhuluma ngokudluliswa kwedatha kusetshenziswa i-S2S, isibonelo esisodwa sibizwa ngokuthi iklayenti, iseva yesibili. Iklayenti lithumela idatha, iseva ithola. Izindlela ezimbili zokumisa ukudluliswa kwedatha phakathi kwazo:

push. Kusukela kusibonelo seklayenti, idatha ithunyelwa kusetshenziswa i-Remote Process Group (RPG). Esikhathini seseva, idatha itholwa kusetshenziswa Imbobo Yokufaka

Donsa. Iseva ithola idatha isebenzisa i-RPG, iklayenti lithumela lisebenzisa Imbobo ye-Output.

Ukugeleza kokukhishwa kugcinwa ku-Apache Registry.

I-Apache NiFi Registry ingaphansi kwe-Apache NiFi ehlinzeka ngethuluzi lokugcinwa kokugeleza nokulawula inguqulo. Uhlobo lwe-GIT. Ulwazi mayelana nokufaka, ukumisa nokusebenza nerejista ingatholakala ku imibhalo esemthethweni. Ukugeleza kokugcina kuhlanganiswa kube yiqembu lenqubo futhi kugcinwe kuleli fomu kurejista. Sizobuyela kulokhu kamuva esihlokweni.

Ekuqaleni, lapho u-N eyinombolo encane, ukugeleza kulethwa futhi kubuyekezwa mathupha ngesikhathi esamukelekile.

Kodwa njengoba i-N ikhula, izinkinga ziba ziningi:

kuthatha isikhathi esengeziwe ukubuyekeza ukugeleza. Udinga ukungena kuwo wonke amaseva
Amaphutha okubuyekeza isifanekiso ayenzeka. Lapha bakubuyekeze, kodwa lapha bakhohlwa
amaphutha omuntu lapho enza inani elikhulu lemisebenzi efanayo

Konke lokhu kusiletha eqinisweni lokuthi sidinga ukwenza inqubo ngokuzenzakalelayo. Ngizame izindlela ezilandelayo zokuxazulula le nkinga:

Sebenzisa i-MiNiFi esikhundleni se-NiFi
NiFi CLI
I-NiPyAPI

Ukusebenzisa i-MiNiFi

I-Apache MiNiFy - iphrojekthi engaphansi ye-Apache NiFi. I-MiNiFy iyi-ejenti ehlangene esebenzisa amaphrosesa afanayo njenge-NiFi, ekuvumela ukuthi udale ukugeleza okufanayo njengaku-NiFi. Imvelo engasindi ye-ejenti ifinyelelwa, phakathi kwezinye izinto, yiqiniso lokuthi i-MiNiFy ayinayo isikhombimsebenzisi esibonakalayo sokucushwa kokugeleza. Ukuntuleka kwe-graphical interface ku-MiNiFy kusho ukuthi kuyadingeka ukuxazulula inkinga yokuletha ukugeleza ku-minifi. Njengoba i-MiNiFy isetshenziswa ngenkuthalo ku-IOT, kunezingxenye eziningi futhi inqubo yokuletha ukugeleza kuzimo zokugcina ze-minifi idinga ukuzenzekelayo. Umsebenzi ojwayelekile, akunjalo?

Enye i-subproject izosiza ukuxazulula le nkinga - Iseva ye-MiNiFi C2. Lo mkhiqizo uhloselwe ukuba indawo emaphakathi ekwakhiweni kokukhishwa kokucushwa. Indlela yokumisa imvelo - echazwe ku lesi sihloko Kunolwazi olwanele ngo-Habré lokuxazulula inkinga. I-MiNiFi, ngokubambisana neseva ye-C2, ibuyekeza ngokuzenzakalelayo ukucushwa kwayo. Okuwukuphela kwe-drawback yale ndlela ukuthi kufanele udale izifanekiso ku-C2 Server; ukuzibophezela okulula ekubhaliseni akwanele.

Inketho echazwe esihlokweni esingenhla iyasebenza futhi akunzima ukuyisebenzisa, kodwa akumelwe sikhohlwe okulandelayo:

I-Minifi ayinawo wonke ama-processor avela ku-nifi
Izinguqulo ze-Minifi processor zisala ngemuva kwezinguqulo ze-NiFi processor.

Ngesikhathi sokubhala, inguqulo yakamuva ye-NiFi ingu-1.9.2. Inguqulo yakamuva ye-MiNiFi processor ingu-1.7.0. Amaprosesa angangezwa ku-MiNiFi, kodwa ngenxa yokungafani kwenguqulo phakathi kwe-NiFi ne-MiNiFi processors, lokhu kungase kungasebenzi.

NiFi CLI

Ukwahlulela incazelo ithuluzi kuwebhusayithi esemthethweni, leli ithuluzi lokuzenzakalela ukusebenzisana phakathi kwe-NiFI ne-NiFi Registry emkhakheni wokulethwa kokugeleza noma ukuphathwa kwenqubo. Ukuze uqalise, udinga ukulanda leli thuluzi. kusuka lapha.

Yethula insiza

./bin/cli.sh
           _     ___  _
 Apache   (_)  .' ..](_)   ,
 _ .--.   __  _| |_  __    )
[ `.-. | [  |'-| |-'[  |  /  
|  | | |  | |  | |   | | '    '
[___||__][___][___] [___]',  ,'
                           `'
          CLI v1.9.2

Type 'help' to see a list of available commands, use tab to auto-complete.

Ukuze silayishe ukugeleza okudingekayo okuvela kurejista, sidinga ukwazi izihlonzi zebhakede (isihlonzi sebhakede) kanye nokugeleza ngokwako (isihlonzi sokugeleza). Le datha ingatholwa nge-cli noma ku-interface yewebhu yokubhalisa ye-NiFi. Ku-interface yewebhu ibonakala kanje:

Ukusebenzisa i-CLI lokhu kwenziwa:

#> registry list-buckets -u http://nifi-registry:18080

#   Name             Id                                     Description
-   --------------   ------------------------------------   -----------
1   test_bucket   709d387a-9ce9-4535-8546-3621efe38e96   (empty)

#> registry list-flows -b 709d387a-9ce9-4535-8546-3621efe38e96 -u http://nifi-registry:18080

#   Name           Id                                     Description
-   ------------   ------------------------------------   -----------
1   test_flow   d27af00a-5b47-4910-89cd-9c664cd91e85

Siqala ukungenisa iqembu lenqubo kurejista:

#> nifi pg-import -b 709d387a-9ce9-4535-8546-3621efe38e96 -f d27af00a-5b47-4910-89cd-9c664cd91e85 -fv 1 -u http://nifi:8080

7f522a13-016e-1000-e504-d5b15587f2f3

Iphuzu elibalulekile ukuthi noma isiphi isibonelo se-nifi singacaciswa njengomsingathi lapho sikhiphela khona iqembu lenqubo.

Iqembu lenqubo lengezwe namaphrosesa amisiwe, adinga ukuqaliswa

#> nifi pg-start -pgid 7f522a13-016e-1000-e504-d5b15587f2f3 -u http://nifi:8080

Kuhle, ama-processor aseqalile. Kodwa-ke, ngokuya ngemibandela yomsebenzi, sidinga izimo ze-NiFi ukuze sithumele idatha kwezinye izimo. Ake sicabange ukuthi ukhethe indlela ye-Push yokudlulisa idatha kuseva. Ukuze uhlele ukudluliswa kwedatha, udinga ukunika amandla ukudluliswa kwedatha ku-Remote Process Group (RPG) eyengeziwe, eseyifakiwe kakade ekugelezeni kwethu.

Embhalweni ku-CLI nakweminye imithombo, angizange ngithole indlela yokuvumela ukudluliswa kwedatha. Uma wazi ukuthi ungakwenza kanjani lokhu, sicela ubhale kumazwana.

Njengoba sine-bash futhi sesilungele ukuya ekugcineni, sizothola indlela yokuphuma! Ungasebenzisa i-NiFi API ukuxazulula le nkinga. Masisebenzise indlela elandelayo, thatha i-ID ezibonelweni ezingenhla (esimweni sethu yi-7f522a13-016e-1000-e504-d5b15587f2f3). Incazelo yezindlela ze-NiFi API lapha.

Emzimbeni udinga ukudlula i-JSON, kanje:

{
    "revision": {
	    "clientId": "value",
	    "version": 0,
	    "lastModifier": "value"
	},
    "state": "value",
    "disconnectedNodeAcknowledged": true
}

Amapharamitha adinga ukugcwaliswa ukuze asebenze:
isimo — isimo sokudlulisa idatha. Iyatholakala: TRANSMITTING ukuze unike amandla ukudluliswa kwedatha, STOPED ukuze ukhubaze
Version - Inguqulo yeprosesa

inguqulo izozenzakalela ibe ngu-0 uma idaliwe, kodwa le mingcele ingatholwa kusetshenziswa indlela

Kubalandeli be-bash scripts, le ndlela ingase ibonakale ifanelekile, kodwa inzima kancane kimi - imibhalo ye-bash ayiyona intandokazi yami. Indlela elandelayo iyathakazelisa futhi ilula ngokubona kwami.

I-NiPyAPI

I-NiPyAPI iwumtapo wezincwadi wePython wokuxhumana nezimo ze-NiFi. Ikhasi lamadokhumenti iqukethe ulwazi oludingekayo lokusebenza nomtapo wolwazi. Ukuqala okusheshayo kuchazwe ku iphrojekthi ku github.

Umbhalo wethu wokukhipha ukucushwa wuhlelo olukwiPython. Asiqhubekele ekubhaleni amakhodi.
Setha izilungiselelo ukuze siqhubeke nomsebenzi. Sizodinga amapharamitha alandelayo:

nipyapi.config.nifi_config.host = 'http://nifi:8080/nifi-api' #путь до nifi-api инстанса, на котором разворачиваем process group
nipyapi.config.registry_config.host = 'http://nifi-registry:18080/nifi-registry-api' #путь до nifi-registry-api registry
nipyapi.config.registry_name = 'MyBeutifulRegistry' #название registry, как будет называться в инстансе nifi
nipyapi.config.bucket_name = 'BucketName' #название bucket, из которого подтягиваем flow
nipyapi.config.flow_name = 'FlowName' #название flow, которое подтягиваем

Okulandelayo ngizofaka amagama ezindlela zalo mtapo wolwazi, ezichazwe lapha.

Xhuma ukubhalisa kusibonelo se-nifi usebenzisa

nipyapi.versioning.create_registry_client

Kulesi sinyathelo, ungakwazi futhi ukwengeza isheke ukuthi ukubhalisa sekungeziwe esibonelweni; kulokhu ungasebenzisa indlela.

nipyapi.versioning.list_registry_clients

Sithola ibhakede ukuze siseshwe ngokuqhubekayo kubhasikidi

nipyapi.versioning.get_registry_bucket

Sisebenzisa ibhakede elitholiwe, sibheka ukugeleza

nipyapi.versioning.get_flow_in_bucket

Okulandelayo, kubalulekile ukuqonda ukuthi leli qembu lenqubo selingeziwe yini. Iqembu leNqubo libekwa ngokuvumelana nezixhumanisi futhi isimo singase sivele lapho ingxenye yesibili ibekwe phezulu phezu kweyodwa. Ngihlolile, lokhu kungenzeka :) Ukuze sithole wonke amaqembu enqubo engeziwe sisebenzisa indlela

nipyapi.canvas.list_all_process_groups

Singaqhubeka nokusesha, isibonelo, ngegama.

Ngeke ngichaze inqubo yokubuyekeza ithempulethi, ngizosho kuphela ukuthi uma amaphrosesa engeziwe enguqulweni entsha yesifanekiso, khona-ke azikho izinkinga ngokuba khona kwemilayezo emigqeni. Kodwa uma amaphrosesa asusiwe, khona-ke izinkinga zingase ziphakame (i-nifi ayikuvumeli ukuthi ususe iphrosesa uma umugqa womlayezo uqoqwe phambi kwayo). Uma unentshisekelo yokuthi ngiyixazulule kanjani le nkinga, ngicela ungibhalele futhi sizoxoxa ngalolu daba. Oxhumana nabo ekupheleni kwesihloko. Masiqhubekele esinyathelweni sokwengeza iqembu lenqubo.

Lapho ngisusa iphutha kuskripthi, ngithole okukhethekile ukuthi inguqulo yakamuva yokugeleza ayidonswa ngaso sonke isikhathi, ngakho-ke ngincoma ukuthi uhlole le nguqulo kuqala:

nipyapi.versioning.get_latest_flow_ver

Sebenzisa iqembu lenqubo:

nipyapi.versioning.deploy_flow_version

Siqala ama-processor:

nipyapi.canvas.schedule_process_group

Ku-block mayelana ne-CLI kwakubhalwe ukuthi ukudluliswa kwedatha akuvunyelwe ngokuzenzakalelayo eqenjini lenqubo ekude? Lapho ngisebenzisa iskripthi, ngihlangabezane nale nkinga nami. Ngaleso sikhathi, angikwazanga ukuqala ukudluliswa kwedatha ngisebenzisa i-API futhi nganquma ukubhalela umthuthukisi welabhulali ye-NiPyAPI futhi ngicele iseluleko/usizo. Umthuthukisi wangiphendula, saxoxa ngenkinga futhi wabhala ukuthi udinga isikhathi "sokuhlola okuthile". Bese kuthi, ezinsukwini ezimbalwa kamuva, kufike incwadi lapho kubhalwe khona umsebenzi ngePython oxazulula inkinga yami yokwethulwa !!! Ngaleso sikhathi, inguqulo ye-NiPyAPI yayingu-0.13.3 futhi, yiqiniso, kwakungekho lutho olunjalo. Kodwa enguqulweni engu-0.14.0, eyakhululwa kamuva nje, lo msebenzi wawusuvele ufakiwe emtatsheni wezincwadi. Hlanganani,

nipyapi.canvas.set_remote_process_group_transmission

Ngakho-ke, sisebenzisa umtapo wezincwadi we-NiPyAPI, sixhume indawo yokubhalisa, sakhipha ukugeleza, futhi saqala namaphrosesa nokudluliswa kwedatha. Ngemuva kwalokho ungahlanganisa ikhodi, wengeze zonke izinhlobo zokuhlola, ukugawula, futhi yilokho kuphela. Kodwa leyo indaba ehluke ngokuphelele.

Kuzinketho zokuzenzakalela engizicabangele, ezokugcina zibonakale kimina zisebenza kahle kakhulu. Okokuqala, lena kuseyikhodi ye-python, ongashumeka kuyo ikhodi yokusiza futhi usebenzise zonke izinzuzo zolimi lokuhlela. Okwesibili, iphrojekthi ye-NiPyAPI ithuthuka ngenkuthalo futhi uma kunezinkinga ungabhalela unjiniyela. Okwesithathu, i-NiPyAPI iseyithuluzi elivumelana nezimo kakhulu lokuxhumana ne-NiFi ekuxazululeni izinkinga eziyinkimbinkimbi. Isibonelo, ekunqumeni ukuthi ingabe imigqa yemilayezo manje ayinalutho yini ekuhambeni nokuthi ingabe iqembu lenqubo lingabuyekezwa yini.

Yilokho kuphela. Ngichaze izindlela ezi-3 zokulethwa kokugeleza okuzenzakalelayo ku-NiFi, izingibe umthuthukisi angase ahlangabezane nazo, futhi nginikeze ikhodi yokusebenza yokulethwa okuzenzakalelayo. Uma unentshisekelo kulesi sihloko njengami - bhala!

Source: www.habr.com

Ukuzenzakalela kokulethwa kokugeleza ku-Apache NiFi

Ukusebenzisa i-MiNiFi

NiFi CLI

I-NiPyAPI

Engeza amazwana Отменить ответ