Hello vose!

Basa nderekutevera - kune kuyerera, kunoratidzwa mumufananidzo uri pamusoro, iyo inoda kuendeswa kune N maseva ane Apache NiFi. Kuyerera bvunzo - faira riri kugadzirwa uye rinotumirwa kune imwe NiFi muenzaniso. Kufambiswa kwedata kunoitika uchishandisa iyo NiFi Saiti kune Saiti protocol.

NiFi Saiti kune Saiti (S2S) inzira yakachengeteka, inogadziriswa nyore yekufambisa data pakati peNiFi zviitiko. Mashandiro eS2S, ona zvinyorwa uye zvakakosha kuti usakanganwa kugadzirisa iyo NiFi muenzaniso kubvumira S2S, ona pano.

Muzviitiko zvatiri kutaura nezve kuendesa data uchishandisa S2S, imwe muenzaniso inonzi mutengi, yechipiri sevha. Mutengi anotumira data, sevha inogamuchira. Nzira mbiri dzekugadzirisa kuendesa data pakati pavo:

kusunda. Kubva kumuenzaniso wemutengi, data inotumirwa uchishandisa Remote Process Group (RPG). Pamuenzaniso weseva, data inogamuchirwa uchishandisa Input Port

kweva. Sevha inogamuchira data ichishandisa RPG, mutengi anotumira achishandisa Output port.

Kuyerera kwekuburitsa kunochengetwa muApache Registry.

Apache NiFi Registry ipurojekiti yeApache NiFi inopa chishandiso chekuyerera kuchengetedza uye vhezheni kutonga. Imwe mhando yeGIT. Ruzivo nezve kuisa, kugadzirisa uye kushanda neregistry inogona kuwanikwa mukati zvinyorwa zvepamutemo. Kuyerera kwekuchengetedza kunosanganiswa kuita boka rekuita uye kuchengetwa mune iyi fomu mune registry. Tichazodzokera kune izvi gare gare muchinyorwa.

Pakutanga, kana N iri nhamba diki, kuyerera kunounzwa uye kugadziridzwa nemaoko munguva inogamuchirwa.

Asi sezvo N inokura, matambudziko anowedzera kuwanda:

zvinotora nguva yakawanda kugadzirisa kuyerera. Iwe unofanirwa kupinda mumaseva ese
Zvikanganiso zvekuvandudza matemplate zvinoitika. Apa vakazvigadzirisa, asi apa vakanganwa
kukanganisa kwevanhu pavanenge vachiita nhamba huru yemabasa akafanana

Zvese izvi zvinotiunza kune chokwadi chekuti isu tinofanirwa kuita otomatiki maitiro. Ndakaedza nzira dzinotevera dzekugadzirisa dambudziko iri:

Shandisa MiNiFi panzvimbo yeNiFi
NiFi CLI
NiPyAPI

Kushandisa MiNiFi

Apache MiNiFy - subproject yeApache NiFi. MiNiFy icompact agent inoshandisa ma processors akafanana neNiFi, ichikubvumidza kuti ugadzire kuyerera kwakafanana nekwaNiFi. Huremu hwemumiriri hunowanikwa, pakati pezvimwe zvinhu, nenyaya yekuti MiNiFy haina graphical interface yekuyerera kuyerera. Kushaikwa kweiyo graphical interface muMiNiFy zvinoreva kuti zvinodikanwa kugadzirisa dambudziko rekuendesa kuyerera kune minifi. Sezvo MiNiFy ichinyatso shandiswa muIOT, kune zvakawanda zvikamu uye maitiro ekuendesa kuyerera kune yekupedzisira minifi zviitiko zvinoda otomatiki. Basa rinozivikanwa, handiti?

Imwe subproject ichabatsira kugadzirisa dambudziko iri - MiNiFi C2 Server. Ichi chigadzirwa chakagadzirirwa kuve chepakati poindi mukugadziriswa kwekuburitsa zvivakwa. Maitiro ekugadzirisa nharaunda - inotsanangurwa mukati ichi chinyorwa Pane ruzivo rwakakwana pamusoro paHabré kugadzirisa dambudziko. MiNiFi, pamwe chete neC2 server, inogadzirisa kugadzirisa kwayo. Iyo chete dhizaini yeiyi nzira ndeyekuti iwe unofanirwa kugadzira matemplate paC2 Server; kuzvipira kuri nyore kune registry hakuna kukwana.

Sarudzo inotsanangurwa munyaya iri pamusoro iri kushanda uye haina kuoma kuita, asi hatifanire kukanganwa zvinotevera:

Minifi haina ese ma processors kubva nifi
Minifi processor shanduro dzinosara kumashure kweNiFi processor shanduro.

Panguva yekunyora, yazvino vhezheni yeNiFi ndeye 1.9.2. Iyo yazvino MiNiFi processor vhezheni ndeye 1.7.0. Ma processor anogona kuwedzerwa kuMiNiFi, asi nekuda kwekusiyana kweshanduro pakati peNiFi neMiNiFi processors, izvi zvinogona kusashanda.

NiFi CLI

Kutonga na tsananguro chishandiso pawebhusaiti yepamutemo, ichi chishandiso chekugadzirisa otomatiki kupindirana pakati peNiFI neNiFi Registry mumunda wekuyerera kuendesa kana process management. Kuti utange, unofanirwa kudhawunirodha chishandiso ichi. kubva pano.

Tanga utility

./bin/cli.sh
           _     ___  _
 Apache   (_)  .' ..](_)   ,
 _ .--.   __  _| |_  __    )
[ `.-. | [  |'-| |-'[  |  /  
|  | | |  | |  | |   | | '    '
[___||__][___][___] [___]',  ,'
                           `'
          CLI v1.9.2

Type 'help' to see a list of available commands, use tab to auto-complete.

Kuti isu titakure kuyerera kunodiwa kubva ku registry, isu tinofanirwa kuziva zviziviso zvebhaketi (bucket identifier) uye kuyerera kwayo (kuyerera identifier). Iyi data inogona kuwanikwa kuburikidza necli kana muNiFi registry web interface. Muwebhu interface inoita seizvi:

Kushandisa CLI izvi zvinoitwa:

#> registry list-buckets -u http://nifi-registry:18080

#   Name             Id                                     Description
-   --------------   ------------------------------------   -----------
1   test_bucket   709d387a-9ce9-4535-8546-3621efe38e96   (empty)

#> registry list-flows -b 709d387a-9ce9-4535-8546-3621efe38e96 -u http://nifi-registry:18080

#   Name           Id                                     Description
-   ------------   ------------------------------------   -----------
1   test_flow   d27af00a-5b47-4910-89cd-9c664cd91e85

Tinotanga kuunza boka rekuita kubva kune registry:

#> nifi pg-import -b 709d387a-9ce9-4535-8546-3621efe38e96 -f d27af00a-5b47-4910-89cd-9c664cd91e85 -fv 1 -u http://nifi:8080

7f522a13-016e-1000-e504-d5b15587f2f3

Chinhu chakakosha ndechekuti chero nifi muenzaniso inogona kutsanangurwa semugamuchiri watinotenderedza boka rekuita.

Boka rekuita rakawedzerwa nema processor akamira, anoda kutangwa

#> nifi pg-start -pgid 7f522a13-016e-1000-e504-d5b15587f2f3 -u http://nifi:8080

Zvakanaka, ma processor atanga. Nekudaro, zvinoenderana nemamiriro ebasa, isu tinoda zviitiko zveNiFi kutumira data kune mamwe mamiriro. Ngatifungei kuti wasarudza iyo Push nzira yekuendesa data kune server. Kuti uronge kuendesa data, unofanirwa kugonesa kuendesa data pane yakawedzera Remote Process Group (RPG), iyo yakatoverengerwa mukuyerera kwedu.

Mune zvinyorwa muCLI uye mamwe masosi, ini handina kuwana nzira yekugonesa kuendesa data. Kana iwe uchiziva kuita izvi, ndapota nyora mumashoko.

Sezvo isu tine bash uye isu takagadzirira kuenda kumagumo, isu tichawana nzira yekubuda! Unogona kushandisa NiFi API kugadzirisa dambudziko iri. Ngatishandisei nzira inotevera, tora ID kubva pamienzaniso iri pamusoro (munyaya yedu ndeye 7f522a13-016e-1000-e504-d5b15587f2f3). Tsanangudzo yeNiFi API nzira pano.

Mumuviri unofanirwa kupfuura JSON, seizvi:

{
    "revision": {
	    "clientId": "value",
	    "version": 0,
	    "lastModifier": "value"
	},
    "state": "value",
    "disconnectedNodeAcknowledged": true
}

Parameters inoda kuzadzwa mukati kuti ishande:
mamiriro - data kutamisa mamiriro. Inowanikwa: TRANSMITTING kugonesa kuendesa data, YAMIRA kudzima
mhando - processor shanduro

vhezheni inozomira ku0 kana yagadzirwa, asi aya ma paramita anogona kuwanikwa uchishandisa nzira

Kune vateveri ve bash scripts, iyi nzira inogona kuita seyakakodzera, asi zvakatiomera kwandiri - bash zvinyorwa hazvisi zvandinoda. Iyo inotevera nzira inonakidza uye yakanakira mumaonero angu.

NiPyAPI

NiPyAPI iraibhurari yePython yekudyidzana neNiFi zviitiko. Peji yezvinyorwa rine mashoko anodiwa pakushanda neraibhurari. Kurumidza kutanga kunotsanangurwa mukati dhizaini pa github.

Yedu script yekuburitsa iyo gadziriso chirongwa muPython. Ngatienderere mberi kune coding.
Isu takagadzira configs kune rimwe basa. Tichada zvinotevera parameters:

nipyapi.config.nifi_config.host = 'http://nifi:8080/nifi-api' #путь до nifi-api инстанса, на котором разворачиваем process group
nipyapi.config.registry_config.host = 'http://nifi-registry:18080/nifi-registry-api' #путь до nifi-registry-api registry
nipyapi.config.registry_name = 'MyBeutifulRegistry' #название registry, как будет называться в инстансе nifi
nipyapi.config.bucket_name = 'BucketName' #название bucket, из которого подтягиваем flow
nipyapi.config.flow_name = 'FlowName' #название flow, которое подтягиваем

Tevere ndichaisa mazita emaitiro eraibhurari iyi, anotsanangurwa pano.

Batanidza registry kune iyo nifi muenzaniso uchishandisa

nipyapi.versioning.create_registry_client

Pane iyi nhanho, iwe unogona zvakare kuwedzera cheki kuti registry yatowedzerwa kumuenzaniso; kune izvi unogona kushandisa nzira.

nipyapi.versioning.list_registry_clients

Isu tinowana bhaketi rekuwedzera kutsvaga kwekuyerera mudengu

nipyapi.versioning.get_registry_bucket

Tichishandisa bhakiti yakawanikwa, tinotarisa kuyerera

nipyapi.versioning.get_flow_in_bucket

Zvadaro, zvakakosha kuti unzwisise kana boka iri rekuita rakatowedzerwa. Boka reProcess rinoiswa maererano nezvirongwa uye mamiriro anogona kuitika kana chikamu chechipiri chaiswa pamusoro pechimwe. Ndakatarisa, izvi zvinogona kuitika :) Kuti tiwane ese akawedzera maitiro mapoka tinoshandisa nzira

nipyapi.canvas.list_all_process_groups

Tinogona kuwedzera kutsvaga, semuenzaniso, nemazita.

Ini handisi kuzotsanangura maitiro ekuvandudza template, ini ndinongotaura kuti kana ma processor akawedzerwa mushanduro nyowani yetemplate, saka hapana matambudziko nekuvapo kwemeseji mumitsetse. Asi kana ma processors akabviswa, ipapo matambudziko anogona kumuka (nifi haakubvumiri kuti ubvise processor kana mutsara wemeseji wakaungana pamberi payo). Kana muchifarira kuti ndakagadzirisa sei dambudziko iri, ndapota ndinyorerei uye tichakurukura nyaya iyi. Mazita pakupera kwechinyorwa. Ngatienderei padanho rekuwedzera boka rekuita.

Pakugadziridza script, ndakasangana neyakasiyana kuti yazvino vhezheni yekuyerera haiwanzo kudhonzwa kumusoro, saka ini ndinokurudzira kutarisa iyi vhezheni kutanga:

nipyapi.versioning.get_latest_flow_ver

Deploy process group:

nipyapi.versioning.deploy_flow_version

Tinotanga ma processors:

nipyapi.canvas.schedule_process_group

Muchivharo pamusoro peCLI zvakanyorwa kuti kuendesa data hakugone kugoneswa muboka rekure kure? Pakuita script, ndakasanganawo nedambudziko iri. Panguva iyoyo, ndakanga ndisingakwanisi kutanga kutumira data ndichishandisa API uye ndakasarudza kunyorera kumugadziri weraibhurari yeNiPyAPI uye kukumbira mazano / rubatsiro. Mugadziri akandipindura, takakurukura dambudziko uye akanyora kuti aida nguva ye "kutarisa chimwe chinhu". Uyezve, mazuva mashoma gare gare, tsamba inosvika umo basa rakanyorwa muPython rinogadzirisa dambudziko rangu rekutanga !!! Panguva iyoyo, shanduro yeNiPyAPI yaiva 0.13.3 uye, zvechokwadi, pakanga pasina chinhu chakadaro. Asi mushanduro 0.14.0, iyo yakabudiswa munguva pfupi yapfuura, basa iri rakanga ratoiswa muraibhurari. Sangana,

nipyapi.canvas.set_remote_process_group_transmission

Saka, tichishandisa raibhurari yeNiPyAPI, takabatanidza registry, takatenderedza kuyerera, uye takatotanga ma processor uye kuendesa data. Ipapo iwe unogona kusanganisa iyo kodhi, wedzera ese marudzi echeki, kutema matanda, uye ndizvo zvese. Asi iyo inyaya yakasiyana zvachose.

Pasarudzo dze otomatiki dzandakafunga, yekupedzisira yairatidzika kwandiri kuti ndiyo inoshanda zvakanyanya. Chekutanga, iyi ichiri python kodhi, mauri iwe unogona kunyudza yekubatsira chirongwa kodhi uye kutora mukana wese mabhenefiti emutauro wechirongwa. Chechipiri, iyo NiPyAPI purojekiti iri kushingaira kusimudzira uye kana paine matambudziko unogona kunyorera kumugadziri. Chechitatu, NiPyAPI ichiri chishandiso chinoshanduka chekudyidzana neNiFi mukugadzirisa matambudziko akaomarara. Semuyenzaniso, mukuona kana mitsara yemeseji yave isina chinhu mukuyerera uye kana boka rekuita rinogona kuvandudzwa.

Ndizvo zvose. Ndakatsanangura nzira nhatu dzekuita otomatiki kuyerera kuyerera muNiFi, misungo ingasangana nemugadziri, uye yakapa kodhi yekushanda yekuzvitakura. Kana iwe uchifarira nyaya iyi sezvandiri - nyora!

Source: www.habr.com

Otomatiki yekuyerera kuburitsa muApache NiFi

Kushandisa MiNiFi

NiFi CLI

NiPyAPI

Voeg kukanzura mhinduro