Hello aliyense!

Ntchitoyi ili motere - pali kuyenda, komwe kukuwonetsedwa pachithunzi pamwambapa, komwe kumayenera kuperekedwa kwa ma seva a N ndi Apache NiFi. Kuyesa kwa Flow - fayilo ikupangidwa ndikutumizidwa ku mtundu wina wa NiFi. Kutumiza kwa data kumachitika pogwiritsa ntchito protocol ya NiFi Site to Site.

NiFi Site to Site (S2S) ndi njira yotetezeka, yosavuta kusinthika yosamutsa deta pakati pa zochitika za NiFi. Momwe S2S imagwirira ntchito, onani zolemba ndipo ndikofunikira kuti musaiwale kukonza chitsanzo cha NiFi kuti mulole S2S, onani apa.

Nthawi zomwe tikukamba za kusamutsa deta pogwiritsa ntchito S2S, nthawi imodzi imatchedwa kasitomala, seva yachiwiri. Wothandizira amatumiza deta, seva imalandira. Njira ziwiri zosinthira kusamutsa kwa data pakati pawo:

1. Kankhani. Kuchokera kwa kasitomala, deta imatumizidwa pogwiritsa ntchito Remote Process Group (RPG). Pachitsanzo cha seva, deta imalandiridwa pogwiritsa ntchito Input Port
2. Kokani. Seva imalandira deta pogwiritsa ntchito RPG, kasitomala amatumiza pogwiritsa ntchito Output port.

Kuthamanga kwa kutuluka kumasungidwa mu Apache Registry.

Apache NiFi Registry ndi gawo la Apache NiFi lomwe limapereka chida chosungira komanso kuwongolera mtundu. Mtundu wa GIT. Zambiri zokhudzana ndi kukhazikitsa, kukonza ndi kugwira ntchito ndi registry zitha kupezeka mu zolemba zovomerezeka. Kuyenda kwa yosungirako kumaphatikizidwa mu gulu la ndondomeko ndikusungidwa mu mawonekedwe awa mu registry. Tidzabweranso ku izi pambuyo pake m'nkhani.

Pachiyambi, pamene N ndi chiwerengero chochepa, kutuluka kumaperekedwa ndikusinthidwa pamanja mu nthawi yovomerezeka.

Koma pamene N ikukula, mavuto amachulukirachulukira:

1. zimatenga nthawi yochulukirapo kuti muwonjezere kuyenda. Muyenera kulowa mu ma seva onse
2. Zolakwika zosintha ma template zimachitika. Apa adasintha, koma apa adayiwala
3. zolakwika za anthu pochita ntchito zambiri zofanana

Zonsezi zikutifikitsa ku mfundo yakuti tiyenera automate ndondomeko. Ndinayesa njira zotsatirazi kuti ndithetse vutoli:

1. Gwiritsani ntchito MiNiFi m'malo mwa NiFi
2. Ndi CLI
3. NiPyAPI

Kugwiritsa ntchito MiNiFi

Apache MiNiFy - gawo la Apache NiFi. MiNiFy ndi othandizira omwe amagwiritsa ntchito mapurosesa omwewo monga NiFi, kukulolani kuti mupange maulendo omwewo monga mu NiFi. Chikhalidwe chopepuka cha wothandizira chimapindula, mwa zina, chifukwa chakuti MiNiFy ilibe mawonekedwe owonetserako kusintha koyenda. Kusowa kwa mawonekedwe owonetserako ku MiNiFy kumatanthauza kuti ndikofunikira kuthetsa vuto lopereka kutuluka kwa minifi. Popeza MiNiFy ikugwiritsidwa ntchito mwakhama ku IOT, pali zigawo zambiri ndipo ndondomeko yoperekera maulendo omaliza a minifi iyenera kukhala yokha. Ntchito yodziwika bwino, sichoncho?

Ntchito ina ikuthandizira kuthetsa vutoli - MiNiFi C2 Server. Chogulitsachi chapangidwa kuti chikhale chigawo chapakati pamapangidwe oyambira. Momwe mungasinthire chilengedwe - zomwe zafotokozedwa mu nkhaniyi Pali zambiri zokwanira za Habré kuti athetse vutoli. MiNiFi, molumikizana ndi seva ya C2, imangosintha makonzedwe ake. Chotsalira chokha cha njirayi ndikuti muyenera kupanga ma templates pa C2 Server; kudzipereka kosavuta ku registry sikokwanira.

Njira yomwe yafotokozedwa m'nkhaniyi ikugwira ntchito ndipo si yovuta kuigwiritsa ntchito, koma tisaiwale zotsatirazi:

1. Minifi ilibe mapurosesa onse ochokera ku nifi
2. Mitundu ya purosesa ya Minifi imatsalira kumbuyo kwa purosesa ya NiFi.

Panthawi yolemba, mtundu waposachedwa wa NiFi ndi 1.9.2. Mtundu waposachedwa wa purosesa wa MiNiFi ndi 1.7.0. Mapurosesa akhoza kuwonjezeredwa ku MiNiFi, koma chifukwa cha kusiyana kwa mtundu pakati pa NiFi ndi MiNiFi processors, izi sizingagwire ntchito.

Ndi CLI

Kuweruza kufotokoza chida patsamba lovomerezeka, ichi ndi chida chosinthira kuyanjana pakati pa NiFI ndi NiFi Registry pagawo la kutumiza kapena kuwongolera njira. Kuti muyambe, muyenera kukopera chida ichi. kuchokera pano.

Yambitsani zothandiza

./bin/cli.sh
           _     ___  _
 Apache   (_)  .' ..](_)   ,
 _ .--.   __  _| |_  __    )
[ `.-. | [  |'-| |-'[  |  /  
|  | | |  | |  | |   | | '    '
[___||__][___][___] [___]',  ,'
                           `'
          CLI v1.9.2

Type 'help' to see a list of available commands, use tab to auto-complete.

Kuti tithe kutsitsa zomwe zikufunika kuchokera ku registry, tiyenera kudziwa zizindikiritso za ndowa (chizindikiritso cha ndowa) ​​ndikuyenda komweko (chizindikiritso chotuluka). Izi zitha kupezeka kudzera pa cli kapena pa intaneti ya NiFi registry. Mu mawonekedwe a intaneti zikuwoneka motere:

Pogwiritsa ntchito CLI izi zachitika:

#> registry list-buckets -u http://nifi-registry:18080

#   Name             Id                                     Description
-   --------------   ------------------------------------   -----------
1   test_bucket   709d387a-9ce9-4535-8546-3621efe38e96   (empty)

#> registry list-flows -b 709d387a-9ce9-4535-8546-3621efe38e96 -u http://nifi-registry:18080

#   Name           Id                                     Description
-   ------------   ------------------------------------   -----------
1   test_flow   d27af00a-5b47-4910-89cd-9c664cd91e85

Timayamba kuitanitsa gulu lazinthu kuchokera ku registry:

#> nifi pg-import -b 709d387a-9ce9-4535-8546-3621efe38e96 -f d27af00a-5b47-4910-89cd-9c664cd91e85 -fv 1 -u http://nifi:8080

7f522a13-016e-1000-e504-d5b15587f2f3

Mfundo yofunika ndi yakuti chitsanzo chilichonse cha nifi chikhoza kufotokozedwa ngati wolandira omwe timayendetsa gululo.

Gulu la ndondomeko lowonjezeredwa ndi mapurosesa oyimitsidwa, ayenera kuyambitsidwa

#> nifi pg-start -pgid 7f522a13-016e-1000-e504-d5b15587f2f3 -u http://nifi:8080

Chabwino, mapurosesa ayamba. Komabe, molingana ndi ntchitoyo, timafunikira zochitika za NiFi kuti titumize deta kuzinthu zina. Tiyerekeze kuti mwasankha Push njira kusamutsa deta ku seva. Kuti mukonzekere kusamutsa deta, muyenera kuloleza kusamutsa deta pagulu lowonjezera la Remote Process Group (RPG), lomwe laphatikizidwa kale mumayendedwe athu.

M'zolembedwa mu CLI ndi magwero ena, sindinapeze njira yothandizira kusamutsa deta. Ngati mukudziwa momwe mungachitire izi, chonde lembani mu ndemanga.

Popeza tili ndi bash ndipo takonzeka kupita kumapeto, tipeza njira yotulukira! Mutha kugwiritsa ntchito NiFi API kuthetsa vutoli. Tiyeni tigwiritse ntchito njira zotsatirazi, tengani ID kuchokera ku zitsanzo pamwambapa (kwa ife ndi 7f522a13-016e-1000-e504-d5b15587f2f3). Kufotokozera kwa njira za NiFi API apa.

M'thupi muyenera kudutsa JSON, motere:

{
    "revision": {
	    "clientId": "value",
	    "version": 0,
	    "lastModifier": "value"
	},
    "state": "value",
    "disconnectedNodeAcknowledged": true
}

Ma Parameters omwe amafunika kudzazidwa kuti agwire ntchito:
Boma - kusamutsa deta. Zilipo: TRANSMITTING kuti athe kusamutsa deta, STOPED kuti azimitsa
Baibulo - purosesa mtundu

mtundu udzasintha kukhala 0 ukapangidwa, koma magawowa atha kupezeka pogwiritsa ntchito njirayo

Kwa mafani a bash scripts, njirayi ingawoneke ngati yoyenera, koma ndizovuta kwa ine - zolemba za bash sindimakonda. Njira yotsatira ndiyosangalatsa komanso yabwino m'malingaliro anga.

NiPyAPI

NiPyAPI ndi laibulale ya Python yolumikizana ndi zochitika za NiFi. Tsamba lazolemba lili ndi chidziwitso chofunikira chogwirira ntchito ndi laibulale. Chiyambi chachangu chikufotokozedwa mu polojekiti pa github.

Zolemba zathu zosinthira kasinthidwe ndi pulogalamu ya Python. Tiyeni tipitirire ku coding.
Timapanga ma configs kuti tigwire ntchito zina. Tidzafunika ma parameter awa:

nipyapi.config.nifi_config.host = 'http://nifi:8080/nifi-api' #путь до nifi-api инстанса, на котором разворачиваем process group
nipyapi.config.registry_config.host = 'http://nifi-registry:18080/nifi-registry-api' #путь до nifi-registry-api registry
nipyapi.config.registry_name = 'MyBeutifulRegistry' #название registry, как будет называться в инстансе nifi
nipyapi.config.bucket_name = 'BucketName' #название bucket, из которого подтягиваем flow
nipyapi.config.flow_name = 'FlowName' #название flow, которое подтягиваем

Kenako ndiyika mayina a njira za laibulale iyi, zomwe zafotokozedwa apa.

Lumikizani registry ku chitsanzo cha nifi pogwiritsa ntchito

nipyapi.versioning.create_registry_client

Pa sitepe iyi, mukhoza kuwonjezera cheke kuti zolembera zawonjezedwa kale pa chitsanzo; pakuti izi mungagwiritse ntchito njira.

nipyapi.versioning.list_registry_clients

Timapeza chidebe kuti tifufuzenso zotuluka mudengu

nipyapi.versioning.get_registry_bucket

Pogwiritsa ntchito chidebe chopezeka, timayang'ana kutuluka

nipyapi.versioning.get_flow_in_bucket

Chotsatira, ndikofunika kumvetsetsa ngati gulu la ndondomekoyi lawonjezeredwa kale. Gulu la Njira limayikidwa molingana ndi makonzedwe ndipo vuto likhoza kuchitika pamene gawo lachiwiri likuyikidwa pamwamba pa chimodzi. Ndinayang'ana, izi zikhoza kuchitika :) Kuti tipeze magulu onse owonjezera ndondomeko timagwiritsa ntchito njira

nipyapi.canvas.list_all_process_groups

Titha kusaka mopitilira, mwachitsanzo, ndi dzina.

Sindidzalongosola ndondomeko yokonzanso template, ndingonena kuti ngati mapurosesa awonjezeredwa mumtundu watsopano wa template, ndiye kuti palibe mavuto ndi kupezeka kwa mauthenga pamizere. Koma ngati mapurosesa achotsedwa, ndiye kuti mavuto angabwere (nifi sakulolani kuchotsa purosesa ngati mzere wa uthenga wasonkhanitsa patsogolo pake). Ngati muli ndi chidwi ndi momwe ndinathetsera vutoli, chonde lembani kwa ine ndipo tidzakambirana nkhaniyi. Othandizira kumapeto kwa nkhaniyi. Tiyeni tipitirire ku sitepe yowonjezera gulu la ndondomeko.

Ndikakonza script, ndidapeza chodabwitsa kuti mawonekedwe aposachedwa samakokedwa nthawi zonse, ndiye ndikupangira kuti muyang'ane kaye mtundu uwu:

nipyapi.versioning.get_latest_flow_ver

Deploy process group:

nipyapi.versioning.deploy_flow_version

Timayamba ma processor:

nipyapi.canvas.schedule_process_group

Mu chipika chokhudza CLI zidalembedwa kuti kusamutsa deta sikungothandizidwa pagulu lakutali? Pokhazikitsa script, ndinakumananso ndi vuto ili. Panthawiyo, sindinathe kuyambitsa kusamutsa deta pogwiritsa ntchito API ndipo ndinaganiza zolembera wopanga laibulale ya NiPyAPI ndikupempha malangizo / chithandizo. Wopanga mapulogalamuyo adandiyankha, tidakambirana za vutoli ndipo adalemba kuti amafunikira nthawi "yoyang'ana china chake". Ndiyeno, patatha masiku angapo, kalata imabwera momwe ntchito inalembedwa mu Python yomwe imathetsa vuto langa loyambitsa !!! Panthawiyo, NiPyAPI version inali 0.13.3 ndipo, ndithudi, panalibe chotere. Koma mu Baibulo 0.14.0, amene anamasulidwa posachedwapa, ntchito imeneyi anali kale m'gulu laibulale. Kumanani,

nipyapi.canvas.set_remote_process_group_transmission

Chifukwa chake, pogwiritsa ntchito laibulale ya NiPyAPI, tidalumikiza zolembera, kutulutsa kutuluka, komanso kuyambitsa mapurosesa ndi kusamutsa deta. Ndiye inu mukhoza kupesa kachidindo, kuwonjezera mitundu yonse ya macheke, kudula mitengo, ndipo ndizo zonse. Koma imeneyo ndi nkhani yosiyana kotheratu.

Mwa njira zopangira zokha zomwe ndidaziganizira, yomaliza idawoneka kwa ine yothandiza kwambiri. Choyamba, iyi ikadali kachidindo ka python, momwe mungathe kuyikamo kachidindo ka pulogalamu yothandizira ndikugwiritsa ntchito mwayi wonse wa chilankhulo cha pulogalamuyo. Kachiwiri, pulojekiti ya NiPyAPI ikukula mwachangu ndipo pakagwa mavuto mutha kulembera woyambitsa. Chachitatu, NiPyAPI akadali chida chosinthika kwambiri cholumikizirana ndi NiFi pakuthana ndi zovuta. Mwachitsanzo, pozindikira ngati mizere ya uthengawo ilibe kanthu komanso ngati gululo litha kusinthidwa.

Ndizomwezo. Ndidafotokoza njira za 3 zopangira ma mayendedwe oyenda mu NiFi, misampha yomwe wopanga angakumane nayo, ndikupereka kachidindo kogwirira ntchito kuti azitumiza zokha. Ngati muli ndi chidwi ndi mutuwu ngati ine - lembani!

Source: www.habr.com