Hello kowa da kowa!
Aikin shine kamar haka - akwai kwarara, wanda aka gabatar a hoton da ke sama, wanda ke buƙatar fitar da shi zuwa sabobin N tare da . Gwajin yawo - ana ƙirƙirar fayil kuma ana aika zuwa wani misalin NiFi. Canja wurin bayanai yana faruwa ta amfani da Shafin NiFi zuwa ka'idar Yanar Gizo.
Shafin NiFi zuwa Yanar Gizo (S2S) amintacciyar hanya ce mai sauƙin daidaitawa don canja wurin bayanai tsakanin al'amuran NiFi. Yadda S2S ke aiki, duba kuma yana da mahimmanci kar a manta don saita misalin NiFi don ba da damar S2S, duba .
A lokuta da muke magana game da canja wurin bayanai ta amfani da S2S, misali ɗaya ana kiransa abokin ciniki, uwar garken na biyu. Abokin ciniki yana aika bayanai, uwar garken yana karɓa. Hanyoyi biyu don daidaita musayar bayanai tsakanin su:
- tura. Daga misalin abokin ciniki, ana aika bayanai ta amfani da Rukunin Tsari Mai Nisa (RPG). A misalin uwar garken, ana karɓar bayanai ta amfani da Port Input
- ja. Sabar tana karɓar bayanai ta amfani da RPG, abokin ciniki yana aika ta amfani da tashar fitarwa.
Ana adana kwarara don mirginewa a cikin Registry Apache.
Apache NiFi Registry jigo ne na Apache NiFi wanda ke ba da kayan aiki don ma'ajiyar kwarara da sarrafa sigar. Wani irin GIT. Ana iya samun bayanai game da shigarwa, daidaitawa da aiki tare da yin rajista a ciki . Ana haɗa kwarara don ajiya cikin ƙungiyar tsari kuma ana adana su a cikin wannan fom a cikin wurin yin rajista. Za mu dawo kan wannan a gaba a cikin labarin.
A farkon, lokacin da N ƙaramin lamba ne, ana isar da kwarara kuma ana sabunta shi da hannu a cikin lokacin karɓuwa.
Amma yayin da N ke girma, matsalolin suna da yawa:
- yana ɗaukar ƙarin lokaci don sabunta kwararar. Kuna buƙatar shiga cikin duk sabobin
- Kurakurai na sabunta samfuri suna faruwa. Anan suka sabunta shi, amma a nan sun manta
- kurakurai na ɗan adam lokacin yin babban adadin ayyuka iri ɗaya
Duk wannan ya kawo mu ga gaskiyar cewa muna buƙatar sarrafa tsarin. Na gwada hanyoyin da za a magance wannan matsalar:
- Yi amfani da MiNiFi maimakon NiFi
- Farashin CLI
- NiPyAPI
Amfani da MiNiFi
- aikin Apache NiFi. MiNiFy ƙaramin wakili ne wanda ke amfani da na'urori masu sarrafawa iri ɗaya kamar NiFi, yana ba ku damar ƙirƙirar kwarara iri ɗaya kamar na NiFi. Ana samun nauyin nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in nau'in) yana samuwa,a tsakanin sauran abubuwa,ta hanyar gaskiyar cewa MiNiFy ba shi da ƙirar hoto don daidaitawar kwarara. Rashin haɗin hoto a cikin MiNiFy yana nufin cewa ya zama dole don magance matsalar isar da kwarara zuwa minifi. Tunda ana amfani da MiNiFy sosai a cikin IOT, akwai abubuwa da yawa kuma ana buƙatar aiwatar da aikin isar da kwarara zuwa ƙaramar ƙarami ta ƙarshe. Aikin da aka saba, daidai?
Wani aikin zai taimaka wajen magance wannan matsala - MiNiFi C2 Server. An yi nufin wannan samfurin ya zama wurin tsakiya a cikin tsarin ƙirar ƙira. Yadda za a daidaita muhalli - aka bayyana a ciki Akwai isassun bayanai akan Habré don magance matsalar. MiNiFi, tare da haɗin gwiwar uwar garken C2, tana sabunta tsarinta ta atomatik. Babban koma baya na wannan hanyar shine dole ne ka ƙirƙiri samfuri akan C2 Server; ƙaddamarwa mai sauƙi ga yin rajista bai isa ba.
Zaɓin da aka bayyana a cikin labarin da ke sama yana aiki kuma ba wuya a aiwatar da shi ba, amma kada mu manta da waɗannan abubuwa:
- Minifi ba shi da duk na'urori masu sarrafawa daga nifi
- Nau'in sarrafa minifi na baya bayan nau'ikan processor na NiFi.
A lokacin rubutawa, sabon sigar NiFi shine 1.9.2. Sabuwar sigar mai sarrafa ta MiNiFi ita ce 1.7.0. Ana iya ƙara na'urori masu sarrafawa zuwa MiNiFi, amma saboda rarrabuwar kawuna tsakanin na'urori masu sarrafa NiFi da na MiNiFi, wannan bazai yi aiki ba.
Farashin CLI
Kuna hukunta by kayan aiki akan gidan yanar gizon hukuma, wannan kayan aiki ne don sarrafa ma'amala tsakanin NiFI da NiFi Registry a fagen isar da kwarara ko sarrafa tsari. Don farawa, kuna buƙatar zazzage wannan kayan aikin. .
Kaddamar da mai amfani
./bin/cli.sh
_ ___ _
Apache (_) .' ..](_) ,
_ .--. __ _| |_ __ )
[ `.-. | [ |'-| |-'[ | /
| | | | | | | | | | ' '
[___||__][___][___] [___]', ,'
`'
CLI v1.9.2
Type 'help' to see a list of available commands, use tab to auto-complete.
Domin mu loda kwararan da ake buƙata daga wurin yin rajista, muna buƙatar sanin abubuwan gano guga (mai gano guga) da kwarara da kanta (mai gano kwarara). Ana iya samun wannan bayanan ta hanyar cli ko a cikin mahaɗin yanar gizo na rajista na NiFi. A cikin mahallin gidan yanar gizon yana kama da haka:
Amfani da CLI ana yin haka:
#> registry list-buckets -u http://nifi-registry:18080
# Name Id Description
- -------------- ------------------------------------ -----------
1 test_bucket 709d387a-9ce9-4535-8546-3621efe38e96 (empty)
#> registry list-flows -b 709d387a-9ce9-4535-8546-3621efe38e96 -u http://nifi-registry:18080
# Name Id Description
- ------------ ------------------------------------ -----------
1 test_flow d27af00a-5b47-4910-89cd-9c664cd91e85
Muna fara shigo da rukunin tsari daga rajista:
#> nifi pg-import -b 709d387a-9ce9-4535-8546-3621efe38e96 -f d27af00a-5b47-4910-89cd-9c664cd91e85 -fv 1 -u http://nifi:8080
7f522a13-016e-1000-e504-d5b15587f2f3
Wani muhimmin batu shi ne cewa kowane misali na nifi za a iya ayyana shi azaman mai watsa shiri wanda muke mirgine rukunin tsari zuwa gare shi.
Ƙungiyoyin tsari waɗanda aka ƙara tare da masu sarrafawa da aka dakatar, suna buƙatar farawa
#> nifi pg-start -pgid 7f522a13-016e-1000-e504-d5b15587f2f3 -u http://nifi:8080
Mai girma, masu sarrafawa sun fara. Koyaya, bisa ga sharuɗɗan aikin, muna buƙatar misalin NiFi don aika bayanai zuwa wasu lokuta. Bari mu ɗauka cewa kun zaɓi hanyar turawa don canja wurin bayanai zuwa uwar garken. Domin tsara canja wurin bayanai, kuna buƙatar kunna canja wurin bayanai akan ƙarar Rukunin Tsari Mai Nisa (RPG), wanda an riga an haɗa shi cikin kwararar mu.
A cikin takaddun da ke cikin CLI da sauran hanyoyin, ban sami hanyar ba da damar canja wurin bayanai ba. Idan kun san yadda ake yin wannan, da fatan za a rubuta a cikin sharhi.
Tunda muna da bash kuma muna shirye mu tafi zuwa ƙarshe, zamu sami mafita! Kuna iya amfani da NiFi API don magance wannan matsalar. Bari mu yi amfani da hanya mai zuwa, ɗauki ID daga misalan da ke sama (a cikin yanayinmu shine 7f522a13-016e-1000-e504-d5b15587f2f3). Bayanin hanyoyin NiFi API .
A cikin jiki kuna buƙatar wuce JSON, kamar haka:
{
"revision": {
"clientId": "value",
"version": 0,
"lastModifier": "value"
},
"state": "value",
"disconnectedNodeAcknowledged": true
}
Sigar da ake buƙatar cikewa don yin aiki:
jihar - halin canja wurin bayanai. Akwai: MASA don ba da damar canja wurin bayanai, TSAYA don kashewa
version - processor version
sigar zata tsohuwa zuwa 0 lokacin da aka ƙirƙira, amma ana iya samun waɗannan sigogi ta amfani da hanyar
Ga masu sha'awar rubutun bash, wannan hanyar na iya zama kamar dacewa, amma yana da ɗan wahala a gare ni - rubutun bash ba na fi so ba. Hanya ta gaba ta fi ban sha'awa da dacewa a ganina.
NiPyAPI
NiPyAPI ɗakin karatu ne na Python don hulɗa tare da misalan NiFi. ya ƙunshi mahimman bayanai don aiki tare da ɗakin karatu. An bayyana farawa mai sauri a ciki na github.
Rubutun mu don fitar da tsarin tsari shiri ne a Python. Bari mu matsa zuwa codeing.
Mun kafa saiti don ƙarin aiki. Za mu buƙaci sigogi masu zuwa:
nipyapi.config.nifi_config.host = 'http://nifi:8080/nifi-api' #путь до nifi-api инстанса, на котором разворачиваем process group
nipyapi.config.registry_config.host = 'http://nifi-registry:18080/nifi-registry-api' #путь до nifi-registry-api registry
nipyapi.config.registry_name = 'MyBeutifulRegistry' #название registry, как будет называться в инстансе nifi
nipyapi.config.bucket_name = 'BucketName' #название bucket, из которого подтягиваем flow
nipyapi.config.flow_name = 'FlowName' #название flow, которое подтягиваем
Na gaba zan saka sunayen hanyoyin wannan ɗakin karatu, waɗanda aka bayyana .
Haɗa rajista zuwa misalin nifi ta amfani da
nipyapi.versioning.create_registry_clientA wannan matakin, zaku iya ƙara duba cewa an riga an ƙara rajista zuwa misali; don wannan zaku iya amfani da hanyar.
nipyapi.versioning.list_registry_clientsMun sami guga don ƙarin bincike don kwarara cikin kwandon
nipyapi.versioning.get_registry_bucketYin amfani da guga da aka samo, muna neman kwarara
nipyapi.versioning.get_flow_in_bucketNa gaba, yana da mahimmanci a gane ko an riga an ƙara wannan rukunin tsari. Ana sanya ƙungiyar tsari bisa ga daidaitawa kuma yanayi na iya tasowa lokacin da aka ɗora sashi na biyu a saman ɗaya. Na duba, wannan na iya faruwa :) Don samun duk ƙarin ƙungiyoyin tsari muna amfani da hanyar
nipyapi.canvas.list_all_process_groupsZa mu iya ƙara bincika, misali, da suna.
Ba zan bayyana tsarin sabunta samfurin ba, zan ce kawai idan an ƙara masu sarrafawa a cikin sabon sigar samfuri, to babu matsala tare da kasancewar saƙo a cikin layi. Amma idan an cire na'urori masu sarrafawa, to matsaloli na iya tasowa (nifi baya barin ku cire processor idan layin saƙo ya taru a gabansa). Idan kuna sha'awar yadda na magance wannan matsalar, don Allah ku rubuto mani mu tattauna wannan batu. Lambobin sadarwa a ƙarshen labarin. Bari mu ci gaba zuwa matakin ƙara ƙungiyar tsari.
Lokacin da ake gyara rubutun, na ci karo da wani takamaiman cewa sabon sigar kwarara ba koyaushe ake jan shi ba, don haka ina ba da shawarar fara duba wannan sigar:
nipyapi.versioning.get_latest_flow_verAika rukunin tsari:
nipyapi.versioning.deploy_flow_versionMuna fara masu sarrafawa:
nipyapi.canvas.schedule_process_groupA cikin toshe game da CLI an rubuta cewa ba a kunna canja wurin bayanai ta atomatik a cikin rukunin tsarin nesa ba? Lokacin aiwatar da rubutun, na ci karo da wannan matsalar kuma. A lokacin, na kasa fara canja wurin bayanai ta amfani da API kuma na yanke shawarar rubutawa ga mai haɓaka ɗakin karatu na NiPyAPI kuma in nemi shawara/taimako. Mai haɓakawa ya amsa mani, mun tattauna matsalar kuma ya rubuta cewa yana buƙatar lokaci don "duba wani abu". Bayan haka, bayan kwana biyu, wasiƙa ta zo inda aka rubuta wani aiki da Python wanda zai magance matsalar ƙaddamar da ni !!! A wancan lokacin, sigar NiPyAPI ta kasance 0.13.3 kuma, ba shakka, babu wani abu makamancin haka. Amma a cikin sigar 0.14.0, wanda aka saki kwanan nan, an riga an haɗa wannan aikin a cikin ɗakin karatu. hadu,
nipyapi.canvas.set_remote_process_group_transmissionDon haka, ta amfani da ɗakin karatu na NiPyAPI, mun haɗa wurin yin rajista, mun fitar da kwarara, har ma mun fara sarrafawa da canja wurin bayanai. Sa'an nan za ka iya tsefe lambar, ƙara kowane irin cak, shiga, kuma shi ke nan. Amma wannan labari ne kwata-kwata.
Daga cikin zaɓuɓɓukan sarrafa kansa da na yi la'akari, na ƙarshe ya zama kamar mafi inganci a gare ni. Da fari dai, wannan har yanzu python code ne, wanda zaku iya shigar da lambar shirin taimako kuma ku ci gajiyar duk fa'idodin yaren shirye-shirye. Abu na biyu, aikin NiPyAPI yana haɓaka sosai kuma idan akwai matsaloli zaku iya rubutawa ga mai haɓakawa. Na uku, NiPyAPI har yanzu kayan aiki ne mai sassauƙa don mu'amala da NiFi wajen warware matsaloli masu rikitarwa. Misali, wajen tantance ko layukan saƙon yanzu babu kowa a cikin yawo da kuma ko za a iya sabunta ƙungiyar aiwatarwa.
Shi ke nan. Na bayyana hanyoyin 3 don sarrafa isar da kwararar ruwa ta atomatik a cikin NiFi, ramukan da mai haɓakawa zai iya fuskanta, da kuma samar da lambar aiki don sarrafa isarwa ta atomatik. Idan kuna sha'awar wannan batu kamar yadda nake -
source: www.habr.com