Buku lakuti "Kafka Mitsinje ikugwira ntchito. Mapulogalamu ndi ma microservices a ntchito zenizeni zenizeni"

Moni, okhala ku Khabro! Bukhuli ndi loyenera kwa wopanga aliyense amene akufuna kumvetsetsa kachulukidwe ka ulusi. Kumvetsetsa mapulogalamu omwe amagawidwa kudzakuthandizani kumvetsetsa bwino Kafka ndi Kafka Mitsinje. Zingakhale zabwino kudziwa dongosolo la Kafka palokha, koma izi sizofunikira: Ndikukuuzani zonse zomwe mukufuna. Madivelopa odziwa bwino a Kafka komanso oyambira nawonso aphunzira kupanga mapulogalamu osangalatsa amitsinje pogwiritsa ntchito laibulale ya Kafka Streams m'bukuli. Madivelopa apakatikati komanso apamwamba a Java omwe amadziwa kale malingaliro ngati serialization aphunzira kugwiritsa ntchito luso lawo kupanga mapulogalamu a Kafka Streams. Magwero a bukhuli amalembedwa mu Java 8 ndipo amagwiritsa ntchito kwambiri Java 8 lambda expression syntax, kotero kudziwa momwe mungagwirire ntchito ndi lambda (ngakhale m'chinenero china) kudzathandiza.

Kadule. 5.3. Kuphatikizika ndi mawindo ntchito

Mugawoli, tipitilira kufufuza magawo omwe amalonjeza kwambiri a Kafka Streams. Pakadali pano tafotokoza izi za Kafka Streams:

• kupanga topology processing;
• kugwiritsa ntchito state in stream applications;
• kuthandizira kulumikizana kwa data;
• kusiyana pakati pa mitsinje ya zochitika (KStream) ndi mitsinje yosintha (KTable).

Mu zitsanzo zotsatirazi tibweretsa zinthu zonsezi pamodzi. Muphunziranso za windowsing, chinthu china chachikulu chosinthira mapulogalamu. Chitsanzo chathu choyamba chidzakhala chophatikiza chosavuta.

5.3.1. Kuphatikizika kwa kugulitsa masheya ndi gawo lamakampani

Kuphatikizika ndi magulu ndi zida zofunika kwambiri mukamagwira ntchito ndikusamutsa deta. Kuwunika kwa zolemba zomwe zimalandilidwa nthawi zambiri sikukwanira. Kuti mutenge zambiri zowonjezera kuchokera ku deta, m'pofunika kusonkhanitsa ndi kuziphatikiza.

Muchitsanzo ichi, muvala zovala za wamalonda wamasiku omwe akuyenera kuyang'anira kuchuluka kwa malonda amakampani m'mafakitale angapo. Makamaka, mumakonda makampani asanu omwe ali ndi gawo lalikulu kwambiri pamsika uliwonse.

Kuphatikizika koteroko kudzafuna njira zingapo zotsatirazi kuti mumasulire deta mu mawonekedwe omwe mukufuna (kulankhula momveka bwino).

1. Pangani gwero lotengera mitu yomwe imasindikiza zambiri zamalonda amasheya. Tidzayenera kupanga mapu amtundu wa StockTransaction ku chinthu chamtundu wa ShareVolume. Mfundo ndi yakuti chinthu cha StockTransaction chili ndi metadata yogulitsa, koma timangofunika deta yokhudzana ndi chiwerengero cha magawo omwe akugulitsidwa.
2. Gulu la ShareVolume data ndi chizindikiro cha stock. Mukasanjidwa ndi chizindikiro, mutha kugwetsa detayi m'magulu ang'onoang'ono a kuchuluka kwa malonda. Ndizofunikira kudziwa kuti njira ya KStream.groupBy imabweretsanso chitsanzo cha mtundu wa KGroupedStream. Ndipo mutha kupeza chitsanzo cha KTable powonjezera njira ya KGroupedStream.reduce.

Kodi mawonekedwe a KGroupedStream ndi chiyani

Njira za KStream.groupBy ndi KStream.groupByKey zimabweretsanso chitsanzo cha KGroupedStream. KGroupedStream ndi chiwonetsero chapakatikati cha zochitika zambiri zitatha kugawidwa ndi makiyi. Izo sizinapangidwe kuti zigwire ntchito mwachindunji ndi izo. M'malo mwake, KGroupedStream imagwiritsidwa ntchito pophatikiza, zomwe nthawi zonse zimabweretsa KTable. Ndipo popeza zotsatira za ntchito zophatikizira ndi KTable ndipo amagwiritsa ntchito sitolo ya boma, ndizotheka kuti si zosintha zonse zomwe zimatumizidwa patsogolo.

Njira ya KTable.groupBy imabweretsanso KGroupedTable yofananira - chiwonetsero chapakatikati cha zosintha, zophatikizidwanso ndi kiyi.

Tiyeni tipume pang'ono ndikuyang'ana mkuyu. 5.9, zomwe zikuwonetsa zomwe tapeza. Topology iyi iyenera kukhala yodziwika bwino kwa inu.

Tiyeni tsopano tione kachidindo ka topology iyi (itha kupezeka mu fayilo src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Listing 5.2).

Khodi yoperekedwayo imasiyanitsidwa ndi kufupika kwake komanso kuchuluka kwa zochita zomwe zimachitika m'mizere ingapo. Mutha kuzindikira china chatsopano mu gawo loyamba la builder.stream njira: mtengo wamtundu wa enum AutoOffsetReset.EARLIEST (palinso ZOCHITIKA), khazikitsani kugwiritsa ntchito Consumed.withOffsetResetPolicy njira. Mtundu wowerengerawu ukhoza kugwiritsidwa ntchito kutchula njira yosinthiranso KStream kapena KTable iliyonse ndipo imatsogola pa njira yosinthira kuchokera pakusintha.

GuluByKey ndi GuluBy

Mawonekedwe a KStream ali ndi njira ziwiri zosungiramo zolemba: GroupByKey ndi GroupBy. Onse amabweretsa KGroupedTable, ndiye mwina mukuganiza kuti pali kusiyana kotani pakati pawo komanso kuti mugwiritse ntchito iti?

Njira ya GroupByKey imagwiritsidwa ntchito pamene makiyi a KStream ali opanda kanthu. Ndipo chofunika kwambiri, mbendera "imafuna kugawanitsanso" sinakhazikitsidwe.

Njira ya GroupBy imaganiza kuti mwasintha makiyi amagulu, kotero mbendera yogawanitsa yakhazikitsidwa kuti ikhale yowona. Kuchita majowina, kuphatikizika, ndi zina zotere pambuyo pa njira ya GroupBy kupangitsa kuti tigawidwenso.
Mwachidule: Ngati n'kotheka, muyenera kugwiritsa ntchito GroupByKey osati GroupBy.

Ndizodziwikiratu zomwe mapuValues ​​​​ndi njira za guluBy amachita, ndiye tiyeni tiwone njira yowerengera () (yomwe imapezeka mu src/main/java/bbejeck/model/ShareVolume.java) (Mndandanda 5.3).

Njira ya ShareVolume.sum imabwezeretsa kuchuluka kwa kuchuluka kwa malonda ogulitsa, ndipo zotsatira za mawerengedwe onsewa ndi chinthu cha KTable. . Tsopano mukumvetsa udindo wa KTable. Zinthu za ShareVolume zikafika, chinthu chofananira cha KTable chimasunga zosintha zaposachedwa. Ndikofunika kukumbukira kuti zosintha zonse zikuwonetsedwa mu shareVolumeKTable yapitayi, koma si onse omwe amatumizidwa patsogolo.

Kenako timagwiritsa ntchito KTable iyi kuti tiphatikize (ndi kuchuluka kwa magawo omwe amagulitsidwa) kuti tifike kumakampani asanu omwe ali ndi magawo ambiri omwe amagulitsidwa pamsika uliwonse. Zochita zathu m'nkhaniyi zidzakhala zofanana ndi za gulu loyamba.

1. Chitani gulu linaPantchito kuti mugawane zinthu za ShareVolume ndi makampani.
2. Yambani kufotokoza mwachidule zinthu za ShareVolume. Nthawi ino chinthu chophatikiza ndi mzere wokhazikika wokhazikika. Pamzera wokhazikikawu, makampani asanu okha omwe ali ndi magawo ambiri ogulitsidwa ndi omwe amasungidwa.
3. Lembani mizere kuchokera ku ndime yapitayi kupita ku mtengo wamtengo wapatali ndikubwezerani masheya asanu omwe amagulitsidwa kwambiri ndi chiwerengero ndi makampani.
4. Lembani zotsatira mumtundu wa chingwe ku mutuwo.

Mku. Chithunzi 5.10 chikuwonetsa graph ya data flow topology. Monga mukuonera, kuzungulira kwachiwiri kwa processing ndikosavuta.

Tsopano popeza tamvetsetsa bwino kapangidwe kake kozungulira kwachiwiri, titha kutembenukira ku gwero lake (mupeza mu fayilo src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Listing 5.4) .

Choyambitsa ichi chili ndi kusintha kwa FixedQueue. Ichi ndi chinthu chokhazikika chomwe ndi adapta ya java.util.TreeSet yomwe imagwiritsidwa ntchito potsata zotsatira zapamwamba za N pakutsika kwa magawo omwe amagulitsidwa.

Mwawona kale mafoni a guluBy ndi mapValues, kotero sitilowamo (tikutcha njira ya KTable.toStream chifukwa njira ya KTable.print yachotsedwa). Koma simunawone mtundu wa KTable wa aggregate() pano, ndiye tikhala kanthawi kochepa tikukambirana izi.

Monga mukukumbukira, chomwe chimapangitsa KTable kukhala yosiyana ndikuti zolemba zomwe zili ndi makiyi omwewo zimatengedwa ngati zosintha. KTable ikusintha cholowa chakale ndi chatsopano. Kuphatikizika kumachitika chimodzimodzi: zolembedwa zaposachedwa zokhala ndi makiyi omwewo zimaphatikizidwa. Zolemba zikafika, zimawonjezeredwa ku kalasi ya FixedSizePriorityQueue pogwiritsa ntchito adder (gawo lachiwiri mu kuyitana kwa njira yonse), koma ngati mbiri ina ilipo kale ndi fungulo lomwelo, ndiye kuti mbiri yakale imachotsedwa pogwiritsa ntchito subtractor (parameter yachitatu mu njira ya aggregate kuitana).

Izi zikutanthauza kuti chophatikiza chathu, FixedSizePriorityQueue, sichiphatikiza zinthu zonse ndi kiyi imodzi, koma imasunga kuchuluka kwamitundu yogulitsidwa kwambiri ya N. Chilichonse cholowa chili ndi chiwerengero cha magawo omwe agulitsidwa mpaka pano. KTable ikupatsirani zambiri za magawo amakampani omwe akugulitsidwa kwambiri pakadali pano, osafunikira kuphatikizika kwakusintha kulikonse.

Tinaphunzira kuchita zinthu ziwiri zofunika kwambiri:

• magulu mu KTable ndi kiyi wamba;
• gwirani ntchito zothandiza monga kusonkhanitsa ndi kusonkhanitsa pamagulu awa.

Kudziwa momwe mungachitire izi ndikofunikira kuti mumvetsetse tanthauzo la zomwe zikuyenda kudzera mu pulogalamu ya Kafka Streams ndikumvetsetsa zomwe zimanyamula.

Tasonkhanitsanso mfundo zazikuluzikulu zomwe takambirana kale m'bukuli. Mu Chaputala 4, tidakambirana momwe kusalolera zolakwika, dziko lakwawo kuli kofunikira pa pulogalamu yotsatsira. Chitsanzo choyamba m'mutu uno chikuwonetsa chifukwa chake dziko lanu ndi lofunika kwambiri - limakupatsani mwayi wosunga zomwe mwawona kale. Kufikira kwanuko kumapewa kuchedwa kwa netiweki, zomwe zimapangitsa kuti pulogalamuyo igwire bwino ntchito komanso kuti isavutike.

Mukamagwira ntchito yosonkhanitsa kapena kuphatikizira, muyenera kutchula dzina la sitolo ya boma. Ntchito za rollup ndi aggregation zimabweretsa chitsanzo cha KTable, ndipo KTable imagwiritsa ntchito kusungirako boma kuti isinthe zotsatira zakale ndi zatsopano. Monga momwe mwawonera, si zosintha zonse zomwe zimatumizidwa, ndipo izi ndizofunikira chifukwa ntchito zophatikizira zidapangidwa kuti zipereke chidziwitso chachidule. Ngati simugwiritsa ntchito dera lanu, KTable itumiza zotsatira zonse zophatikiza ndi zotuluka.

Kenako, tiwona momwe zimagwirira ntchito monga kuphatikizira mkati mwa nthawi inayake - zomwe zimatchedwa mawindo.

5.3.2. Zochita za chiwindi

M'gawo lapitalo, tidayambitsa sliding convolution ndi kuphatikiza. Ntchitoyi idachulukitsa mosalekeza kuchuluka kwa malonda a masheya, kutsatiridwa ndi kuphatikizika kwa magawo asanu omwe agulitsidwa kwambiri pakusinthana.

Nthawi zina kuphatikizika kosalekeza kotereku ndikuwonjezera zotsatira ndikofunikira. Ndipo nthawi zina muyenera kuchita maopareshoni pakapita nthawi. Mwachitsanzo, werengerani kuchuluka kwa malonda omwe adapangidwa ndi ma sheya akampani inayake mphindi 10 zapitazi. Kapena ndi ogwiritsa ntchito angati adadina chikwangwani chatsopano chotsatsa mphindi 15 zapitazi. Pulogalamuyi imatha kuchita izi kangapo, koma ndi zotsatira zomwe zimagwira ntchito pakanthawi kochepa (mawindo anthawi).

Kuwerengera kusinthana ndi wogula

Muchitsanzo chotsatira, tiwona zochitika zamalonda pakati pa amalonda angapo, kaya ndi mabungwe akuluakulu kapena anthu anzeru azandalama.

Pali zifukwa ziwiri zomwe mungatsatire izi. Chimodzi mwa izo ndikufunika kudziwa zomwe atsogoleri amsika akugula / kugulitsa. Ngati osewera akuluwa ndi osunga ndalama otsogola awona mwayi, ndizomveka kutsatira njira yawo. Chifukwa chachiwiri ndi chikhumbo chofuna kuwona zisonyezo zilizonse zamalonda osagwirizana ndi malamulo. Kuti muchite izi, muyenera kusanthula kulumikizana kwa spikes zazikulu zogulitsa ndi zolemba zofunika kwambiri.

Kutsata koteroko kumakhala ndi izi:

• kupanga mtsinje wowerengera kuchokera pamutu wamalonda;
• kuyika m'magulu marekodi obwera ndi ID ya wogula ndi chizindikiro cha masheya. Kuitana guluBy njira kumabweretsa chitsanzo cha KGroupedStream kalasi;
• Njira ya KGroupedStream.windowedBy imabweza mtsinje wa data womwe uli ndi zenera la nthawi, lomwe limalola kusonkhanitsa mazenera. Kutengera mtundu wazenera, mwina TimeWindowedKStream kapena SessionWindowedKStream imabwezedwa;
• chiwerengero cha transaction pa ntchito yophatikiza. Kuthamanga kwa deta kwawindo kumatsimikizira ngati zolemba zina zimaganiziridwa mu chiwerengero ichi;
• kulemba zotsatira pamutu kapena kuzitulutsa ku konsoli panthawi ya chitukuko.

Topology ya pulogalamuyi ndi yosavuta, koma chithunzi chomveka bwino chingakhale chothandiza. Tiyeni tione mkuyu. 5.11.

Kenako, tiwona magwiridwe antchito a zenera ndi nambala yofananira.

Mitundu yamawindo

Pali mitundu itatu ya mazenera ku Kafka Mitsinje:

• nthawi;
• "kugwa" (kugwa);
• kutsetsereka/kulumpha.

Zomwe mungasankhe zimatengera zomwe bizinesi yanu ikufuna. Mazenera opunthwa ndi kudumpha amakhala ndi nthawi yochepa, pamene mazenera a gawo amachepetsedwa ndi zochitika za ogwiritsa ntchito-nthawi ya gawo (ma) imatsimikiziridwa ndi momwe wogwiritsa ntchitoyo amachitira. Chinthu chachikulu kukumbukira ndikuti mitundu yonse yazenera imachokera ku masitampu a deti/nthawi ya zolembera, osati nthawi yadongosolo.

Kenako, timakhazikitsa topology yathu ndi mtundu uliwonse wazenera. Khodi yathunthu idzaperekedwa pachitsanzo choyamba; kwa mitundu ina ya windows palibe chomwe chidzasinthe kupatula mtundu wa ntchito yazenera.

Mawindo a gawo

Mawindo a gawo ndi osiyana kwambiri ndi mawindo ena onse. Amachepetsedwa osati mochulukira ndi nthawi monga momwe wogwiritsa ntchito amachitira (kapena zochita za bungwe lomwe mukufuna kutsatira). Mazenera agawo amagawidwa ndi nthawi zosagwira ntchito.

Chithunzi 5.12 chikuwonetsa lingaliro la gawo windows. Gawo laling'ono lidzaphatikizana ndi gawo kumanzere kwake. Ndipo gawo lakumanja lidzakhala losiyana chifukwa likutsatira nthawi yayitali yosagwira ntchito. Mazenera agawo amatengera zochita za ogwiritsa ntchito, koma gwiritsani ntchito masitampu a deti/nthawi kuchokera pazolembedwa kuti muwone gawo lomwe gawolo likhala.

Kugwiritsa ntchito mawindo a gawo kuti mufufuze zochitika za stock

Tiyeni tigwiritse ntchito mazenera a gawo kuti tipeze zambiri zokhudzana ndi kusinthana. Kukhazikitsa kwa mazenera a gawo kukuwonetsedwa mu Listing 5.5 (yomwe ingapezeke mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKTableJoinExample.java).

Mwawona kale machitidwe ambiri mu topology iyi, ndiye palibe chifukwa chowunikiranso apa. Koma palinso zinthu zingapo zatsopano pano, zomwe tikambirana tsopano.

Opaleshoni iliyonse ya guluBy imachita mtundu wina wa ntchito yophatikiza (kuphatikiza, kusonkhanitsa, kapena kuwerengera). Mutha kuchita zophatikizika ndi chiwopsezo chonse, kapena kuphatikizira kwazenera, zomwe zimatengera ma rekodi mkati mwa zenera lanthawi lomwe latchulidwa.

Khodi mu Listing 5.5 imawerengera kuchuluka kwa zochitika mkati mwa gawo windows. Mku. 5.13 Zochita izi zimawunikidwa pang'onopang'ono.

Poyimba kuti windowedBy(SessionWindows.with(twentySeconds).mpaka(FifteenMinutes)) timapanga zenera la gawo lomwe limakhala ndi nthawi yosagwira ntchito ya masekondi 20 ndi kulimbikira kwa mphindi 15. Kusagwira ntchito kwa masekondi a 20 kumatanthauza kuti pulogalamuyo iphatikiza zolowera zilizonse zomwe zikufika mkati mwa masekondi 20 kuchokera kumapeto kapena kuyamba kwa gawo lomwe lilipo mu gawo lapano (logwira).

Kenako, timatchula ntchito yophatikiza yomwe iyenera kuchitidwa pawindo la gawo - apa, kuwerengera. Ngati cholowa chikugwera kunja kwa zenera losagwira ntchito (mbali iliyonse ya tsiku/nthawi ya sitampu), pulogalamuyo imapanga gawo latsopano. Kusunga nthawi kumatanthauza kusunga gawo kwa nthawi yochuluka komanso kulola deta mochedwa yomwe imapitirira nthawi yomwe gawolo silikugwira ntchito koma ikhoza kulumikizidwabe. Kuphatikiza apo, chiyambi ndi kutha kwa gawo latsopanolo chifukwa cha kuphatikiza kumayenderana ndi sitampu yakale komanso yaposachedwa kwambiri.

Tiyeni tiwone zolemba zingapo kuchokera mu njira yowerengera kuti tiwone momwe magawo amagwirira ntchito (Table 5.1).

Zolemba zikafika, timayang'ana magawo omwe alipo ndi fungulo lomwelo, nthawi yomaliza yocheperako kuposa tsiku / sitampu yamakono - nthawi yosagwira ntchito, ndi nthawi yoyambira yokulirapo kuposa nthawi yapano / sitampu + nthawi yosagwira ntchito. Potengera izi, zolemba zinayi kuchokera patebulo. 5.1 amaphatikizidwa mu gawo limodzi motere.

1. Lembani 1 ifika poyamba, kotero nthawi yoyambira ndi yofanana ndi nthawi yotsiriza ndipo ndi 00:00:00.

2. Kenako, kulowa 2 kumabwera, ndipo timayang'ana magawo omwe samatha kale kuposa 23:59:55 ndikuyamba pasanathe 00:00:35. Timapeza zolemba 1 ndikuphatikiza magawo 1 ndi 2. Timatenga nthawi yoyambira gawo 1 (poyamba) ndi nthawi yomaliza ya gawo 2 (pambuyo pake), kotero kuti gawo lathu latsopano liyambe pa 00:00:00 ndikutha pa 00: 00:15.

3. Record 3 ifika, timayang'ana magawo pakati pa 00:00:30 ndi 00:01:10 ndipo osapeza. Onjezani gawo lachiwiri la kiyi 123-345-654,FFBE, kuyambira ndi kutha 00:00:50.

4. Lembani 4 ikufika ndipo tikuyang'ana magawo pakati pa 23:59:45 ndi 00:00:25. Nthawi ino magawo onse 1 ndi 2 akupezeka. Magawo onse atatu akuphatikizidwa kukhala amodzi, ndi nthawi yoyambira 00:00:00 ndi nthawi yomaliza ya 00:00:15.

Kuchokera pazomwe zafotokozedwa m'gawoli, ndikofunikira kukumbukira ma nuances otsatirawa:

• magawo si mazenera okhazikika. Kutalika kwa gawo kumatsimikiziridwa ndi zochitika mkati mwa nthawi yoperekedwa;
• Masitampu a deti/nthawi mu data amatsimikizira ngati chochitikacho chikugwera mkati mwa gawo lomwe lidalipo kapena munthawi yopanda ntchito.

Kenako tikambirana mtundu wotsatira wazenera - mazenera "ogwetsa".

Mawindo a "kuthamanga".

Mawindo akupunthwa amatenga zochitika zomwe zimagwera mkati mwa nthawi inayake. Tangoganizani kuti mukufunika kujambula zochitika zonse za kampani inayake pamasekondi 20 aliwonse, kuti mutenge zochitika zonse panthawiyo. Pamapeto pa nthawi ya 20-sekondi, zenera likuzungulira ndikusunthira kumalo atsopano a 20-sekondi imodzi. Chithunzi 5.14 chikuwonetsa izi.

Monga mukuwonera, zochitika zonse zomwe zidalandilidwa mumasekondi 20 omaliza zimaphatikizidwa pazenera. Pamapeto pa nthawiyi, zenera latsopano limapangidwa.

Mndandanda wa 5.6 umasonyeza kachidindo kamene kamasonyeza kugwiritsa ntchito mawindo opunthwa kuti atenge malonda a masheya masekondi aliwonse a 20 (omwe amapezeka mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Ndi kusintha kwakung'ono kumeneku kuyitanira kwa TimeWindows.of njira, mutha kugwiritsa ntchito zenera lakugwa. Chitsanzochi sichimatchula njira yofikira (), kotero kuti nthawi yosunga yokhazikika ya maola 24 idzagwiritsidwa ntchito.

Pomaliza, ndi nthawi yoti mupite ku zomaliza zazenera - "hopping" mazenera.

Mawindo otsetsereka ("kudumpha")

Mawindo otsetsereka / odumphira ndi ofanana ndi mawindo akugwa, koma ndi kusiyana pang'ono. Mazenera otsetsereka samadikirira mpaka kumapeto kwa nthawiyo asanapange zenera latsopano kuti akonze zochitika zaposachedwa. Amayamba kuwerengera kwatsopano pakadutsa nthawi yodikirira yocheperako kuposa nthawi yazenera.

Kuti tifotokoze kusiyana pakati pa kugwa ndi kulumpha mazenera, tiyeni tibwererenso ku chitsanzo cha kuwerengera malonda a masheya. Cholinga chathu ndikuwerengera kuchuluka kwa zomwe zachitika, koma sitikufuna kudikirira nthawi yonseyo tisanasinthe kauntala. M'malo mwake, tidzasintha kauntala pakanthawi kochepa. Mwachitsanzo, tidzawerengerabe kuchuluka kwa zomwe zachitika masekondi 20 aliwonse, koma sinthani kauntala masekondi 5 aliwonse, monga momwe tawonetsera mkuyu. 5.15. Pankhaniyi, timathera ndi mazenera atatu otsatila omwe ali ndi deta yochulukirapo.

Mndandanda wa 5.7 ukuwonetsa khodi yofotokozera mazenera otsetsereka (opezeka mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Zenera lakugwa litha kusinthidwa kukhala zenera lodumphira powonjezera kuyimba kwa advanceBy() njira. Mu chitsanzo chowonetsedwa, nthawi yopulumutsira ndi mphindi 15.

Munawona m'gawoli momwe mungachepetsere zotsatira zophatikiza nthawi mawindo. Makamaka, ndikufuna kuti mukumbukire zinthu zitatu zotsatirazi kuchokera mgawoli:

• kukula kwa mazenera a gawo kumachepetsedwa osati ndi nthawi, koma ndi ntchito za ogwiritsa ntchito;
• "Kugwa" mazenera amapereka chithunzithunzi cha zochitika mkati mwa nthawi yoperekedwa;
• Kutalika kwa mazenera odumpha kumakhazikika, koma amasinthidwa pafupipafupi ndipo akhoza kukhala ndi zolembera zodutsana m'mawindo onse.

Kenako, tiphunzira momwe mungasinthire KTable kubwerera ku KStream kuti mulumikizane.

5.3.3. Kulumikiza KStream ndi KTable zinthu

Mu Chaputala 4, tidakambirana kulumikiza zinthu ziwiri za KStream. Tsopano tiyenera kuphunzira kulumikiza KTable ndi KStream. Izi zingakhale zofunikira pazifukwa zosavuta zotsatirazi. KStream ndi mtsinje wa zolemba, ndipo KTable ndi mtsinje wa zosintha zojambulidwa, koma nthawi zina mungafune kuwonjezera zina pa mtsinje rekodi pogwiritsa ntchito zosintha kuchokera KTable.

Tiyeni titenge zambiri pazambiri zomwe zachitika pakusinthanitsa masheya ndikuziphatikiza ndi nkhani zamakampani omwe akhudzidwa. Izi ndi zomwe muyenera kuchita kuti mukwaniritse izi chifukwa cha code yomwe muli nayo kale.

1. Sinthani chinthu cha KTable chokhala ndi data pa kuchuluka kwa malonda a masheya kukhala KStream, ndikutsatiridwa ndikusintha makiyi ndi kiyi yosonyeza gawo lamakampani lomwe likugwirizana ndi chizindikiro ichi.
2. Pangani chinthu cha KTable chomwe chimawerenga deta kuchokera pamutu wokhala ndi nkhani za stock exchange. KTable yatsopanoyi idzagawika m'magulu amakampani.
3. Lumikizani zosintha zankhani ndi zambiri za kuchuluka kwa kusinthana kwamasheya ndi makampani.

Tsopano tiyeni tiwone momwe tingagwiritsire ntchito ndondomekoyi.

Sinthani KTable kukhala KStream

Kuti musinthe KTable kukhala KStream muyenera kuchita zotsatirazi.

1. Imbani njira ya KTable.toStream().
2. Poyimba njira ya KStream.map, sinthani kiyiyo ndi dzina lamakampani, kenako pezani chinthu cha TransactionSummary kuchokera pawindo la Windowed.

Tidzagwirizanitsa ntchito izi motere (code ingapezeke mu fayilo src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Listing 5.8).

Chifukwa tikuchita ntchito ya KStream.map, chitsanzo cha KStream chobwezedwa chimagawidwanso chokha chikagwiritsidwa ntchito polumikizana.

Tamaliza kutembenuza, kenako tifunika kupanga chinthu cha KTable chowerengera nkhani zamasheya.

Kupanga KTable kwa nkhani zamasheya

Mwamwayi, kupanga chinthu cha KTable kumatenga mzere umodzi wokha wa code (code ingapezeke mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Mndandanda 5.9).

Ndizofunikira kudziwa kuti palibe zinthu za Serde zomwe zimafunikira kufotokozedwa, popeza chingwe cha Serdes chimagwiritsidwa ntchito pazokonda. Komanso, pogwiritsa ntchito kuwerengetsa KWAMBIRI, tebulo limadzazidwa ndi zolemba pachiyambi pomwe.

Tsopano tikhoza kupita ku sitepe yomaliza - kugwirizana.

Kulumikiza zosintha zankhani ndi data yowerengera ndalama

Kupanga kulumikizana sikovuta. Tidzagwiritsa ntchito kujowina kumanzere ngati palibe nkhani zamakampani zomwe zikufunika (code yofunikira ingapezeke mufayilo src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Listing 5.10).

Wothandizira wa leftJoin uyu ndiwosavuta. Mosiyana ndi majowina mu Chaputala 4, njira ya JoinWindow sigwiritsidwa ntchito chifukwa pochita nawo KStream-KTable, pali cholowa chimodzi chokha mu KTable pa kiyi iliyonse. Kulumikizana koteroko sikuli ndi nthawi: zolembazo zili mu KTable kapena palibe. Mapeto ake: kugwiritsa ntchito zinthu za KTable mutha kulemeretsa KStream ndi zambiri zosinthidwa pafupipafupi.

Tsopano tiwona njira yabwino kwambiri yolemeretsa zochitika kuchokera ku KStream.

5.3.4. GlobalKTable zinthu

Monga mukuonera, pakufunika kukulitsa mitsinje ya zochitika kapena kuwonjezera nkhani kwa iwo. Mu Chaputala 4 munawona kulumikizana pakati pa zinthu ziwiri za KStream, ndipo mu gawo lapitalo mudawona kulumikizana pakati pa KStream ndi KTable. Pazochitika zonsezi, ndikofunikira kugawanso mtsinje wa deta pamene mukujambula makiyi amtundu watsopano kapena mtengo. Nthawi zina kugawanitsa kumachitika momveka bwino, ndipo nthawi zina Kafka Mitsinje imangochita zokha. Kugawanitsanso ndikofunikira chifukwa makiyi asintha ndipo zolemba ziyenera kutha m'magawo atsopano, apo ayi kugwirizana sikudzakhala kosatheka (izi zinakambidwa mu Mutu 4, mu gawo la "Re-partitioning data" mu ndime 4.2.4).

Kugawanitsanso kuli ndi mtengo wake

Kugawanso kumafuna ndalama - ndalama zowonjezera zopangira mitu yapakatikati, kusunga deta yobwereza pamutu wina; zimatanthauzanso kuchuluka kwa latency chifukwa cholemba ndi kuwerenga kuchokera pamutuwu. Kuphatikiza apo, ngati mukufuna kujowina mbali zingapo kapena milingo imodzi, muyenera kulumikiza zolumikizira, kupanga mapu ndi makiyi atsopano, ndikuyambitsanso kugawanso.

Kulumikiza kumagulu ang'onoang'ono a data

Nthawi zina, kuchuluka kwa data yolumikizidwa yomwe iyenera kulumikizidwa ndi yaying'ono, kotero kuti makope athunthu amatha kukwanira mdera lanu pamfundo iliyonse. Pazifukwa ngati izi, Kafka Mitsinje imapereka kalasi ya GlobalKTable.

Zochitika za GlobalKTable ndizopadera chifukwa pulogalamuyo imabwereza deta yonse kumalo aliwonse. Ndipo popeza zidziwitso zonse zilipo pa node iliyonse, palibe chifukwa chogawanitsa mtsinjewo ndi kiyi ya data kuti ipezeke ku magawo onse. Mutha kupanganso zolumikizira zopanda pake pogwiritsa ntchito zinthu za GlobalKTable. Tiyeni tibwererenso ku chimodzi mwa zitsanzo zam'mbuyomu kuti tiwonetse izi.

Kulumikiza zinthu za KStream ku zinthu za GlobalKTable

Mundime 5.3.2, tidachita kuphatikizika kwazenera kwa ogula. Zotsatira za kuphatikizika uku zidawoneka motere:

{customerId='074-09-3705', stockTicker='GUTM'}, 17
{customerId='037-34-5184', stockTicker='CORK'}, 16

Ngakhale kuti zotsatira izi zidakwaniritsa cholinga, zikadakhala zothandiza kwambiri ngati dzina la kasitomala ndi dzina lakampani lonse zikadawonetsedwanso. Kuti muwonjezere dzina la kasitomala ndi dzina la kampani, mutha kujowina wamba, koma muyenera kupanga makiyi awiri ndikugawanso. Ndi GlobalKTable mutha kupewa mtengo wantchito zotere.

Kuti tichite izi, tidzagwiritsa ntchito chinthu cha countStream kuchokera ku Listing 5.11 (code yogwirizana ingapezeke mu src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) ndikugwirizanitsa ndi zinthu ziwiri za GlobalKTable.

Takambirana kale izi, kotero sindibwereza. Koma ndikuzindikira kuti code yomwe ili mu toStream().mapu ntchito imatengedwa kukhala chinthu chogwira ntchito m'malo mwa mawu amkati a lambda pofuna kuwerengedwa.

Chotsatira ndicho kulengeza zochitika ziwiri za GlobalKTable (code yomwe ikuwonetsedwa ikhoza kupezeka mu fayilo src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Mndandanda wa 5.12).

Chonde dziwani kuti mayina amutu amafotokozedwa pogwiritsa ntchito mitundu yowerengeka.

Tsopano popeza tili ndi zigawo zonse zokonzeka, zonse zomwe zatsala ndikulemba ndondomeko yolumikizira (yomwe ingapezeke mu fayilo src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Listing 5.13).

Ngakhale pali magawo awiri mu code iyi, amamangidwa chifukwa palibe zotsatira zake zomwe zimagwiritsidwa ntchito mosiyana. Zotsatira zikuwonetsedwa kumapeto kwa ntchito yonse.

Mukayendetsa ntchito yojowina pamwambapa, mupeza zotsatira monga izi:

{customer='Barney, Smith' company="Exxon", transactions= 17}

Chofunikira sichinasinthe, koma zotsatirazi zikuwoneka bwino kwambiri.

Ngati muwerengera mpaka Chaputala 4, mwawona kale mitundu ingapo yolumikizira ikugwira ntchito. Zalembedwa mu tebulo. 5.2. Gome ili likuwonetsa kuthekera kolumikizana ndi mtundu 1.0.0 wa Kafka Streams; China chake chitha kusintha pazotulutsa zamtsogolo.

Kuti titsirize, tiyeni tibwerezenso zoyambira: mutha kulumikiza mayendedwe a zochitika (KStream) ndikusintha mitsinje (KTable) pogwiritsa ntchito madera akumaloko. Mwinanso, ngati kukula kwa deta yofotokozera sikuli kwakukulu, mungagwiritse ntchito chinthu cha GlobalKTable. GlobalKTables imabwereza magawo onse kumalo aliwonse a Kafka Streams, kuonetsetsa kuti deta yonse ikupezeka mosasamala kanthu kuti fungulo likugwirizana ndi liti.

Kenako tiwona mawonekedwe a Kafka Mitsinje, chifukwa chake titha kuwona kusintha kwa boma popanda kugwiritsa ntchito deta kuchokera pamutu wa Kafka.

5.3.5. Dziko lofunsidwa

Tachita kale ntchito zingapo zokhudza boma ndipo nthawi zonse timatulutsa zotsatira ku kontrakitala (zolinga zachitukuko) kapena kuzilemba pamutu (zofuna kupanga). Mukalemba zotsatira pamutu, muyenera kugwiritsa ntchito kasitomala wa Kafka kuti muwone.

Kuwerenga zomwe zalembedwa pamitu iyi zitha kuonedwa ngati mtundu wamalingaliro owoneka bwino. Pazolinga zathu, titha kugwiritsa ntchito tanthauzo la mawonekedwe opangidwa kuchokera ku Wikipedia: “...chinthu chankhokwe chomwe chili ndi zotsatira za funso. Mwachitsanzo, ikhoza kukhala kope lakutali lakutali, kapena kachigawo kakang'ono ka mizere ndi/kapena mizati ya tebulo kapena lowani nawo zotsatira, kapena chidule cha tebulo chopezedwa mwa kuphatikiza” (https://en.wikipedia.org/wiki /Zowoneka_zowoneka).

Mitsinje ya Kafka imakupatsaninso mwayi wofunsa mafunso m'masitolo a boma, kukulolani kuti muwerenge molunjika malingaliro opangidwa ndi thupi. Ndikofunika kuzindikira kuti funso ku sitolo ya boma ndi ntchito yowerengera yokha. Izi zimatsimikizira kuti simuyenera kuda nkhawa kuti mwangozi mupangitsa kuti dziko likhale losagwirizana pomwe pulogalamu yanu ikukonza data.

Kutha kufunsa mwachindunji masitolo aboma ndikofunikira. Izi zikutanthauza kuti mutha kupanga mapulogalamu a dashboard osayamba kutenga deta kuchokera kwa ogula a Kafka. Zimawonjezeranso mphamvu yogwiritsira ntchito, chifukwa chakuti palibe chifukwa cholemberanso deta:

• chifukwa cha malo a deta, amatha kupezeka mwamsanga;
• kubwereza kwa deta kumathetsedwa, popeza sikunalembedwe ku yosungirako kunja.

Chachikulu chomwe ndikufuna kuti mukumbukire ndikuti mutha kufunsa mwachindunji kuchokera mkati mwa pulogalamu yanu. Mwayi umene izi zimakupatsani simunganene mopambanitsa. M'malo mogwiritsa ntchito zambiri kuchokera ku Kafka ndikusunga zolemba mu nkhokwe ya pulogalamuyo, mutha kufunsa masitolo aboma ndi zotsatira zomwezo. Mafunso achindunji ku malo ogulitsa amatanthawuza ma code ochepa (palibe ogula) ndi mapulogalamu ochepa (palibe chifukwa cha tebulo la database kuti asunge zotsatira).

Takambirana zambiri mumutu uno, kotero tisiya zokambirana zathu zokhudzana ndi masitolo aboma pakadali pano. Koma musade nkhawa: mu Chaputala 9, tipanga pulogalamu yosavuta yapadashboard yokhala ndi mafunso. Idzagwiritsa ntchito zitsanzo za mitu iyi ndi yapitayi kuti iwonetse mafunso okhudzana ndi momwe mungawonjezere ku mapulogalamu a Kafka Streams.

Chidule

• Zinthu za KStream zimayimira zochitika zambiri, zofananira ndi zoyika mu database. Zinthu za KTable zimayimira mitsinje yosintha, monga zosintha pa database. Kukula kwa chinthu cha KTable sikukula, zolemba zakale zimasinthidwa ndi zatsopano.
• Zinthu za KTable ndizofunikira pakuphatikiza.
• Pogwiritsa ntchito mazenera, mutha kugawa deta yophatikizidwa mu zidebe za nthawi.
• Chifukwa cha zinthu za GlobalKTable, mutha kupeza zolozera kulikonse mukugwiritsa ntchito, mosasamala kanthu za magawo.
• Kulumikizana pakati pa KStream, KTable ndi GlobalKTable zinthu ndizotheka.

Pakadali pano, tayang'ana kwambiri pakumanga mapulogalamu a Kafka Streams pogwiritsa ntchito KStream DSL yapamwamba. Ngakhale njira yapamwamba imakulolani kuti mupange mapulogalamu abwino komanso achidule, kugwiritsa ntchito kumayimira malonda. Kugwira ntchito ndi DSL KStream kumatanthauza kukulitsa chidule cha code yanu pochepetsa kuwongolera. M'mutu wotsatira, tiwona API yogwira ntchito yotsika ndikuyesa malonda ena. Mapulogalamuwa adzakhala aatali kuposa momwe analili kale, koma tidzatha kupanga pafupifupi node iliyonse yomwe tingafune.

→ Zambiri za bukuli zitha kupezeka pa tsamba la osindikiza

→ Kwa Habrozhiteli 25% kuchotsera pogwiritsa ntchito kuponi - Kafka Mitsinje

→ Pakulipira buku la pepala, buku lamagetsi lidzatumizidwa ndi imelo.

Source: www.habr.com