Littafin "Kafka Streams in Action. Aikace-aikace da microservices don aiki na ainihi"

Sannu, mazauna Khabro! Wannan littafin ya dace da duk mai haɓakawa da ke son fahimtar sarrafa zaren. Fahimtar shirye-shiryen da aka rarraba zai taimaka muku mafi fahimtar Kafka da Rafukan Kafka. Zai yi kyau a san tsarin Kafka kanta, amma wannan ba lallai ba ne: Zan gaya muku duk abin da kuke buƙata. Kwararrun masu haɓaka Kafka da novice iri ɗaya za su koyi yadda ake ƙirƙirar aikace-aikacen sarrafa rafi mai ban sha'awa ta amfani da ɗakin karatu na Kafka Streams a cikin wannan littafin. Matsakaici da masu haɓaka Java sun riga sun saba da ra'ayoyi kamar serialization za su koyi yin amfani da ƙwarewarsu don ƙirƙirar aikace-aikacen Rafukan Kafka. An rubuta lambar tushe na littafin a cikin Java 8 kuma yana yin amfani da mahimmanci na Java 8 lambda magana syntax, don haka sanin yadda ake aiki da ayyukan lambda (har ma a cikin wani harshe na shirye-shirye) zai zo da amfani.

Bangaren. 5.3. Tari da ayyukan taga

A cikin wannan sashe, za mu ci gaba don bincika mafi kyawun sassa na Rafukan Kafka. Ya zuwa yanzu mun rufe bangarori masu zuwa na Rafukan Kafka:

• ƙirƙirar topology aiki;
• yin amfani da yanayi a aikace-aikacen yawo;
• yin hanyoyin haɗin rafi na bayanai;
• bambance-bambance tsakanin rafukan taron (KStream) da sabunta rafukan (KTable).

A cikin misalan da ke gaba za mu kawo dukkan waɗannan abubuwa tare. Za ku kuma koyi game da taga, wani babban fasalin aikace-aikacen yawo. Misalinmu na farko zai zama tarawa mai sauƙi.

5.3.1. Tarin tallace-tallacen hannun jari ta bangaren masana'antu

Haɗawa da tarawa kayan aiki ne masu mahimmanci yayin aiki tare da bayanan yawo. Gwajin bayanan mutum ɗaya yayin da ake karɓar su galibi bai isa ba. Don cire ƙarin bayani daga bayanai, ya zama dole a haɗa su da haɗa su.

A cikin wannan misalin, zaku sanya suturar ɗan kasuwa na rana wanda ke buƙatar bin diddigin adadin tallace-tallace na hannun jari na kamfanoni a masana'antu da yawa. Musamman, kuna sha'awar kamfanoni biyar tare da tallace-tallace mafi girma a kowace masana'antu.

Irin wannan tarawa zai buƙaci matakai da yawa masu zuwa don fassara bayanai cikin sigar da ake so (magana a gaba ɗaya).

1. Ƙirƙiri tushen tushen jigo wanda ke buga danyen bayanan ciniki. Dole ne mu tsara wani abu na nau'in StockTransaction zuwa wani abu na nau'in ShareVolume. Ma'anar ita ce abin StockTransaction ya ƙunshi metadata tallace-tallace, amma muna buƙatar bayanai kawai game da adadin hannun jarin da ake sayarwa.
2. Rukunin Rarraba Ƙirar ƙira ta alamar hannun jari. Da zarar an haɗa su ta alama, zaku iya ruguje wannan bayanan zuwa jumloli na tallace-tallacen hannun jari. Yana da kyau a lura cewa hanyar KStream.groupBy tana dawo da misalin nau'in KGroupedStream. Kuma zaku iya samun misalin KTable ta ƙara kiran hanyar KGroupedStream.reduce.

Menene KGroupedStream dubawa

Hanyoyin KStream.groupBy da KStream.groupByKey suna dawo da misalin KGroupedStream. KGroupedStream shine matsakaicin wakilci na rafi na abubuwan da suka faru bayan haɗawa ta maɓalli. Ba a yi nufin komai ba don aiki kai tsaye tare da shi. Madadin haka, ana amfani da KGroupedStream don ayyukan tarawa, wanda koyaushe yana haifar da KTable. Kuma tun da sakamakon ayyukan tarawa shine KTable kuma suna amfani da kantin sayar da kayayyaki na jihar, yana yiwuwa ba duk sabuntawa ba ne a sakamakon haka an aika da kara saukar da bututun.

Hanyar KTable.groupBy tana dawo da irin wannan KGroupedTable - matsakaicin wakilci na rafi na sabuntawa, wanda aka sake tattarawa ta maɓalli.

Bari mu ɗan ɗan huta mu dubi Fig. 5.9, wanda ke nuna abin da muka cim ma. Wannan topology ya kamata ya riga ya saba muku sosai.

Yanzu bari mu kalli lambar don wannan topology (ana iya samunta a cikin fayil src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Jeri 5.2).

An bambanta lambar da aka ba ta ta gajeriyar sa da babban ƙarar ayyukan da aka yi a cikin layi da yawa. Kuna iya lura da wani sabon abu a cikin siga na farko na hanyar ginin ginin. Ana iya amfani da wannan nau'in ƙidayar don ƙididdige dabarun sake saitin saiti don kowane KStream ko KTable kuma yana ɗaukar fifiko akan zaɓin sake saitin saiti daga daidaitawa.

RukuniByKey da RukuniBy

Ƙididdigar KStream yana da hanyoyi guda biyu don tattara bayanan: GroupByKey da GroupBy. Dukansu suna dawo da KGroupedTable, don haka kuna iya mamakin menene bambanci tsakanin su da lokacin amfani da wanne?

Ana amfani da hanyar GroupByKey lokacin da maɓallai a cikin KStream sun riga sun zama marasa fanko. Kuma mafi mahimmanci, ba a taɓa saita tutar "yana buƙatar sake rarrabawa" ba.

Hanyar GroupBy tana ɗaukan cewa kun canza maɓallan haɗawa, don haka an saita tutar rabuwa zuwa gaskiya. Yin haɗin gwiwa, tarawa, da sauransu bayan hanyar GroupBy zai haifar da sake rabuwa ta atomatik.
Takaitawa: Duk lokacin da zai yiwu, yakamata kuyi amfani da GroupByKey maimakon GroupBy.

Ya bayyana a sarari abin da taswiraValues ​​da rukuniBy hanyoyin suke yi, don haka bari mu dubi hanyar jimla () (wanda aka samo a src/main/java/bbejeck/model/ShareVolume.java) (Jeri 5.3).

Hanyar ShareVolume.sum tana mayar da jimlar yawan tallace-tallacen tallace-tallace, kuma sakamakon dukkanin jerin lissafin shine KTable. . Yanzu kun fahimci rawar da KTable ke takawa. Lokacin da abubuwan ShareVolume suka isa, abin KTable daidai yana adana sabon sabuntawa na yanzu. Yana da mahimmanci a tuna cewa duk abubuwan sabuntawa suna nunawa a cikin sharewar da ta gabataVolumeKTable, amma ba duka ana aika su gaba ba.

Sannan muna amfani da wannan KTable don tarawa (ta adadin hannun jarin da aka yi ciniki) don isa ga kamfanoni biyar waɗanda ke da mafi girman adadin hannun jarin da aka yi ciniki a kowace masana'anta. Ayyukanmu a cikin wannan yanayin za su kasance daidai da waɗanda aka yi don tarawar farko.

1. Yi wani rukuniTa aiki don haɗa abubuwan ShareVolume daidaikun masana'antu.
2. Fara taƙaita abubuwan ShareVolume. Wannan karon abin haɗakarwa shine ƙayyadadden jerin gwanon fifiko. A cikin wannan tsayayyen jerin gwano, kamfanoni biyar ne kawai ke da mafi yawan hannun jarin da aka sayar.
3. Taswirar jerin layi daga sakin layi na baya zuwa ƙimar kirtani kuma mayar da manyan hajoji biyar da aka fi ciniki ta lamba ta masana'antu.
4. Rubuta sakamakon a cikin sigar kirtani zuwa batun.

A cikin siffa. Hoto 5.10 yana nuna jadawali kwararar bayanai. Kamar yadda kake gani, zagaye na biyu na sarrafawa abu ne mai sauƙi.

Yanzu da muka fahimci tsarin wannan zagaye na biyu na sarrafawa, za mu iya juya zuwa lambar tushe (za ku same shi a cikin fayil src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Jerin 5.4) .

Wannan mai farawa yana ƙunshe da tsayayyen Queue. Wannan wani abu ne na al'ada wanda shine adaftan java.util.TreeSet wanda ake amfani dashi don bin manyan sakamakon N a cikin tsari na saukowa na hannun jari da aka yi ciniki.

Kun riga kun ga kiran rukuniBy da taswirar ƙira, don haka ba za mu shiga cikin waɗannan ba (muna kiran hanyar KTable.toStream saboda hanyar KTable.print ta ƙare). Amma ba ku ga nau'in tara () na KTable ba tukuna, don haka za mu ɗan ɗauki ɗan lokaci muna tattauna hakan.

Kamar yadda kuke tunawa, abin da ke sa KTable ya bambanta shine cewa ana ɗaukar rikodin tare da maɓalli iri ɗaya sabuntawa. KTable ya maye gurbin tsohuwar shigarwa da sabo. Tari yana faruwa a irin wannan hanya: ana tattara sabbin bayanan da ke da maɓalli iri ɗaya. Lokacin da rikodi ya zo, ana ƙara shi zuwa misali ajin FixedSizePriorityQueue ta amfani da adder ( siga na biyu a cikin hanyar haɗin kira), amma idan wani rikodin ya riga ya kasance tare da maɓalli iri ɗaya, to ana cire tsohon rikodin ta amfani da mai ragewa (parameter na uku a ciki). da tarin hanyar kira).

Wannan duk yana nufin cewa mai tara mu, FixedSizePriorityQueue, baya tara duk dabi'u tare da maɓalli ɗaya, amma yana adana jimlar motsi na adadin N mafi yawan hannun jarin da aka yi ciniki. Kowane shigarwa mai shigowa ya ƙunshi jimlar adadin hannun jarin da aka sayar ya zuwa yanzu. KTable zai ba ku bayani game da waɗanne hannayen jarin kamfanoni ne aka fi siyar da su a halin yanzu, ba tare da buƙatar tarawa na kowane sabuntawa ba.

Mun koyi yin abubuwa biyu masu muhimmanci:

• ƙimar rukuni a cikin KTable ta maɓalli gama gari;
• aiwatar da ayyuka masu amfani kamar naɗawa da tarawa akan waɗannan ƙididdiga masu tarin yawa.

Sanin yadda ake aiwatar da waɗannan ayyuka yana da mahimmanci don fahimtar ma'anar bayanan da ke motsawa ta aikace-aikacen Kafka Streams da fahimtar irin bayanan da yake ɗauka.

Mun kuma tattara wasu mahimman ra'ayoyin da aka tattauna a baya a wannan littafin. A cikin Babi na 4, mun tattauna yadda rashin haƙuri, ƙaramar hukuma ke da mahimmanci ga aikace-aikacen yawo. Misali na farko a wannan babin ya nuna dalilin da ya sa jihar ke da muhimmanci sosai—yana ba ka damar ci gaba da bin diddigin bayanan da ka riga ka gani. Samun shiga cikin gida yana guje wa jinkirin hanyar sadarwa, yana sa aikace-aikacen ya zama mai ƙwazo da juriya da kuskure.

Lokacin yin kowane naɗawa ko aikin tarawa, dole ne ka saka sunan kantin sayar da jigo. Ayyukan naɗawa da tarawa suna dawo da misalin KTable, kuma KTable yana amfani da ma'ajin jiha don maye gurbin tsoffin sakamakon da sababbi. Kamar yadda kuka gani, ba duk sabuntawa ba ne aka saukar da bututun, kuma wannan yana da mahimmanci saboda an tsara ayyukan tarawa don samar da taƙaitaccen bayani. Idan ba ku yi amfani da jaha na gida ba, KTable zai tura duk sakamakon tattarawa da tattarawa.

Na gaba, za mu duba aiwatar da ayyuka kamar tarawa cikin ƙayyadadden lokaci - abin da ake kira ayyukan taga.

5.3.2. Ayyukan taga

A cikin sashin da ya gabata, mun gabatar da juzu'in zamewa da tarawa. Aikace-aikacen ya ci gaba da jujjuyawar tallace-tallacen hajoji wanda ya biyo baya tare da tara mafi yawan hajoji biyar da aka yi ciniki akan musayar.

Wani lokaci irin wannan ci gaba da tarawa da jujjuya sakamako ya zama dole. Kuma wani lokacin kuna buƙatar aiwatar da ayyuka na ɗan lokaci kawai. Misali, lissafta yawan mu'amalar musaya da aka yi tare da hannun jari na wani kamfani a cikin mintuna 10 na ƙarshe. Ko masu amfani nawa ne suka danna sabon tutar talla a cikin mintuna 15 na ƙarshe. Aikace-aikace na iya yin irin waɗannan ayyuka sau da yawa, amma tare da sakamakon da ya shafi ƙayyadadden lokaci kawai (windows lokaci).

Ƙididdigar musayar mu'amala ta mai siye

A misali na gaba, za mu bibiyi hada-hadar hannun jari a tsakanin 'yan kasuwa da yawa-ko dai manyan kungiyoyi ko masu kudi masu kaifin basira.

Akwai dalilai guda biyu masu yiwuwa na wannan bin diddigin. Ɗaya daga cikinsu shine buƙatar sanin abin da shugabannin kasuwa ke saya / sayarwa. Idan waɗannan manyan 'yan wasa da ƙwararrun masu saka hannun jari sun ga dama, yana da ma'ana don bin dabarun su. Dalili na biyu shi ne sha'awar gano duk wata alama mai yuwuwar cinikin cikin gida ba bisa ka'ida ba. Don yin wannan, kuna buƙatar yin nazarin alaƙar manyan tallace-tallacen tallace-tallace tare da sakin latsa mai mahimmanci.

Irin wannan bin diddigin ya ƙunshi matakai masu zuwa:

• ƙirƙirar rafi don karantawa daga batun hayar-ma'amala;
• haɗa bayanan masu shigowa ta ID na mai siye da alamar hannun jari. Kira ƙungiyarTa hanya yana dawo da misalin ajin KGroupedStream;
• Hanyar KGroupedStream.windowedBy tana dawo da rafin bayanai iyakance ga taga lokaci, wanda ke ba da damar tara tagar. Dangane da nau'in taga, ko dai an dawo da TimeWindowedKStream ko SessionWindowedKStream;
• ƙididdigar ma'amala don aikin tarawa. Gudun bayanan da aka rufe ta taga yana ƙayyade ko an yi la'akari da takamaiman rikodin a cikin wannan ƙidaya;
• rubuta sakamako zuwa wani batu ko fitar da su zuwa na'ura wasan bidiyo yayin haɓakawa.

Topology na wannan aikace-aikacen yana da sauƙi, amma bayyanannen hotonsa zai taimaka. Bari mu kalli Fig. 5.11.

Na gaba, za mu dubi ayyukan ayyukan taga da lambar da ta dace.

Nau'in taga

Akwai windows iri uku a cikin Rafukan Kafka:

• zaman lokaci;
• "tumbling" (tumbling);
• zamiya/hopping.

Wanne za a zaɓa ya dogara da buƙatun kasuwancin ku. Tumbling da tsalle windows suna da iyakacin lokaci, yayin da windows zaman ke iyakance ta ayyukan mai amfani—tsawon lokacin (s) yana ƙayyade ta yadda mai amfani yake aiki kawai. Babban abin da za a tuna shi ne cewa duk nau'ikan taga suna dogara ne akan tambarin kwanan wata / lokaci na shigarwar, ba lokacin tsarin ba.

Na gaba, muna aiwatar da topology ɗinmu tare da kowane nau'in taga. Za a ba da cikakkiyar lambar kawai a misali na farko; ga sauran nau'ikan windows babu abin da zai canza sai nau'in aikin taga.

Gilashin zama

Gilashin zama sun sha bamban da sauran nau'ikan tagogi. Ba a iyakance su ba da lokaci sosai kamar ta aikin mai amfani (ko ayyukan mahaɗan da kuke son waƙa). Gilashin zama yana iyakance ta lokutan rashin aiki.

Hoto 5.12 yana kwatanta manufar windows zaman. Karamin zaman zai hade tare da zaman zuwa hagunsa. Kuma zaman da ke hannun dama zai zama daban saboda ya biyo bayan dogon lokaci na rashin aiki. Dogon zama yana dogara ne akan ayyukan mai amfani, amma yi amfani da tambarin kwanan wata/lokaci daga shigarwar don sanin wane zaman ne shigarwar.

Amfani da windows zaman don bin diddigin ma'amalar haja

Bari mu yi amfani da tagogin zama don ɗaukar bayanai game da mu'amalar musayar. Ana nuna aiwatar da windows na zama a cikin Lissafi 5.5 (wanda za'a iya samuwa a cikin src/main/java/bbejeck/chapter_5/CountingWindowingAndKTableJoinExample.java).

Kun riga kun ga yawancin ayyukan da ake gudanarwa a cikin wannan topology, don haka babu buƙatar sake duba su anan. Amma kuma akwai sabbin abubuwa da yawa a nan, waɗanda za mu tattauna yanzu.

Duk wani aiki na rukuniBy yawanci yana yin wasu nau'ikan ayyukan tarawa (ƙara, juzu'i, ko ƙidayawa). Kuna iya yin ko dai tarawa tare da jimlar gudu, ko tagar taga, wanda ke yin la'akari da bayanan cikin ƙayyadadden taga.

Lambar da ke cikin Lissafi 5.5 tana ƙidaya adadin ma'amaloli a cikin windows zaman. A cikin siffa. 5.13 ana nazarin waɗannan ayyukan mataki-mataki.

Ta hanyar kiran windowedBy(SessionWindows.with(20Seconds)) har zuwa (Minuti goma sha biyar) zamu ƙirƙiri taga zaman tare da tazarar rashin aiki na daƙiƙa 15 da tazarar dagewa na mintuna 20. Tazarar rashin aiki na daƙiƙa 20 yana nufin cewa aikace-aikacen zai haɗa da duk wani shigarwa da ya zo cikin daƙiƙa XNUMX na ƙarshen ko farkon zaman na yanzu cikin zaman (aiki) na yanzu.

Na gaba, mun ƙayyade abin da aikin tarawa ya buƙaci a yi a cikin taga zaman - a wannan yanayin, ƙidaya. Idan shigarwa mai shigowa ta faɗi a wajen taga rashin aiki (kowanne gefen tambarin kwanan wata/lokaci), aikace-aikacen yana ƙirƙirar sabon zama. Tazarar riƙewa yana nufin kiyaye zama na ɗan lokaci kuma yana ba da damar bayanan marigayi waɗanda suka wuce lokacin rashin aiki na zaman amma har yanzu ana iya haɗa su. Bugu da ƙari, farawa da ƙarshen sabon zaman da aka samu sakamakon haɗuwa sun dace da farkon da sabon tambarin kwanan wata/lokaci.

Bari mu kalli ƴan shigarwar daga hanyar ƙidayar don ganin yadda zaman ke aiki (Table 5.1).

Lokacin da bayanai suka zo, muna neman zaman da ke da maɓalli iri ɗaya, ƙarshen lokacin ƙasa da tambarin kwanan wata/lokaci na yanzu - tazarar rashin aiki, da lokacin farawa mafi girma fiye da kwanan wata/tambarin lokaci na yanzu + tazarar rashin aiki. Yin la'akari da wannan, shigarwa huɗu daga tebur. 5.1 an haɗa su cikin zama ɗaya kamar haka.

1. Rikodi 1 ya zo da farko, don haka lokacin farawa daidai yake da lokacin ƙarshe kuma shine 00:00:00.

2. Na gaba, shigarwa 2 ya zo, kuma muna neman zaman da ya ƙare ba a baya fiye da 23:59:55 ba kuma farawa ba daga baya ba fiye da 00:00:35. Mun sami rikodin 1 kuma muna haɗa zaman 1 da 2. Muna ɗaukar lokacin farawa na zama na 1 (da farko) da ƙarshen lokacin 2 (daga baya), don sabon zaman mu ya fara a 00: 00: 00 kuma ya ƙare a 00: 00:15.

3. Rikodi 3 ya zo, muna neman zaman tsakanin 00:00:30 da 00:01:10 kuma ba mu sami wani ba. Ƙara zama na biyu don maɓallin 123-345-654,FFBE, farawa da ƙarewa a 00:00:50.

4. Rikodi 4 ya zo kuma muna neman zaman tsakanin 23:59:45 da 00:00:25. Wannan lokacin ana samun duka zaman 1 da 2. Dukan zaman guda uku an haɗa su cikin ɗaya, with a start time of 00:00:00 da ƙarshen lokacin 00:00:15.

Daga abin da aka bayyana a cikin wannan sashe, yana da daraja tunawa da waɗannan mahimman nuances:

• zaman ba tsayayyen tagogi ba ne. Tsawon lokacin zaman yana ƙaddara ta aikin a cikin wani lokacin da aka ba da shi;
• Tambarin kwanan wata/lokaci a cikin bayanan yana ƙayyade ko taron ya faɗi a cikin zaman da ake da shi ko kuma lokacin zaman banza.

Na gaba za mu tattauna nau'in taga na gaba - windows "tumbling".

"Tsarin" windows

Tumbling taga yana ɗaukar abubuwan da suka faɗi cikin wani ɗan lokaci. Ka yi tunanin cewa kana buƙatar kama duk ma'amalar hannun jari na wani kamfani kowane sakan 20, don haka za ka tattara duk abubuwan da suka faru a lokacin. A ƙarshen tazarar daƙiƙa 20, taga yana jujjuya kuma yana matsawa zuwa sabon tazarar kallo na daƙiƙa 20. Hoto 5.14 ya kwatanta wannan yanayin.

Kamar yadda kake gani, duk abubuwan da aka karɓa a cikin daƙiƙa 20 na ƙarshe ana haɗa su a cikin taga. A ƙarshen wannan lokacin, an ƙirƙiri sabuwar taga.

Lissafin 5.6 yana nuna lambar da ke nuna amfani da tagogi masu taurin kai don ɗaukar hada-hadar hannun jari kowane daƙiƙa 20 (wanda aka samo a src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Tare da wannan ɗan ƙaramin canji zuwa kiran hanyar TimeWindows.na, zaku iya amfani da taga tumbling. Wannan misalin baya kiran hanyar har sai(), don haka za'a yi amfani da tazarar riƙon tsoho na sa'o'i 24.

A ƙarshe, lokaci ya yi don matsawa zuwa ƙarshen zaɓuɓɓukan taga - windows "hopping".

Gilashin zamiya ("tsalle")

Gilashin zamewa/hopping suna kama da tagogin tudu, amma tare da ɗan bambanci. Gilashin zamewa ba sa jira har zuwa ƙarshen tazarar lokaci kafin ƙirƙirar sabuwar taga don aiwatar da abubuwan da suka faru kwanan nan. Suna fara sabon lissafin bayan tazarar jira ƙasa da tsawon lokacin taga.

Don kwatanta bambance-bambancen da ke tsakanin tagogi da tsalle-tsalle, bari mu koma ga misalin kirga mu'amalar haja. Burin mu har yanzu shine mu ƙidaya adadin ma'amaloli, amma ba ma so mu jira adadin lokaci gaba ɗaya kafin mu sabunta ma'ajin. Madadin haka, za mu sabunta ma'aunin a ɗan gajeren lokaci. Misali, har yanzu za mu kirga adadin ma’amaloli kowane dakika 20, amma sabunta ma’aunin kowane daƙiƙa 5, kamar yadda aka nuna a cikin siffa. 5.15. A wannan yanayin, mun ƙare tare da windows sakamako uku tare da bayanan da suka mamaye.

Lissafin 5.7 yana nuna lambar don ma'anar windows masu zamewa (wanda aka samo a src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Ana iya jujjuya taga tagulla zuwa tagar hopping ta ƙara kira zuwa hanyar gabaBy(). A cikin misalin da aka nuna, tazarar ceto shine mintuna 15.

Kun ga a cikin wannan sashin yadda ake iyakance sakamakon tarawa zuwa windows lokaci. Musamman ma, ina so ku tuna abubuwa guda uku kamar haka:

• girman girman windows yana iyakance ba ta lokacin lokaci ba, amma ta aikin mai amfani;
• Gilashin “tumbling” suna ba da bayyani na abubuwan da suka faru a cikin ƙayyadaddun lokaci;
• Tsawon lokacin tsalle-tsalle yana da ƙayyadaddun lokaci, amma ana sabunta su akai-akai kuma yana iya ƙunsar shigarwar masu mamayewa a cikin dukkan windows.

Na gaba, za mu koyi yadda ake canza KTable zuwa KStream don haɗi.

5.3.3. Haɗa KStream da KTable abubuwa

A cikin Babi na 4, mun tattauna haɗa abubuwa biyu na KStream. Yanzu dole mu koyi yadda ake haɗa KTable da KStream. Ana iya buƙatar wannan don dalilai masu sauƙi masu zuwa. KStream rafi ne na rikodin, kuma KTable rafi ne na sabunta rikodin, amma wani lokacin kuna iya ƙara ƙarin mahallin zuwa rafin rikodin ta amfani da sabuntawa daga KTable.

Bari mu ɗauki bayanai kan adadin musayar hannun jari kuma mu haɗa su tare da labaran musayar hannun jari don masana'antu masu dacewa. Ga abin da kuke buƙatar yi don cimma wannan idan aka ba da lambar da kuke da ita.

1. Mayar da wani abu na KTable tare da bayanai akan adadin ma'amalar haja zuwa KStream, sannan maye gurbin maɓalli tare da maɓalli mai nuna ɓangaren masana'antu wanda ya dace da wannan alamar haja.
2. Ƙirƙiri wani abu na KTable wanda ke karanta bayanai daga wani batu tare da labaran musayar hannun jari. Wannan sabon KTable za a rarraba ta bangaren masana'antu.
3. Haɗa sabuntawar labarai tare da bayani kan adadin musayar hannun jari ta bangaren masana'antu.

Yanzu bari mu ga yadda za a aiwatar da wannan shirin aiki.

Maida KTable zuwa KStream

Don canza KTable zuwa KStream kuna buƙatar yin haka.

1. Kira hanyar KTable.toStream().
2. Ta hanyar kiran hanyar KStream.map, maye gurbin maɓalli tare da sunan masana'antu, sa'an nan kuma dawo da abin da ake kira TransactionSummary daga misalin Windowed.

Za mu haɗa waɗannan ayyuka tare kamar haka (ana iya samun lambar a cikin fayil ɗin src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Jeri 5.8).

Saboda muna yin aikin KStream.map, misalin KStream da aka dawo ana sake raba shi ta atomatik lokacin da aka yi amfani da shi a cikin haɗin gwiwa.

Mun kammala tsarin jujjuyawar, na gaba muna buƙatar ƙirƙirar abu na KTable don karanta labarai na jari.

Ƙirƙirar KTable don labaran jari

Abin farin ciki, ƙirƙirar abin KTable yana ɗaukar layi ɗaya kawai na lamba (ana iya samun lambar a src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Jeri 5.9).

Yana da mahimmanci a lura cewa babu abubuwan Serde da ake buƙata don ƙayyade, tunda ana amfani da kirtani Serdes a cikin saitunan. Hakanan, ta amfani da ƙididdige FARKO, tebur yana cike da bayanai a farkon farkon.

Yanzu za mu iya ci gaba zuwa mataki na ƙarshe - haɗi.

Haɗa sabuntawar labarai tare da ƙidayar ma'amala

Ƙirƙirar haɗi ba shi da wahala. Za mu yi amfani da haɗin hagu idan babu labaran jari don masana'antun da suka dace (ana iya samun lambar da ake bukata a cikin fayil src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Jeri 5.10).

Wannan ma'aikacin leftJoin yana da sauƙi. Ba kamar haɗin kai a Babi na 4 ba, ba a amfani da hanyar JoinWindow saboda lokacin yin haɗin KStream-KTable, shigarwa ɗaya ne kawai a cikin KTable ga kowane maɓalli. Irin wannan haɗin ba a iyakance a cikin lokaci ba: rikodin yana cikin KTable ko babu. Babban ƙarshe: ta amfani da abubuwan KTable zaku iya wadatar da KStream tare da ƙarancin sabunta bayanan tunani akai-akai.

Yanzu za mu kalli ingantacciyar hanya don wadatar abubuwa daga KStream.

5.3.4. Abubuwan GlobalKTable

Kamar yadda kuke gani, akwai buƙatar haɓaka rafukan taron ko ƙara mahallin zuwa gare su. A cikin Babi na 4 kun ga haɗin kai tsakanin abubuwa biyu na KStream, kuma a cikin sashin da ya gabata kun ga haɗin tsakanin KStream da KTable. A duk waɗannan lokuta, ya zama dole a sake raba raƙuman bayanai lokacin da ake tsara maɓallan zuwa wani sabon nau'i ko ƙima. Wani lokaci reparting ana yin shi a bayyane, kuma wani lokacin Kafka Streams yana yin ta ta atomatik. Sake rarrabawa yana da mahimmanci saboda maɓallan sun canza kuma dole ne rikodin su ƙare a cikin sababbin sassan, in ba haka ba haɗin ba zai yiwu ba (an tattauna wannan a Babi na 4, a cikin sashin "Sake rarraba bayanai" a cikin sashe na 4.2.4).

Sake rarrabawa yana da tsada

Sake rarrabawa yana buƙatar farashi - ƙarin farashin albarkatun don ƙirƙirar batutuwa masu tsaka-tsaki, adana kwafin bayanai a cikin wani batu; kuma yana nufin ƙara jinkiri saboda rubutu da karatu daga wannan batu. Bugu da ƙari, idan kuna buƙatar haɗawa a cikin fiye da ɗaya fanni ko girma, dole ne ku ɗaure haɗin haɗin, taswirar bayanan tare da sababbin maɓallai, sannan sake aiwatar da tsarin sake rarrabawa.

Haɗa zuwa ƙananan bayanan bayanai

A wasu lokuta, ƙarar bayanan tunani da za a haɗa ba su da ɗanɗano, don haka cikakkun kwafinsa na iya dacewa da gida cikin sauƙi a kowane kulli. Don yanayi irin wannan, Kafka Streams yana ba da ajin GlobalKTable.

Alamomin GlobalKTable na musamman ne saboda aikace-aikacen yana kwafin duk bayanai zuwa kowane kumburi. Kuma tun da duk bayanan suna nan akan kowane kulli, babu buƙatar raba rafin taron ta hanyar maɓallin bayanai ta yadda ya kasance ga dukkan ɓangarori. Hakanan zaka iya yin haɗin mara waya ta amfani da abubuwan GlobalKTable. Bari mu koma ɗaya daga cikin misalan da suka gabata don nuna wannan fasalin.

Haɗa abubuwan KStream zuwa abubuwan GlobalKTable

A cikin sashe na 5.3.2, mun aiwatar da tari ta taga na mu'amalar musanya ta masu siye. Sakamakon wannan taro yayi kama da haka:

{customerId='074-09-3705', stockTicker='GUTM'}, 17
{customerId='037-34-5184', stockTicker='CORK'}, 16

Duk da yake waɗannan sakamakon sun yi amfani da manufar, da zai fi amfani idan an nuna sunan abokin ciniki da cikakken sunan kamfani. Don ƙara sunan abokin ciniki da sunan kamfani, kuna iya yin haɗin kai na yau da kullun, amma kuna buƙatar yin taswirar maɓalli biyu da sake rarrabawa. Tare da GlobalKTable za ku iya guje wa farashin irin waɗannan ayyuka.

Don yin wannan, za mu yi amfani da abin countStream daga Lissafin 5.11 (ana iya samun madaidaicin lambar a src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) kuma mu haɗa shi zuwa abubuwa na GlobalKTable guda biyu.

Mun riga mun tattauna wannan a baya, don haka ba zan maimaita ba. Amma na lura cewa lambar da ke cikin toStream () aikin taswira an ɓoye shi cikin wani abu mai aiki maimakon lambda na layi don kare iya karantawa.

Mataki na gaba shine bayyana misalai biyu na GlobalKTable (ana iya samun lambar da aka nuna a cikin fayil src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Jeri 5.12).

Lura cewa ana bayyana sunayen jigo ta amfani da ƙididdiga iri.

Yanzu da muke da duk abubuwan da aka shirya, abin da ya rage shine rubuta lambar don haɗin (wanda za'a iya samuwa a cikin fayil ɗin src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Jerin 5.13).

Ko da yake akwai ƙungiyoyi biyu a cikin wannan lambar, an ɗaure su saboda ba a yi amfani da sakamakonsu daban ba. Ana nuna sakamakon a ƙarshen dukan aikin.

Lokacin da kuka gudanar da aikin haɗin gwiwa na sama, zaku sami sakamako kamar haka:

{customer='Barney, Smith' company="Exxon", transactions= 17}

Mahimmancin bai canza ba, amma waɗannan sakamakon sun fi fitowa fili.

Idan kun ƙidaya zuwa Babi na 4, kun riga kun ga nau'ikan haɗin gwiwa da yawa suna aiki. An jera su a cikin tebur. 5.2. Wannan tebur yana nuna damar haɗin kai kamar nau'in 1.0.0 na Rafukan Kafka; Wani abu na iya canzawa a cikin fitowar gaba.

Don tattara abubuwa, bari mu sake fasalin abubuwan yau da kullun: zaku iya haɗa rafukan taron (KStream) da sabunta rafukan (KTable) ta amfani da jihar gida. A madadin, idan girman bayanan bayanan bai yi girma ba, kuna iya amfani da abin GlobalKTable. GlobalKTables suna kwafi duk ɓangarori zuwa kowane kullin aikace-aikacen Rafukan Kafka, yana tabbatar da cewa duk bayanan suna samuwa ba tare da la'akari da wane ɓangaren maɓalli ya dace ba.

Na gaba za mu ga fasalin Kafka Streams, godiya ga wanda za mu iya lura da canje-canje na jihohi ba tare da cinye bayanai daga batun Kafka ba.

5.3.5. Halin da ake nema

Mun riga mun aiwatar da ayyuka da yawa da suka shafi jihar kuma koyaushe muna fitar da sakamakon zuwa na'ura mai ba da hanya tsakanin hanyoyin sadarwa (don dalilai na haɓakawa) ko rubuta su zuwa wani batu (don dalilai na samarwa). Lokacin rubuta sakamako zuwa wani batu, dole ne ka yi amfani da mabukaci na Kafka don duba su.

Ana iya ɗaukar bayanan karanta waɗannan batutuwa a matsayin nau'in ra'ayi na zahiri. Don dalilanmu, za mu iya amfani da ma'anar ra'ayi na zahiri daga Wikipedia: “... abu na zahiri da ke ɗauke da sakamakon tambaya. Misali, yana iya zama kwafin bayanan nesa, ko juzu'in layuka da/ko ginshiƙan tebur ko haɗa sakamako, ko taƙaitaccen tebur da aka samu ta hanyar tarawa" (https://en.wikipedia.org/wiki /Materialized_view).

Kafka Streams kuma yana ba ku damar gudanar da tambayoyin mu'amala akan shagunan jihohi, yana ba ku damar karanta waɗannan ra'ayoyin zahiri. Yana da mahimmanci a lura cewa tambayar zuwa kantin sayar da kayayyaki aiki ne na karantawa kawai. Wannan yana tabbatar da cewa ba lallai ne ku damu ba da gangan yin rashin daidaituwa a jihar yayin da aikace-aikacenku ke sarrafa bayanai.

Ikon tambayar shagunan jihar kai tsaye yana da mahimmanci. Wannan yana nufin cewa zaku iya ƙirƙirar aikace-aikacen dashboard ba tare da fara samo bayanai daga mabukacin Kafka ba. Hakanan yana ƙara haɓaka aikin aikace-aikacen, saboda gaskiyar cewa babu buƙatar sake rubuta bayanai:

• godiya ga wurin wurin bayanan, ana iya isa gare su da sauri;
• Ana kawar da kwafin bayanai, tunda ba a rubuta shi zuwa ma'ajiyar waje ba.

Babban abin da nake so ku tuna shine zaku iya yin bayani kai tsaye daga cikin aikace-aikacenku. Ba za a iya faɗi damar da wannan damar ke ba ku ba. Maimakon cinye bayanai daga Kafka da adana bayanai a cikin ma'ajin bayanai don aikace-aikacen, zaku iya tambayar shagunan jihohi tare da sakamako iri ɗaya. Tambayoyi kai tsaye zuwa shagunan jihohi suna nufin ƙarancin lamba (babu mabukaci) da ƙasan software (babu buƙatar tebur bayanai don adana sakamakon).

Mun yi bayani kadan a cikin wannan babin, don haka za mu bar zancenmu na tambayoyin mu'amala da shagunan jihohi a yanzu. Amma kada ku damu: a Babi na 9, za mu ƙirƙiri aikace-aikacen dashboard mai sauƙi tare da tambayoyin hulɗa. Zai yi amfani da wasu misalan wannan da surori da suka gabata don nuna tambayoyin hulɗa da yadda za ku iya ƙara su zuwa aikace-aikacen Rafukan Kafka.

Takaitaccen

• Abubuwan KStream suna wakiltar rafukan abubuwan da suka faru, kwatankwacin sakawa cikin bayanan bayanai. Abubuwan KTable suna wakiltar rafukan ɗaukaka, kama da sabuntawa zuwa bayanan bayanai. Girman abin KTable ba ya girma, ana maye gurbin tsoffin bayanan da sababbi.
• Ana buƙatar abubuwan KTable don ayyukan tarawa.
• Yin amfani da ayyukan taga, zaku iya raba haɗe-haɗen bayanai zuwa buckets na lokaci.
• Godiya ga abubuwan GlobalKTable, zaku iya samun damar bayanan tunani a ko'ina cikin aikace-aikacen, ba tare da la'akari da rarrabawa ba.
• Haɗi tsakanin KStream, KTable da GlobalKTable abubuwa mai yiwuwa ne.

Ya zuwa yanzu, mun mai da hankali kan gina aikace-aikacen Rafukan Kafka ta amfani da babban matakin KStream DSL. Kodayake babban matakin yana ba ku damar ƙirƙirar shirye-shirye masu tsafta da ƙayyadaddun tsari, yin amfani da shi yana wakiltar ciniki. Yin aiki tare da DSL KStream yana nufin haɓaka taƙaitaccen lambar ku ta rage ƙimar sarrafawa. A babi na gaba, za mu kalli API ɗin ƙaramar mai kula da node kuma mu gwada sauran cinikin. Shirye-shiryen za su yi tsayi fiye da yadda suke a da, amma za mu iya ƙirƙirar kusan kowane kullin mai sarrafa da za mu iya buƙata.

→ Ana iya samun ƙarin bayani game da littafin a gidan yanar gizon mawallafi

→ Don Habrozhiteli 25% rangwame ta amfani da coupon - Kafka Streams

→ Bayan biyan kuɗin sigar takarda na littafin, za a aika da littafin lantarki ta imel.

source: www.habr.com