Bhuku rinonzi "Kafka Streams in Action. Zvishandiso uye microservices yebasa renguva chaiyo"

Mhoroi, vagari veKhabro! Iri bhuku rakakodzera chero mugadziri anoda kunzwisisa shinda kugadzirisa. Kunzwisisa yakagoverwa programming ichakubatsira iwe kunzwisisa zviri nani Kafka uye Kafka Streams. Zvingave zvakanaka kuziva iyo Kafka chimiro pachayo, asi izvi hazvidiwi: Ini ndichakuudza zvese zvaunoda. Vane ruzivo veKafka vanogadzira uye novices zvakafanana vachadzidza kugadzira inonakidza rukova kugadzirisa maapplication vachishandisa Kafka Streams raibhurari mubhuku rino. Vepakati uye vepamberi vanogadzira Java vanotojairana nemaconcepts serialization vachadzidza kushandisa hunyanzvi hwavo kugadzira Kafka Streams application. Iro bhuku rekushandisa kodhi rakanyorwa muJava 8 uye rinoshandisa zvakanyanya Java 8 lambda kutaura syntax, saka kuziva maitiro ekushanda nelambda mabasa (kunyangwe mune mumwe mutauro wechirongwa) kuchauya zvine mutsindo.

Excerpt. 5.3. Aggregation uye windowsing mashandiro

Muchikamu chino, tichaenderera mberi nekuongorora zvikamu zvinonyanya kuvimbiswa zveKafka Streams. Parizvino takafukidza zvinotevera zvikamu zveKafka Streams:

• kugadzira topology yekugadzirisa;
• kushandisa nyika mukutepfenyura maapplication;
• kuita data stream connections;
• misiyano pakati pezviitiko nzizi (KStream) nekuvandudza hova (KTable).

Mumienzaniso inotevera tichaunza zvinhu izvi zvese pamwechete. Iwe zvakare unozodzidza nezve windowsing, chimwe chinhu chikuru chekushambadzira maapplication. Muenzaniso wedu wekutanga uchave uri nyore kuunganidza.

5.3.1. Kuunganidzwa kwekutengeswa kwemasheya neindasitiri chikamu

Aggregation uye mapoka maturusi akakosha paunenge uchishanda nekutepfenyura data. Kuongororwa kwemarekodhi ega ega sezvaanogamuchirwa kazhinji hakuna kukwana. Kuti utore rumwe ruzivo kubva kune data, zvinodikanwa kuunganidza uye kusanganisa ivo.

Mumuenzaniso uyu, iwe uchapfekedza chipfeko chemutengesi wezuva anoda kuteedzera huwandu hwekutengesa kwemasheya emakambani mumaindasitiri akati wandei. Kunyanya, iwe unofarira makambani mashanu ane hombe kutengeswa kwemasheya muindasitiri yega yega.

Kuunganidzwa kwakadaro kunoda matanho akati wandei anotevera kuti ashandure data muchimiro chaunoda (kutaura mune zvakajairika).

1. Gadzira chinyorwa-chinobva mumusoro chinoburitsa ruzivo rwekutengesa masheya. Isu tichafanirwa kumepu chinhu chemhando StockTransaction kune chinhu cherudzi ShareVolume. Pfungwa ndeyokuti StockTransaction chinhu chine metadata yekutengesa, asi isu tinongoda data pamusoro pehuwandu hwezvikamu zviri kutengeswa.
2. Group ShareVolume data nestock chiratidzo. Kana yangoiswa muboka nechiratidzo, unogona kudonhedza iyi data kuita subtotals yemasheya ekutengesa mavhoriyamu. Zvakakosha kuziva kuti nzira yeKStream.groupBy inodzosa muenzaniso werudzi rweKGroupedStream. Uye iwe unogona kuwana muenzaniso weKTable nekuwedzera kudaidza iyo KGroupedStream.reduce nzira.

Chii chinonzi KGroupedStream interface

Nzira dzeKStream.groupBy neKStream.groupByKey dzinodzorera muenzaniso weKGroupedStream. KGroupedStream inomiririra yepakati yerukova rwezviitiko mushure mekuisa mapoka nemakiyi. Hazvina zvachose zvakagadzirirwa kushanda zvakananga nazvo. Pane kudaro, KGroupedStream inoshandiswa pakuunganidza mashandiro, ayo anogara achikonzera KTable. Uye sezvo mhedzisiro yeaggregation mashandiro ari KTable uye vanoshandisa chitoro chehurumende, zvinogoneka kuti hazvisi zvese zvinogadziridzwa semhedzisiro zvinotumirwa zvakare pasi pombi.

Iyo KTable.groupBy nzira inodzosa yakafanana KGroupedTable - inomiririra yepakati yerukova rwezvigadziriso, zvakabatanidzwa nekiyi.

Ngatimbozororai zvishoma uye titarise Fig. 5.9, inoratidza zvatakawana. Iyi topology inofanirwa kunge yatove yakajairika kwauri.

Ngatitarisei kodhi yeiyi topology (inogona kuwanikwa mufaira src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Listing 5.2).

Kodhi yakapihwa inosiyaniswa nekupfupika kwayo uye huwandu hukuru hwezviito zvinoitwa mumitsetse yakati wandei. Iwe unogona kuona chimwe chinhu chitsva mune yekutanga parameter ye builder.stream method: kukosha kwe enum type AutoOffsetReset.EARLIEST (kunewo RATEST), yakagadzirirwa kushandisa Consumed.withOffsetResetPolicy nzira. Mhando yekuverenga iyi inogona kushandiswa kutsanangura nzira yekumisikidza patsva yeKStream yega yega kana KTable uye inotungamira pamusoro peiyo offset reset sarudzo kubva pakumisikidzwa.

GroupByKey uye GroupBy

Iyo KStream interface ine nzira mbiri dzekuisa marekodhi: GroupByKey uye GroupBy. Ose ari maviri anodzosera KGroupedTable, saka unogona kunge uchinetseka kuti musiyano uripi pakati pavo uye nguva yekushandisa ipi?

Nzira yeGroupByKey inoshandiswa kana makiyi muKStream atove asina chinhu. Uye zvinonyanya kukosha, iyo "inoda kupatsanurwazve" mureza hauna kumboiswa.

Iyo GroupBy nzira inofungidzira kuti wachinja makiyi emapoka, saka mureza wekugovera unoiswa kuti uve chokwadi. Kuita majoints, aggregations, etc. mushure meGroupBy nzira inozokonzera kugadzikazve otomatiki.
Pfupiso: Pese pazvinogoneka, unofanira kushandisa GroupByKey pane GroupBy.

Zvakajeka zvinoitwa nemapValues ​​​​uye bokaBy nzira, saka ngatitarisei iyo sum() nzira (inowanikwa mu src/main/java/bbejeck/model/ShareVolume.java) (Listing 5.3).

Iyo ShareVolume.sum nzira inodzosera iyo inomhanya yehuwandu hwekutengesa vhoriyamu, uye mhedzisiro yeketani yese yekuverenga chinhu cheKTable. . Zvino iwe unonzwisisa basa rinoitwa neKTable. Kana ShareVolume zvinhu zvasvika, iyo inoenderana KTable chinhu inochengeta yazvino yekuvandudza. Izvo zvakakosha kuyeuka kuti zvese zvinogadziridzwa zvinoratidzwa mune yapfuura shareVolumeKTable, asi kwete ese anotumirwa mberi.

Isu tinobva tashandisa iyi KTable kuunganidza (nenhamba yezvikamu zvakatengeswa) kusvika kumakambani mashanu ane mavhoriyamu epamusoro emasheya anotengeswa muindasitiri yega yega. Zviito zvedu munyaya iyi zvichafanana neizvo zvekutanga kuunganidza.

1. Ita rimwe bokaNekushanda kuunganidza munhu wega ShareVolume zvinhu neindasitiri.
2. Tanga kupfupisa ShareVolume zvinhu. Panguva ino chinhu chekuunganidza ndicho mutsara wehukuru hwakatarwa. Mumutsara wakasarudzika uyu, makambani mashanu chete ane huwandu hukuru hwemasheya akatengeswa anochengetwa.
3. Mepu mitsetse kubva pandima yapfuura kune kukosha kwetambo uye dzorera shanu dzepamusoro dzakatengeswa zvakanyanya nehuwandu neindasitiri.
4. Nyora zvabuda mutambo fomu kune musoro wenyaya.

Mumufananidzo. Mufananidzo 5.10 inoratidza iyo data flow topology graph. Sezvauri kuona, iyo yechipiri kutenderera kwekugadzirisa iri nyore.

Iye zvino zvatava nekunzwisisa kwakajeka kwechimiro chechikamu chechipiri chekugadzirisa, tinogona kutendeukira kune iyo kodhi kodhi (iwe uchaiwana mufaira src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Listing 5.4) .

Iyi yekutanga ine yakagadziriswaQueue inoshanduka. Ichi chinhu chakajairwa chinova adapta yejava.util.TreeSet inoshandiswa kuronda mibairo yeN yepamusoro mukudzikira kwezvikamu zvinotengeswa.

Wakatoona bokaBy uye mapValues ​​kufona, saka isu hatipinde mune idzodzo (tiri kudaidza nzira yeKTable.toStream nekuti nzira yeKTable.print yakadzikiswa). Asi iwe hausati waona iyo KTable vhezheni yeaggregate() parizvino, saka tichapedza nguva shoma tichikurukura izvozvo.

Sezvaunorangarira, chii chinoita kuti KTable isiyane ndechekuti marekodhi ane makiyi akafanana anoonekwa sekuvandudzwa. KTable inotsiva yekare yekupinda neitsva. Aggregation inoitika nenzira yakafanana: zvinyorwa zvitsva zvine kiyi imwechete zvakaunganidzwa. Kana rekodhi rasvika, rinowedzerwa kune FixedSizePriorityQueue kirasi muenzaniso uchishandisa adder (yechipiri parameter mu aggregate nzira yekufona), asi kana imwe rekodhi yatovepo nekiyi imwechete, ipapo rekodhi rekare rinobviswa uchishandisa subtractor (yechitatu parameter mu. iyo aggregate nzira yekufona).

Izvi zvese zvinoreva kuti aggregator yedu, FixedSizePriorityQueue, haiunganidzi ese kukosha nekiyi imwechete, asi inochengetedza huwandu hunofamba hwehuwandu hweN inonyanya kutengeswa mhando dzemasheya. Imwe neimwe inopinda ine nhamba yese yezvikamu zvakatengeswa kusvika zvino. KTable inokupa ruzivo rwekuti ndeapi makambani emasheya ari kunyanya kutengeswa, pasina kuda kuunganidzwa kwega yega yega.

Takadzidza kuita zvinhu zviviri zvakakosha:

• kukosha kweboka muKTable nekiyi yakajairwa;
• ita mabasa anobatsira akadai sekukungurusa uye kuunganidza pane izvi zvakaiswa mumapoka.

Kuziva kuita mashandiro aya kwakakosha kuti unzwisise zvinoreva data inofamba kuburikidza neKafka Streams application uye kunzwisisa kuti inotakura ruzivo rwupi.

Taunzawo pamwe chete mamwe epfungwa dzakakosha dzakurukurwa pakutanga mubhuku rino. MuChitsauko 4, takakurukura kuti kushivirira-kushivirira, mamiriro enzvimbo kwakakosha sei pakushambadzira application. Muenzaniso wekutanga muchitsauko chino wakaratidza chikonzero nei nyika yemuno yakakosha-inokupa kugona kuchengeta ruzivo rwawakamboona. Kuwanikwa kwenzvimbo kunodzivirira kunonoka kwenetiweki, zvichiita kuti application iwedzere kuita uye kusakanganiswa.

Paunenge uchiita chero rollup kana kuunganidza oparesheni, unofanirwa kutsanangura zita rechitoro chehurumende. Iyo rollup uye aggregation mashandiro anodzosera muenzaniso weKTable, uye KTable inoshandisa chengetedzo yenyika kutsiva zvekare mhedzisiro nemitsva. Sezvawaona, hazvisi zvese zvinogadziridzwa zvinotumirwa pasi pombi, uye izvi zvakakosha nekuti mabasa ekubatanidza akagadzirirwa kuburitsa ruzivo rwechidimbu. Kana ukasashandisa dunhu renzvimbo, KTable inoendesa mberi zvese zvakaunganidzwa uye rollup mhinduro.

Tevere, isu tichatarisa kuita maoparesheni akadai sekuunganidza mukati meimwe nguva yenguva - inonzi mahwindo ekuita.

5.3.2. Hwindi mabasa

Muchikamu chakapfuura, takaunza kutsvedza convolution uye kuunganidza. Chikumbiro chakaita kuenderera mberi kwehuwandu hwekutengeswa kwemasheya, kuchiteverwa nekuunganidzwa kwemasheya mashanu anotengeswa zvakanyanya pakuchinjana.

Dzimwe nguva kuenderera mberi kwekuunganidza uye roll-up yemhedzisiro inodiwa. Uye dzimwe nguva unofanirwa kuita maoparesheni chete pane imwe nguva yakatarwa. Semuyenzaniso, verenga kuti mangani matransactions akaitwa nemasheya eimwe kambani mumaminetsi gumi apfuura. Kana kuti vangani vashandisi vakadzvanya pabanner nyowani yekushambadza mumaminetsi gumi nemashanu apfuura. Chikumbiro chinogona kuita mashandiro akadai kakawanda, asi nemhedzisiro inoshanda kune yakatarwa nguva yenguva (nguva windows).

Kuverenga kuchinjanisa kutengeserana nemutengi

Mumuenzaniso unotevera, isu tichateedzera mastock transaction kune akawanda vatengesi-angave masangano makuru kana akangwara ega emari.

Pane zvikonzero zviviri zvinogoneka zvekutevera uku. Imwe yacho kudiwa kwekuziva izvo vatungamiriri vemusika vari kutenga / kutengesa. Kana vatambi vakuru ava uye vanoisa mari vakanyanya kuona mukana, zvine musoro kutevera zano ravo. Chikonzero chechipiri chishuwo chekuona chero zviratidzo zvinogoneka zvekutengesa kwemukati zvisiri pamutemo. Kuti uite izvi, iwe unozofanirwa kuongorora kuwirirana kwemahombe ekutengesa spikes ane zvakakosha kuburitswa kwenhau.

Kutevera kwakadaro kunosanganisira matanho anotevera:

• kugadzira rwizi rwekuverenga kubva kune stock-transactions musoro;
• kuisa mapoka marekodhi nevatengi ID uye chiratidzo chemasheya. Kufonera bokaNenzira kunodzosa muenzaniso wekirasi yeKGroupedStream;
• Iyo KGroupedStream.windowedBy nzira inodzosa kuyerera kwedata kune imwe nguva hwindo, iyo inobvumira kuunganidzwa kwehwindo. Zvichienderana nerudzi rwehwindo, ingave TimeWindowedKStream kana SessionWindowedKStream inodzoswa;
• transaction count yeaggregation operation. Kuyerera kwedhata kwakavharidzirwa kunotarisisa kana rekodhi yakati inoverengerwa mukuverenga uku;
• kunyora mhinduro kune imwe nyaya kana kuiburitsa kune iyo console panguva yekusimudzira.

Iyo topology yechishandiso ichi iri nyore, asi mufananidzo wakajeka wayo ungabatsira. Ngatitarisei Fig. 5.11.

Zvadaro, tichatarisa kushanda kwekushanda kwehwindo uye kodhi inoenderana.

Mahwindo marudzi

Kune marudzi matatu emahwindo muKafka Streams:

• sessional;
• “kudonha” (kudonha);
• kutsvedza/kusvetuka.

Ndeipi yekusarudza zvinoenderana nezvinodiwa zvebhizinesi rako. Kudonha uye kusvetuka windows inguva-inogumira, nepo mahwindo echikamu anoganhurirwa nebasa remushandisi-nguva yechikamu (s) inotarwa chete nemabatiro anoita mushandisi. Chinhu chikuru chekurangarira ndechekuti marudzi ese emahwindo anoenderana nezuva/nguva zvitambi zvezvinyorwa, kwete nguva yehurongwa.

Tevere, isu tinoshandisa topology yedu neimwe yemhando dzehwindo. Iyo kodhi yakazara ichapihwa chete mumuenzaniso wekutanga; kune mamwe marudzi emahwindo hapana chinochinja kunze kwerudzi rwekushanda kwehwindo.

Session windows

Session mahwindo akasiyana zvakanyanya nemamwe marudzi ese emahwindo. Iwo anoganhurirwa kwete zvakanyanya nenguva senge nebasa remushandisi (kana chiitiko chesangano chaungade kutevedzera). Session windows inoganhurwa nenguva dzekusaita basa.

Mufananidzo 5.12 unoratidza pfungwa yechikamu windows. Chikamu chidiki chinozobatana nechikamu kuruboshwe. Uye chikamu chiri kurudyi chichave chakaparadzana nekuti chinotevera nguva refu yekusaita basa. Session windows yakavakirwa pakuita kwemushandisi, asi shandisa zvitambi zvemazuva/nguva kubva pane zvinonyorwa kuti uone kuti chikamu ndechepi.

Kushandisa sesheni windows kutevedzera stock transaction

Ngatishandisei mahwindo echikamu kutora ruzivo nezvekuchinjana kwekuchinjana. Kuitwa kwesesheni windows kunoratidzwa muKunyora 5.5 (iyo inogona kuwanikwa mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKTableJoinExample.java).

Iwe watoona akawanda maoparesheni mune iyi topology, saka hapana chikonzero chekuvatarisa zvakare pano. Asi kune zvakare akati wandei zvinhu zvitsva pano, izvo zvatichakurukura zvino.

Chero bokaBy oparesheni rinowanzoita imwe mhando yeaggregation oparesheni (kuunganidza, rollup, kana kuverenga). Iwe unogona kuita chero kuunganidza kuunganidza nehuwandu hwekumhanya, kana hwindo kuunganidzwa, izvo zvinotora muakaundi marekodhi mukati menguva yakatarwa hwindo.

Iyo kodhi muChinyorwa 5.5 inoverenga nhamba yekutengeserana mukati mechikamu windows. Mumufananidzo. 5.13 zviito izvi zvinoongororwa nhanho nhanho.

Nekufonera windowedBy(SessionWindows.with(twentySeconds).kusvika(fifteenMinutes)) tinogadzira hwindo reseshini ine nguva yekusaita basa yemasekonzi makumi maviri uye nekufamba kwemaminetsi gumi nemashanu. Nguva isina basa yemasekonzi makumi maviri inoreva kuti chikumbiro chinosanganisa chero chinopinda chinosvika mukati memasekonzi makumi maviri ekupera kana kutanga kwechikamu chazvino muchikamu chazvino (chinoshanda).

Tevere, isu tinotsanangura kuti ndeipi mashandiro ekubatanidza anofanirwa kuitwa muhwindo rechikamu - mune iyi kesi, kuverenga. Kana inopinda ichipinda ichiwira kunze kwehwindo rekusaita (rimwe divi rezuva / nguva chitambi), chishandiso chinogadzira chikamu chitsva. Nguva yekuchengeta inoreva kuchengetedza nguva yenguva yakati uye inobvumira data yanonoka inoenderera kupfuura nguva yekusaita chikamu asi inogona kubatanidzwa. Pamusoro pezvo, kutanga uye kupera kwechikamu chitsva chinobva mukubatanidzwa kunoenderana neyekutanga uye yazvino chitambi chemazuva / nguva.

Ngatitarisei zvinyorwa zvishoma kubva munzira yekuverenga kuti tione kuti zvikamu zvinoshanda sei (Tafura 5.1).

Kana marekodhi asvika, tinotarisa zvirongwa zviripo nekiyi imwe chete, nguva yekupedzisira isingasviki zuva razvino / chitambi chenguva - kusaita basa, uye nguva yekutanga yakakura kupfuura yazvino zuva / nguva chitambi + kusashanda nguva. Tichifunga izvi, zvinyorwa zvina kubva patafura. 5.1 inosanganiswa kuita chikamu chimwe seinotevera.

1. Rekodha 1 inosvika pakutanga, saka nguva yekutanga yakaenzana nenguva yekupedzisira uye 00:00:00.

2. Tevere, kupinda 2 kunosvika, uye tinotarisa zvirongwa zvinopera kwete kare kupfuura 23:59:55 uye kutanga kwete gare gare 00:00:35. Tinowana rekodhi 1 uye tinobatanidza zvikamu 1 uye 2. Tinotora nguva yekutanga yechikamu 1 (kare) uye nguva yekupedzisira yechikamu 2 (gare gare), kuitira kuti purogiramu yedu itsva itange pa 00: 00: 00 uye inopera pa00: 00:15.

3. Rekodha 3 inosvika, tinotarisa zvirongwa pakati pe00:00:30 uye 00:01:10 uye hatiwani chero. Wedzera chikamu chechipiri chekiyi 123-345-654,FFBE, kutanga uye kupera na 00:00:50.

4. Rekodha 4 inosvika uye tiri kutsvaga zvirongwa pakati pa23:59:45 na00:00:25. Panguva ino zvose zviri zviviri zvikamu 1 uye 2. Zvirongwa zvose zviri zvitatu zvinosanganiswa kuita chimwe, nenguva yekutanga 00:00:00 uye nguva yekupedzisira ye 00:00:15.

Kubva pane zvinotsanangurwa muchikamu chino, zvakakosha kuyeuka zvinotevera zvakakosha nuances:

• zvikamu hazvina kusimbiswa-saizi windows. Nguva yechikamu inotarwa nechiitwa mukati menguva yakatarwa;
• Iyo yemazuva / nguva zvitambi mune data inotarisisa kuti chiitiko chinowira mukati mechikamu chiripo here kana panguva isina basa.

Zvadaro tichakurukura rudzi runotevera rwehwindo - "kudonha" mahwindo.

"Kudonha" mahwindo

Mahwindo ekudonha anotora zviitiko zvinowira mukati meimwe nguva yenguva. Fungidzira kuti iwe unofanirwa kutora ese mastock transaction eimwe kambani masekonzi makumi maviri ega ega, saka unounganidza zvese zviitiko panguva iyoyo yenguva. Pakupera kwe20-second interval, hwindo rinotenderera pamusoro uye rinoenda kune imwe itsva 20-sekondi yekucherechedza nguva. Mufananidzo 20 unoratidza mamiriro ezvinhu aya.

Sezvauri kuona, zviitiko zvese zvakagamuchirwa mumasekonzi makumi maviri ekupedzisira zvinosanganisirwa pahwindo. Pakupera kwenguva ino yenguva, hwindo idzva rinogadzirwa.

Kunyora 5.6 kunoratidza kodeti inoratidza kushandiswa kwemahwindo ekudonha kutora kutengeswa kwemasheya ose e20 seconds (inowanikwa mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Nekuchinja kudiki uku kuTimeWindows.of method call, unogona kushandisa hwindo rinodonha. Uyu muenzaniso haudaidze iyo kusvika () nzira, saka iyo yekusagadzika yekuvharira yemaawa makumi maviri nemana ichashandiswa.

Pakupedzisira, inguva yekuenda kune yekupedzisira yezvisarudzo zvehwindo - "hopping" windows.

Kutsvedza ("kusvetuka") mahwindo

Mahwindo ekutsvedza/kusvetuka akafanana nemahwindo anodonha, asi aine musiyano mudiki. Kutsvedza windows usamirire kusvika pakupera kwenguva usati wagadzira hwindo nyowani kugadzirisa zviitiko zvichangoburwa. Vanotanga kuverenga kutsva mushure menguva yekumirira isingasviki nguva yehwindo.

Kuenzanisira mutsauko pakati pekudonha nekusvetuka mahwindo, ngatidzokere kumuenzaniso wekuverenga mastock exchange transaction. Chinangwa chedu chichiri chekuverenga huwandu hwekutengeserana, asi isu hatidi kumirira huwandu hwese hwenguva tisati tagadziridza counter. Pane kudaro, isu tichagadzirisa counter munguva pfupi. Semuenzaniso, ticharamba tichiverenga nhamba yekutengeserana masekonzi makumi maviri ega ega, asi gadziridza counter masekonzi mashanu ega ega, sezvakaratidzwa muFig. 20. Muchiitiko ichi, tinopedzisira tave nemahwindo matatu emhedzisiro ane data rinopindirana.

Chinyorwa 5.7 chinoratidza kodhi yekutsanangura mahwindo ekutsvedza (inowanikwa mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Hwindo rinodonha rinogona kushandurwa kuita hwindo rekusvetuka nekuwedzera kufona kune advanceBy() nzira. Mumuenzaniso wakaratidzwa, nguva yekuchengetedza ndeye maminitsi gumi nemashanu.

Iwe wakaona muchikamu chino maitiro ekudzikamisa mibairo yekuunganidza kune nguva windows. Kunyanya, ndinoda kuti urangarire zvinhu zvitatu zvinotevera kubva muchikamu chino:

• saizi yechikamu windows inogumira kwete nenguva, asi nebasa remushandisi;
• "kudonha" mahwindo anopa mhedziso yezviitiko mukati menguva yakatarwa;
• Nguva yekusvetuka mahwindo yakagadziriswa, asi inovandudzwa nguva nenguva uye inogona kunge iine zvinopindirana mumahwindo ese.

Tevere, isu tichadzidza nzira yekushandura KTable kudzokera kuKStream yekubatanidza.

5.3.3. Kubatanidza KStream uye KTable zvinhu

MuChitsauko 4, takakurukura kubatanidza zvinhu zviviri zveKStream. Iye zvino tinofanira kudzidza nzira yekubatanidza KTable uye KStream. Izvi zvinogona kudikanwa nekuda kwechikonzero chinotevera chiri nyore. KStream irukova rwemarekodhi, uye KTable irukova rwemarekodhi ekugadzirisa, asi dzimwe nguva ungangoda kuwedzera mamwe mamiriro kune rekodhi rukova uchishandisa zvigadziriso kubva kuKTable.

Ngatitorei data pamusoro pehuwandu hwekutsinhana kwemasheya uye tivasanganise nenhau dzekutsinhana kwemasheya kune maindasitiri akakodzera. Hezvino izvo zvaunoda kuita kuti uwane izvi uchipihwa iyo kodhi yauinayo kare.

1. Shandura chinhu cheKTable chine data pamusoro pehuwandu hwekutengesa kwemasheya kuita KStream, ichiteverwa nekutsiva kiyi nekiyi inoratidza chikamu cheindasitiri chinoenderana nechiratidzo chemasheya.
2. Gadzira chinhu cheKTable chinoverenga data kubva mumusoro une stock exchange nhau. Iyi KTable nyowani ichaiswa muchikamu neindasitiri chikamu.
3. Batanidza nhau dzakagadziridzwa neruzivo rwehuwandu hwekutsinhana kwemasheya neindasitiri chikamu.

Zvino ngationei kuti tingaite sei chirongwa ichi chekuita.

Shandura KTable kuita KStream

Kushandura KTable kuita KStream unofanirwa kuita zvinotevera.

1. Fonera iyo KTable.toStream() nzira.
2. Nekudana nzira yeKStream.map, tsiva kiyi nezita reindasitiri, wobva watora chinhu cheTransactionSummary kubva paWindowed muenzaniso.

Tichabatanidza mabasa aya pamwe chete sezvinotevera (code inogona kuwanikwa mufaira src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Listing 5.8).

Nekuti tiri kuita KStream.map oparesheni, yakadzoserwa KStream muenzaniso inopatsanurwa otomatiki kana ichinge yashandiswa mukubatanidza.

Isu tapedza maitiro ekutendeuka, chinotevera isu tinoda kugadzira KTable chinhu chekuverenga stock news.

Kugadzirwa kweKTable kune stock news

Sezvineiwo, kugadzira chinhu cheKTable chinongotora mutsara mumwechete wekodhi (iyo kodhi inogona kuwanikwa mu src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Listing 5.9).

Zvakakosha kuziva kuti hapana zvinhu zveSerde zvinofanirwa kutsanangurwa, sezvo tambo Serdes inoshandiswa muzvirongwa. Zvakare, nekushandisa iyo EARLIEST enumeration, tafura inozadzwa nemarekodhi pakutanga chaiko.

Iye zvino tinogona kuenderera kune danho rekupedzisira - kubatana.

Kubatanidza zvigadziriso zvenhau nedata rekutengesa kuverenga

Kugadzira kubatana hakusi kwakaoma. Tichashandisa kujoinwa kuruboshwe kana pasina nhau dzemasheya kuindasitiri inokodzera (iyo kodhi inodiwa inogona kuwanikwa mufaira src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Listing 5.10).

Iyi leftJoin opareta iri nyore. Kusiyana nemajoini muChitsauko 4, nzira yeJoinWindow haishandiswe nekuti kana uchiita kujoinha kweKStream-KTable, pane kungopinda muKTable pakiyi yega yega. Kubatana kwakadaro hakuna kuganhurirwa munguva: rekodhi ingave muKTable kana isipo. Mhedziso huru: kushandisa KTable zvinhu iwe unogona kupfumisa KStream ine isingawanzo gadziridzwa referensi data.

Iye zvino tichatarisa nzira inoshanda yekupfumisa zviitiko kubva kuKStream.

5.3.4. GlobalKTable zvinhu

Sezvauri kuona, pane chinodiwa kupfumisa zviitiko nzizi kana kuwedzera mamiriro kwavari. MuChitsauko 4 wakaona kubatana pakati pezvinhu zviviri zveKStream, uye muchikamu chapfuura wakaona kubatana pakati peKStream neKTable. Muzviitiko zvese izvi, zvinodikanwa kugovera zvakare kuyerera kwedata paunenge uchigadzira makiyi emhando nyowani kana kukosha. Dzimwe nguva repartitioning inoitwa zvakajeka, uye dzimwe nguva Kafka Streams inozviita otomatiki. Kuparadzanisazve kunokosha nokuti makiyi akachinja uye zvinyorwa zvinofanira kuguma muzvikamu zvitsva, kana zvisina kudaro kubatana kunenge kusingabviri (izvi zvakakurukurwa muChitsauko 4, muchikamu che "Re-partitioning data" muchikamu 4.2.4).

Re-partitioning ine mari

Kupatsanura zvakare kunoda mari - yekuwedzera mari yekushandisa kugadzira misoro yepakati, kuchengetedza yakapetwa data mune imwe musoro; zvinoreva zvakare kuwedzera latency nekuda kwekunyora nekuverenga kubva pane ino musoro. Pamusoro pezvo, kana iwe uchida kujoinha kupfuura chikamu chimwe chete kana chiyero, iwe unofanirwa kubatanidza majoini, mepu marekodhi nemakiyi matsva, uye womhanyisa maitiro ekugovera zvakare.

Kubatanidza kune madiki dataset

Mune zvimwe zviitiko, vhoriyamu yedata rereferenzi yekubatanidza idiki, saka makopi akazara ayo anogona kukwana nyore munharaunda pane imwe neimwe node. Kune mamiriro akadai, Kafka Streams inopa iyo GlobalKTable kirasi.

GlobalKTable zviitiko zvakasarudzika nekuti chishandiso chinodzokorora data rese kune imwe neimwe yemanodhi. Uye sezvo data rese riripo pane imwe neimwe node, hapana chikonzero chekugovanisa chiitiko rwizi nereferenzi data kiyi kuitira kuti iwanikwe kune ese mapartitions. Iwe unogona zvakare kugadzira makiyi ekujoinha uchishandisa GlobalKTable zvinhu. Ngatidzokerei kune mumwe wemienzaniso yapfuura kuratidza chimiro ichi.

Kubatanidza zvinhu zveKStream kune GlobalKTable zvinhu

Muchikamu 5.3.2, takaita kuunganidzwa kwehwindo kwekuchinjana kwevatengi. Mhedzisiro yekuunganidzwa uku yakaita seizvi:

{customerId='074-09-3705', stockTicker='GUTM'}, 17
{customerId='037-34-5184', stockTicker='CORK'}, 16

Nepo izvi zvabuda zvichishandira chinangwa, zvingadai zvakanyanya kubatsira dai zita remutengi uye zita rekambani rakazara rairatidzwawo. Kuti uwedzere zita remutengi uye zita rekambani, unogona kuita zvakajairwa kujoinha, asi iwe uchafanirwa kuita maviri kiyi mepu uye kupatsanura zvakare. NeGlobalKTable unogona kudzivirira mutengo wekuita kwakadaro.

Kuti tiite izvi, isu tichashandisa iyo countStream chinhu kubva Kunyorwa 5.11 (iyo inoenderana kodhi inogona kuwanikwa mu src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) uye kuibatanidza kune zviviri GlobalKTable zvinhu.

Takambokurukura izvi, saka handizozvidzokorora. Asi ndinocherechedza kuti kodhi iri muToStream() basa remepu rinotorwa kuita chinhu chinoshanda pachinzvimbo cheinline lambda kutaura nekuda kwekuverenga.

Nhanho inotevera ndeyekuzivisa zviitiko zviviri zveGlobalKTable (iyo kodhi inoratidzwa inogona kuwanikwa mufaira src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Listing 5.12).

Ndokumbira utarise kuti mazita enyaya anotsanangurwa pachishandiswa marudzi akanyorwa.

Iye zvino zvatava nezvikamu zvose zvakagadzirirwa, zvose zvinosara ndezvekunyora kodhi yekubatanidza (iyo inogona kuwanikwa mufaira src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Listing 5.13).

Kunyangwe paine maviri anojoinha mune iyi kodhi, iwo akasungwa nekuti hapana mhedzisiro yavo inoshandiswa zvakasiyana. Migumisiro inoratidzwa pakupera kwebasa rose.

Kana iwe uchimhanyisa iyo iri pamusoro yekujoinha oparesheni, iwe unowana mibairo seizvi:

{customer='Barney, Smith' company="Exxon", transactions= 17}

Essence haina kuchinja, asi izvi zvinobuda zvinotaridzika zvakanyanya kujeka.

Kana iwe ukaverenga kusvika kuChitsauko 4, iwe watoona akati wandei marudzi ekubatanidza ari muchiito. Akanyorwa patafura. 5.2. Tafura iyi inoratidza kugona kwekubatanidza sevhezheni 1.0.0 yeKafka Streams; Chimwe chinhu chinogona kuchinja mune ramangwana rinoburitswa.

Kuputira zvinhu, ngatidzokorore izvo zvekutanga: unogona kubatanidza hova dzezviitiko (KStream) uye kugadzirisa hova (KTable) uchishandisa yenzvimbo. Neimwe nzira, kana saizi yedata rereferenzi isiri yakakura kwazvo, unogona kushandisa GlobalKTable chinhu. GlobalKTables inodzokorora zvikamu zvese kune imwe neimwe yeKafka Streams application node, kuve nechokwadi chekuti data rese riripo zvisinei kuti kiyi inoenderana nechikamu chipi.

Tevere tichaona iyo Kafka Streams chimiro, nekuda kwatinogona kuona shanduko yenyika pasina kushandisa data kubva kuKafka musoro.

5.3.5. Queryable state

Isu takatoita akati wandei mashandiro anosanganisira nyika uye tinogara tichiburitsa mibairo kune iyo console (nekuda kwezvinangwa zvekusimudzira) kana kuinyorera kune dingindira (nechinangwa chekugadzira). Paunenge uchinyora mhinduro kune musoro, unofanirwa kushandisa Kafka mutengi kuti uzvione.

Kuverenga data kubva mumisoro iyi kunogona kutorwa serudzi rwemaonero enyama. Nezvinangwa zvedu, tinogona kushandisa tsananguro yemaonero akaitwa kubva kuWikipedia: “...chinhu chedhatabhesi chemuviri chine zvabuda pamubvunzo. Semuenzaniso, inogona kunge iri kopi yenzvimbo yedata iri kure, kana chikamu chidiki chemitsara uye/kana makoramu etafura kana kujoinwa zvabuda, kana tafura yepfupiso inowanikwa kuburikidza nekuunganidza” (https://en.wikipedia.org/wiki /Materialized_view).

Kafka Streams zvakare inobvumidza iwe kuti umhanye mibvunzo inopindirana pazvitoro zvehurumende, ichikubvumidza kuti uverenge zvakananga aya maonerwo enyama. Zvakakosha kuziva kuti mubvunzo kuchitoro chehurumende ndeyekuverenga-chete kushanda. Izvi zvinovimbisa kuti haufanirwe kunetseka nezve netsaona kuita kuti nyika isaenderane apo application yako iri kugadzirisa data.

Iko kugona kubvunza zvakananga zvitoro zvehurumende kwakakosha. Izvi zvinoreva kuti iwe unogona kugadzira dashboard application pasina kutanga watora data kubva kumutengi weKafka. Iyo inowedzerawo kugona kwechishandiso, nekuda kwekuti hapana chikonzero chekunyora data zvakare:

• nekuda kwenzvimbo yedata, vanogona kukurumidza kuwanikwa;
• kudzokororwa kwedata kunobviswa, sezvo isina kunyorwa kune ekunze kuchengetedza.

Chinhu chikuru chandinoda kuti uyeuke ndechekuti unogona kubvunza zvakananga nyika kubva mukati mekushandisa kwako. Mikana inokupa iwe haugone kuwedzeredzwa. Panzvimbo pekushandisa data kubva kuKafka uye kuchengetedza marekodhi mudhatabhesi rekushandisa, unogona kubvunza zvitoro zvehurumende nemhedzisiro yakafanana. Yakananga mibvunzo kuzvitoro zvehurumende inoreva kodhi shoma (hapana mutengi) uye shoma software (hapana kudikanwa kwetafura yedatabase kuchengetedza mhinduro).

Tataura zvakawanda muchitsauko chino, saka tichasiya nhaurirano yedu yemibvunzo ine chekuita nezvitoro zvehurumende parizvino. Asi usazvinetse: muChitsauko 9, tichagadzira yakapfava dashboard application ine mibvunzo inopindirana. Ichashandisa mimwe yemienzaniso kubva pane izvi uye zvitsauko zvakapfuura kuratidza mibvunzo inopindirana uye kuti iwe ungaiwedzera sei kuKafka Streams application.

Summary

• KStream zvinhu zvinomiririra hova dzezviitiko, zvakafanana nekuisa mudhatabhesi. Zvinhu zveKTable zvinomiririra nzizi dzekuvandudza, senge zvigadziriso kune dhatabhesi. Ukuru hwechinhu cheKTable hachikure, zvinyorwa zvekare zvinotsiviwa nezvitsva.
• KTable zvinhu zvinodiwa pakuunganidza mashandiro.
• Uchishandisa mashandiro ehwindo, unogona kupatsanura data rakaunganidzwa kuita mabhakiti enguva.
• Kutenda kune GlobalKTable zvinhu, unogona kuwana referensi data chero kupi mukushandisa, zvisinei nekugovaniswa.
• Kubatana pakati peKStream, KTable uye GlobalKTable zvinhu zvinogoneka.

Parizvino, takatarisana nekuvaka Kafka Streams application tichishandisa iyo yepamusoro-level KStream DSL. Kunyangwe iyo yepamusoro-yepamusoro nzira inobvumidza iwe kugadzira yakatsvinda uye yakapfupika zvirongwa, kuishandisa inomiririra kutengeserana-kure. Kushanda neDSL KStream zvinoreva kuwedzera kupfupika kwekodhi yako nekudzikisa dhigirii rekutonga. Muchitsauko chinotevera, tichatarisa iyo yakaderera-level handler node API uye edza zvimwe zvekutengesa-offs. Zvirongwa zvichange zvakareba kupfuura zvazvaive kare, asi isu tichakwanisa kugadzira chero yekubata node yatingade.

→ Mamwe mashoko pamusoro pebhuku anogona kuwanikwa pa muparidzi webhusaiti

→ YeHabrozhiteli 25% kuderedzwa uchishandisa kopani - Kafka Streams

→ Pakubhadhara kwebepa rebhuku rebhuku, bhuku remagetsi rinotumirwa ne-e-mail.

Source: www.habr.com