Phau ntawv "Kafka Kwj hauv Kev Ua Haujlwm. Cov ntawv thov thiab microservices rau kev ua haujlwm tiag tiag "

Nyob zoo, cov neeg nyob hauv Khabro! Phau ntawv no haum rau txhua tus tsim tawm uas xav nkag siab txog kev ua cov xov. Kev nkag siab txog kev faib tawm yuav pab koj nkag siab zoo dua Kafka thiab Kafka kwj. Nws yuav zoo dua kom paub Kafka lub moj khaum nws tus kheej, tab sis qhov no tsis tsim nyog: ​​Kuv yuav qhia koj txhua yam koj xav tau. Kev paub Kafka cov tsim tawm thiab cov novices ib yam nkaus yuav kawm paub tsim cov kwj deg ua cov ntawv thov siv lub tsev qiv ntawv Kafka Stream hauv phau ntawv no. Cov neeg tsim khoom nruab nrab thiab qib siab Java twb paub txog cov ntsiab lus zoo li serialization yuav kawm siv lawv cov kev txawj los tsim cov ntawv thov Kafka Stream. Phau ntawv lub hauv paus code yog sau nyob rau hauv Java 8 thiab ua rau kev siv tseem ceeb ntawm Java 8 lambda qhia syntax, yog li paub yuav ua li cas ua hauj lwm nrog lambda zog (txawm hais tias nyob rau hauv lwm yam lus programming) yuav los nyob rau hauv ke.

Tshaj tawm. 5.3. Aggregation thiab windowing ua haujlwm

Hauv seem no, peb yuav txav mus los tshawb txog qhov zoo tshaj plaws ntawm Kafka kwj. Txog tam sim no peb tau npog cov hauv qab no ntawm Kafka kwj:

• tsim cov txheej txheem topology;
• siv lub xeev hauv kev siv streaming;
• ua cov ntaub ntawv kwj kev sib txuas;
• Qhov sib txawv ntawm qhov kev tshwm sim ntws (KStream) thiab hloov tshiab kwj (KTable).

Hauv cov piv txwv hauv qab no peb yuav coj tag nrho cov ntsiab lus no ua ke. Koj tseem yuav kawm txog windowsing, lwm qhov zoo ntawm cov ntawv thov streaming. Peb thawj qhov piv txwv yuav yog ib qho yooj yim aggregation.

5.3.1. Aggregation ntawm Tshuag muag los ntawm kev lag luam sector

Aggregation thiab pab pawg yog cov cuab yeej tseem ceeb thaum ua haujlwm nrog cov ntaub ntawv streaming. Kev tshuaj xyuas ntawm tus kheej cov ntaub ntawv raws li lawv tau txais feem ntau tsis txaus. Txhawm rau rho tawm cov ntaub ntawv ntxiv los ntawm cov ntaub ntawv, nws yog qhov tsim nyog los pab pawg thiab sib xyaw ua ke.

Hauv qhov piv txwv no, koj yuav muab cov khaub ncaws ntawm ib hnub tub luam uas yuav tsum tau taug qab qhov muag ntim ntawm cov tuam txhab lag luam hauv ntau qhov kev lag luam. Tshwj xeeb, koj txaus siab rau tsib lub tuam txhab uas muaj kev muag khoom loj tshaj plaws hauv txhua qhov kev lag luam.

Xws li kev sib sau yuav xav tau ob peb kauj ruam hauv qab no los txhais cov ntaub ntawv rau hauv daim ntawv xav tau (hais lus dav dav).

1. Tsim cov ncauj lus raws li qhov chaw tshaj tawm cov ntaub ntawv raw khoom lag luam. Peb yuav tsum tau kos ib yam khoom ntawm hom StockTransaction rau ib yam khoom ntawm hom ShareVolume. Lub ntsiab lus yog tias cov khoom lag luam StockTransaction muaj cov ntaub ntawv muag muag, tab sis peb tsuas yog xav tau cov ntaub ntawv hais txog tus lej ntawm cov khoom muag.
2. Pawg ShareVolume cov ntaub ntawv los ntawm cov cim Tshuag. Thaum muab pawg los ntawm cov cim, koj tuaj yeem tsoo cov ntaub ntawv no rau hauv subtotals ntawm cov khoom muag muag. Nws tsim nyog sau cia tias KStream.groupBy txoj kev rov qab ua piv txwv ntawm hom KGroupedStream. Thiab koj tuaj yeem tau txais KTable piv txwv los ntawm kev hu xov tooj ntxiv rau KGroupedStream.reduce txoj kev.

KGroupedStream interface yog dab tsi

Txoj kev KStream.groupBy thiab KStream.groupByKey rov qab ua piv txwv ntawm KGroupedStream. KGroupedStream yog tus sawv cev nruab nrab ntawm cov kwj ntawm cov xwm txheej tom qab muab pawg los ntawm cov yuam sij. Nws tsis yog txhua yam npaj rau kev ua haujlwm ncaj qha nrog nws. Hloov chaw, KGroupedStream yog siv rau kev ua haujlwm sib sau ua ke, uas ib txwm ua rau KTable. Thiab txij li qhov tshwm sim ntawm kev ua haujlwm sib sau ua ke yog KTable thiab lawv siv lub khw muag khoom hauv lub xeev, nws yog qhov ua tau tias tsis yog txhua qhov kev hloov kho tshiab raws li qhov tshwm sim raug xa mus ntxiv rau hauv cov kav dej.

KTable.groupBy txoj kev rov qab zoo li KGroupedTable - qhov nruab nrab sawv cev ntawm cov kwj ntawm kev hloov tshiab, rov ua pawg los ntawm qhov tseem ceeb.

Cia peb so luv luv thiab saib Fig. 5.9, uas qhia peb tau ua tiav. Qhov topology no yuav tsum tau paub zoo rau koj.

Tam sim no cia peb saib cov cai rau qhov topology no (nws tuaj yeem pom nyob rau hauv cov ntaub ntawv src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Sau npe 5.2).

Cov cai muab tau qhov txawv los ntawm nws cov brevity thiab cov ntim loj ntawm kev ua tau ua nyob rau hauv ob peb kab. Tej zaum koj yuav pom tej yam tshiab nyob rau hauv thawj parameter ntawm builder.stream txoj kev: ib tug nqi ntawm cov enum hom AutoOffsetReset.EARLIEST (tseem muaj ib tug LATEST), teem siv lub Consumed.withOffsetResetPolicy txoj kev. Hom kev suav sau no tuaj yeem siv los qhia txog qhov offset reset lub tswv yim rau txhua KStream lossis KTable thiab ua ntej tshaj qhov kev xaiv rov pib dua los ntawm kev teeb tsa.

GroupByKey thiab GroupBy

KStream interface muaj ob txoj hauv kev rau kev sib sau cov ntaub ntawv: GroupByKey thiab GroupBy. Ob leeg rov qab KGroupedTable, yog li koj yuav xav paub qhov txawv ntawm lawv thiab thaum twg siv qhov twg?

Txoj kev GroupByKey yog siv thaum cov yuam sij hauv KStream twb tsis yog khoob. Thiab qhov tseem ceeb tshaj plaws, tus chij "yuav tsum tau rov muab faib" tsis tau teeb tsa.

Txoj kev GroupBy xav tias koj tau hloov cov yuam sij pab pawg, yog li tus chij repartition tau teem rau qhov tseeb. Kev sib koom ua ke, sib sau ua ke, thiab lwm yam. tom qab GroupBy txoj kev yuav ua rau tsis siv neeg rov muab faib.
Cov ntsiab lus: Thaum twg ua tau, koj yuav tsum siv GroupByKey es tsis yog GroupBy.

Nws yog qhov tseeb tias mapValues ​​thiab groupBy txoj kev ua li cas, yog li cia peb saib cov sum() txoj kev (pom nyob rau hauv src/main/java/bbejeck/model/ShareVolume.java) (Lusing 5.3).

Txoj kev ShareVolume.sum rov qab tag nrho cov khoom muag ntawm cov khoom muag, thiab cov txiaj ntsig ntawm tag nrho cov saw ntawm kev suav yog KTable khoom. . Tam sim no koj nkag siab lub luag haujlwm KTable ua si. Thaum ShareVolume cov khoom tuaj txog, cov khoom siv KTable khaws cov hloov tshiab tam sim no. Nws yog ib qho tseem ceeb uas yuav tsum nco ntsoov tias tag nrho cov kev hloov tshiab tau cuam tshuam rau yav dhau los shareVolumeKTable, tab sis tsis yog txhua tus raug xa mus ntxiv.

Tom qab ntawd peb siv KTable no los sib sau ua ke (los ntawm tus lej ntawm cov kev sib pauv pauv) kom tuaj txog ntawm tsib lub tuam txhab uas muaj cov khoom lag luam ntau tshaj plaws hauv txhua qhov kev lag luam. Peb qhov kev nqis tes ua hauv qhov no yuav zoo ib yam li cov thawj sib sau ua ke.

1. Ua lwm pab pawg los ntawm kev ua haujlwm rau pab pawg neeg ShareVolume khoom los ntawm kev lag luam.
2. Pib summarizing ShareVolume khoom. Lub sijhawm no cov khoom sib sau ua ke yog qhov tseem ceeb ntawm qhov tseem ceeb. Nyob rau hauv qhov teeb meem-loj queue no, tsuas yog tsib lub tuam txhab uas muag ntau tshaj plaws yog khaws cia.
3. Daim ntawv qhia cov kab los ntawm kab lus dhau los mus rau ib txoj hlua tus nqi thiab xa rov qab tsib feem ntau cov khoom lag luam los ntawm tus lej los ntawm kev lag luam.
4. Sau cov txiaj ntsig hauv kab ntawv rau lub ntsiab lus.

Hauv daim duab. Daim duab 5.10 qhia cov ntaub ntawv ntws topology graph. Raws li koj tuaj yeem pom, qhov thib ob ntawm kev ua haujlwm yog qhov yooj yim heev.

Tam sim no peb muaj kev nkag siab meej txog cov qauv ntawm qhov thib ob ntawm kev ua tiav, peb tuaj yeem tig mus rau nws qhov chaws (koj yuav pom nws hauv cov ntaub ntawv src/main/java/bbejeck/chapter_5/AggregationsAndReducingExample.java) (Sau npe 5.4) .

Qhov kev pib pib no muaj qhov sib txawv ntawm qhov sib txawv. Qhov no yog ib qho khoom siv kev cai uas yog ib qho adapter rau java.util.TreeSet uas yog siv los taug qab N cov txiaj ntsig saum toj kawg nkaus hauv kev txiav txim qis ntawm kev sib pauv pauv.

Koj twb pom cov groupBy thiab mapValues ​​hu, yog li peb yuav tsis mus rau cov (peb hu rau txoj kev KTable.toStream vim hais tias txoj kev KTable.print yog deprecated). Tab sis koj tseem tsis tau pom KTable version ntawm aggregate() tsis tau, yog li peb yuav siv sij hawm me ntsis los tham txog qhov ntawd.

Raws li koj nco qab, dab tsi ua rau KTable txawv yog tias cov ntaub ntawv nrog tib tus yuam sij raug suav tias yog hloov tshiab. KTable hloov qhov qub nkag nrog ib qho tshiab. Aggregation tshwm sim nyob rau hauv ib txoj kev zoo sib xws: cov ntaub ntawv tshiab kawg nrog tib tus yuam sij yog sib sau ua ke. Thaum cov ntaub ntawv tuaj txog, nws tau ntxiv rau hauv chav kawm FixedSizePriorityQueue piv txwv siv tus adder (qhov thib ob parameter nyob rau hauv txoj kev sib sau ua ke), tab sis yog tias lwm cov ntaub ntawv twb muaj nrog tib tus yuam sij, ces cov ntaub ntawv qub raug tshem tawm siv lub subtractor (thib peb parameter hauv aggregate method hu).

Qhov no tag nrho txhais tau hais tias peb cov aggregator, FixedSizePriorityQueue, tsis suav tag nrho cov txiaj ntsig nrog ib tus yuam sij, tab sis khaws cov nyiaj tau los ntawm qhov ntau ntawm N feem ntau cov khoom lag luam. Txhua qhov nkag nkag muaj tag nrho cov lej ntawm cov khoom muag tam sim no. KTable yuav muab cov ntaub ntawv qhia rau koj txog cov tuam txhab lag luam twg tam sim no cov khoom lag luam tshaj plaws, tsis tas yuav tsum muaj kev sib sau ua ke ntawm txhua qhov hloov tshiab.

Peb kawm ua ob yam tseem ceeb:

• pab pawg muaj nuj nqis hauv KTable los ntawm ib qho tseem ceeb;
• ua cov haujlwm tseem ceeb xws li kev sib sau ua ke thiab sib sau ua ke ntawm cov txiaj ntsig ntawm pawg no.

Kev paub yuav ua li cas ua cov haujlwm no yog qhov tseem ceeb rau kev nkag siab lub ntsiab lus ntawm cov ntaub ntawv txav los ntawm Kafka Stream daim ntawv thov thiab nkag siab txog cov ntaub ntawv nws nqa.

Peb kuj tau coj los ua ke qee cov ntsiab lus tseem ceeb uas tau tham ua ntej hauv phau ntawv no. Hauv Tshooj 4, peb tau tham txog yuav ua li cas ua txhaum, lub xeev hauv cheeb tsam tseem ceeb rau daim ntawv thov streaming. Thawj qhov piv txwv hauv tshooj no tau qhia tias vim li cas lub xeev hauv cheeb tsam tseem ceeb heev - nws tso cai rau koj taug qab cov ntaub ntawv koj twb pom lawm. Kev nkag mus hauv zos tsis txhob ncua kev sib txuas hauv network, ua rau daim ntawv thov ua tau zoo dua thiab ua yuam kev.

Thaum ua ib qho kev sib tw lossis kev sib sau ua ke, koj yuav tsum qhia lub npe ntawm lub xeev lub khw. Kev sib sau ua ke thiab kev sib sau ua ke rov qab ua piv txwv KTable, thiab KTable siv lub xeev cia los hloov cov txiaj ntsig qub nrog cov tshiab. Raws li koj tau pom, tsis yog txhua qhov kev hloov kho tshiab raug xa mus rau hauv cov kav dej, thiab qhov no yog qhov tseem ceeb vim tias kev ua haujlwm sib sau ua ke tau tsim los tsim cov ntsiab lus. Yog tias koj tsis siv lub xeev hauv zos, KTable yuav xa mus rau tag nrho cov kev sib sau ua ke thiab cov txiaj ntsig sib sau ua ke.

Tom ntej no, peb yuav saib txog kev ua haujlwm xws li kev sib sau ua ke hauv ib lub sijhawm tshwj xeeb - hu ua windowing operations.

5.3.2. Qhov rai ua haujlwm

Hauv seem dhau los, peb tau qhia txog zawv zawg convolution thiab aggregation. Daim ntawv thov tau ua ib qho txuas ntxiv ntawm kev muag khoom tom qab los ntawm kev sib sau ntawm tsib feem ntau cov khoom lag luam ntawm kev sib pauv.

Qee lub sij hawm xws li kev sib sau ua ke tas mus li thiab ua tiav cov txiaj ntsig yog qhov tsim nyog. Thiab qee zaum koj yuav tsum ua haujlwm tsuas yog nyob rau lub sijhawm. Piv txwv li, xam pes tsawg qhov kev pauv pauv tau ua nrog kev sib koom ntawm ib lub tuam txhab tshwj xeeb hauv 10 feeb dhau los. Los yog pes tsawg tus neeg siv nyem rau ntawm daim ntawv tshaj tawm tshiab hauv 15 feeb dhau los. Ib daim ntawv thov tuaj yeem ua cov haujlwm no ntau zaus, tab sis nrog cov txiaj ntsig uas tsuas yog siv rau lub sijhawm teev tseg (lub sijhawm windows).

suav cov kev pauv pauv los ntawm cov neeg yuav khoom

Hauv qhov piv txwv tom ntej, peb yuav taug qab cov khoom lag luam thoob plaws ntau tus neeg lag luam - txawm tias cov koom haum loj lossis cov neeg txawj ntse nyiaj txiag.

Muaj ob qho laj thawj rau qhov kev taug qab no. Ib qho ntawm lawv yog qhov xav tau kom paub tias cov thawj coj ua lag luam yuav yuav / muag dab tsi. Yog tias cov neeg ua si loj thiab cov neeg ua lag luam zoo tshaj plaws pom lub sijhawm, nws ua rau kev txiav txim siab ua raws li lawv lub tswv yim. Qhov thib ob yog vim li cas yog lub siab xav pom tej yam tshwm sim ntawm kev ua lag luam tsis raug cai. Txhawm rau ua qhov no, koj yuav tsum tau txheeb xyuas qhov sib raug zoo ntawm kev muag khoom loj nrog cov xov xwm tseem ceeb.

Cov kev taug qab zoo li no muaj cov kauj ruam hauv qab no:

• tsim cov kwj deg rau kev nyeem los ntawm cov ntsiab lus ntawm cov khoom lag luam;
• pab pawg cov ntaub ntawv tuaj los ntawm tus neeg yuav khoom ID thiab cov cim Tshuag. Hu rau groupBy method rov ua piv txwv ntawm KGroupedStream chav kawm;
• KGroupedStream.windowedBy txoj kev xa rov qab cov ntaub ntawv kwj txwv rau lub sijhawm qhov rai, uas tso cai rau kev sib sau ua ke windows. Nyob ntawm hom qhov rais, xws li TimeWindowedKStream lossis SessionWindowedKStream rov qab los;
• kev suav suav rau kev ua haujlwm sib sau ua ke. Cov ntaub ntawv windowed ntws txiav txim seb puas muaj cov ntaub ntawv tshwj xeeb raug suav nrog hauv cov suav no;
• sau cov txiaj ntsig rau ib lub ncauj lus lossis muab lawv tso rau hauv lub console thaum txhim kho.

Lub topology ntawm daim ntawv thov no yooj yim, tab sis ib daim duab meej ntawm nws yuav pab tau. Cia wb mus saib daim duab. 5.11.

Tom ntej no, peb yuav saib cov kev ua haujlwm ntawm lub qhov rais ua haujlwm thiab cov lej sib raug.

Qhov rai hom

Muaj peb hom qhov rais hauv Kafka kwj:

• kev sib ntsib;
• "tuag";
• zawv zawg / hopping.

Qhov twg xaiv yog nyob ntawm koj qhov kev xav tau ntawm kev lag luam. Tumbling thiab dhia qhov rais yog lub sij hawm txwv, thaum lub qhov rais kev sib kho yog txwv los ntawm cov neeg siv kev ua si - lub sij hawm ntawm kev sib kho (s) tsuas yog txiav txim los ntawm lub active tus neeg siv. Qhov tseem ceeb tshaj plaws uas yuav tsum nco ntsoov yog tias txhua hom qhov rais yog raws hnub tim / lub sij hawm thwj ntawm kev nkag, tsis yog lub sij hawm system.

Tom ntej no, peb siv peb cov topology nrog txhua hom qhov rais. Cov lej tiav yuav raug muab tsuas yog hauv thawj qhov piv txwv; rau lwm hom windows tsis muaj dab tsi yuav hloov pauv tsuas yog hom kev ua haujlwm ntawm lub qhov rais.

Session windows

Qhov rai qhov rais sib txawv heev ntawm txhua hom qhov rais. Lawv txwv tsis pub ntau npaum li lub sijhawm raws li kev ua haujlwm ntawm tus neeg siv (lossis cov haujlwm ntawm lub koomhaum uas koj xav taug qab). Kev sib tham qhov rais yog delimited los ntawm lub sij hawm ntawm inactivity.

Daim duab 5.12 qhia txog lub tswvyim ntawm kev sib kho qhov rais. Lub rooj sib tham me me yuav ua ke nrog qhov kev sib kho rau sab laug. Thiab qhov kev sib kho ntawm sab xis yuav raug cais vim nws ua raws li lub sijhawm ntev ntawm kev tsis ua haujlwm. Kev sib tham qhov rais yog raws li cov neeg siv cov dej num, tab sis siv cov hnub / lub sij hawm thwj los ntawm cov ntawv nkag los txiav txim seb qhov kev sib ntsib twg nkag mus.

Siv lub qhov rais sib kho los taug qab cov khoom lag luam

Cia peb siv lub qhov rais sib tham los ntes cov ntaub ntawv hais txog kev pauv pauv. Qhov kev siv ntawm kev sib kho qhov rais tau pom nyob rau hauv Cov Npe 5.5 (uas tuaj yeem pom hauv src/main/java/bbejeck/chapter_5/CountingWindowingAndKTableJoinExample.java).

Koj twb pom feem ntau ntawm cov haujlwm hauv qhov topology no, yog li tsis tas yuav tshuaj xyuas lawv ntawm no dua. Tab sis kuj tseem muaj ob peb yam tshiab ntawm no, uas tam sim no peb yuav tham txog.

Ib pawg ntawm kev ua haujlwm feem ntau ua qee yam kev ua haujlwm sib sau ua ke (ua ke, sib sau, lossis suav). Koj tuaj yeem ua ob qho kev sib sau ua ke nrog kev khiav tag nrho, lossis qhov rais aggregation, uas coj mus rau hauv tus account cov ntaub ntawv nyob rau hauv lub sij hawm teev qhov rais.

Cov cai hauv Cov Ntawv Teev Npe 5.5 suav cov lej ntawm kev hloov pauv hauv lub qhov rais sib tham. Hauv daim duab. 5.13 cov kev nqis tes ua no tau txheeb xyuas ib qib zuj zus.

Los ntawm kev hu rau windowedBy(SessionWindows.with(twentySeconds).txog(fifteenMinutes)) peb tsim ib lub qhov rais kev sib kho nrog ib tug inactivity lub sij hawm ntawm 20 vib nas this thiab ib tug persistence ncua ntawm 15 feeb. Lub sijhawm tsis ua haujlwm ntawm 20 vib nas this txhais tau hais tias daim ntawv thov yuav suav nrog txhua qhov kev nkag uas tuaj txog hauv 20 vib nas this ntawm qhov kawg lossis pib ntawm qhov kev sib tham tam sim no rau hauv qhov kev sib tham tam sim no.

Tom ntej no, peb qhia meej tias qhov kev sib sau ua haujlwm yuav tsum tau ua nyob rau hauv qhov kev sib kho qhov rai - qhov no, suav. Yog tias qhov kev nkag nkag tau poob rau sab nraum lub qhov rais tsis ua haujlwm (sab twg ntawm lub hnub / lub sijhawm stamp), daim ntawv thov tsim kev sib tham tshiab. Retention interval txhais tau hais tias tuav ib qho kev sib kho rau qee lub sijhawm thiab tso cai rau cov ntaub ntawv lig uas txuas ntxiv mus rau lub sijhawm tsis ua haujlwm tab sis tseem tuaj yeem txuas nrog. Tsis tas li ntawd, qhov pib thiab xaus ntawm qhov kev sib kho tshiab uas tshwm sim los ntawm kev sib koom ua ke sib haum rau hnub tim thiab lub sij hawm tshiab tshaj plaws.

Cia peb saib ob peb qhov kev nkag los ntawm txoj kev suav kom pom cov kev sib tham ua haujlwm li cas (Table 5.1).

Thaum cov ntaub ntawv tuaj txog, peb saib rau cov ntu uas twb muaj lawm nrog tib tus yuam sij, lub sijhawm kawg tsawg dua li hnub tim / lub sijhawm tam sim no - lub sijhawm tsis ua haujlwm, thiab lub sijhawm pib ntau dua li hnub tim / lub sijhawm tam sim no + lub sijhawm tsis ua haujlwm. Noj qhov no rau hauv tus account, plaub nkag los ntawm lub rooj. 5.1 tau muab tso rau hauv ib qho kev sib tham raws li hauv qab no.

1. Cov ntaub ntawv 1 tuaj txog ua ntej, yog li lub sijhawm pib yog sib npaug rau lub sijhawm kawg thiab yog 00:00:00.

2. Tom ntej no, nkag 2 tuaj txog, thiab peb tab tom nrhiav rau ntu uas xaus tsis pub dhau 23:59:55 thiab pib tsis pub dhau 00:00:35. Peb pom cov ntaub ntawv 1 thiab muab cov kev sib tham 1 thiab 2. Peb siv lub sijhawm pib ntawm kev sib kho 1 (ua ntej) thiab lub sijhawm kawg ntawm kev sib kho 2 (tom qab), kom peb qhov kev sib tham tshiab pib ntawm 00: 00: 00 thiab xaus ntawm 00: 00:15 ib.

3. Cov ntaub ntawv 3 tuaj txog, peb nrhiav kev sib tham ntawm 00:00:30 thiab 00:01:10 thiab tsis pom muaj. Ntxiv qhov kev sib tham thib ob rau tus yuam sij 123-345-654, FFBE, pib thiab xaus ntawm 00:00:50.

4. Cov ntaub ntawv 4 tuaj txog thiab peb tab tom nrhiav kev sib tham ntawm 23:59:45 txog 00:00:25. Lub sijhawm no ob qhov kev sib tham 1 thiab 2 tau pom, Tag nrho peb ntu tau muab tso ua ke, nrog lub sijhawm pib ntawm 00:00:00 thiab xaus sijhawm ntawm 00:00:15.

Los ntawm cov lus piav qhia hauv tshooj no, nws tsim nyog nco ntsoov cov nuances tseem ceeb hauv qab no:

• Cov kev sib tham tsis yog qhov rais qhov loj. Lub sijhawm ntawm kev sib tham yog txiav txim siab los ntawm kev ua haujlwm hauv lub sijhawm;
• Hnub tim / lub sij hawm stamps nyob rau hauv cov ntaub ntawv txiav txim seb qhov kev tshwm sim ntog nyob rau hauv ib qho kev sib kho uas twb muaj lawm los yog thaum lub sij hawm tsis ua hauj lwm.

Tom ntej no peb yuav tham txog lwm hom qhov rais - "tumbling" windows.

"Tumbling" windows

Tumbling windows ntes cov xwm txheej uas poob hauv ib lub sijhawm. Xav txog tias koj yuav tsum tau ntes tag nrho cov khoom lag luam ntawm ib lub tuam txhab txhua 20 vib nas this, yog li koj sau tag nrho cov xwm txheej thaum lub sijhawm ntawd. Thaum kawg ntawm 20-thib ob lub sijhawm, lub qhov rais dov dhau thiab txav mus rau qhov tshiab 20-thib ob lub sijhawm soj ntsuam. Daim duab 5.14 qhia txog qhov xwm txheej no.

Raws li koj tuaj yeem pom, txhua qhov xwm txheej tau txais hauv 20 vib nas this dhau los suav nrog hauv lub qhov rais. Thaum kawg ntawm lub sijhawm no, lub qhov rais tshiab raug tsim.

Cov npe 5.6 qhia tau hais tias cov cai uas qhia tau hais tias siv qhov rais tumbling los ntes cov khoom lag luam txhua 20 vib nas this (pom hauv src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Nrog rau qhov kev hloov me me no rau TimeWindows.of txoj kev hu, koj tuaj yeem siv lub qhov rais tumbling. Qhov piv txwv no tsis hu ua kom txog thaum () txoj kev, yog li lub sijhawm khaws cia ntawm 24 teev yuav raug siv.

Thaum kawg, nws yog lub sijhawm txav mus rau qhov kawg ntawm qhov kev xaiv qhov rais - "hopping" windows.

zawv zawg ("dhia") windows

zawv zawg / hopping qhov rais zoo ib yam li cov qhov rais tumbling, tab sis muaj qhov sib txawv me ntsis. zawv zawg qhov rais tsis txhob tos kom txog thaum kawg ntawm lub sijhawm luv ua ntej tsim lub qhov rais tshiab los ua cov txheej xwm tsis ntev los no. Lawv pib suav tshiab tom qab lub sijhawm tos tsawg dua lub sijhawm lub qhov rais.

Txhawm rau piav qhia qhov sib txawv ntawm qhov sib tw thiab dhia qhov rais, cia peb rov qab mus rau qhov piv txwv ntawm kev suav cov kev pauv pauv hloov pauv. Peb lub hom phiaj tseem yog suav cov lej ntawm kev hloov pauv, tab sis peb tsis xav tos tag nrho lub sijhawm ua ntej hloov kho lub txee. Hloov chaw, peb yuav hloov kho lub txee ntawm lub sijhawm luv. Piv txwv li, peb tseem yuav suav tus lej ntawm kev hloov pauv txhua 20 vib nas this, tab sis hloov kho lub txee txhua 5 vib nas this, raws li qhia hauv daim duab. 5.15. Hauv qhov no, peb xaus nrog peb qhov txiaj ntsig qhov rais nrog cov ntaub ntawv sib tshooj.

Sau npe 5.7 qhia cov cai rau kev txhais cov qhov rais zawv zawg (pom hauv src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java).

Lub qhov rais tumbling tuaj yeem hloov mus rau lub qhov rais hopping los ntawm kev ntxiv hu rau advanceBy() txoj kev. Hauv qhov piv txwv qhia, lub sijhawm txuag yog 15 feeb.

Koj pom hauv ntu no yuav ua li cas txwv kev sib sau ua ke rau lub sijhawm windows. Tshwj xeeb, kuv xav kom koj nco qab peb yam hauv qab no los ntawm ntu no:

• qhov loj ntawm kev sib kho qhov rais yog txwv tsis pub dhau lub sij hawm, tab sis los ntawm cov neeg siv kev ua si;
• "tumbling" windows muab cov ntsiab lus ntawm cov xwm txheej hauv lub sijhawm;
• Lub sijhawm dhia lub qhov rais yog tsau, tab sis lawv tau hloov kho tas li thiab tej zaum yuav muaj cov ntawv sib tshooj hauv txhua qhov rais.

Tom ntej no, peb yuav kawm yuav ua li cas hloov KTable rov qab mus rau KStream rau kev sib txuas.

5.3.3. Txuas KStream thiab KTable cov khoom

Hauv Tshooj 4, peb tau tham txog kev txuas ob lub KStream khoom. Tam sim no peb yuav tsum kawm yuav ua li cas txuas KTable thiab KStream. Qhov no tej zaum yuav xav tau rau cov laj thawj yooj yim hauv qab no. KStream yog cov kwj ntawm cov ntaub ntawv, thiab KTable yog cov kwj ntawm cov ntaub ntawv hloov tshiab, tab sis qee zaum koj yuav xav ntxiv cov ntsiab lus ntxiv rau cov kwj dej uas siv cov kev hloov tshiab los ntawm KTable.

Cia peb muab cov ntaub ntawv ntawm tus lej ntawm cov khoom lag luam pauv pauv thiab muab lawv nrog cov xov xwm sib pauv khoom lag luam rau cov lag luam uas cuam tshuam. Nov yog yam koj yuav tsum tau ua kom ua tiav qhov no muab cov cai uas koj twb muaj lawm.

1. Hloov cov khoom KTable nrog cov ntaub ntawv ntawm tus lej ntawm cov khoom lag luam hauv KStream, tom qab ntawd hloov tus yuam sij nrog tus yuam sij qhia txog kev lag luam kev lag luam sib raug rau cov cim Tshuag.
2. Tsim cov khoom KTable uas nyeem cov ntaub ntawv los ntawm cov ncauj lus nrog cov xov xwm sib pauv khoom lag luam. KTable tshiab no yuav raug cais los ntawm kev lag luam sector.
3. Txuas cov xov xwm hloov tshiab nrog cov ntaub ntawv ntawm tus lej ntawm kev sib pauv pauv khoom lag luam los ntawm kev lag luam sector.

Tam sim no cia peb saib yuav ua li cas los ua txoj haujlwm no.

Konvertiere KTable in KWS

Txhawm rau hloov KTable rau KStream koj yuav tsum ua cov hauv qab no.

1. Hu rau KTable.toStream() txoj kev.
2. Los ntawm kev hu rau KStream.map txoj hauv kev, hloov tus yuam sij nrog lub npe kev lag luam, thiab tom qab ntawd muab cov khoom lag luam Summary los ntawm Windowed piv txwv.

Peb yuav muab cov haujlwm no ua ke ua ke raws li hauv qab no (cov cai tuaj yeem pom hauv cov ntaub ntawv src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Tsev Npe 5.8).

Vim tias peb tab tom ua haujlwm KStream.map, qhov rov qab KStream piv txwv yog rov muab faib ua haujlwm thaum nws siv hauv kev sib txuas.

Peb tau ua tiav cov txheej txheem hloov dua siab tshiab, tom ntej no peb yuav tsum tsim cov khoom KTable rau kev nyeem cov xov xwm Tshuag.

Tsim KTable rau cov xov xwm Tshuag

Hmoov zoo, tsim cov khoom KTable tsuas yog siv ib kab ntawm cov lej (cov lej tuaj yeem pom hauv src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Sau npe 5.9).

Nws yog ib qho tsim nyog sau cia tias tsis muaj Serde cov khoom yuav tsum tau teev tseg, txij li txoj hlua Serdes tau siv rau hauv qhov chaw. Tsis tas li ntawd, los ntawm kev siv EARLIEST enumeration, lub rooj tau sau nrog cov ntaub ntawv thaum pib.

Tam sim no peb tuaj yeem txav mus rau theem kawg - kev sib txuas.

Txuas cov xov xwm hloov tshiab nrog cov ntaub ntawv suav suav

Tsim kev sib txuas tsis yog qhov nyuaj. Peb yuav siv sab laug koom nrog yog tias tsis muaj cov xov xwm hauv Tshuag rau cov kev lag luam cuam tshuam (cov cai tsim nyog tuaj yeem pom nyob rau hauv cov ntaub ntawv src/main/java/bbejeck/chapter_5/CountingWindowingAndKtableJoinExample.java) (Sau npe 5.10).

Tus neeg teb xov tooj leftJoin no yooj yim heev. Tsis zoo li cov koom nrog hauv Tshooj 4, txoj kev JoinWindow tsis siv vim tias thaum ua KStream-KTable koom nrog, tsuas muaj ib qho nkag hauv KTable rau txhua tus yuam sij. Qhov kev sib txuas no tsis txwv rau lub sijhawm: cov ntaub ntawv yog nyob rau hauv KTable lossis tsis tuaj. Lub ntsiab lus xaus: siv KTable cov khoom koj tuaj yeem txhawb nqa KStream nrog cov ntaub ntawv hloov tshiab tsis tu ncua.

Tam sim no peb yuav saib txoj hauv kev zoo dua los txhawb cov xwm txheej los ntawm KStream.

5.3.4 ib. GlobalKTable khoom

Raws li koj tuaj yeem pom, muaj qhov xav tau los txhawb cov xwm txheej ntws los yog ntxiv cov ntsiab lus rau lawv. Hauv Tshooj 4 koj pom cov kev sib txuas ntawm ob yam khoom KStream, thiab hauv ntu dhau los koj pom kev sib txuas ntawm KStream thiab KTable. Nyob rau hauv tag nrho cov no, nws yog ib qho tsim nyog yuav tsum rov muab faib cov ntaub ntawv kwj thaum daim ntawv qhia cov yuam sij rau hom tshiab lossis tus nqi. Qee lub sij hawm repartitioning yog ua tiav, thiab qee zaum Kafka Stream ua nws tau. Kev faib rov qab yog qhov tsim nyog vim tias cov yuam sij tau hloov pauv thiab cov ntaub ntawv yuav tsum xaus rau hauv ntu tshiab, txwv tsis pub kev sib txuas yuav ua tsis tau (qhov no tau tham hauv Tshooj 4, hauv ntu "Re-partitioning data" hauv ntu 4.2.4).

Re-partitioning muaj nqi

Rov muab faib yuav tsum tau them nqi - cov nqi ntxiv rau kev tsim cov ntsiab lus nruab nrab, khaws cov ntaub ntawv sib npaug hauv lwm lub ncauj lus; nws kuj txhais tau hais tias nce latency vim kev sau ntawv thiab nyeem los ntawm cov ncauj lus no. Tsis tas li ntawd, yog tias koj xav tau koom nrog ntau dua ib qho lossis qhov loj me, koj yuav tsum tau txuas cov kev koom nrog, kos duab cov ntaub ntawv nrog cov yuam sij tshiab, thiab khiav cov txheej txheem rov muab faib dua.

Txuas rau cov ntaub ntawv me me

Qee zaum, qhov ntim ntawm cov ntaub ntawv siv los txuas yog me me, yog li ua tiav cov ntawv luam ntawm nws tuaj yeem yooj yim haum hauv zos ntawm txhua qhov. Rau cov xwm txheej zoo li no, Kafka Stream muab cov chav kawm GlobalKTable.

Cov xwm txheej GlobalKTable yog qhov tshwj xeeb vim tias daim ntawv thov rov ua dua txhua cov ntaub ntawv rau txhua qhov ntawm cov nodes. Thiab txij li tag nrho cov ntaub ntawv tam sim no nyob rau ntawm txhua qhov, tsis tas yuav muab faib cov xwm txheej los ntawm kev siv cov ntaub ntawv tseem ceeb kom nws muaj rau txhua qhov kev faib tawm. Koj tuaj yeem ua kom tsis muaj key sib koom nrog siv GlobalKTable cov khoom. Cia peb rov qab mus rau ib qho piv txwv yav dhau los los ua kom pom cov yam ntxwv no.

Txuas KStream cov khoom rau GlobalKTable cov khoom

Hauv ntu 5.3.2, peb tau ua qhov sib xyaw ua ke ntawm kev sib pauv pauv los ntawm cov neeg yuav khoom. Cov txiaj ntsig ntawm qhov kev sib sau no tau pom qee yam zoo li no:

{customerId='074-09-3705', stockTicker='GUTM'}, 17
{customerId='037-34-5184', stockTicker='CORK'}, 16

Thaum cov txiaj ntsig no tau txais lub hom phiaj, nws yuav muaj txiaj ntsig zoo dua yog tias tus neeg siv khoom lub npe thiab lub tuam txhab npe tag nrho tau tshwm sim. Txhawm rau ntxiv cov neeg siv khoom lub npe thiab lub tuam txhab npe, koj tuaj yeem ua qhov sib koom ua ke, tab sis koj yuav tsum tau ua ob qhov tseem ceeb tshaj plaws thiab rov muab faib. Nrog GlobalKTable koj tuaj yeem zam cov nqi ntawm cov haujlwm no.

Txhawm rau ua qhov no, peb yuav siv cov khoom countStream los ntawm Kev Sau Npe 5.11 (cov lej sib raug tuaj yeem pom hauv src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) thiab txuas rau ob lub GlobalKTable khoom.

Peb twb tau tham txog qhov no ua ntej, yog li kuv yuav tsis rov hais dua. Tab sis kuv nco ntsoov tias cov cai hauv toStream().map muaj nuj nqi yog abstracted rau hauv ib qho khoom muaj nuj nqi es tsis txhob ntawm inline lambda qhia rau lub hom phiaj ntawm kev nyeem ntawv.

Cov kauj ruam tom ntej yog tshaj tawm ob qhov xwm txheej ntawm GlobalKTable (cov cai qhia pom tuaj yeem pom hauv cov ntaub ntawv src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Tsev Npe 5.12).

Thov nco ntsoov tias cov npe ntawm cov ncauj lus tau piav qhia los ntawm cov npe suav nrog.

Tam sim no peb muaj tag nrho cov khoom npaj, txhua yam uas tseem tshuav yog sau cov cai rau kev sib txuas (uas tuaj yeem pom hauv cov ntaub ntawv src/main/java/bbejeck/chapter_5/GlobalKTableExample.java) (Tshooj 5.13).

Txawm hais tias muaj ob qhov sib koom ua ke hauv cov cai no, lawv raug kaw vim tias tsis muaj lawv cov txiaj ntsig raug siv cais. Cov txiaj ntsig tau tshwm sim thaum kawg ntawm tag nrho cov haujlwm.

Thaum koj khiav qhov kev koom ua ke saum toj no, koj yuav tau txais cov txiaj ntsig zoo li no:

{customer='Barney, Smith' company="Exxon", transactions= 17}

Lub ntsiab lus tseem tsis tau hloov pauv, tab sis cov txiaj ntsig no pom meej dua.

Yog tias koj suav mus rau Tshooj 4, koj twb pom ntau hom kev sib txuas hauv kev nqis tes ua. Lawv tau teev nyob rau hauv lub rooj. 5.2. Cov lus no qhia txog kev muaj peev xwm sib txuas raws li version 1.0.0 ntawm Kafka Stream; Tej yam yuav hloov pauv hauv kev tso tawm yav tom ntej.

Txhawm rau qhwv txhua yam, cia peb rov qab cov ntsiab lus: koj tuaj yeem txuas cov xwm txheej ntws (KStream) thiab hloov kho cov kwj (KTable) siv lub xeev hauv zos. Xwb, yog tias qhov loj ntawm cov ntaub ntawv siv tsis loj dhau, koj tuaj yeem siv GlobalKTable khoom. GlobalKTables replicate tag nrho cov partitions rau txhua Kafka Stream daim ntawv thov node, kom ntseeg tau tias tag nrho cov ntaub ntawv muaj nyob txawm hais tias qhov kev faib qhov tseem ceeb sib raug rau.

Tom ntej no peb yuav pom Kafka Stream feature, ua tsaug uas peb tuaj yeem soj ntsuam lub xeev cov kev hloov pauv yam tsis tas siv cov ntaub ntawv los ntawm lub ncauj lus Kafka.

5.3.5. Queryable xeev

Peb twb tau ua ntau yam haujlwm uas cuam tshuam nrog lub xeev thiab ib txwm tso tawm cov txiaj ntsig mus rau lub console (rau kev txhim kho lub hom phiaj) lossis sau rau hauv cov ncauj lus (rau cov hom phiaj tsim khoom). Thaum sau cov txiaj ntsig rau lub ntsiab lus, koj yuav tsum siv Kafka cov neeg siv khoom los saib lawv.

Kev nyeem cov ntaub ntawv los ntawm cov ncauj lus no tuaj yeem suav hais tias yog hom kev pom zoo. Rau peb lub hom phiaj, peb tuaj yeem siv lub ntsiab lus ntawm kev pom los ntawm Wikipedia: "... lub cev database khoom uas muaj cov txiaj ntsig ntawm cov lus nug. Piv txwv li, nws tuaj yeem yog cov ntawv luam tawm hauv zos ntawm cov ntaub ntawv nyob deb, lossis ib pawg ntawm cov kab thiab / lossis kab ntawm lub rooj lossis koom nrog cov txiaj ntsig, lossis cov lus xaus tau los ntawm kev sib sau "(https://en.wikipedia.org/wiki /Materialized_view).

Kafka Stream tseem tso cai rau koj los khiav cov lus nug sib tham hauv lub xeev cov khw muag khoom, tso cai rau koj ncaj qha nyeem cov ntsiab lus no. Nws yog ib qho tseem ceeb uas yuav tsum nco ntsoov tias cov lus nug mus rau lub xeev lub khw muag khoom yog kev nyeem ntawv nkaus xwb. Qhov no ua kom ntseeg tau tias koj tsis tas yuav txhawj xeeb txog kev ua yuam kev ua lub xeev tsis sib haum thaum koj daim ntawv thov ua cov ntaub ntawv.

Lub peev xwm los nug ncaj qha rau lub xeev cov khw muag khoom yog qhov tseem ceeb. Qhov no txhais tau tias koj tuaj yeem tsim cov ntawv thov dashboard yam tsis tas yuav tsum xub nqa cov ntaub ntawv los ntawm Kafka cov neeg siv khoom. Nws kuj tseem ua rau kom muaj txiaj ntsig ntawm daim ntawv thov, vim tias tsis tas yuav sau cov ntaub ntawv ntxiv:

• ua tsaug rau qhov chaw ntawm cov ntaub ntawv, lawv tuaj yeem nkag tau sai;
• duplication ntawm cov ntaub ntawv yog tshem tawm, vim nws tsis yog sau rau sab nraud cia.

Qhov tseem ceeb kuv xav kom koj nco ntsoov yog tias koj tuaj yeem nug ncaj qha los ntawm koj daim ntawv thov. Cov cib fim no muab rau koj tsis tuaj yeem overstated. Tsis txhob siv cov ntaub ntawv los ntawm Kafka thiab khaws cov ntaub ntawv hauv cov ntaub ntawv rau daim ntawv thov, koj tuaj yeem nug cov khw muag khoom hauv xeev nrog cov txiaj ntsig zoo ib yam. Cov lus nug ncaj qha mus rau lub xeev cov khw muag khoom txhais tau tias tsawg tus lej (tsis muaj cov neeg siv khoom) thiab tsawg dua software (tsis xav tau lub rooj database los khaws cov txiaj ntsig).

Peb tau npog me ntsis hauv av hauv tshooj no, yog li peb yuav tawm peb qhov kev sib tham ntawm cov lus nug sib tham tawm tsam lub xeev cov khw muag khoom tam sim no. Tab sis tsis txhob txhawj: hauv Tshooj 9, peb yuav tsim ib daim ntawv thov dashboard yooj yim nrog cov lus nug sib tham. Nws yuav siv qee qhov piv txwv los ntawm tshooj no thiab yav dhau los los qhia cov lus nug sib tham thiab seb koj tuaj yeem ntxiv lawv li cas rau Kafka Stream daim ntawv thov.

Txoj kev xaus

• KStream cov khoom sawv cev cov kwj ntawm cov xwm txheej, piv rau kev ntxig rau hauv cov ntaub ntawv. KTable cov khoom sawv cev rau cov kwj hloov tshiab, zoo li kev hloov tshiab rau cov ntaub ntawv. Qhov loj ntawm cov khoom KTable tsis loj hlob, cov ntaub ntawv qub raug hloov los ntawm cov tshiab.
• KTable cov khoom yuav tsum tau ua rau kev sib sau ua ke.
• Siv cov haujlwm windowing, koj tuaj yeem faib cov ntaub ntawv sib sau ua ke rau hauv lub thoob lub sijhawm.
• Ua tsaug rau GlobalKTable cov khoom, koj tuaj yeem nkag mus rau cov ntaub ntawv siv nyob qhov twg hauv daim ntawv thov, tsis hais txog kev faib tawm.
• Kev sib txuas ntawm KStream, KTable thiab GlobalKTable cov khoom yog ua tau.

Txog tam sim no, peb tau tsom mus rau tsim cov ntawv thov Kafka Stream siv cov qib siab KStream DSL. Txawm hais tias txoj hauv kev qib siab tso cai rau koj los tsim cov kev pabcuam zoo thiab cov ntsiab lus, siv nws sawv cev rau kev lag luam tawm. Ua hauj lwm nrog DSL KStream txhais tau tias ua kom lub ntsiab lus ntawm koj cov cai los ntawm kev txo cov qib ntawm kev tswj. Nyob rau hauv tshooj tom ntej, peb yuav saib cov qib qis tus tuav API thiab sim ua lwm yam kev lag luam tawm. Cov kev pab cuam yuav ntev dua li lawv tau ua dhau los, tab sis peb yuav tuaj yeem tsim yuav luag txhua tus neeg tuav haujlwm uas peb xav tau.

→ Xav paub ntxiv txog phau ntawv tuaj yeem nrhiav tau ntawm tus tshaj tawm lub vev xaib

→ Rau Habrozhiteli 25% luv nqi siv daim coupon - Kafka ntws

→ Thaum them nyiaj rau daim ntawv luam ntawm phau ntawv, phau ntawv hluav taws xob yuav raug xa los ntawm e-mail.

Tau qhov twg los: www.hab.com