{"id":32632,"date":"2019-10-31T21:48:08","date_gmt":"2019-10-31T18:48:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/ne-tolko-obrabotka-kak-my-sdelali-iz-kafka-streams-raspredelennuyu-bazu-dannyh-i-chto-iz-etogo-vyshlo\/"},"modified":"2019-10-31T21:48:08","modified_gmt":"2019-10-31T18:48:08","slug":"ne-tolko-obrabotka-kak-my-sdelali-iz-kafka-streams-raspredelennuyu-bazu-dannyh-i-chto-iz-etogo-vyshlo","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/ne-tolko-obrabotka-kak-my-sdelali-iz-kafka-streams-raspredelennuyu-bazu-dannyh-i-chto-iz-etogo-vyshlo","title":{"rendered":"Jo vet\u00ebm p\u00ebrpunim: Si e transformuam Kafka Streams n\u00eb nj\u00eb baz\u00eb t\u00eb dh\u00ebnash t\u00eb shp\u00ebrndar\u00eb dhe \u00e7far\u00eb doli prej saj","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

P\u00ebrsh\u00ebndetje, Habr!<\/p>\n

Ju kujtojm\u00eb se pas librit p\u00ebr Kafka<\/a><\/noindex> ne l\u00ebshuam nj\u00eb punim t\u00eb pakrahasuesh\u00ebm mbi bibliotek\u00ebn Kafka Streams API<\/a><\/noindex>. <\/p>\n

\"Jo
\n
\nDerisa komiteti vet\u00ebm po e shqyrton kufijt\u00eb e mund\u00ebsive t\u00eb k\u00ebtij instrumenti t\u00eb fuqish\u00ebm. S\u00eb fundmi, doli nj\u00eb artikull, me p\u00ebrkthimin e t\u00eb cilit d\u00ebshirojm\u00eb t'ju njohim. Autori ndan p\u00ebrvoj\u00ebn e tij se si t\u00eb shnd\u00ebrrohet Kafka Streams n\u00eb nj\u00eb depo t\u00eb shp\u00ebrndar\u00eb t\u00eb t\u00eb dh\u00ebnave. Lexim t\u00eb k\u00ebndsh\u00ebm!
\n<\/a><\/noindex>
\nBiblioteka Apache Kafka Streams<\/a><\/noindex> p\u00ebrdoret n\u00eb t\u00eb gjith\u00eb bot\u00ebn n\u00eb nd\u00ebrmarrje p\u00ebr p\u00ebrpunimin e shp\u00ebrndar\u00eb t\u00eb rrjedhave mbi Apache Kafka. Nj\u00eb nga aspektet e n\u00ebnvler\u00ebsuara t\u00eb k\u00ebtij framework \u00ebsht\u00eb se ai lejon ruajtjen e gjendjes lokale, e cila krijohet mbi baz\u00ebn e p\u00ebrpunimit t\u00eb rrjedhave.<\/p>\n

N\u00eb k\u00ebt\u00eb artikull, do t\u00eb flas p\u00ebr m\u00ebnyr\u00ebn se si n\u00eb kompanin\u00eb ton\u00eb arrit\u00ebm ta shfryt\u00ebzojm\u00eb me p\u00ebrfitim k\u00ebt\u00eb mund\u00ebsi n\u00eb zhvillimin e nj\u00eb produkti p\u00ebr sigurin\u00eb e aplikacioneve n\u00eb re. Me ndihm\u00ebn e Kafka Streams, ne krijuam mikrosh\u00ebrbime me gjendje t\u00eb ndar\u00eb, secili prej t\u00eb cil\u00ebve na sh\u00ebrben si nj\u00eb burim i besuesh\u00ebm dhe me disponueshm\u00ebri t\u00eb lart\u00eb p\u00ebr informacionin e sakt\u00eb mbi gjendjen e objekteve n\u00eb sistem. Ky \u00ebsht\u00eb nj\u00eb hap p\u00ebrpara p\u00ebr ne n\u00eb lidhje me q\u00ebndrueshm\u00ebrin\u00eb dhe leht\u00ebsin\u00eb e mb\u00ebshtetjes.<\/p>\n

N\u00ebse jeni t\u00eb interesuar p\u00ebr nj\u00eb qasje alternative, e cila lejon p\u00ebrdorimin e nj\u00eb baze t\u00eb dh\u00ebnash qendrore p\u00ebr mb\u00ebshtetje t\u00eb gjendjes formale t\u00eb objekteve tuaja \u2013 lexoni, do t\u00eb jet\u00eb interesante... <\/p>\n

Pse e konsideruam se erdhi koha t\u00eb ndryshojm\u00eb qasjet tona n\u00eb pun\u00ebn me gjendjen e ndar\u00eb <\/b><\/p>\n

Na nevojitej t\u00eb mb\u00ebshtesim gjendjen e objekteve t\u00eb ndryshme, duke u mb\u00ebshtetur n\u00eb raportet e agjent\u00ebve (p.sh.: a ishte faqja n\u00eb sulm)? Deri n\u00eb kalimin n\u00eb Kafka Streams, shpesh kemi mb\u00ebshtetur menaxhimin e gjendjes n\u00eb nj\u00eb baz\u00eb t\u00eb dh\u00ebnash qendrore (+ API sh\u00ebrbimi). Ky qasje ka disa disavantazhe: n\u00eb situata t\u00eb dh\u00ebnash intensive<\/a><\/noindex> mb\u00ebshtetjeja e koherenc\u00ebs dhe sinkronizimit shnd\u00ebrrohet n\u00eb nj\u00eb sfid\u00eb t\u00eb v\u00ebrtet\u00eb. Baza e t\u00eb dh\u00ebnave mund t\u00eb b\u00ebhet nj\u00eb ngushtic\u00eb, ose mund t\u00eb p\u00ebrfundoj\u00eb n\u00eb nj\u00eb gjendje garash<\/a><\/noindex> dhe t\u00eb vuaj\u00eb nga parashikueshm\u00ebria.<\/p>\n

\"Jo
\n
\nIlustrimi 1: nj\u00eb skenar tipik i ndarjes s\u00eb gjendjes, i hasur para kalimit n\u00eb
\nKafka dhe Kafka Streams: agjent\u00ebt raportojn\u00eb p\u00ebrfaq\u00ebsimet e tyre p\u00ebrmes API-s\u00eb, gjendja e p\u00ebrdit\u00ebsuar llogaritet p\u00ebrmes baz\u00ebs qendrore t\u00eb t\u00eb dh\u00ebnave <\/i><\/p>\n

Njoftohuni me Kafka Streams \u2013 tani \u00ebsht\u00eb e leht\u00eb t\u00eb krijoni mikrosh\u00ebrbime me gjendje t\u00eb ndar\u00eb <\/b><\/p>\n

Rreth nj\u00eb vit m\u00eb par\u00eb, ne vendos\u00ebm ta rishqyrtonim me kujdes skenar\u00ebt tan\u00eb t\u00eb pun\u00ebs me gjendjen e ndar\u00eb, p\u00ebr t\u00eb zgjidhur disa probleme. Menj\u00ebher\u00eb vendos\u00ebm t\u00eb provojm\u00eb Kafka Streams \u2013 \u00ebsht\u00eb e njohur sa e shkall\u00ebzueshme, e disponueshme dhe rezistente ndaj d\u00ebshtimeve, dhe sa e pasur \u00ebsht\u00eb funksionaliteti i saj p\u00ebr p\u00ebrpunimin e rrjedhave (p\u00ebrfshir\u00eb transformimet, duke ruajtur gjendjen). Pik\u00ebrisht ajo q\u00eb na duhej, pa p\u00ebrmendur se sa e pjekur dhe e besueshme \u00ebsht\u00eb sistemi i nd\u00ebrrimit t\u00eb mesazheve q\u00eb ndodhet n\u00eb Kafka.<\/p>\n

\u00c7do nj\u00ebri nga mikrosh\u00ebrbimet q\u00eb kemi krijuar me ruajtjen e gjendjes \u00ebsht\u00eb nd\u00ebrtuar mbi nj\u00eb instanc\u00eb t\u00eb Kafka Streams me nj\u00eb topologji mjaft t\u00eb thjesht\u00eb. Ai p\u00ebrb\u00ebhej nga 1) burimi 2) procesori me nj\u00eb depo t\u00eb p\u00ebrhershme t\u00eb \u00e7el\u00ebsave dhe vlerave 3) skema:<\/p>\n

\"Jo
\n
\nIlustrimi 2: topologjia e parazgjedhur e instancave tona t\u00eb rrjedhave p\u00ebr mikrosh\u00ebrbimet me ruajtjen e gjendjes. Vini re: k\u00ebtu ka gjithashtu nj\u00eb depo q\u00eb p\u00ebrmban metadat\u00eb mbi planifikimin. <\/i><\/p>\n

Me k\u00ebt\u00eb qasje t\u00eb re, agjent\u00ebt krijojn\u00eb mesazhe q\u00eb d\u00ebrgohen n\u00eb topikun fillestar, nd\u00ebrsa konsumator\u00ebt \u2013 le t\u00eb themi, sh\u00ebrbimi i njoftimeve me post\u00eb \u2013 pranojn\u00eb gjendjen e ndar\u00eb t\u00eb llogaritur p\u00ebrmes skem\u00ebs (topikut t\u00eb daljes).<\/p>\n

\"Jo
\n
\nIlustrimi 3: nj\u00eb shembull i ri i rrjedh\u00ebs s\u00eb detyrave p\u00ebr nj\u00eb skenar me mikrosh\u00ebrbime t\u00eb ndara: 1) agjenti krijon nj\u00eb mesazh q\u00eb d\u00ebrgohet n\u00eb topikun fillestar t\u00eb Kafka; 2) mikrosh\u00ebrbimi me gjendjen e ndar\u00eb (duke p\u00ebrdorur Kafka Streams) e p\u00ebrpunon at\u00eb dhe regjistron gjendjen e llogaritur n\u00eb topikun p\u00ebrfundimtar t\u00eb Kafka; pas k\u00ebsaj, 3) konsumator\u00ebt pranojn\u00eb gjendjen e re.<\/i><\/p>\n

Hej, dhe kjo depo e integruar e \u00e7el\u00ebsave dhe vlerave n\u00eb t\u00eb v\u00ebrtet\u00eb \u00ebsht\u00eb shum\u00eb e dobishme!<\/b><\/p>\n

Si\u00e7 u p\u00ebrmend m\u00eb sip\u00ebr, topologjia jon\u00eb me gjendjen e ndar\u00eb p\u00ebrmban nj\u00eb depo t\u00eb \u00e7el\u00ebsave dhe vlerave. Gjet\u00ebm disa opsione p\u00ebr ta p\u00ebrdorur at\u00eb, dhe dy prej tyre jan\u00eb p\u00ebrshkruar m\u00eb posht\u00eb.<\/p>\n

Opsioni #1: p\u00ebrdorimi i depos s\u00eb \u00e7el\u00ebsave dhe vlerave gjat\u00eb llogaritjeve<\/b><\/i><\/p>\n

Depo i par\u00eb i \u00e7el\u00ebsave dhe vlerave p\u00ebrmbante t\u00eb dh\u00ebna ndihm\u00ebse q\u00eb na duheshin p\u00ebr llogaritje. P\u00ebr shembull, n\u00eb disa raste, gjendja e ndar\u00eb p\u00ebrcaktohej n\u00eb baz\u00eb t\u00eb principit t\u00eb \"shumic\u00ebs s\u00eb votave\". N\u00eb depo mund t\u00eb mbaheshin t\u00eb gjitha raportet e fundit t\u00eb agent\u00ebve mbi gjendjen e nj\u00eb objekti t\u00eb caktuar. Pastaj, duke marr\u00eb nj\u00eb raport t\u00eb ri nga ndonj\u00eb agent, ne mund t\u00eb ruanim at\u00eb, t\u00eb nxirrnim nga depo raportet e t\u00eb gjith\u00eb agjenteve t\u00eb tjer\u00eb mbi gjendjen e t\u00eb nj\u00ebjtit objekt dhe t\u00eb p\u00ebrs\u00ebritnim llogaritjen.
\nN\u00eb ilustrimin 4 m\u00eb posht\u00eb tregohet se si ne hap\u00ebm aksesin n\u00eb depo t\u00eb \u00e7el\u00ebsave dhe vlerave p\u00ebr metod\u00ebn e p\u00ebrpunimit t\u00eb procesorit, n\u00eb m\u00ebnyr\u00eb q\u00eb t\u00eb mund t\u00eb p\u00ebrprocessonim nj\u00eb mesazh t\u00eb ri.<\/p>\n

\"Jo
\n
\nIlustrimi 4: hapja e aksesit n\u00eb depo t\u00eb \u00e7el\u00ebsave dhe vlerave p\u00ebr metod\u00ebn e p\u00ebrpunimit t\u00eb procesorit (pas k\u00ebsaj, n\u00eb \u00e7do skenar q\u00eb punon me gjendje t\u00eb ndar\u00eb, nevojitet implementimi i metod\u00ebs doProcess<\/code>)<\/i><\/p>\n

Varianti #2: krijimi i API CRUD mbi Kafka Streams<\/b><\/i><\/p>\n

Pas vendosjes s\u00eb rrjedh\u00ebs son\u00eb bazike t\u00eb detyrave, filluam t\u00eb provonim t\u00eb shkruanim nj\u00eb API RESTful CRUD p\u00ebr mikro-sh\u00ebrbimet tona me gjendje t\u00eb ndar\u00eb. Ne donim q\u00eb t\u00eb ishte e mundur t\u00eb nxirrnim gjendjen e disa ose t\u00eb gjith\u00eb objekteve, si dhe t\u00eb vendosnim ose fshija gjendjen e nj\u00eb objekti (kjo \u00ebsht\u00eb e dobishme p\u00ebr mb\u00ebshtetje n\u00eb an\u00ebn server).<\/p>\n

P\u00ebr t\u00eb mb\u00ebshtetur t\u00eb gjith\u00eb API Get State, sa her\u00eb q\u00eb na nevojitej t\u00eb rip\u00ebrcaktonim gjendjen gjat\u00eb p\u00ebrpunimit, ne e ruanim at\u00eb n\u00eb depo t\u00eb integruar t\u00eb \u00e7el\u00ebsave dhe vlerave. N\u00eb k\u00ebt\u00eb rast, b\u00ebhet mjaft e thjesht\u00eb t\u00eb implementosh nj\u00eb API t\u00eb till\u00eb duke p\u00ebrdorur nj\u00eb instanc\u00eb t\u00eb vet\u00ebm t\u00eb Kafka Streams, si\u00e7 tregohet n\u00eb listimin e m\u00ebposht\u00ebm:<\/p>\n

\"Jo
\n
\nIlustrimi 5: p\u00ebrdorimi i depo t\u00eb integruar t\u00eb \u00e7el\u00ebsave dhe vlerave p\u00ebr marrjen e gjendjes paraprakisht t\u00eb llogaritur t\u00eb nj\u00eb objekti <\/i><\/p>\n

P\u00ebrdit\u00ebsimi i gjendjes s\u00eb nj\u00eb objekti p\u00ebrmes API gjithashtu nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb implementohet. N\u00eb thelb, p\u00ebr k\u00ebt\u00eb nevojitet vet\u00ebm t\u00eb krijosh nj\u00eb prodhues Kafka dhe me t\u00eb t\u00eb b\u00ebsh nj\u00eb shkruarje, q\u00eb p\u00ebrmban gjendjen e re. K\u00ebshtu sigurohet q\u00eb t\u00eb gjitha mesazhet e gjeneruara p\u00ebrmes API do t\u00eb p\u00ebrpunohen pik\u00ebrisht ashtu si ato q\u00eb vijn\u00eb nga prodhues t\u00eb tjer\u00eb (p.sh., agjent\u00ebve).<\/p>\n

\"Jo
\n
\nIlustrimi 6: gjendjen e nj\u00eb objekti mund ta caktosh p\u00ebrmes nj\u00eb prodhuesi Kafka <\/i><\/p>\n

Nj\u00eb komplikim i vog\u00ebl: Kafka ka shum\u00eb parti<\/b><\/p>\n

M\u00eb pas, donim t\u00eb shp\u00ebrndanim ngarkes\u00ebn e p\u00ebrpunimit dhe t\u00eb p\u00ebrmir\u00ebsonim disponueshm\u00ebrin\u00eb duke ofruar nj\u00eb grup mikrosh\u00ebrbimesh me gjendje t\u00eb p\u00ebrbashk\u00ebt p\u00ebr \u00e7do skenar. Konfigurimi doli t\u00eb ishte mjaft i thjesht\u00eb: pasi e konfiguruan t\u00eb gjith\u00eb instancat q\u00eb t\u00eb punonin me t\u00eb nj\u00ebjtin ID t\u00eb aplikacionit (dhe me t\u00eb nj\u00ebjtin server t\u00eb ngarkes\u00ebs fillestare), pothuajse gjith\u00e7ka tjet\u00ebr u b\u00eb automatikisht. Ne gjithashtu caktuam q\u00eb \u00e7do tem\u00eb burimi do t\u00eb p\u00ebrb\u00ebhej nga disa pjes\u00eb, p\u00ebr t'i dh\u00ebn\u00eb \u00e7do instance nj\u00eb n\u00ebn-grup t\u00eb k\u00ebtyre pjes\u00ebve.<\/p>\n

D\u00ebshiroj gjithashtu t\u00eb p\u00ebrmend se k\u00ebtu \u00ebsht\u00eb e zakonshme t\u00eb b\u00ebhet nj\u00eb kopje rezerv\u00eb e magazin\u00ebs s\u00eb gjendjes, n\u00eb m\u00ebnyr\u00eb q\u00eb, p\u00ebr shembull, n\u00eb rast rikuperimi nga d\u00ebshtimi, t\u00eb mund t\u00eb transferohet kjo kopje n\u00eb nj\u00eb instanc\u00eb tjet\u00ebr. P\u00ebr \u00e7do magazin\u00eb gjendjeje n\u00eb Kafka Streams krijohet nj\u00eb tem\u00eb e replikueshme me nj\u00eb regjist\u00ebr t\u00eb ndryshimeve (n\u00eb t\u00eb cilin ndjeken p\u00ebrdit\u00ebsimet lokale). K\u00ebshtu, Kafka siguron vazhdimisht magazin\u00ebn e gjendjes. Prandaj, n\u00eb rast d\u00ebshtimi t\u00eb nj\u00eb instanc\u00ebs s\u00eb caktuar, magazina e gjendjes s\u00eb Kafka Streams mund t\u00eb rikuperohet shpejt n\u00eb nj\u00eb instanc\u00eb tjet\u00ebr, ku do t\u00eb transferohen pjes\u00ebt p\u00ebrkat\u00ebse. Testet tona treguan se kjo b\u00ebhet n\u00eb pak sekonda edhe n\u00ebse magazina p\u00ebrmban miliona regjistrime.<\/p>\n

Duke kaluar nga nj\u00eb mikrosh\u00ebrbim me gjendje t\u00eb p\u00ebrbashk\u00ebt n\u00eb nj\u00eb grup mikrosh\u00ebrbimesh, realizimi i Get State API b\u00ebhet nj\u00eb sfid\u00eb m\u00eb e madhe. N\u00eb situat\u00ebn e re, magazina e gjendjes p\u00ebr \u00e7do mikrosh\u00ebrbim p\u00ebrmban vet\u00ebm nj\u00eb pjes\u00eb t\u00eb panoram\u00ebs s\u00eb p\u00ebrgjithshme (ato objekte, \u00e7el\u00ebsat e t\u00eb cilave jan\u00eb mapuar n\u00eb nj\u00eb pjes\u00eb t\u00eb caktuar). Duhej t\u00eb p\u00ebrcaktonim se n\u00eb cil\u00ebn instanc\u00eb ndodhej gjendja e objektit q\u00eb na nevojitej, dhe ne e b\u00ebm\u00eb k\u00ebt\u00eb mbi baz\u00ebn e metadatave t\u00eb rrjedhave, si\u00e7 tregohet m\u00eb posht\u00eb:<\/p>\n

\"Jo
\n
\nIlustrimi 7: me ndihm\u00ebn e metadatave t\u00eb rrjedhave ne p\u00ebrcaktojm\u00eb se nga cil\u00eb instanc\u00eb duhet t\u00eb k\u00ebrkojm\u00eb gjendjen e objektit n\u00eb fjal\u00eb; nj\u00eb qasje e till\u00eb \u00ebsht\u00eb p\u00ebrdorur me GET ALL API.<\/i><\/p>\n

Konkluzionet kryesore<\/b><\/p>\n

Magazin\u00ebn e gjendjeve n\u00eb Kafka Streams de-facto mund t\u00eb sh\u00ebrbej\u00eb si nj\u00eb baz\u00eb t\u00eb dh\u00ebnash t\u00eb shp\u00ebrndara, <\/p>\n