{"id":30778,"date":"2019-10-31T21:37:23","date_gmt":"2019-10-31T18:37:23","guid":{"rendered":"https:\/\/prohoster.info\/blog\/opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka\/"},"modified":"2019-10-31T21:37:23","modified_gmt":"2019-10-31T18:37:23","slug":"opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka","title":{"rendered":"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Co m\u016f\u017ee p\u0159im\u011bt tak velkou spole\u010dnost jako Lamoda se zaveden\u00fdm procesem a des\u00edtkami vz\u00e1jemn\u011b propojen\u00fdch slu\u017eeb k radik\u00e1ln\u00ed zm\u011bn\u011b p\u0159\u00edstupu? Motivace m\u016f\u017ee b\u00fdt zcela r\u016fzn\u00e1: od legislativn\u00ed po inherentn\u00ed touhu v\u0161ech program\u00e1tor\u016f experimentovat.<\/p>\n<p>To v\u0161ak neznamen\u00e1, \u017ee nelze o\u010dek\u00e1vat dal\u0161\u00ed v\u00fdhody. O tom, v \u010dem konkr\u00e9tn\u011b lze z\u00edskat, pokud zavede events-driven API na Kafka, promluv\u00ed Sergey Zaika (<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/users\/fewald\/\" class=\"user_link\">fewald<\/a><\/noindex>). O zku\u0161enostech a zaj\u00edmav\u00fdch objevech bude tak\u00e9 ur\u010dit\u011b \u0159e\u010d \u2013 experimenty se bez nich neobejdou.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/7ab959ab45ec5c6565b35b18b361c0ea.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<em>Upozorn\u011bn\u00ed: Tento \u010dl\u00e1nek je zalo\u017een na materi\u00e1lech ze setk\u00e1n\u00ed, kter\u00e9 Sergey vedl v listopadu 2018 na HighLoad++. \u017div\u00e9 zku\u0161enosti Lamody s Kafka p\u0159il\u00e1kaly poslucha\u010de stejn\u011b jako jin\u00e9 p\u0159edn\u00e1\u0161ky z programu. Domn\u00edv\u00e1me se, \u017ee je to skv\u011bl\u00fd p\u0159\u00edklad toho, \u017ee je v\u017edy mo\u017en\u00e9 a nezbytn\u00e9 naj\u00edt sp\u0159\u00edzn\u011bn\u00e9 du\u0161e, a organiz\u00e1to\u0159i HighLoad++ budou nad\u00e1le usilovat o vytv\u00e1\u0159en\u00ed p\u0159\u00edjemn\u00e9 atmosf\u00e9ry pro to.<\/em><br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>O procesu<\/h2>\n<p>\nLamoda je velk\u00e1 e-commerce platforma, kter\u00e1 m\u00e1 vlastn\u00ed kontaktn\u00ed centrum, doru\u010dovac\u00ed slu\u017ebu (a \u0159adu partnersk\u00fdch), fotostudio, obrovsk\u00fd sklad a to v\u0161e funguje na vlastn\u00edm softwaru. Existuje n\u011bkolik des\u00edtek zp\u016fsob\u016f platby, b2b-partne\u0159i, kte\u0159\u00ed mohou vyu\u017e\u00edvat \u010d\u00e1st nebo v\u0161echny tyto slu\u017eby a cht\u011bj\u00ed zn\u00e1t aktu\u00e1ln\u00ed informace o sv\u00fdch produktech. Krom\u011b toho, Lamoda m\u00e1 pobo\u010dky ve t\u0159ech zem\u00edch mimo RF, kde to funguje trochu jinak. Celkov\u011b existuje pravd\u011bpodobn\u011b v\u00edce ne\u017e sto zp\u016fsob\u016f, jak konfigurovat novou objedn\u00e1vku, kter\u00e1 mus\u00ed b\u00fdt zpracov\u00e1na ur\u010dit\u00fdm zp\u016fsobem. To v\u0161e funguje s pomoc\u00ed des\u00edtek slu\u017eeb, kter\u00e9 vz\u00e1jemn\u011b komunikuj\u00ed ne v\u017edy z\u0159ejm\u00fdm zp\u016fsobem. D\u00e1le existuje centr\u00e1ln\u00ed syst\u00e9m, jeho\u017e hlavn\u00ed odpov\u011bdnost\u00ed jsou stavy objedn\u00e1vek. Naz\u00fdv\u00e1me jej BOB, pracuji s n\u00edm.<\/p>\n<h2>Refund Tool s events-driven API <\/h2>\n<p>\nSlovo events-driven je obvykl\u00e9, podrobn\u011bji si za chv\u00edli objasn\u00edme, co t\u00edm m\u00e1me na mysli. Za\u010dnu kontextem, ve kter\u00e9m jsme se rozhodli vyzkou\u0161et p\u0159\u00edstup events-driven API na Kafka. <\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/3bfdce47dd8420fc63d84645e76de647.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV ka\u017ed\u00e9m obchod\u011b, krom\u011b objedn\u00e1vek, za kter\u00e9 z\u00e1kazn\u00edci plat\u00ed, nast\u00e1vaj\u00ed okam\u017eiky, kdy je od obchodu vy\u017eadov\u00e1no vr\u00e1tit pen\u00edze, proto\u017ee z\u00e1kazn\u00edkovi zbo\u017e\u00ed nevyhovuje. Tento pom\u011brn\u011b kr\u00e1tk\u00fd proces: ov\u011b\u0159ujeme informace, pokud je to nutn\u00e9, a p\u0159ev\u00e1d\u00edme pen\u00edze. <\/p>\n<p>Ale vracen\u00ed se zkomplikovalo v d\u016fsledku zm\u011bny legislativy a museli jsme pro to implementovat samostatnou mikroservisu.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/0e358df5476e7448976e0f1147a103fb.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa\u0161e motivace:<\/p>\n<ol>\n<li><strong>Z\u00e1kon FZ-54<\/strong>\u00a0\u2014 stru\u010dn\u011b \u0159e\u010deno, z\u00e1kon vy\u017eaduje, aby se ka\u017ed\u00e1 finan\u010dn\u00ed operace, a\u0165 u\u017e jde o vracen\u00ed nebo p\u0159ij\u00edm\u00e1n\u00ed, nahl\u00e1sila da\u0148ov\u00e9mu \u00fa\u0159adu v pom\u011brn\u011b kr\u00e1tk\u00e9m SLA v \u0159\u00e1du minut. My jako e-commerce prov\u00e1d\u00edme pom\u011brn\u011b hodn\u011b operac\u00ed. Technicky to znamen\u00e1 novou zodpov\u011bdnost (a tedy novou slu\u017ebu) a \u00fapravy ve v\u0161ech zapojen\u00fdch syst\u00e9mech.<\/li>\n<li><strong>BOB split<\/strong>\u00a0\u2014 intern\u00ed projekt spole\u010dnosti na zbaven\u00ed BOBu velk\u00e9ho mno\u017estv\u00ed nepodstatn\u00fdch zodpov\u011bdnost\u00ed a sn\u00ed\u017een\u00ed jeho celkov\u00e9 slo\u017eitosti.<\/li>\n<\/ol>\n<p>\n<img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/d3ecf9961bdb372fc5f84ee9389f73ef.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNa tomto sch\u00e9matu jsou nakresleny hlavn\u00ed syst\u00e9my Lamoda. V sou\u010dasnosti v\u011bt\u0161ina z nich p\u0159edstavuje sp\u00ed\u0161e <strong>souhv\u011bzd\u00ed 5-10 mikroservis kolem zmen\u0161uj\u00edc\u00edho se monolitu<\/strong>. Pomalu rostou, ale sna\u017e\u00edme se je zmen\u0161it, proto\u017ee nasazen\u00ed vy\u010dlen\u011bn\u00e9ho fragmentu uprost\u0159ed je d\u011bsiv\u00e9 \u2013 nelze dopustit, aby spadl. V\u0161echny v\u00fdm\u011bny (\u0161ipky) mus\u00edme rezervovat a po\u010d\u00edtat s t\u00edm, \u017ee jak\u00fdkoliv z nich m\u016f\u017ee b\u00fdt nedostupn\u00fd.<\/p>\n<p>V BOBu je tak\u00e9 pom\u011brn\u011b hodn\u011b v\u00fdm\u011bn: syst\u00e9my plateb, doru\u010den\u00ed, notifikace atd. <\/p>\n<p>Technicky BOB je:<\/p>\n<ul>\n<li>~150k \u0159\u00e1dk\u016f k\u00f3du + ~100k \u0159\u00e1dk\u016f test\u016f;<\/li>\n<li>php7.2 + Zend 1 &amp; Symfony Components 3;<\/li>\n<li>&gt;100 API &amp; ~50 v\u00fdchoz\u00edch integrac\u00ed;<\/li>\n<li>4 zem\u011b se svou obchodn\u00ed logikou. <\/li>\n<\/ul>\n<p>\nNasazen\u00ed BOBu je drah\u00e9 a bolestiv\u00e9, mno\u017estv\u00ed k\u00f3du a \u00fakol\u016f, kter\u00e9 \u0159e\u0161\u00ed, je takov\u00e9, \u017ee nikdo nem\u016f\u017ee m\u00edt v\u0161e v hlav\u011b najednou. Obecn\u011b \u0159e\u010deno, je mnoho d\u016fvod\u016f, pro\u010d ho zjednodu\u0161it.<\/p>\n<h2>Proces vracen\u00ed<\/h2>\n<p>\nP\u016fvodn\u011b jsou do procesu zapojeny dv\u011b syst\u00e9my: BOB a Payment. Nyn\u00ed se p\u0159id\u00e1vaj\u00ed dal\u0161\u00ed dva:<\/p>\n<ul>\n<li>Fiscalization Service, kter\u00e1 se postar\u00e1 o probl\u00e9my s fiskalizac\u00ed a komunikaci s extern\u00edmi slu\u017ebami.<\/li>\n<li>Refund Tool, do kter\u00e9ho se jednodu\u0161e p\u0159ev\u00e1d\u011bj\u00ed nov\u00e9 v\u00fdm\u011bny, aby se nezv\u011bt\u0161oval BOB.<\/li>\n<\/ul>\n<p>\nNyn\u00ed proces vypad\u00e1 takto:<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/13c02975881ad35c61304053c604cda3.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<ol>\n<li>Do BOBu p\u0159ich\u00e1z\u00ed \u017e\u00e1dost o vr\u00e1cen\u00ed pen\u011bz.<\/li>\n<li>BOB o tom informuje Refund Tool.<\/li>\n<li>Refund Tool \u0159\u00edk\u00e1 Payment: \u201eVra\u0165 pen\u00edze\u201c.<\/li>\n<li>Payment vrac\u00ed pen\u00edze.<\/li>\n<li>Refund Tool a BOB si navz\u00e1jem synchronizuj\u00ed stavy, proto\u017ee oba to zat\u00edm pot\u0159ebuj\u00ed. Dosud nejsme p\u0159ipraveni pln\u011b p\u0159ej\u00edt na Refund Tool, proto\u017ee v BOBu existuje UI, reporty pro \u00fa\u010detnictv\u00ed a v\u016fbec spousta dat, kter\u00e1 tak jednodu\u0161e nep\u0159enesete. Mus\u00edme sed\u011bt na dvou \u017eidl\u00edch.<\/li>\n<li>Odch\u00e1z\u00ed \u017e\u00e1dost o fiskalizaci.<\/li>\n<\/ol>\n<p>\nNakonec jsme vytvo\u0159ili na Kafka ur\u010ditou ud\u00e1lostn\u00ed sb\u011brnici \u2013 event-bus, na kter\u00e9 se v\u0161echno to\u010dilo. Hur\u00e1, te\u010f m\u00e1me jednotn\u00fd bod selh\u00e1n\u00ed (sarkasmus).<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/674edd7972998b4985071f5250612c7e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fdhody a nev\u00fdhody jsou docela z\u0159ejm\u00e9. Ud\u011blali jsme sb\u011brnici, tak\u017ee nyn\u00ed v\u0161echny slu\u017eby na n\u00ed z\u00e1vis\u00ed. To usnad\u0148uje n\u00e1vrh, ale p\u0159in\u00e1\u0161\u00ed do syst\u00e9mu jednotn\u00fd bod selh\u00e1n\u00ed. Kdy\u017e spadne Kafka, proces se zastav\u00ed.<\/p>\n<h2>Co je to API \u0159\u00edzen\u00e9 ud\u00e1lostmi <\/h2>\n<p>\nDobr\u00e1 odpov\u011b\u010f na tuto ot\u00e1zku je v prezentaci Martina Fowlerea (GOTO 2017) <noindex><a rel=\"nofollow\" href=\"https:\/\/youtu.be\/STKCRSUsyPO\">\u201eMnoho v\u00fdznam\u016f architektury \u0159\u00edzen\u00e9 ud\u00e1lostmi\u201c<\/a><\/noindex>. <\/p>\n<p>Stru\u010dn\u011b, co jsme ud\u011blali:<\/p>\n<ol>\n<li>Zabalili jsme v\u0161echny asynchronn\u00ed v\u00fdm\u011bny p\u0159es <strong>\u00falo\u017ei\u0161t\u011b ud\u00e1lost\u00ed<\/strong>. M\u00edsto toho, abychom informovali p\u0159es s\u00ed\u0165 ka\u017ed\u00e9ho zainteresovan\u00e9ho spot\u0159ebitele o zm\u011bn\u011b stavu, zapisujeme do centralizovan\u00e9ho \u00falo\u017ei\u0161t\u011b ud\u00e1lost o zm\u011bn\u011b stavu a zainteresovan\u00ed spot\u0159ebitel\u00e9 si z n\u011bj \u010dtou v\u0161e, co se objev\u00ed.<\/li>\n<li>Ud\u00e1lost v tomto p\u0159\u00edpad\u011b je ozn\u00e1men\u00ed (<strong>notifications<\/strong>) o tom, \u017ee se n\u011bco n\u011bkde zm\u011bnilo. Nap\u0159\u00edklad se zm\u011bnil stav objedn\u00e1vky. Spot\u0159ebitel, kter\u00e9mu z\u00e1le\u017e\u00ed na n\u011bjak\u00fdch doprovodn\u00fdch datech ke zm\u011bn\u011b stavu, kter\u00e1 nejsou v ozn\u00e1men\u00ed, si m\u016f\u017ee zjistit jejich stav s\u00e1m.<\/li>\n<li>Maxim\u00e1ln\u00ed varianta \u2013 plnohodnotn\u00e9 event sourcing, <strong>p\u0159enos stavu<\/strong>, p\u0159i kter\u00e9m ud\u00e1lost obsahuje v\u0161echny informace pot\u0159ebn\u00e9 k zpracov\u00e1n\u00ed: odkud a do jak\u00e9ho stavu se p\u0159e\u0161lo, jak se data p\u0159esn\u011b zm\u011bnila atd. Ot\u00e1zkou je jen opr\u00e1vn\u011bnost a objem informac\u00ed, kter\u00e9 si m\u016f\u017eete dovolit uchov\u00e1vat.<\/li>\n<\/ol>\n<p>\nV r\u00e1mci spu\u0161t\u011bn\u00ed Refund Tool jsme pou\u017eili t\u0159et\u00ed variantu. To zjednodu\u0161ilo zpracov\u00e1n\u00ed ud\u00e1lost\u00ed, proto\u017ee nebylo pot\u0159eba vyhled\u00e1vat podrobn\u00e9 informace, nav\u00edc vylou\u010dilo sc\u00e9n\u00e1\u0159, kdy ka\u017ed\u00e1 nov\u00e1 ud\u00e1lost vyvol\u00e1v\u00e1 vlnu up\u0159es\u0148uj\u00edc\u00edch GET po\u017eadavk\u016f od spot\u0159ebitel\u016f.<\/p>\n<p>Slu\u017eba Refund Tool <strong>nen\u00ed zat\u00ed\u017eena<\/strong>, tak\u017ee Kafka tam sp\u00ed\u0161e slou\u017e\u00ed jako pokus ne\u017e nezbytnost. Nemysl\u00edm si, \u017ee by byl byznys r\u00e1d, kdyby se slu\u017eba vracen\u00ed pen\u011bz stala projektem s vysokou z\u00e1t\u011b\u017e\u00ed.<\/p>\n<h4>Asynchronn\u00ed v\u00fdm\u011bna AS IS<\/h4>\n<p>\nPro asynchronn\u00ed v\u00fdm\u011bny obvykle PHP odd\u011blen\u00ed pou\u017e\u00edv\u00e1 RabbitMQ. Shrom\u00e1\u017ed\u00ed data pro po\u017eadavek, daj\u00ed je do fronty a koncov\u00fd u\u017eivatel stejn\u00e9 slu\u017eby je p\u0159e\u010detl a poslal (nebo neposlal). Pro samotn\u00e9 API Lamoda aktivn\u011b vyu\u017e\u00edv\u00e1 Swagger. Navrhujeme API, popisujeme ho ve Swaggeru, generujeme klientsk\u00fd a serverov\u00fd k\u00f3d. Je\u0161t\u011b pou\u017e\u00edv\u00e1me m\u00edrn\u011b roz\u0161\u00ed\u0159en\u00fd JSON RPC 2.0. <\/p>\n<p>N\u011bkde se pou\u017e\u00edvaj\u00ed esb sb\u011brnice, n\u011bkdo \u017eije na activeMQ, ale celkov\u011b, <strong>RabbitMQ \u2014 standard<\/strong>.<\/p>\n<h4>Async exchange TO BE<\/h4>\n<p>\nP\u0159i navrhov\u00e1n\u00ed v\u00fdm\u011bny p\u0159es events-bus se objevuje analogie. Budouc\u00ed v\u00fdm\u011bnu dat popisujeme podobn\u00fdm zp\u016fsobem pomoc\u00ed popisu struktury eventu. Form\u00e1t yaml, k\u00f3dovou generaci jsme museli ud\u011blat sami, gener\u00e1tor podle specifikace vytv\u00e1\u0159\u00ed DTO a u\u010d\u00ed klienty a servery s nimi pracovat. Generace prob\u00edh\u00e1 ve dvou jazyc\u00edch \u2014 <strong>golang a php<\/strong>. To n\u00e1m umo\u017e\u0148uje udr\u017eovat knihovny v souladu. Gener\u00e1tor je naps\u00e1n v golangu a dostal jm\u00e9no gogi.<\/p>\n<p>Event-sourcing na Kafka je b\u011b\u017en\u00e1 v\u011bc. Existuje \u0159e\u0161en\u00ed od hlavn\u00ed enterprise verze Kafka Confluent, je tu tak\u00e9 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/zalando\/nakadi\">nakadi<\/a><\/noindex>, \u0159e\u0161en\u00ed od na\u0161ich \u201ebratr\u016f\u201c v dom\u00e9nov\u00e9 oblasti Zalando. Na\u0161\u00ed <strong>motivac\u00ed za\u010d\u00edt s vanilla Kafka<\/strong>\u00a0je udr\u017eet \u0159e\u0161en\u00ed zdarma, dokud se definitivn\u011b nerozhodneme, zda ho budeme \u0161iroce pou\u017e\u00edvat, a tak\u00e9 si ponechat prostor pro man\u00e9vrov\u00e1n\u00ed a dal\u0161\u00ed pr\u00e1ce: chceme podporu sv\u00e9ho <strong>JSON RPC 2.0<\/strong>, gener\u00e1tory pro oba jazyky a uvid\u00edme, co je\u0161t\u011b. <\/p>\n<p>Ironi\u00ed je, \u017ee i v tak \u0161\u0165astn\u00e9m p\u0159\u00edpad\u011b, kdy existuje p\u0159ibli\u017en\u011b analogick\u00fd podnik Zalando, kter\u00e9 ud\u011blalo p\u0159ibli\u017en\u011b podobn\u00e9 \u0159e\u0161en\u00ed, ho nem\u016f\u017eeme efektivn\u011b vyu\u017e\u00edt. <\/p>\n<p>Architektonicky p\u0159i spu\u0161t\u011bn\u00ed je vzor n\u00e1sleduj\u00edc\u00ed: \u010dteme p\u0159\u00edmo z Kafka, ale zapisujeme pouze p\u0159es events-bus. Pro \u010dten\u00ed z Kafka je k dispozici mnoho hotov\u00e9ho: broke\u0159i, vyva\u017eova\u010de a je v\u00edce m\u00e9n\u011b p\u0159ipravena na horizont\u00e1ln\u00ed \u0161k\u00e1lov\u00e1n\u00ed, co\u017e chceme zachovat. Z\u00e1pis jsme se rozhodli zabalit p\u0159es jeden Gateway aka Events-bus, a to je d\u016fvod.<\/p>\n<h3>Events-bus<\/h3>\n<p>\nNebo autobus ud\u00e1lost\u00ed. To je jednodu\u0161e stateless http gateway, kter\u00fd na sebe bere n\u011bkolik d\u016fle\u017eit\u00fdch rol\u00ed:<\/p>\n<ul>\n<li><strong>Validace produ\u010dn\u00edho procesu<\/strong>\u00a0\u2014 ov\u011b\u0159ujeme, \u017ee ud\u00e1losti vyhovuj\u00ed na\u0161\u00ed specifikaci.<\/li>\n<li><strong>Hlavn\u00ed syst\u00e9m pro ud\u00e1losti<\/strong>, tedy to je hlavn\u00ed a jedin\u00fd syst\u00e9m ve spole\u010dnosti, kter\u00fd odpov\u00edd\u00e1 na ot\u00e1zku, jak\u00e9 ud\u00e1losti s jak\u00fdmi strukturami se pova\u017euj\u00ed za platn\u00e9. Do validace pat\u0159\u00ed jednodu\u0161e typy dat a enums pro p\u0159\u00edsnou specifikaci obsahu. <\/li>\n<li><strong>Hashov\u00e1 funkce<\/strong> pro shardov\u00e1n\u00ed \u2014 struktura zpr\u00e1vy Kafka je key-value a zde se podle hash od key po\u010d\u00edt\u00e1, kam to um\u00edstit.<\/li>\n<\/ul>\n<p><\/p>\n<h3>Pro\u010d<\/h3>\n<p>\nPracujeme ve velk\u00e9 spole\u010dnosti s nastaven\u00fdmi procesy. Pro\u010d n\u011bco m\u011bnit? <strong>To je experiment<\/strong>, a o\u010dek\u00e1v\u00e1me, \u017ee z\u00edsk\u00e1me n\u011bkolik v\u00fdhod.<\/p>\n<h4>1:n+1 v\u00fdm\u011bny (jeden k mnoha)<\/h4>\n<p>\nS Kafka je velmi snadn\u00e9 p\u0159ipojit nov\u00e9 spot\u0159ebitele k API. <\/p>\n<p>P\u0159edstavte si, \u017ee m\u00e1te adres\u00e1\u0159, kter\u00fd je t\u0159eba udr\u017eovat aktu\u00e1ln\u00ed v n\u011bkolika syst\u00e9mech najednou (a tak\u00e9 v n\u011bjak\u00fdch nov\u00fdch). D\u0159\u00edve jsme vym\u00fd\u0161leli bundle, kter\u00fd realizoval set-API, a hlavn\u00edmu syst\u00e9mu jsme oznamovali adresy p\u0159\u00edjemc\u016f. Nyn\u00ed hlavn\u00ed syst\u00e9m pos\u00edl\u00e1 aktualizace do t\u00e9matu a v\u0161ichni, koho to zaj\u00edm\u00e1, \u010dtou. Objevil se nov\u00fd syst\u00e9m \u2013 p\u0159ihl\u00e1sili jsme ho k t\u00e9matu. Ano, tak\u00e9 bundle, ale jednodu\u0161\u0161\u00ed.<\/p>\n<p>V p\u0159\u00edpad\u011b refund-tool, co\u017e je vlastn\u011b kus BOB, je pro n\u00e1s pohodln\u00e9 dr\u017eet je synchronizovan\u00e9 p\u0159es Kafka. Payment \u0159\u00edk\u00e1, \u017ee pen\u00edze byly vr\u00e1ceny: BOB a RT o tom v\u011bd\u011bli, zm\u011bnili si stavy, Fiscalization Service o tom v\u011bd\u011bl a vydal \u00fa\u010dtenku.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/b01b22a333b58e87aeef0c52d40e6960.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nM\u00e1me v pl\u00e1nu vytvo\u0159it jednotnou slu\u017ebu Notifications Service, kter\u00e1 by informovala klienta o novink\u00e1ch kolem jeho objedn\u00e1vek\/vratek. V sou\u010dasnosti je tato odpov\u011bdnost rozpt\u00fdlena mezi syst\u00e9my. Budeme schopni nau\u010dit Notifications Service zachycovat z Kafka relevantn\u00ed informace a reagovat na n\u011b (a vypnout tato ozn\u00e1men\u00ed v ostatn\u00edch syst\u00e9mech). Nebude t\u0159eba \u017e\u00e1dn\u00e9 nov\u00e9 p\u0159\u00edm\u00e9 v\u00fdm\u011bny.<\/p>\n<h4>Data-driven<\/h4>\n<p>\nInformace mezi syst\u00e9my se st\u00e1v\u00e1 transparentn\u00edmi \u2013 jak\u00fdkoli \u201ekrvav\u00fd enterprise\u201c byste m\u011bli a jak objemn\u00fd by v\u00e1\u0161 backlog nebyl. V Lamod\u011b je odd\u011blen\u00ed Data Analytics, kter\u00e9 shroma\u017e\u010fuje data ze syst\u00e9m\u016f a p\u0159etv\u00e1\u0159\u00ed je do znovu pou\u017eiteln\u00fdch form\u00e1t\u016f, jak pro podnik\u00e1n\u00ed, tak pro inteligentn\u00ed syst\u00e9my. Kafka umo\u017e\u0148uje rychle jim poskytnout velk\u00e9 mno\u017estv\u00ed dat a udr\u017eovat tento informa\u010dn\u00ed tok aktu\u00e1ln\u00ed.<\/p>\n<h4>Replication log<\/h4>\n<p>\nZpr\u00e1vy nezmiz\u00ed po p\u0159e\u010dten\u00ed, jako v RabbitMQ. Kdy\u017e ud\u00e1lost obsahuje dostatek informac\u00ed pro zpracov\u00e1n\u00ed, m\u00e1me historii posledn\u00edch zm\u011bn objektu a, pokud si p\u0159ejeme, mo\u017enost tyto zm\u011bny aplikovat.<\/p>\n<p>Doba uchov\u00e1v\u00e1n\u00ed replication log z\u00e1vis\u00ed na intenzit\u011b z\u00e1pisu do tohoto t\u00e9matu, Kafka umo\u017e\u0148uje flexibiln\u011b nastavit limity pro dobu uchov\u00e1v\u00e1n\u00ed a objem dat. Pro intenzivn\u00ed t\u00e9mata je d\u016fle\u017eit\u00e9, aby v\u0161ichni p\u0159\u00edjemci stihli p\u0159e\u010d\u00edst informace d\u0159\u00edve, ne\u017e zmiz\u00ed, i v p\u0159\u00edpad\u011b kr\u00e1tkodob\u00e9 nefunk\u010dnosti. Obvykle se poda\u0159\u00ed uchov\u00e1vat data za\u00a0<strong>jednotky dn\u00ed<\/strong>, co\u017e je zcela dosta\u010duj\u00edc\u00ed pro podporu. <\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/0e08dd384155289123ebee96430c2370.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nD\u00e1le je tu trochu shrnut\u00ed dokumentace pro ty, kdo nejsou s Kafkou obezn\u00e1meni (obr\u00e1zek je tak\u00e9 z dokumentace)<\/p>\n<p>V AMQP existuj\u00ed fronty: zapisujeme zpr\u00e1vy do fronty pro konzumenta. Obvykle jednu frontu zpracov\u00e1v\u00e1 jeden syst\u00e9m s jednou a tou samou obchodn\u00ed logikou. Pokud je pot\u0159eba informovat n\u011bkolik syst\u00e9m\u016f, lze aplikaci nau\u010dit zapisovat do n\u011bkolika front nebo nastavit exchange s mechanikou fanout, kter\u00e1 je sama klonuje.<\/p>\n<p>V Kafka existuje podobn\u00e1 abstrakce <em>topic<\/em>, do kter\u00e9 zapisujete zpr\u00e1vy, ale ty nezmiz\u00ed po p\u0159e\u010dten\u00ed. Ve v\u00fdchoz\u00edm nastaven\u00ed, p\u0159i p\u0159ipojen\u00ed k Kafka, dostanete v\u0161echny zpr\u00e1vy, a tak\u00e9 m\u00e1te mo\u017enost ulo\u017eit m\u00edsto, na kter\u00e9m jste se zastavili. To znamen\u00e1, \u017ee \u010dtete sekven\u010dn\u011b, m\u016f\u017eete neozna\u010dit zpr\u00e1vu jako p\u0159e\u010dtenou, ale ulo\u017eit id, od kter\u00e9ho pak pokra\u010dujete ve \u010dten\u00ed. Id, na kter\u00e9m jste se zastavili, se naz\u00fdv\u00e1 offset a mechanika je commit offset. <\/p>\n<p>Odpov\u00eddaj\u00edc\u00edm zp\u016fsobem lze implementovat r\u016fzn\u00e9 logiky. Nap\u0159\u00edklad, n\u00e1\u0161 BOB existuje ve 4 instanc\u00edch pro r\u016fzn\u00e9 zem\u011b - Lamoda je v Rusku, Kazachst\u00e1nu, Ukrajin\u011b a B\u011blorusku. Proto\u017ee se nasazuj\u00ed odd\u011blen\u011b, maj\u00ed trochu sv\u00e9 konfigurace a svou obchodn\u00ed logiku. V zpr\u00e1v\u011b uv\u00e1d\u00edme, k jak\u00e9 zemi se vztahuje. Ka\u017ed\u00fd konzument BOB v ka\u017ed\u00e9 zemi \u010dte s r\u016fzn\u00fdmi groupId, a pokud se zpr\u00e1va na n\u011bj nevztahuje, p\u0159esko\u010d\u00ed ji, tedy ihned commitne offset +1. Pokud stejn\u00fd topic \u010dte n\u00e1\u0161 Payment Service, d\u011bl\u00e1 to s odd\u011blenou skupinou, a proto se offsety nep\u0159ekr\u00fdvaj\u00ed.<\/p>\n<p><b>Po\u017eadavky na ud\u00e1losti:<\/b><\/p>\n<ul>\n<li><strong>\u00daplnost dat. <\/strong>Cht\u011bli bychom, aby v ud\u00e1losti bylo dostatek dat, aby mohla b\u00fdt zpracov\u00e1na. <\/li>\n<\/ul>\n<p><\/p>\n<ul>\n<li><strong>Integrita. <\/strong>Delegujeme Events-bus kontrole toho, \u017ee ud\u00e1lost je konzistentn\u00ed a m\u016f\u017ee b\u00fdt zpracov\u00e1na.<\/li>\n<li><strong>Po\u0159ad\u00ed je d\u016fle\u017eit\u00e9. <\/strong>V p\u0159\u00edpad\u011b vr\u00e1cen\u00ed jsme nuceni pracovat s histori\u00ed. U notifikac\u00ed nen\u00ed po\u0159ad\u00ed d\u016fle\u017eit\u00e9, pokud jsou to homogenn\u00ed notifikace, e-mail bude stejn\u00fd bez ohledu na to, kter\u00fd objedn\u00e1vka dorazila jako prvn\u00ed. V p\u0159\u00edpad\u011b vr\u00e1cen\u00ed existuje jasn\u00fd proces; pokud zm\u011bn\u00edte po\u0159ad\u00ed, vzniknou v\u00fdjimky, refundace se nevytvo\u0159\u00ed nebo nezpracuje \u2013 dostaneme se do jin\u00e9ho stavu.<\/li>\n<li><strong>Konzistence. <\/strong>M\u00e1me \u00falo\u017ei\u0161t\u011b a nyn\u00ed m\u00edsto API vytv\u00e1\u0159\u00edme ud\u00e1losti. Pot\u0159ebujeme zp\u016fsob, jak rychle a levn\u011b p\u0159ed\u00e1vat do na\u0161ich slu\u017eeb informace o nov\u00fdch ud\u00e1lostech a o zm\u011bn\u00e1ch v ji\u017e existuj\u00edc\u00edch. Toho dosahujeme pomoc\u00ed spole\u010dn\u00e9 specifikace v samostatn\u00e9m git repozit\u00e1\u0159i a k\u00f3dov\u00fdch gener\u00e1tor\u016f. Proto jsou u n\u00e1s klienti a servery v r\u016fzn\u00fdch slu\u017eb\u00e1ch slad\u011bny.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Kafka v Lamoda<\/h2>\n<p>\nM\u00e1me t\u0159i instalace Kafka: <\/p>\n<ol>\n<li>Logy;<\/li>\n<li>R&amp;D;<\/li>\n<li>Events-bus.<\/li>\n<\/ol>\n<p>\nDnes mluv\u00edme pouze o posledn\u00edm bod\u011b. V events-busu m\u00e1me ne p\u0159\u00edli\u0161 velk\u00e9 instalace \u2013 3 brokery (servery) a celkem 27 t\u00e9mat. Obvykle jedno t\u00e9ma \u2013 to je jeden proces. Ale to je subtiln\u00ed bod a te\u010f se ho dotkneme.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/f398852689b31429cc97b4cbffcabab5.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV\u00fd\u0161e je graf rps. Proces vratek je ozna\u010den tyrkysovou \u010darou (ano, ano, ta, co le\u017e\u00ed na ose X), a r\u016f\u017eovou \u2013 proces aktualizace obsahu. <\/p>\n<p>Katalog Lamoda obsahuje miliony produkt\u016f, p\u0159i\u010dem\u017e data se neust\u00e1le aktualizuj\u00ed. Jedny kolekce vych\u00e1zej\u00ed z m\u00f3dy, na jejich m\u00edsto jsou vyd\u00e1v\u00e1ny nov\u00e9, v katalogu neust\u00e1le p\u0159ib\u00fdvaj\u00ed nov\u00e9 modely. Sna\u017e\u00edme se p\u0159edpov\u011bd\u011bt, co bude zaj\u00edmat na\u0161e z\u00e1kazn\u00edky z\u00edtra, proto neust\u00e1le nakupujeme nov\u00e9 v\u011bci, fotografujeme je a aktualizujeme v\u00fdlohu. <\/p>\n<p>R\u016f\u017eov\u00e9 hroty \u2013 to jsou aktualizace produktu, to znamen\u00e1 zm\u011bny v produktech. Je vid\u011bt, \u017ee kluci fotografovali, fotografovali, a pak najednou! \u2013 nahr\u00e1li bal\u00ed\u010dek ud\u00e1lost\u00ed.<\/p>\n<h2>P\u0159\u00edpadov\u00e9 studie Lamoda Events<\/h2>\n<p>\nPostavenou architekturu pou\u017e\u00edv\u00e1me pro takov\u00e9 operace:<\/p>\n<ul>\n<li><strong>Sledov\u00e1n\u00ed stav\u016f vratek<\/strong>: call-to-action a sledov\u00e1n\u00ed stav\u016f ze v\u0161ech zapojen\u00fdch syst\u00e9m\u016f. Platba, stavy, fiskalizace, notifikace. Zde jsme vyzkou\u0161eli p\u0159\u00edstup, vytvo\u0159ili n\u00e1stroje, shrom\u00e1\u017edili v\u0161echny chyby, napsali dokumentaci a pov\u011bd\u011bli koleg\u016fm, jak to pou\u017e\u00edvat.<\/li>\n<li><strong>Aktualizace karet produkt\u016f: <\/strong>konfigurace, meta-data, charakteristiky. \u010cte jeden syst\u00e9m (kter\u00fd zobrazuje), ale zapisuj\u00ed jich n\u011bkolik.<\/li>\n<li><strong>Email, push a sms<\/strong>: objedn\u00e1vka byla shrom\u00e1\u017ed\u011bna, objedn\u00e1vka dorazila, vr\u00e1tka byla p\u0159ijata atd., je jich hodn\u011b. <\/li>\n<li><strong>Z\u00e1soby, aktualizace skladu<\/strong>\u00a0\u2013 kvantitativn\u00ed aktualizace n\u00e1zv\u016f, prost\u011b \u010d\u00edsla: p\u0159ijet\u00ed na sklad, vr\u00e1tka. Je nutn\u00e9, aby v\u0161echny syst\u00e9my spojen\u00e9 s rezervac\u00ed zbo\u017e\u00ed pracovaly s co nejaktu\u00e1ln\u011bj\u0161\u00edmi daty. Nyn\u00ed je syst\u00e9m aktualizace z\u00e1sob docela slo\u017eit\u00fd, Kafka n\u00e1m umo\u017en\u00ed jej zjednodu\u0161it.<\/li>\n<li><strong>Anal\u00fdza dat<\/strong> (R&amp;D-odd\u011blen\u00ed), ML-n\u00e1stroje, anal\u00fdza, statistika. Chceme, aby informace byly p\u0159ehledn\u00e9 \u2013 k tomu je Kafka dob\u0159e vhodn\u00e1.<\/li>\n<\/ul>\n<p>\nTe\u010f p\u0159ich\u00e1z\u00ed zaj\u00edmav\u011bj\u0161\u00ed \u010d\u00e1st o zku\u0161enostech a zaj\u00edmav\u00fdch objevech, kter\u00e9 se objevily za posledn\u00edch \u0161est m\u011bs\u00edc\u016f.<\/p>\n<h2>Probl\u00e9my n\u00e1vrhu<\/h2>\n<p>\nP\u0159edstavme si, \u017ee chceme ud\u011blat novou v\u011bc \u2013 nap\u0159\u00edklad p\u0159ev\u00e9st cel\u00fd proces dod\u00e1n\u00ed na Kafka. V sou\u010dasnosti je \u010d\u00e1st procesu implementov\u00e1na v Order Processing v BOB. P\u0159ed p\u0159ed\u00e1n\u00edm objedn\u00e1vky do slu\u017eby dod\u00e1n\u00ed, p\u0159em\u00edst\u011bn\u00edm do meziskladu a dal\u0161\u00edmi z\u00e1le\u017eitostmi stoj\u00ed statusov\u00e1 model. Existuje cel\u00fd monolit, dokonce dva, plus spousta API v\u011bnovan\u00fdch dod\u00e1n\u00ed. V\u011bd\u00ed o dod\u00e1n\u00ed daleko v\u00edce. <\/p>\n<p>Zd\u00e1 se, \u017ee se jedn\u00e1 o podobn\u00e9 oblasti, ale pro Order Processing v BOB a pro dodac\u00ed syst\u00e9m se stavy li\u0161\u00ed. Nap\u0159\u00edklad n\u011bkter\u00e9 kur\u00fdrn\u00ed slu\u017eby neodes\u00edlaj\u00ed mezistavy a pouze kone\u010dn\u00e9: \"doru\u010deno\" nebo \"ztraceno\". Jin\u00e9, naopak, velmi podrobn\u011b informuj\u00ed o pohybu zbo\u017e\u00ed. Ka\u017ed\u00fd m\u00e1 sv\u00e1 pravidla validace: pro n\u011bkoho je platn\u00fd e-mail, znamen\u00e1 to, \u017ee bude zpracov\u00e1n; pro jin\u00e9 je neplatn\u00fd, ale objedn\u00e1vka bude i tak zpracov\u00e1na, proto\u017ee existuje telefon pro kontakt, a n\u011bkdo jin\u00fd \u0159ekne, \u017ee takovou objedn\u00e1vku v\u016fbec zpracov\u00e1vat nebudou.<\/p>\n<h3>Tok dat<\/h3>\n<p>\nV p\u0159\u00edpad\u011b Kafky vyvst\u00e1v\u00e1 ot\u00e1zka organizace toku dat. Tato \u00faloha souvis\u00ed s v\u00fdb\u011brem strategie podle n\u011bkolika bod\u016f, projdeme si je v\u0161echny.<\/p>\n<h4>Do jednoho t\u00e9matu nebo do r\u016fzn\u00fdch?<\/h4>\n<p>\nM\u00e1me specifikaci ud\u00e1losti. V BOB p\u00ed\u0161eme, \u017ee je t\u0159eba doru\u010dit tuto objedn\u00e1vku a uv\u00e1d\u00edme: \u010d\u00edslo objedn\u00e1vky, jej\u00ed slo\u017een\u00ed, n\u011bkter\u00e9 SKU a \u010d\u00e1rov\u00e9 k\u00f3dy atd. Kdy\u017e zbo\u017e\u00ed doraz\u00ed na sklad, dod\u00e1n\u00ed m\u016f\u017ee z\u00edskat stavy, \u010dasov\u00e9 zna\u010dky a v\u0161e pot\u0159ebn\u00e9. Ale d\u00e1le chceme v BOB z\u00edsk\u00e1vat aktualizace k t\u011bmto dat\u016fm. Vznik\u00e1 n\u00e1m zp\u011btn\u00fd proces z\u00edsk\u00e1v\u00e1n\u00ed dat z dod\u00e1n\u00ed. Je to ta sam\u00e1 ud\u00e1lost? Nebo se jedn\u00e1 o samostatnou v\u00fdm\u011bnu, kter\u00e1 si zaslou\u017e\u00ed samostatn\u00e9 t\u00e9ma?<\/p>\n<p>Pravd\u011bpodobn\u011b budou hodn\u011b podobn\u00e9, a poku\u0161en\u00ed ud\u011blat jedno t\u00e9ma je opodstatn\u011bn\u00e9, proto\u017ee samostatn\u00e9 t\u00e9ma znamen\u00e1 samostatn\u00e9 spot\u0159ebitele, samostatn\u00e9 konfigurace, samostatnou generaci toho v\u0161eho. Ale nen\u00ed to jist\u00e9.<\/p>\n<h4>Nov\u00e9 pole nebo nov\u00e1 ud\u00e1lost?<\/h4>\n<p>\nAle pokud pou\u017eijeme ty sam\u00e9 ud\u00e1losti, objev\u00ed se jin\u00fd probl\u00e9m. Nap\u0159\u00edklad ne v\u0161echny doru\u010dovac\u00ed syst\u00e9my dok\u00e1\u017eou vygenerovat takov\u00e9 DTO, kter\u00e9 bude schopn\u00e9 generovat BOB. Pos\u00edl\u00e1me jim id, a oni je neuchov\u00e1vaj\u00ed, proto\u017ee je nepot\u0159ebuj\u00ed, a z pohledu zah\u00e1jen\u00ed procesu event-bus je toto pole povinn\u00e9. <\/p>\n<p>Pokud stanov\u00edme pro event-bus pravidlo, \u017ee toto pole je povinn\u00e9, mus\u00edme v BOB nebo v handleru po\u010d\u00e1te\u010dn\u00ed ud\u00e1losti nastavit dal\u0161\u00ed valida\u010dn\u00ed pravidla. Validace se za\u010d\u00edn\u00e1 \u0161\u00ed\u0159it po slu\u017eb\u011b \u2013 to nen\u00ed moc pohodln\u00e9.<\/p>\n<p>Dal\u0161\u00edm probl\u00e9mem je poku\u0161en\u00ed inkrement\u00e1ln\u00edho v\u00fdvoje. \u0158\u00edkaj\u00ed n\u00e1m, \u017ee je pot\u0159eba n\u011bco p\u0159idat do ud\u00e1losti, a mo\u017en\u00e1, pokud se na to dob\u0159e zamysl\u00edme, m\u011blo by to b\u00fdt zvl\u00e1\u0161tn\u00ed ud\u00e1lost\u00ed. Ale v na\u0161\u00ed sch\u00e9matu je zvl\u00e1\u0161tn\u00ed ud\u00e1lost\u00ed zvl\u00e1\u0161tn\u00ed t\u00e9ma. Zvl\u00e1\u0161tn\u00ed t\u00e9ma \u2013 to je cel\u00fd ten proces, kter\u00fd jsem popsal v\u00fd\u0161e. U v\u00fdvoj\u00e1\u0159e vznik\u00e1 poku\u0161en\u00ed jednodu\u0161e p\u0159idat do JSON sch\u00e9matu dal\u0161\u00ed pole a znovu vygenerovat.<\/p>\n<p>V p\u0159\u00edpad\u011b refunds jsme tak za p\u016fl roku do\u0161li k ud\u00e1losti ud\u00e1lost\u00ed. M\u011bli jsme jedno meta-ud\u00e1lost, kter\u00e1 se naz\u00fdv\u00e1 refund update, kde bylo pole type, popisuj\u00edc\u00ed, v \u010dem p\u0159esn\u011b tento update spo\u010d\u00edv\u00e1. Z toho byly \"skv\u011bl\u00e9\" sp\u00edna\u010de s valid\u00e1tory, kter\u00e9 \u0159\u00edkaly, jak je t\u0159eba tuto ud\u00e1lost s t\u00edmto typem validovat.<\/p>\n<h4>Verzov\u00e1n\u00ed ud\u00e1lost\u00ed<\/h4>\n<p>\nPro validaci zpr\u00e1v v Kafka lze pou\u017e\u00edt <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.confluent.io\/current\/schema-registry\/docs\/index.html\">Avro<\/a><\/noindex>, ale muselo se na to hned myslet a pou\u017e\u00edt Confluent. V na\u0161em p\u0159\u00edpad\u011b s verzov\u00e1n\u00edm mus\u00edme b\u00fdt opatrn\u00ed. Ne v\u017edy bude mo\u017en\u00e9 p\u0159e\u010d\u00edst zpr\u00e1vy z replication log, proto\u017ee model \"odjel\". V podstat\u011b se sna\u017e\u00edme vytv\u00e1\u0159et verze tak, aby model byl zp\u011btn\u011b kompatibiln\u00ed: nap\u0159\u00edklad ud\u011blat pole do\u010dasn\u011b nepovinn\u00fdm. Pokud jsou rozd\u00edly p\u0159\u00edli\u0161 siln\u00e9, za\u010dneme ps\u00e1t do nov\u00e9ho t\u00e9matu a klienty p\u0159evedeme, kdy\u017e do\u010dtou star\u00e9.<\/p>\n<h4>Z\u00e1ruka po\u0159ad\u00ed \u010dten\u00ed \u010d\u00e1st\u00ed<\/h4>\n<p>\nT\u00e9mata uvnit\u0159 Kafka jsou rozd\u011blena na \u010d\u00e1sti. To nen\u00ed moc d\u016fle\u017eit\u00e9, dokud navrhujeme entity a v\u00fdm\u011bny, ale d\u016fle\u017eit\u00e9 je, kdy\u017e se rozhodujeme, jak to konzumovat a \u0161k\u00e1lovat.<\/p>\n<p>Obvykle zapisujete do jednoho t\u00e9matu v Kafka. Ve v\u00fdchoz\u00edm nastaven\u00ed je vyu\u017eit jeden partition a v\u0161echny zpr\u00e1vy tohoto t\u00e9matu do n\u011bj spadaj\u00ed. Spot\u0159ebitel pak tyto zpr\u00e1vy \u010dte po sob\u011b. P\u0159edstavme si, \u017ee nyn\u00ed pot\u0159ebujete syst\u00e9m roz\u0161\u00ed\u0159it tak, aby zpr\u00e1vy \u010detli dva r\u016fzn\u00ed spot\u0159ebitel\u00e9. Pokud nap\u0159\u00edklad pos\u00edl\u00e1te SMS, m\u016f\u017eete \u0159\u00edct, aby Kafka vytvo\u0159ila dal\u0161\u00ed partition, a Kafka za\u010dne rozd\u011blovat zpr\u00e1vy na dv\u011b \u010d\u00e1sti \u2014 p\u016flku sem, p\u016flku tam. <\/p>\n<p>Jak je Kafka d\u011bl\u00ed? Ka\u017ed\u00e1 zpr\u00e1va m\u00e1 t\u011blo (ve kter\u00e9m uchov\u00e1v\u00e1me JSON) a m\u00e1 kl\u00ed\u010d. K tomuto kl\u00ed\u010di lze p\u0159ipojit hashovac\u00ed funkci, kter\u00e1 ur\u010d\u00ed, do kter\u00e9ho partition zpr\u00e1va spadne.<\/p>\n<p>V na\u0161em p\u0159\u00edpad\u011b s refundacemi je to d\u016fle\u017eit\u00e9, proto\u017ee pokud vezmeme dva partitions, existuje \u0161ance, \u017ee paraleln\u00ed spot\u0159ebitel zpracuje druhou ud\u00e1lost d\u0159\u00edve ne\u017e prvn\u00ed a to by byl probl\u00e9m. Hashovac\u00ed funkce zaru\u010duje, \u017ee zpr\u00e1vy se stejn\u00fdm kl\u00ed\u010dem skon\u010d\u00ed ve stejn\u00e9m partition. <\/p>\n<h4>Ud\u00e1losti vs p\u0159\u00edkazy<\/h4>\n<p>\nTo je dal\u0161\u00ed probl\u00e9m, se kter\u00fdm jsme se setkali. Ud\u00e1lost je n\u011bjak\u00e1 ud\u00e1lost: \u0159\u00edk\u00e1me, \u017ee se n\u011bco n\u011bkde stalo (something_happened), nap\u0159\u00edklad polo\u017eka byla zru\u0161ena nebo do\u0161lo k refundaci. Pokud tyto ud\u00e1losti n\u011bkdo naslouch\u00e1, pak podle \"polo\u017eka byla zru\u0161ena\" vznikne entita refundace a \"do\u0161lo k refundaci\" se zap\u00ed\u0161e n\u011bkam do nastaven\u00ed.<\/p>\n<p>Ale \u010dasto, kdy\u017e navrhujete ud\u00e1losti, p\u0159ece jen nechcete je ps\u00e1t zbyte\u010dn\u011b \u2014 po\u010d\u00edt\u00e1te s t\u00edm, \u017ee je n\u011bkdo bude \u010d\u00edst. Velk\u00fdm poku\u0161en\u00edm je napsat ne something_happened (item_canceled, refund_refunded), ale something_should_be_done. Nap\u0159\u00edklad polo\u017eka je p\u0159ipravena na vr\u00e1cen\u00ed.<\/p>\n<p>Na jedn\u00e9 stran\u011b to nazna\u010duje, jak bude ud\u00e1lost pou\u017eita. Na druh\u00e9 stran\u011b to mnohem m\u00e9n\u011b p\u0159ipom\u00edn\u00e1 norm\u00e1ln\u00ed n\u00e1zev ud\u00e1losti. Nav\u00edc odtud u\u017e je to bl\u00edzko k p\u0159\u00edkazu do_something. Ale nem\u00e1te \u017e\u00e1dnou z\u00e1ruku, \u017ee tuto ud\u00e1lost n\u011bkdo p\u0159e\u010detl; a pokud ji p\u0159e\u010detl, tak \u00fasp\u011b\u0161n\u011b; a pokud ji p\u0159e\u010detl \u00fasp\u011b\u0161n\u011b, tak n\u011bco ud\u011blal, a to n\u011bco pro\u0161lo \u00fasp\u011b\u0161n\u011b. V okam\u017eiku, kdy se ud\u00e1lost st\u00e1v\u00e1 do_something, je pot\u0159ebn\u00e1 zp\u011btn\u00e1 vazba, a to je probl\u00e9m.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/b755d91208092bd9791a41ce4633fb48.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV asynchronn\u00ed komunikaci v RabbitMQ, kdy\u017e p\u0159e\u010dtete zpr\u00e1vu, jdete na http, m\u00e1te odpov\u011b\u010f \u2014 aspo\u0148 \u017ee byla zpr\u00e1va p\u0159ijata. Kdy\u017e zapisujete do Kafka, m\u00e1te zpr\u00e1vu, \u017ee jste zapsali do Kafka, ale o tom, jak byla zpracov\u00e1na, nev\u00edte v\u016fbec nic. <\/p>\n<p>Proto\u017ee jsme v na\u0161em p\u0159\u00edpad\u011b museli zav\u00e9st odpov\u00eddaj\u00edc\u00ed ud\u00e1lost a nastavit monitorov\u00e1n\u00ed tak, aby, pokud do\u0161lo k ur\u010dit\u00e9mu po\u010dtu ud\u00e1lost\u00ed b\u011bhem ur\u010dit\u00e9 doby, m\u011blo tak\u00e9 p\u0159ij\u00edt stejn\u00e9 mno\u017estv\u00ed odpov\u00eddaj\u00edc\u00edch ud\u00e1lost\u00ed. Pokud k tomu nedo\u0161lo, zd\u00e1 se, \u017ee n\u011bco \u0161lo \u0161patn\u011b. Nap\u0159\u00edklad pokud jsme poslali ud\u00e1lost \u201eitem_ready_to_refund\u201c, o\u010dek\u00e1v\u00e1me, \u017ee refundace bude vytvo\u0159ena, klientovi se vr\u00e1t\u00ed pen\u00edze a n\u00e1m p\u0159ijde ud\u00e1lost \u201emoney_refunded\u201c. Ale to nen\u00ed jist\u00e9, proto pot\u0159ebujeme monitorov\u00e1n\u00ed.<\/p>\n<h3>Nuance<\/h3>\n<p>\nExistuje docela z\u0159ejm\u00fd probl\u00e9m: pokud \u010dtete z topiku sekven\u010dn\u011b a m\u00e1te n\u011bjakou \u0161patnou zpr\u00e1vu, consumer spadne a d\u00e1l se nedostanete. Mus\u00edte <strong>zastavit v\u0161echny consumery<\/strong>, potvrdit offset d\u00e1le, aby bylo mo\u017en\u00e9 pokra\u010dovat ve \u010dten\u00ed.<\/p>\n<p>O tom jsme v\u011bd\u011bli, na to jsme se p\u0159ipravili, a stejn\u011b se to stalo. A stalo se to, proto\u017ee ud\u00e1lost byla platn\u00e1 z pohledu events-bus, ud\u00e1lost byla platn\u00e1 z pohledu valida\u010dn\u00edho n\u00e1stroje aplikace, ale nebyla platn\u00e1 z pohledu PostgreSQL, proto\u017ee m\u00e1me v jednom syst\u00e9mu MySQL s UNSIGNED INT, a v pr\u00e1v\u011b napsan\u00e9m syst\u00e9mu byl PostgreSQL prost\u011b s INT. Jeho velikost je o n\u011bco men\u0161\u00ed a Id se neve\u0161lo. Symfony spadlo s v\u00fdjimkou. Samoz\u0159ejm\u011b jsme v\u00fdjimku zachytili, proto\u017ee jsme na to byli p\u0159ipraveni a cht\u011bli jsme potvrdit tento offset, ale p\u0159edt\u00edm jsme cht\u011bli inkrementovat po\u010d\u00edtadlo probl\u00e9m\u016f, kdy\u017e u\u017e byla zpr\u00e1va zpracov\u00e1na ne\u00fasp\u011b\u0161n\u011b. Po\u010d\u00edtadla v tomto projektu tak\u00e9 le\u017e\u00ed v datab\u00e1zi, a Symfony u\u017e uzav\u0159elo komunikaci s datab\u00e1z\u00ed, a druh\u00e1 v\u00fdjimka zabila cel\u00fd proces bez \u0161ance potvrdit offset.<\/p>\n<p>N\u011bjakou dobu slu\u017eba le\u017eela \u2014 na\u0161t\u011bst\u00ed, s Kafka to nen\u00ed tak stra\u0161n\u00e9, proto\u017ee zpr\u00e1vy z\u016fst\u00e1vaj\u00ed. Kdy\u017e se pr\u00e1ce obnov\u00ed, bude mo\u017en\u00e9 je do\u010d\u00edst. To je pohodln\u00e9.<\/p>\n<p>Kafka m\u00e1 mo\u017enost p\u0159es n\u00e1stroje nastavit libovoln\u00fd offset. Ale abyste to ud\u011blali, mus\u00edte zastavit v\u0161echny consumery \u2014 v na\u0161em p\u0159\u00edpad\u011b p\u0159ipravit samostatnou verzi, ve kter\u00e9 nebudou consumery, redeployments. Potom lze v Kafce p\u0159es n\u00e1stroje posunout offset, a zpr\u00e1va projde.<\/p>\n<p>Dal\u0161\u00ed nuance \u2014 <strong>replication log vs rdkafka.so<\/strong>\u00a0\u2014 souvis\u00ed se specifiky na\u0161eho projektu. Pou\u017e\u00edv\u00e1me PHP, a v PHP obvykle v\u0161echny knihovny komunikuj\u00ed s Kafkou prost\u0159ednictv\u00edm repozit\u00e1\u0159e rdkafka.so, a potom n\u00e1sleduje n\u011bjak\u00fd obal. Mo\u017en\u00e1 je to na\u0161e osobn\u00ed pot\u00ed\u017e, ale uk\u00e1zalo se, \u017ee prost\u011b si p\u0159e\u010d\u00edst kousek u\u017e jednou p\u0159e\u010dten\u00e9ho nen\u00ed tak snadn\u00e9. Celkov\u011b jsme m\u011bli softwarov\u00e9 probl\u00e9my.<\/p>\n<p>Pokud se vr\u00e1t\u00edme k vlastnostem pr\u00e1ce s partitions, p\u0159\u00edmo v dokumentaci je uvedeno <strong>consumers &gt;= topic partitions<\/strong>. Ale dozv\u011bd\u011bl jsem se o tom mnohem pozd\u011bji, ne\u017e bych cht\u011bl. Pokud chcete \u0161k\u00e1lovat a m\u00edt dva konzumenty, pot\u0159ebujete alespo\u0148 dv\u011b partitions. To znamen\u00e1, \u017ee pokud m\u00e1te jednu partition, ve kter\u00e9 se nahromadilo 20 tis\u00edc zpr\u00e1v, a ud\u011bl\u00e1te novou, po\u010det zpr\u00e1v se vyrovn\u00e1 brzy ne tak snadno. Proto, abyste m\u011bli dva paraleln\u00ed konzumenty, mus\u00edte se zab\u00fdvat partitions.<\/p>\n<h2>Monitoring<\/h2>\n<p>\nMysl\u00edm, \u017ee podle toho, jak monitorujeme, bude je\u0161t\u011b jasn\u011bj\u0161\u00ed, jak\u00e9 probl\u00e9my existuj\u00ed v sou\u010dasn\u00e9m p\u0159\u00edstupu.<\/p>\n<p>Nap\u0159\u00edklad po\u010d\u00edt\u00e1me, kolik produkt\u016f v datab\u00e1zi ned\u00e1vno zm\u011bnilo status, a podle t\u011bchto zm\u011bn by se m\u011bly ud\u00e1t ud\u00e1losti, a pos\u00edl\u00e1me toto \u010d\u00edslo do na\u0161eho monitorovac\u00edho syst\u00e9mu. Pot\u00e9 z Kafky dost\u00e1v\u00e1me druh\u00e9 \u010d\u00edslo, kolik ud\u00e1lost\u00ed skute\u010dn\u011b bylo zaps\u00e1no. Je z\u0159ejm\u00e9, \u017ee rozd\u00edl mezi t\u011bmito dv\u011bma \u010d\u00edsly by m\u011bl b\u00fdt v\u017edy nulov\u00fd.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/07d8ed08514f2fb97d9019466b96342c.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKrom\u011b toho je t\u0159eba monitorovat, jak si vede producent, jestli events-bus p\u0159ijal zpr\u00e1vy, a jak si vede konzument. Nap\u0159\u00edklad na grafech n\u00ed\u017ee m\u00e1 Refund Tool v\u0161echno v po\u0159\u00e1dku, ale BOB m\u00e1 evidentn\u011b n\u011bjak\u00e9 probl\u00e9my (modr\u00e9 vrcholy).<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/57112dbe70d2b388c53f19c34cca6f63.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJ\u00e1 u\u017e jsem zm\u00ednil consumer-group lag. Ve zjednodu\u0161en\u00e9 podob\u011b jde o po\u010det nep\u0159e\u010dten\u00fdch zpr\u00e1v. Celkov\u011b na\u0161i konzumenti pracuj\u00ed rychle, tak\u017ee lag obvykle \u010din\u00ed 0, ale ob\u010das m\u016f\u017ee nastat kr\u00e1tkodob\u00fd vrchol. Kafka to um\u00ed z krabice, ale je t\u0159eba nastavit ur\u010dit\u00fd interval. <\/p>\n<p>Existuje projekt <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/linkedin\/Burrow\">Burrow<\/a><\/noindex>, kter\u00fd v\u00e1m poskytne v\u00edce informac\u00ed o Kafce. Jednodu\u0161e pomoc\u00ed API vrac\u00ed stav podle consumer-group, jak si tato skupina vede. Krom\u011b OK a Failed existuje tak\u00e9 warning, a m\u016f\u017eete zjistit, \u017ee va\u0161i konzumenti nezvl\u00e1daj\u00ed tempo produkce - nest\u00edhaj\u00ed \u010d\u00edst to, co se zapisuje. Syst\u00e9m je dost chytr\u00fd, je pohodln\u00e9 ho pou\u017e\u00edvat. <\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/cece8495801e187b155487a802e1a35a.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nTakto vypad\u00e1 odpov\u011b\u010f podle API. Zde je skupina bob-live-fifa, partition refund.update.v1, stav OK, lag 0 - posledn\u00ed kone\u010dn\u00fd offset je takov\u00fd.<\/p>\n<p><img decoding=\"async\" alt=\"Zku\u0161enosti s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka\" src=\"\/wp-content\/uploads\/2019\/04\/1538b2ccc390e9b83075f54566f1039b.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nMonitoring <strong>updated_at SLA (stuck)<\/strong> U\u017e jsem o tom mluvil. Nap\u0159\u00edklad, kdy\u017e produkt p\u0159e\u0161el do stavu, \u017ee je p\u0159ipraven k vr\u00e1cen\u00ed. Nastav\u00edme Cron, kter\u00fd \u0159\u00edk\u00e1, \u017ee pokud za 5 minut tento objekt nep\u0159ejde do refundace (pen\u00edze vrac\u00edme p\u0159es platebn\u00ed syst\u00e9my velmi rychle), tak se ur\u010dit\u011b n\u011bco pokazilo a je to jasn\u00fd p\u0159\u00edpad pro podporu. Proto vezmeme cron, kter\u00fd sleduje tyto v\u011bci, a pokud jsou v\u011bt\u0161\u00ed ne\u017e 0, po\u0161le upozorn\u011bn\u00ed.<\/p>\n<p><b>Shrnuto, pou\u017eit\u00ed ud\u00e1lost\u00ed je pohodln\u00e9, kdy\u017e<\/b>:<\/p>\n<ul>\n<li>informace pot\u0159ebuj\u00ed n\u011bkolik syst\u00e9m\u016f;<\/li>\n<li>v\u00fdsledek zpracov\u00e1n\u00ed nen\u00ed d\u016fle\u017eit\u00fd;<\/li>\n<li>ud\u00e1lost\u00ed je m\u00e1lo nebo jsou ud\u00e1losti mal\u00e9. <\/li>\n<\/ul>\n<blockquote><p>Na prvn\u00ed pohled m\u00e1 \u010dl\u00e1nek velmi konkr\u00e9tn\u00ed t\u00e9ma - asynchronn\u00ed API na Kafka, ale v souvislosti s n\u00edm bych okam\u017eit\u011b cht\u011bl doporu\u010dit spoustu v\u011bc\u00ed.<br \/>\nZa prv\u00e9, n\u00e1sleduj\u00edc\u00ed <noindex><a rel=\"nofollow\" href=\"https:\/\/www.highload.ru\/\">HighLoad++<\/a><\/noindex> nen\u00ed t\u0159eba \u010dekat do listopadu, u\u017e v dubnu bude jeho peterbursk\u00e1 verze, a v \u010dervnu se budeme bavit o vysok\u00fdch z\u00e1t\u011b\u017e\u00edch v Novosibirsku.<br \/>\nZa druh\u00e9, autor zpr\u00e1vy Sergey Zaika je \u010dlenem programov\u00e9ho v\u00fdboru na\u0161\u00ed nov\u00e9 konference o \u0159\u00edzen\u00ed znalost\u00ed. <noindex><a rel=\"nofollow\" href=\"https:\/\/knowledgeconf.ru\/2019\">KnowledgeConf<\/a><\/noindex>Konference je jednodenn\u00ed, prob\u011bhne 26. dubna, ale m\u00e1 velmi nabit\u00fd program.<br \/>\nA je\u0161t\u011b v kv\u011btnu bude <noindex><a rel=\"nofollow\" href=\"https:\/\/phprussia.ru\/2019\">PHP Russia<\/a><\/noindex> a\u00a0<noindex><a rel=\"nofollow\" href=\"https:\/\/ritfest.ru\/2019\">RIT++<\/a><\/noindex> (s DevOpsConf v r\u00e1mci) - tam je\u0161t\u011b m\u016f\u017eete navrhnout sv\u00e9 t\u00e9ma, pod\u011blit se o sv\u00e9 zku\u0161enosti a st\u011b\u017eovat si na sv\u00e9 nabit\u00e9 \u0161r\u00e1my.<\/p><\/blockquote>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/oleg-bunin\/blog\/445424\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u0437\u0430\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u0442\u0430\u043a\u0443\u044e \u0431\u043e\u043b\u044c\u0448\u0443\u044e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044e \u043a\u0430\u043a Lamoda \u0441\u00a0\u043e\u0442\u043b\u0430\u0436\u0435\u043d\u043d\u044b\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u043c \u0438\u00a0\u0434\u0435\u0441\u044f\u0442\u043a\u0430\u043c\u0438 \u0432\u0437\u0430\u0438\u043c\u043e\u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0435\u043d\u043d\u043e \u043c\u0435\u043d\u044f\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434? \u041c\u043e\u0442\u0438\u0432\u0430\u0446\u0438\u044f \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c \u0441\u043e\u0432\u0435\u0440\u0448\u0435\u043d\u043d\u043e \u0440\u0430\u0437\u043d\u0430\u044f: \u043e\u0442\u00a0\u0437\u0430\u043a\u043e\u043d\u043e\u0434\u0430\u0442\u0435\u043b\u044c\u043d\u043e\u0439 \u0434\u043e\u00a0\u043f\u0440\u0438\u0441\u0443\u0449\u0435\u0433\u043e \u0432\u0441\u0435\u043c \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u0438\u0441\u0442\u0430\u043c \u0436\u0435\u043b\u0430\u043d\u0438\u044f \u044d\u043a\u0441\u043f\u0435\u0440\u0438\u043c\u0435\u043d\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c. \u041d\u043e\u00a0\u044d\u0442\u043e \u0432\u043e\u0432\u0441\u0435 \u043d\u0435\u00a0\u0437\u043d\u0430\u0447\u0438\u0442, \u0447\u0442\u043e \u043d\u0435\u043b\u044c\u0437\u044f \u0440\u0430\u0441\u0441\u0447\u0438\u0442\u044b\u0432\u0430\u0442\u044c \u043d\u0430\u00a0\u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u0443\u044e \u0432\u044b\u0433\u043e\u0434\u0443. \u0412\u00a0\u0447\u0435\u043c \u043a\u043e\u043d\u043a\u0440\u0435\u0442\u043d\u043e \u043c\u043e\u0436\u043d\u043e \u0432\u044b\u0438\u0433\u0440\u0430\u0442\u044c, \u0435\u0441\u043b\u0438 \u0432\u043d\u0435\u0434\u0440\u0438\u0442\u044c events-driven API \u043d\u0430\u00a0Kafka, \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0435\u0442 \u0421\u0435\u0440\u0433\u0435\u0439 \u0417\u0430\u0438\u043a\u0430 (fewald). \u041f\u0440\u043e \u043d\u0430\u0431\u0438\u0442\u044b\u0435 \u0448\u0438\u0448\u043a\u0438 \u0438\u00a0\u0438\u043d\u0442\u0435\u0440\u0435\u0441\u043d\u044b\u0435 \u043e\u0442\u043a\u0440\u044b\u0442\u0438\u044f \u0442\u043e\u0436\u0435 \u043e\u0431\u044f\u0437\u0430\u0442\u0435\u043b\u044c\u043d\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":22763,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-30778","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u0437\u0430\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u0442\u0430\u043a\u0443\u044e \u0431\u043e\u043b\u044c\u0448\u0443\u044e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044e \u043a\u0430\u043a Lamoda \u0441 \u043e\u0442\u043b\u0430\u0436\u0435\u043d\u043d\u044b\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u043c \u0438 \u0434\u0435\u0441\u044f\u0442\u043a\u0430\u043c\u0438 \u0432\u0437\u0430\u0438\u043c\u043e\u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0435\u043d\u043d\u043e \u043c\u0435\u043d\u044f\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434?\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u044b\u0442 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0441\u0435\u0440\u0432\u0438\u0441\u0430 Refund Tool \u0441 \u0430\u0441\u0438\u043d\u0445\u0440\u043e\u043d\u043d\u044b\u043c API \u043d\u0430 Kafka | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u0437\u0430\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u0442\u0430\u043a\u0443\u044e \u0431\u043e\u043b\u044c\u0448\u0443\u044e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044e \u043a\u0430\u043a Lamoda \u0441 \u043e\u0442\u043b\u0430\u0436\u0435\u043d\u043d\u044b\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u043c \u0438 \u0434\u0435\u0441\u044f\u0442\u043a\u0430\u043c\u0438 \u0432\u0437\u0430\u0438\u043c\u043e\u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0435\u043d\u043d\u043e \u043c\u0435\u043d\u044f\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434?\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:37:23+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:37:23+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Zku\u0161enost s v\u00fdvojem slu\u017eby Refund Tool s asynchronn\u00edm API na Kafka | ProHoster","description":"Co m\u016f\u017ee p\u0159im\u011bt tak velkou spole\u010dnost jako Lamoda s dob\u0159e vypracovan\u00fdmi procesy a des\u00edtkami vz\u00e1jemn\u011b propojen\u00fdch slu\u017eeb z\u00e1sadn\u011b zm\u011bnit p\u0159\u00edstup?","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u044b\u0442 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0441\u0435\u0440\u0432\u0438\u0441\u0430 Refund Tool \u0441 \u0430\u0441\u0438\u043d\u0445\u0440\u043e\u043d\u043d\u044b\u043c API \u043d\u0430 Kafka | ProHoster","og:description":"\u0427\u0442\u043e \u043c\u043e\u0436\u0435\u0442 \u0437\u0430\u0441\u0442\u0430\u0432\u0438\u0442\u044c \u0442\u0430\u043a\u0443\u044e \u0431\u043e\u043b\u044c\u0448\u0443\u044e \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u044e \u043a\u0430\u043a Lamoda \u0441 \u043e\u0442\u043b\u0430\u0436\u0435\u043d\u043d\u044b\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u043e\u043c \u0438 \u0434\u0435\u0441\u044f\u0442\u043a\u0430\u043c\u0438 \u0432\u0437\u0430\u0438\u043c\u043e\u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0435\u043d\u043d\u043e \u043c\u0435\u043d\u044f\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434?","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/opyt-razrabotki-servisa-refund-tool-s-asinhronnym-api-na-kafka","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:37:23+00:00","article:modified_time":"2019-10-31T18:37:23+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30778","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 02:58:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:29:56","updated":"2026-01-21 02:58:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/30778","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=30778"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/30778\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/22763"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=30778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=30778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=30778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}