Indlela i-Kafka eyaba ngayo iqiniso

Sawubona Habr!

Ngisebenza eqenjini leTinkoff, elizakhela esalo isikhungo sezaziso. Ngithuthuka kakhulu ku-Java ngisebenzisa i-Spring boot futhi ngixazulula izinkinga ezahlukahlukene zobuchwepheshe eziphakama kuphrojekthi.

Iningi lama-microservices ethu lixhumana nomunye ngendlela esynchronously ngomthengisi wemiyalezo. Ngaphambili, sasebenzisa i-IBM MQ njengomthengisi, ongasakwazi ukubhekana nomthwalo, kodwa ngesikhathi esifanayo ube neziqinisekiso zokulethwa okuphezulu.

Njengokuthatha indawo, sinikezwe i-Apache Kafka, enamandla amakhulu okukala, kodwa, ngeshwa, idinga indlela yomuntu ngamunye yokucushwa kwezimo ezahlukene. Ngaphezu kwalokho, indlela yokulethwa okungenani kanye esebenza e-Kafka ngokuzenzakalelayo ayizange ivumele ukugcinwa kwezinga elidingekayo lokungaguquguquki ngaphandle kwebhokisi. Okulandelayo, ngizokwabelana ngolwazi lwethu ekucushweni kwe-Kafka, ikakhulukazi, ngizokutshela ukuthi ungalungisa kanjani futhi uphile ngokudiliva kanye kanye.

Ukulethwa okuqinisekisiwe nokunye okwengeziwe

Izilungiselelo okuxoxwe ngazo ngezansi zizosiza ukuvikela inombolo yezinkinga ngezilungiselelo zokuxhuma ezizenzakalelayo. Kodwa okokuqala ngithanda ukunaka ipharamitha eyodwa ezokwenza kube lula ukulungisa iphutha.

Lokhu kuzosiza client.id okoMkhiqizi kanye Nomthengi. Uma uthi nhlá, ungasebenzisa igama lohlelo lokusebenza njengenani, futhi ezimweni eziningi lokhu kuzosebenza. Nakuba isimo lapho uhlelo lokusebenza lusebenzisa Abathengi abambalwa futhi ubanika i-client.id efanayo, kuphumela kusixwayiso esilandelayo:

org.apache.kafka.common.utils.AppInfoParser — Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=kafka.test-0

Uma ufuna ukusebenzisa i-JMX kuhlelo lokusebenza ne-Kafka, lokhu kungaba inkinga. Kulesi simo, kungcono ukusebenzisa inhlanganisela yegama lohlelo lokusebenza futhi, isibonelo, igama lesihloko njengenani le-client.id. Umphumela wokucushwa kwethu ungabonakala ekuphumeni komyalo amaqembu kafka-abathengi kusuka kuzinsiza ezivela ku-Confluent:

Manje ake sibheke isimo sokulethwa komlayezo okuqinisekisiwe. I-Kafka Producer inepharamitha ama-acks, okukuvumela ukuthi ulungiselele ngemuva kokuthi bangaki abavumayo ukuthi umholi weqoqo udinga ukucabangela umlayezo obhalwe ngempumelelo. Le parameter ingathatha amanani alandelayo:

• 0 - ukuvuma ngeke kucatshangelwe.
• U-1 uyipharamitha ezenzakalelayo, ikhophi engu-1 kuphela edingekayo ukuze kuvunywe.
• −1 - ukuvuma kuzo zonke izifaniso ezivumelanisiwe kuyadingeka (ukusethwa kweqoqo min.insync.replicas).

Kumanani asohlwini kuyacaca ukuthi ama-acks alingana no-−1 anikeza isiqinisekiso esiqine kakhulu sokuthi umlayezo ngeke ulahleke.

Njengoba sonke sazi, amasistimu asabalalisiwe awathembekile. Ukuvikela kumaphutha esikhashana, Umdidiyeli we-Kafka unikeza inketho iyazama futhi, okukuvumela ukuthi usethe inombolo yemizamo yokuthumela kabusha ngaphakathi delivery.timeout.ms. Njengoba ipharamitha ephinda izama inenani elizenzakalelayo le-Integer.MAX_VALUE (2147483647), inombolo yokuzama kabusha komlayezo ingalungiswa ngokushintsha kuphela delivery.timeout.ms.

Sibheke ekulethweni kanye ncamashi

Izilungiselelo ezisohlwini zivumela Umkhiqizi wethu ukuthi alethe imilayezo enesiqinisekiso esiphezulu. Manje ake sikhulume ngendlela yokuqinisekisa ukuthi ikhophi eyodwa kuphela yomlayezo ibhalwa esihlokweni se-Kafka? Esimweni esilula, ukwenza lokhu, udinga ukusetha ipharamitha kuMkhiqizi vumela.ukungabi namandla ngokweqiniso. Ukungabi namandla kuqinisekisa ukuthi umlayezo owodwa kuphela obhalwa engxenyeni ethile yesihloko esisodwa. Umbandela wangaphambili wokunika amandla idempotency amanani ama-acks = konke, zama futhi > 0, max.in.flight.requests.per.connection ≤ 5. Uma le mingcele ingacaciswanga unjiniyela, amanani angenhla azosethwa ngokuzenzakalelayo.

Uma i-idempotency imisiwe, kuyadingeka ukuqinisekisa ukuthi imilayezo efanayo igcina ngokuhlukanisa okufanayo ngaso sonke isikhathi. Lokhu kungenziwa ngokusetha ukhiye we-partitioner.class kanye nepharamitha ku-Producer. Ake siqale ngokhiye. Kufanele kufane esithumeni ngasinye. Lokhu kungafinyelelwa kalula ngokusebenzisa noma yimaphi ama-ID ebhizinisi asuka kokuthunyelwe kwangempela. Ipharamitha ye-partitioner.class inenani elizenzakalelayo − I-DefaultPartitioner. Ngalelisu lokuhlukanisa, ngokuzenzakalelayo senza kanje:

• Uma ukwahlukanisa kucaciswe ngokusobala lapho kuthunyelwa umlayezo, khona-ke siyakusebenzisa.
• Uma ukwahlukanisa kungacacisiwe, kodwa ukhiye ucacisiwe, khetha ukwahlukanisa nge-hash yokhiye.
• Uma ukwahlukanisa nokhiye kungacacisiwe, khetha ama-partitions ngamunye ngamunye (round-robin).

Futhi, sebenzisa ukhiye nokuthumela okungenangqondo ngepharamitha max.in.flight.requests.per.connection = 1 ikunikeza ukucutshungulwa komlayezo oqondile ku-Consumer. Kuyafaneleka futhi ukukhumbula ukuthi uma ukulawula kokufinyelela kulungiselelwe kuqoqo lakho, uzodinga amalungelo okubhala ngokungafanele esihlokweni.

Uma ngokuzumayo untula amandla okuthumela okungenangqondo ngokhiye noma ingqondo ehlangothini Lomkhiqizi idinga ukulondoloza ukuvumelana kwedatha phakathi kwezingxenye ezihlukene, khona-ke okwenziwayo kuyosiza. Ngaphezu kwalokho, usebenzisa i-chain transaction, ungakwazi ukuvumelanisa ngokwemibandela irekhodi e-Kafka, isibonelo, nerekhodi ku-database. Ukuze unike amandla ukuthumela okwenziwayo Kumdidiyeli, kufanele kungabi namandla futhi kusethwe ngokungeziwe transactional.id. Uma iqoqo lakho le-Kafka linokulawula kokufinyelela okulungisiwe, irekhodi lokwenziwayo, njengerekhodi elingenalutho, lizodinga izimvume zokubhala, ezinganikezwa ngemaski kusetshenziswa inani eligcinwe ku-transactional.id.

Ngokusemthethweni, noma iyiphi iyunithi yezinhlamvu, njengegama lohlelo lokusebenza, ingasetshenziswa njengesihlonzi sokwenziwe. Kodwa uma wethula izikhathi ezimbalwa zohlelo lokusebenza olufanayo nge-transactional.id efanayo, isenzakalo sokuqala esethulwe sizomiswa ngephutha, njengoba i-Kafka izoyibheka njengenqubo ye-zombie.

org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.

Ukuxazulula le nkinga, sengeza isijobelelo egameni lesicelo ngendlela yegama lomethuleli, esilithola kokuguquguqukayo kwemvelo.

Umkhiqizi ulungisiwe, kodwa okwenziwa ku-Kafka kulawula kuphela ububanzi bomlayezo. Kungakhathalekile isimo sokwenziwe, umlayezo ngokushesha uya esihlokweni, kodwa unezibaluli zesistimu ezengeziwe.

Ukuze uvimbele imilayezo enjalo ukuthi ifundwe Umsebenzisi kusenesikhathi, idinga ukusetha ipharamitha ukuzihlukanisa.leveli ukufunda_inani lokuzibophezela. Umsebenzisi onjalo uzokwazi ukufunda imilayezo okungeyona eyokuthengiselana njengangaphambili, kanye nemilayezo yokwenziwayo kuphela ngemva kokuzibophezela.
Uma usethe wonke amasethingi asohlwini lwangaphambilini, uyobe usulungise kanye ukulethwa kanye nje. Siyakuhalalisela!

Kodwa kunenye futhi i-nuance. I-Transactional.id, esiyilungiselele ngenhla, empeleni iyisiqalo somsebenzi. Kumphathi wokwenziwe, inombolo yokulandelana yengezwa kuyo. Inkomba etholiwe ikhishelwe transactional.id.expiration.ms, elungiselelwe kuqoqo le-Kafka futhi elinenani elizenzakalelayo "lezinsuku ezingu-7". Uma ngalesi sikhathi uhlelo lokusebenza lungakayitholi imilayezo, lapho-ke uzama ukuthumela okwenziwayo okulandelayo uzothola I-InvalidPidMappingException. Umxhumanisi wokwenziwe uzobese ekhipha inombolo entsha yokulandelana kokwenziwe okulandelayo. Nokho, umlayezo ungase ulahleke uma i-InvalidPidMappingException ingaphathwa kahle.

Esikhundleni samathothali

Njengoba ubona, akwanele ukumane uthumele imilayezo e-Kafka. Udinga ukukhetha inhlanganisela yamapharamitha futhi ulungele ukwenza izinguquko ezisheshayo. Kulesi sihloko, ngizamile ukukhombisa ngokuningiliziwe ukusethwa kokulethwa kanye ngachaza izinkinga ezimbalwa nge-client.id kanye nokuhlelwa kwe-transactional.id esihlangabezane nakho. Ngezansi isifinyezo sezilungiselelo Zomkhiqizi kanye Nomthengi.

Umkhiqizi:

1. acks = konke
2. izama futhi > 0
3. vumela.idempotence = iqiniso
4. max.in.flight.requests.per.connection ≤ 5 (1 ukuthumela ngokuhlelekile)
5. transactional.id = ${application-name}-${hostname}

Umthengi:

1. isolation.level = read_committed

Ukuze kuncishiswe amaphutha ezinhlelweni zokusebenza ezizayo, senze okwethu ukugoqa phezu kokucushwa kwentwasahlobo, lapho amanani amanye amapharamitha asohlwini asevele asethiwe.

Nazi izinto ezimbalwa zokuzifundela:

• I-KIP-98 - Impela Kanye Kanye Ukulethwa Nokuthunyelwa Kwemiyalezo
• Incazelo yezilungiselelo

Source: www.habr.com