Indlela iKafka yaba yinto yokwenene

Hayi Habr!

Ndisebenza kwiqela leTinkoff, eliphuhlisa iziko layo lezaziso. Ubukhulu becala ndiphuhlisa kwiJava usebenzisa iSpring boot kunye nokusombulula iingxaki ezahlukeneyo zobugcisa ezivela kwiprojekthi.

Uninzi lwee-microservices zethu zinxibelelana enye kwenye ngokungahambelaniyo nge-broker yomyalezo. Ngaphambili, sasebenzisa i-IBM MQ njenge-broker, eyayingenakukwazi ukujamelana nomthwalo, kodwa ngexesha elifanayo lineziqinisekiso zokuhanjiswa okuphezulu.

Njengokutshintshwa, sanikwa i-Apache Kafka, enokuthi ibe namandla aphezulu, kodwa, ngelishwa, ifuna indlela ephantse ibe ngumntu ngamnye kuqwalaselo lweemeko ezahlukeneyo. Ukongezelela, ubuncinane kanye indlela yokuhambisa esebenza eKafka ngokungagqibekanga ayizange ivumele ukugcina umgangatho ofunekayo wokuhambelana ngaphandle kwebhokisi. Okulandelayo, ndiza kwabelana ngamava ethu kuqwalaselo lwe-Kafka, ngakumbi, ndiya kukuxelela indlela yokumisela kwaye uphile kunye nokuhanjiswa kanye kanye.

Ukuhanjiswa okuqinisekisiweyo nokunye

Izicwangciso ezixoxwe ngezantsi ziya kunceda ukunqanda inani leengxaki ngoseto loqhagamshelo olungagqibekanga. Kodwa okokuqala ndingathanda ukunikela ingqalelo kwiparameter enye eya kuququzelela ulungiso olunokwenzeka.

Oku kuya kunceda client.id kuMvelisi kunye noMthengi. Ekuboneni kokuqala, ungasebenzisa igama lesicelo njengexabiso, kwaye kwiimeko ezininzi oku kuya kusebenza. Nangona imeko xa isicelo sisebenzisa abathengi abaninzi kwaye ubanika i-client.id efanayo, iziphumo zesi silumkiso silandelayo:

org.apache.kafka.common.utils.AppInfoParser — Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=kafka.test-0

Ukuba ufuna ukusebenzisa i-JMX kwisicelo kunye neKafka, oku kunokuba yingxaki. Kule meko, kungcono ukusebenzisa indibaniselwano yegama lesicelo kwaye, umzekelo, igama lesihloko njengexabiso le-client.id. Isiphumo soqwalaselo lwethu sinokubonwa kwisiphumo somyalelo amaqela kafka-abathengi ukusuka kwizinto eziluncedo ezivela kwi-Confluent:

Ngoku makhe sijonge imeko yonikezelo lomyalezo oluqinisekisiweyo. Umvelisi weKafka unepharamitha ii acks, ekuvumela ukuba uqwalasele emva kokuba zingaphi ezivumayo ukuba inkokeli yeqela ifuna ukuqwalasela umyalezo obhalwe ngempumelelo. Le parameter inokuthatha amaxabiso alandelayo:

• 0 - ukuvuma akuyi kuqwalaselwa.
• I-1 yiparameter emiselweyo, kuphela i-replica enye efunekayo ukuvuma.
• −1 — ukuvuma kuzo zonke iikopi ezidityanisiweyo ziyafuneka (ulungiselelo lweqela min.insync.replicas).

Kumaxabiso adwelisiweyo kuyacaca ukuba ii-acks ezilingana no -1 zinika esona siqinisekiso sinamandla sokuba umyalezo awuzukulahleka.

Njengoko sonke sisazi, iinkqubo ezisasazwayo azithembekanga. Ukukhusela kwiziphoso zexeshana, uMvelisi weKafka ubonelela ngokhetho iyazama kwakhona, ekuvumela ukuba usete inani lemizamo yokuthumela kwakhona ngaphakathi delivery.timeout.ms. Ekubeni iparamitha iphinda izame inexabiso elingagqibekanga le-Integer.MAX_VALUE (2147483647), inani lomyalezo ophinda lizamayo lingahlengahlengiswa ngokutshintsha kuphela delivery.timeout.ms.

Sijonge ukuhanjiswa kanye kanye

Iisetingi ezidwelisiweyo zivumela uMvelisi wethu ukuba athumele imiyalezo enesiqinisekiso esiphezulu. Ngoku makhe sithethe malunga nendlela yokuqinisekisa ukuba ikopi enye yomyalezo ibhalwe kwisihloko seKafka? Kweyona meko ilula, ukwenza oku, kufuneka usete iparameter kuMvelisi vumela.ukungabi namandla kwinyani. I-Idempotency iqinisekisa ukuba umyalezo omnye kuphela obhaliweyo kwisahlulo esithile sesihloko esinye. Umqathango wokuvumela ukungakwazi ukuzimela ngamaxabiso ii-acks = zonke, zama kwakhona > 0, max.in.flight.requests.per.connection ≤ 5. Ukuba ezi parameters azichazwanga ngumphuhlisi, amaxabiso angentla aya kumiselwa ngokuzenzekelayo.

Xa i-idempotency iqwalaselwe, kuyimfuneko ukuqinisekisa ukuba imiyalezo efanayo iphelela kwizahlulo ezifanayo rhoqo. Oku kunokwenziwa ngokuseta iqhosha le-partitioner.class kunye nepharamitha kuMvelisi. Masiqale ngesitshixo. Kufuneka kube yinto enye kwisingeniso ngasinye. Oku kunokufumaneka ngokulula ngokusebenzisa naziphi na ii-ID zeshishini ezisuka kwisithuba sokuqala. I-partitioner.class iparameter inexabiso elingagqibekanga - DefaultPartitioner. Ngesi sicwangciso sokwahlulahlula, ngokungagqibekanga senza ngolu hlobo:

• Ukuba ulwahlulo luchazwe ngokucacileyo xa uthumela umyalezo, ngoko siyawusebenzisa.
• Ukuba ulwahlulo aluchazwanga, kodwa isitshixo sicacisiwe, khetha ulwahlulo nge-hash yesitshixo.
• Ukuba isahlulelo kunye nesitshixo asichazwanga, khetha izahlulo nganye nganye (i-robin ejikelezayo).

Kwakhona, usebenzisa isitshixo kunye nokuthumela okungenamandla ngeparameter max.in.flight.requests.per.connection = 1 ikunika inkqubo yomyalezo olungelelanisiweyo kuMthengi. Kukwafanelekile ukuba ukhumbule ukuba ukuba ulawulo lofikelelo luqwalaselwe kwiqela lakho, ke uya kufuna amalungelo okubhala ngokungaziyo isihloko.

Ukuba ngequbuliso uswele amandla okuthumela ngokungenangqondo okanye ingqiqo kwicala loMvelisi ifuna ukugcina ukuhambelana kwedatha phakathi kwezahlulo ezahlukeneyo, ukuthengiselana kuya kuza kuhlangula. Ukongeza, usebenzisa i-chain transaction, unokuvumelanisa ngokwemeko irekhodi eKafka, umzekelo, kunye nerekhodi kwisiseko sedatha. Ukwenza uthumelo lwentengiselwano kuMvelisi, kufuneka lungabinamandla kwaye lusetwe okongeziweyo transaction.id. Ukuba iqela lakho leKafka linolawulo lofikelelo olucwangcisiweyo, ngoko irekhodi yentengiselwano, njengerekhodi elingenamsebenzi, iya kufuna imvume yokubhala, enokunikezelwa ngemaski usebenzisa ixabiso eligcinwe kwi-transactional.id.

Ngokusesikweni, nawuphi na umtya, onjengegama lesicelo, unokusetyenziswa njengesichongi sentengiselwano. Kodwa ukuba uvula iimeko ezininzi zesicelo esifanayo kunye ne-transactional.id efanayo, ngoko umzekelo wokuqala osungulweyo uya kunqunyanyiswa ngempazamo, kuba iKafka iyakuyithatha njengenkqubo ye-zombie.

org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.

Ukusombulula le ngxaki, songeza isimamva kwigama lesicelo ngendlela yegama lenginginya, esilifumana kwizinto eziguquguqukayo zemo engqongileyo.

Umvelisi uqwalaselwe, kodwa ukuthengiselana kwiKafka kulawula kuphela umda womyalezo. Kungakhathaliseki ukuba imeko yentengiselwano, umyalezo ngokukhawuleza uya kwisihloko, kodwa uneempawu ezongezelelweyo zenkqubo.

Ukuthintela loo miyalezo ukuba ifundwe nguMthengi kwangethuba, kufuneka abeke iparamitha inqanaba ukufunda_ixabiso elizibolekileyo. UmThengi onjalo uya kukwazi ukufunda imiyalezo engeyoyentengiselwano njengangaphambili, kunye nemiyalezo yentengiselwano kuphela emva kokuzibophelela.
Ukuba usete zonke iisetingi ezidweliswe ngaphambili, ngoko uqwalasele kanye unikezelo kanye. Sivuyisana nawe!

Kodwa kukho enye nuance ngaphezulu. Transactional.id, esiyiqwalasele ngasentla, sisiqalo sentengiselwano. Kumphathi wentengiselwano, inombolo yolandelelwano yongezwa kuyo. Isazisi esifunyenweyo sikhutshelwa transactional.id.expiration.ms, elungiselelwe kwiqela le-Kafka kwaye inexabiso elingagqibekanga "iintsuku ezisi-7". Ukuba ngeli xesha isicelo asifumananga nayiphi na imiyalezo, ngoko xa uzama itransekshini elandelayo thumela uya kufumana InvalidPidMappingException. Umququzeleli wentengiselwano uya kukhupha inombolo entsha yolandelelwano lwentengiselwano elandelayo. Nangona kunjalo, umyalezo ungalahleka ukuba i-InvalidPidMappingException ayiphathwanga kakuhle.

Endaweni yeetotali

Njengoko ubona, akwanelanga ukuthumela imiyalezo eKafka. Kufuneka ukhethe indibaniselwano yeeparamitha kwaye ulungele ukwenza utshintsho olukhawulezayo. Kweli nqaku, ndizamile ukubonisa ngokweenkcukacha ukuseta kube kanye kanye kwaye ndachaza iingxaki ezininzi nge-client.id kunye noqwalaselo lwetransactional.id esidibene nalo. Apha ngezantsi kukho isishwankathelo sesetingi zoMvelisi kunye noMthengi.

Umvelisi:

1. acks = zonke
2. izama kwakhona > 0
3. nika amandla.ukungakwazi ukuzimela = yinyaniso
4. max.in.flight.requests.per.connection ≤ 5 (1 yokuthumela ngocwangco)
5. transactional.id = ${application-name}-${hostname}

Umthengi:

1. i-isolation.level = read_committed

Ukunciphisa iimpazamo kwizicelo ezizayo, senze eyethu i-wrapper phezu koqwalaselo lwasentwasahlobo, apho amaxabiso ezinye zeeparamitha ezidwelisiweyo sele zisetiwe.

Nazi izixhobo ezimbalwa zokuzifundela:

• I-KIP-98-Ngqo kanye Ukuhanjiswa kunye noThunyelo loMyalezo
• Inkcazo yezicwangciso

umthombo: www.habr.com