Yuav ua li cas Kafka los ua kev muaj tiag

Hlo Habr!

Kuv ua haujlwm ntawm pab pawg Tinkoff, uas tab tom tsim nws lub chaw ceeb toom. Kuv feem ntau txhim kho hauv Java siv lub caij nplooj ntoos hlav khau raj thiab daws teeb meem ntau yam uas tshwm sim hauv qhov project.

Feem ntau ntawm peb cov microservices sib txuas lus nrog ib leeg asynchronously los ntawm cov lus broker. Yav dhau los, peb tau siv IBM MQ ua tus broker, uas tsis tuaj yeem tiv nrog cov khoom thauj, tab sis tib lub sijhawm muaj kev lav zoo siab.

Raws li kev hloov pauv, peb tau muab Apache Kafka, uas muaj peev xwm ua kom muaj peev xwm loj, tab sis, hmoov tsis, yuav tsum muaj ib tus neeg yuav luag rau kev teeb tsa rau cov xwm txheej sib txawv. Tsis tas li ntawd, tsawg kawg ib zaug xa cov txheej txheem uas ua haujlwm hauv Kafka los ntawm lub neej ntawd tsis tso cai rau kev tswj hwm qib uas yuav tsum tau ua kom zoo sib xws ntawm lub thawv. Tom ntej no, kuv yuav qhia peb cov kev paub dhau los hauv Kafka teeb tsa, tshwj xeeb, kuv yuav qhia koj yuav ua li cas teeb tsa thiab nyob nrog raws nraim ib zaug xa khoom.

Guaranteed tus me nyuam thiab lwm yam

Cov kev teeb tsa tau tham hauv qab no yuav pab tiv thaiv ntau yam teeb meem nrog rau qhov kev sib txuas ua ke. Tab sis ua ntej kuv xav them sai sai rau ib qho parameter uas yuav pab txhawb kev debug.

Qhov no yuav pab tau neeg.id rau tus tsim khoom thiab cov neeg siv khoom. Thaum xub thawj siab ib muag, koj tuaj yeem siv lub npe thov raws li tus nqi, thiab feem ntau qhov no yuav ua haujlwm. Txawm hais tias qhov xwm txheej thaum daim ntawv thov siv ntau tus neeg siv khoom thiab koj muab rau lawv tib tus client.id, ua rau cov lus ceeb toom hauv qab no:

org.apache.kafka.common.utils.AppInfoParser — Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=kafka.test-0

Yog tias koj xav siv JMX hauv daim ntawv thov nrog Kafka, qhov no yuav yog qhov teeb meem. Rau cov ntaub ntawv no, nws yog qhov zoo tshaj plaws los siv kev sib txuas ntawm daim ntawv thov npe thiab, piv txwv li, lub npe lub npe raws li tus nqi client.id. Cov txiaj ntsig ntawm peb qhov kev teeb tsa tuaj yeem pom hauv cov lus txib tso tawm kafka-consumer-groups los ntawm kev siv hluav taws xob los ntawm Confluent:

Tam sim no cia saib ntawm qhov xwm txheej rau kev lees paub xa xov. Kafka Producer muaj qhov ntsuas acks ua, uas tso cai rau koj los teeb tsa tom qab muaj pes tsawg tus lees paub cov thawj coj pawg xav tau los txiav txim siab cov lus sau tiav. Qhov parameter no tuaj yeem coj cov txiaj ntsig hauv qab no:

• 0 - Kev lees paub yuav tsis raug txiav txim siab.
• 1 yog lub neej ntawd parameter, tsuas yog 1 replica yuav tsum lees paub.
• −1 — lees paub los ntawm tag nrho cov synchronized replicas yuav tsum tau (cluster teeb min.insync.replicas).

Los ntawm cov nqi teev tseg nws yog qhov tseeb tias acks sib npaug rau −1 muab qhov muaj zog tshaj lav tias cov lus yuav tsis ploj.

Raws li peb txhua tus paub, cov txheej txheem faib tsis tuaj yeem ntseeg tau. Txhawm rau tiv thaiv qhov tsis raug ncua sijhawm, Kafka Tus tsim tawm muab kev xaiv rov sim dua, uas tso cai rau koj los teeb tsa tus naj npawb ntawm resend sim hauv shipping.timeout.ms. Txij li thaum qhov kev rov ua dua tsis tau muaj tus nqi pib ntawm Integer.MAX_VALUE (2147483647), tus naj npawb ntawm cov lus rov tuaj yeem hloov kho los ntawm kev hloov tsuas yog xa.timeout.ms.

Peb tab tom tsiv mus raws nraim ib zaug xa khoom

Cov chaw teev cia tso cai rau peb Tus Tsim Khoom xa cov lus nrog kev lav zoo. Tam sim no peb tham txog yuav ua li cas kom ntseeg tau tias tsuas yog ib daim ntawv luam ntawm cov lus raug sau rau Kafka lub ntsiab lus? Hauv qhov yooj yim tshaj plaws, ua qhov no, koj yuav tsum teeb tsa qhov ntsuas ntawm Tus Tsim Khoom pab.idempotence kom muaj tseeb. Idempotency lav tias tsuas yog ib cov lus sau rau ib qho kev faib tshwj xeeb ntawm ib lub ncauj lus. Lub precondition rau enabling idempotency yog cov nqi acks = tag nrho, rov qab> 0, max.in.flight.requests.per.connection ≤ 5. Yog tias cov kev txwv no tsis tau teev tseg los ntawm tus tsim tawm, cov txiaj ntsig saum toj no yuav raug teeb tsa.

Thaum ideempotency yog configured, nws yog tsim nyog los xyuas kom meej tias tib cov lus xaus nyob rau hauv tib partitions txhua txhua lub sij hawm. Qhov no tuaj yeem ua tiav los ntawm kev teeb tsa partitioner.class key thiab parameter rau Tus Tsim Khoom. Cia peb pib nrog tus yuam sij. Nws yuav tsum yog tib yam rau txhua qhov kev xa tawm. Qhov no tuaj yeem ua tiav tau yooj yim los ntawm kev siv ib qho ntawm cov lag luam IDs los ntawm thawj tus ncej. Lub partitioner.class parameter muaj tus nqi pib − DefaultPartitioner. Nrog rau qhov kev faib tawm tswv yim no, los ntawm lub neej ntawd peb ua zoo li no:

• Yog tias qhov muab faib tau qhia meej meej thaum xa cov lus, ces peb siv nws.
• Yog tias qhov muab faib tsis tau teev tseg, tab sis tus yuam sij tau teev tseg, xaiv qhov muab faib los ntawm hash ntawm tus yuam sij.
• Yog hais tias qhov muab faib thiab qhov tseem ceeb tsis tau teev tseg, xaiv cov partitions ib tug los ntawm ib tug (round-robin).

Tsis tas li ntawd, siv tus yuam sij thiab idempotent xa nrog ib qho parameter max.in.flight.requests.per.connection = 1 muab koj cov lus streamlined txheej txheem ntawm Consumer. Nws tseem tsim nyog nco ntsoov tias yog tias kev tswj hwm kev nkag mus tau teeb tsa ntawm koj pawg, ces koj yuav xav tau txoj cai los sau ntawv rau ib lub ntsiab lus.

Yog tias mam li nco dheev koj tsis muaj peev xwm ntawm idempotent xa los ntawm qhov tseem ceeb los yog lub logic ntawm tus tsim tawm sab yuav tsum tuav cov ntaub ntawv sib xws ntawm cov sib txawv sib txawv, ces kev hloov pauv yuav los cawm. Tsis tas li ntawd, siv cov saw hlau sib pauv, koj tuaj yeem kho cov ntaub ntawv hauv Kafka, piv txwv li, nrog cov ntaub ntawv hauv cov ntaub ntawv. Txhawm rau kom muaj kev hloov pauv xa mus rau Tus Tsim Khoom, nws yuav tsum tsis muaj peev xwm thiab teeb tsa ntxiv muas.id. Yog tias koj pawg Kafka muaj kev tswj hwm kev teeb tsa, tom qab ntawd cov ntaub ntawv hloov pauv, zoo li cov ntaub ntawv tsis txaus ntseeg, yuav xav tau kev tso cai sau ntawv, uas tuaj yeem tso cai los ntawm daim npog ntsej muag siv tus nqi khaws cia hauv kev hloov pauv.id.

Raws li txoj cai, txhua txoj hlua, xws li daim ntawv thov npe, tuaj yeem siv los ua tus cim kev lag luam. Tab sis yog tias koj tso ntau zaus ntawm tib daim ntawv thov nrog tib qhov kev hloov pauv.id, tom qab ntawd thawj qhov piv txwv yuav raug tso tseg nrog qhov yuam kev, txij li Kafka yuav xav tias nws yog txheej txheem zombie.

org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.

Txhawm rau daws qhov teeb meem no, peb ntxiv cov lus txuas ntxiv rau daim ntawv thov npe hauv daim ntawv hostname, uas peb tau txais los ntawm ib puag ncig hloov pauv.

Cov neeg tsim khoom tau teeb tsa, tab sis kev lag luam ntawm Kafka tsuas yog tswj hwm cov lus ntawm cov lus. Txawm li cas los xij ntawm cov xwm txheej pauv, cov lus tam sim ntawd mus rau lub ntsiab lus, tab sis muaj cov yam ntxwv ntxiv.

Txhawm rau tiv thaiv cov lus no los ntawm kev nyeem los ntawm Cov Neeg Siv Khoom ua ntej, nws yuav tsum teeb tsa qhov ntsuas cais.level nyeem_committed tus nqi. Cov neeg siv khoom zoo li no yuav tuaj yeem nyeem cov lus tsis yog kev hloov pauv raws li ua ntej, thiab cov lus hloov pauv tsuas yog tom qab cog lus.
Yog tias koj tau teeb tsa txhua qhov chaw teev tseg ua ntej, ces koj tau teeb tsa raws nraim ib zaug xa khoom. Nrog koj zoo siab!

Tab sis muaj ib qho ntxiv nuance. Transactional.id, uas peb tau teeb tsa saum toj no, yog qhov kev ua lag luam ua ntej. Ntawm tus thawj tswj kev lag luam, ib tus lej sib txuas ntxiv rau nws. Tus ID tau txais yog muab rau muas.id.expiration.ms, uas tau teeb tsa ntawm Kafka pawg thiab muaj tus nqi pib ntawm "7 hnub". Yog tias lub sijhawm no daim ntawv thov tsis tau txais cov lus, tom qab ntawd thaum koj sim cov kev hloov pauv tom ntej koj yuav tau txais InvalidPidMappingException. Tus neeg saib xyuas kev sib pauv yuav muab tus lej tshiab rau qhov kev hloov pauv tom ntej. Txawm li cas los xij, cov lus yuav ploj yog tias InvalidPidMappingException tsis raug tswj kom raug.

Hloov cov txiaj ntsig

Raws li koj tuaj yeem pom, nws tsis txaus los tsuas yog xa cov lus rau Kafka. Koj yuav tsum xaiv qhov sib xyaw ua ke ntawm qhov tsis sib xws thiab npaj kom hloov pauv sai. Hauv tsab xov xwm no, kuv sim ua kom nthuav dav qhov tseeb ib zaug xa khoom teeb tsa thiab piav qhia ntau yam teeb meem nrog tus client.id thiab kev hloov pauv.id teeb tsa uas peb ntsib. Hauv qab no yog cov ntsiab lus ntawm Cov Tsim Khoom thiab Cov Neeg Siv Khoom.

tsim kom muaj:

1. acks = tag nrho
2. rov qab> 0
3. enable.idempotence = tseeb
4. max.in.flight.requests.per.connection ≤ 5 (1 rau kev xa khoom xa tuaj)
5. transactional.id = ${application-name}-${hostname}

Cov neeg siv khoom:

1. isolation.level = nyeem_committed

Txhawm rau txo qhov ua yuam kev hauv cov ntawv thov yav tom ntej, peb ua peb tus kheej wrapper hla lub caij nplooj ntoos hlav teeb tsa, qhov twg cov txiaj ntsig rau qee qhov teev tsis tau teev tseg.

Nov yog ob peb yam khoom siv rau kev kawm tus kheej:

• KIP-98 - Raws nraim ib zaug xa khoom thiab xa ntawv xa mus
• Kev piav qhia ntawm kev teeb tsa

Tau qhov twg los: www.hab.com