Momwe Kafka adakhalira zenizeni

Pa Habr!

Ndimagwira ntchito pa gulu la Tinkoff, lomwe likupanga malo ake azidziwitso. Nthawi zambiri ndimakhala ku Java pogwiritsa ntchito Boot ya Spring ndikuthana ndi zovuta zosiyanasiyana zomwe zimachitika mu polojekiti.

Ambiri mwa ma microservices athu amalumikizana wina ndi mnzake molumikizana ndi uthenga broker. M'mbuyomu, tidagwiritsa ntchito IBM MQ ngati broker, yemwe sakanatha kupirira katunduyo, koma nthawi yomweyo anali ndi zitsimikizo zapamwamba zoperekera.

Monga cholowa m'malo, tidapatsidwa Apache Kafka, yomwe ili ndi kuthekera kwakukulu, koma, mwatsoka, imafuna njira yamunthu payekhapayekha pakusintha kosiyanasiyana. Kuonjezera apo, njira yoperekera kamodzi yomwe imagwira ntchito ku Kafka mwachisawawa sinalole kusunga mlingo wofunikira wa kusasinthasintha kunja kwa bokosi. Chotsatira, ndigawana zomwe takumana nazo mu kasinthidwe ka Kafka, makamaka, ndikuwuzani momwe mungakhazikitsire ndikukhala ndi nthawi yobereka.

Kutumiza kotsimikizika ndi zina zambiri

Zokonda zomwe zafotokozedwa m'munsimu zithandiza kupewa zovuta zingapo ndi zokonda zolumikizira. Koma choyamba ndikufuna kutchera khutu ku parameter imodzi yomwe ingathandize kuthetsa vuto.

Izi zithandiza client.id kwa Wopanga ndi Wogula. Poyang'ana koyamba, mutha kugwiritsa ntchito dzina la pulogalamuyo ngati mtengo, ndipo nthawi zambiri izi zitha kugwira ntchito. Ngakhale momwe pulogalamu imagwiritsa ntchito Ogula angapo ndikuwapatsa kasitomala.id yemweyo, zimabweretsa chenjezo ili:

org.apache.kafka.common.utils.AppInfoParser — Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=kafka.test-0

Ngati mukufuna kugwiritsa ntchito JMX mu pulogalamu ndi Kafka, ndiye kuti izi zitha kukhala vuto. Pachifukwa ichi, ndi bwino kugwiritsa ntchito dzina lophatikizana komanso, mwachitsanzo, dzina la mutu monga mtengo wa client.id. Zotsatira za kasinthidwe kwathu zitha kuwoneka muzotulutsa zamalamulo kafka-ogula-magulu zochokera ku Confluent:

Tsopano tiyeni tiyang'ane zochitika za kuperekedwa kwa uthenga wotsimikizika. Kafka Producer ali ndi parameter acks, zomwe zimakulolani kuti mukonzekere pambuyo pa angati omwe amavomereza kuti mtsogoleri wamagulu akuyenera kuganizira uthenga wolembedwa bwino. Parameter iyi ikhoza kutenga zinthu zotsatirazi:

• 0 - kuvomereza sikungaganizidwe.
• 1 ndiye gawo lokhazikika, chofanizira chimodzi chokha ndichofunikira kuti muvomereze.
• −1 - kuvomereza kuchokera ku zofananira zonse zolumikizidwa ndikofunikira (kukhazikitsa magulu min.insync.replicas).

Kuchokera pazomwe zatchulidwazi zikuwonekeratu kuti ma acks ofanana ndi −1 amapereka chitsimikizo champhamvu kuti uthengawo sudzatayika.

Monga tonse tikudziwira, machitidwe ogawidwa ndi osadalirika. Kuteteza ku zolakwika zosakhalitsa, Kafka Producer amapereka mwayi amayesanso, zomwe zimakulolani kuti muyike chiwerengero cha kuyesanso kutumiza mkati delivery.timeout.ms. Popeza zoyesanso zili ndi mtengo wokhazikika wa Integer.MAX_VALUE (2147483647), chiwerengero cha mauthenga obwereza chikhoza kusinthidwa posintha delivery.timeout.ms yokha.

Tikupita ku kubweretsa kamodzi

Zokonda zomwe zalembedwa zimalola Wopanga wathu kutumiza mauthenga ndi chitsimikizo chapamwamba. Tiyeni tsopano tikambirane momwe tingatsimikizire kuti uthenga umodzi wokha walembedwa pamutu wa Kafka? Muzochitika zosavuta, kuti muchite izi, muyenera kukhazikitsa parameter pa Producer athe.kulephera ku zoona. Idempotency imatsimikizira kuti uthenga umodzi wokha umalembedwa ku gawo lina la mutu umodzi. Chofunikira chothandizira kuti idempotency ndi zofunika acks = zonse, yesaninso > 0, max.in.flight.requests.per.connection ≤ 5. Ngati magawowa sanatchulidwe ndi wopanga, zomwe zili pamwambazi zidzakhazikitsidwa zokha.

Pamene idempotency ikukonzedwa, m'pofunika kuonetsetsa kuti mauthenga omwewo amathera m'magawo omwewo nthawi zonse. Izi zitha kuchitika pokhazikitsa kiyi ya partitioner.class ndi parameter ku Producer. Tiyeni tiyambe ndi kiyi. Ziyenera kukhala zofanana pa kuperekedwa kulikonse. Izi zitha kuchitika mosavuta pogwiritsa ntchito ma ID aliwonse abizinesi kuchokera patsamba loyambirira. Gawo la partitioner.class lili ndi mtengo wokhazikika - DefaultPartitioner. Ndi njira yogawa iyi, mwachisawawa timachita motere:

• Ngati gawolo lidanenedwa momveka bwino potumiza uthengawo, ndiye kuti timagwiritsa ntchito.
• Ngati kugawa sikunatchulidwe, koma fungulo lafotokozedwa, sankhani kugawa ndi hashi ya kiyi.
• Ngati kugawa ndi kiyi sizinatchulidwe, sankhani magawo amodzi ndi amodzi (ozungulira-robin).

Komanso, kugwiritsa ntchito makiyi ndi kutumiza kwachidziwitso ndi parameter max.in.flight.requests.per.connection = 1 imakupatsirani kusintha kwa uthenga pa Consumer. Ndikoyeneranso kukumbukira kuti ngati kuwongolera kofikira kumakhazikitsidwa pagulu lanu, ndiye kuti mudzafunika ufulu wolembera mutu mosasamala.

Ngati mwadzidzidzi mulibe kuthekera kwa kutumiza kwachidziwitso ndi kiyi kapena malingaliro pa mbali ya Wopanga amafuna kusunga kusasinthika kwa data pakati pa magawo osiyanasiyana, ndiye kuti kubweza kudzakupulumutsani. Kuphatikiza apo, pogwiritsa ntchito unyolo, mutha kulunzanitsa mbiri ku Kafka, mwachitsanzo, ndi mbiri yosungira. Kuti mutsegule kutumiza kwa Wopanga, kuyenera kukhala kopanda mphamvu ndikukhazikitsanso transactional.id. Ngati gulu lanu la Kafka lili ndi mwayi wowongolera, ndiye kuti mbiri yakale, ngati mbiri yosadziwika, idzafunika zilolezo zolembera, zomwe zitha kuperekedwa ndi chigoba pogwiritsa ntchito mtengo wosungidwa mu transactional.id.

Mwamwayi, chingwe chilichonse, monga dzina la pulogalamu, chingagwiritsidwe ntchito ngati chizindikiritso cha malonda. Koma ngati muyambitsa maulendo angapo a pulogalamu yomweyi ndi transactional.id yomweyo, ndiye kuti chochitika choyamba chidzayimitsidwa ndi cholakwika, chifukwa Kafka adzawona ngati njira ya zombie.

org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.

Kuti tithane ndi vutoli, timawonjezera suffix ku dzina la pulogalamuyo mu mawonekedwe a hostname, omwe timapeza kuchokera kumitundu yosiyanasiyana ya chilengedwe.

Wopangayo amakonzedwa, koma zochitika pa Kafka zimangoyang'anira kuchuluka kwa uthengawo. Mosasamala kanthu za zomwe zikuchitika, uthengawo umapita kumutu, koma uli ndi machitidwe owonjezera.

Kuletsa mauthenga oterowo kuti asawerengedwe ndi Consumer pasadakhale, iyenera kukhazikitsa parameter kudzipatula.level kuwerenga_kudzipereka mtengo. Wogula wotereyo azitha kuwerenga mauthenga osagwirizana ndi malonda monga kale, ndi mauthenga amalonda pokhapokha atapanga mgwirizano.
Ngati mwakhazikitsa makonda onse omwe atchulidwa kale, ndiye kuti mwakonza ndendende kamodzi kotumizira. Zabwino zonse!

Koma pali kusiyana kwina. Transactional.id, yomwe tidakonza pamwambapa, ndiye maziko oyambira. Pa woyang'anira zochitika, nambala yotsatizana imawonjezedwa kwa iyo. Chizindikiritso cholandiridwa chimaperekedwa kwa transactional.id.expiration.ms, yomwe imakonzedwa pagulu la Kafka ndipo ili ndi mtengo wokhazikika wa "masiku 7". Ngati panthawiyi ntchitoyo sinalandire mauthenga aliwonse, ndiye kuti mukamayesa kutumiza kotsatira mudzalandira InvalidPidMappingException. Wogwirizanitsa ntchitoyo adzapereka nambala yatsopano yotsatizana pazochitika zina. Komabe, uthengawo ukhoza kutayika ngati InvalidPidMappingException sichikuyendetsedwa bwino.

M'malo mwa zotsatira

Monga mukuonera, sikokwanira kungotumiza mauthenga ku Kafka. Muyenera kusankha kuphatikiza magawo ndikukonzekera kusintha mwachangu. Munkhaniyi, ndidayesa kuwonetsa mwatsatanetsatane kukhazikitsidwa kamodzi kokha ndikulongosola zovuta zingapo ndi kasitomala.id ndi masinthidwe a transactional.id omwe tidakumana nawo. Pansipa pali chidule cha zokonda za Opanga ndi Ogula.

Wopanga:

1. zonse = zonse
2. kuyesanso> 0
3. enable.idempotence = zoona
4. max.in.flight.requests.per.connection ≤ 5 (1 yotumiza mwadongosolo)
5. transactional.id = ${application-name}-${hostname}

Ogwiritsa:

1. isolation.level = read_committed

Kuti muchepetse zolakwika pazogwiritsa ntchito m'tsogolomu, tidapanga zomangira zathu pakasinthidwe kakasupe, pomwe zikhalidwe zamagawo ena omwe adatchulidwa zakhazikitsidwa kale.

Nazi zida zingapo zophunzirira nokha:

• KIP-98 - Ndendende Kamodzi Kutumiza ndi Mauthenga Ochitika
• Kufotokozera za zoikamo

Source: www.habr.com