Hai Habr!
Ina aiki a ƙungiyar Tinkoff, wacce ke haɓaka cibiyar sanarwar kanta. Na fi haɓakawa a cikin Java ta amfani da boot boot kuma na magance matsalolin fasaha daban-daban waɗanda ke tasowa a cikin aikin.
Yawancin ƙananan ayyukan mu suna sadarwa tare da juna ta hanyar dillalin saƙo. A baya can, mun yi amfani da IBM MQ a matsayin dillali, wanda ba zai iya jurewa da kaya ba, amma a lokaci guda yana da babban garantin bayarwa.
A matsayin musanya, an ba mu Apache Kafka, wanda ke da yuwuwar zazzagewa, amma, da rashin alheri, yana buƙatar kusan kowane tsari na daidaitawa don yanayi daban-daban. Bugu da ƙari, aƙalla tsarin isarwa sau ɗaya wanda ke aiki a cikin Kafka ta tsohuwa bai ba da izinin kiyaye matakin da ake buƙata na daidaito daga cikin akwatin ba. Na gaba, zan raba kwarewarmu a cikin tsarin Kafka, musamman, zan gaya muku yadda ake daidaitawa da rayuwa tare da isarwa sau ɗaya daidai.
Garantin bayarwa da ƙari
Saitunan da aka tattauna a ƙasa zasu taimaka hana matsaloli da yawa tare da saitunan haɗin kai na asali. Amma da farko ina so in kula da siga guda ɗaya wanda zai sauƙaƙe yuwuwar cire kuskure.
Wannan zai taimaka abokin ciniki.id don Furodusa da Mabukaci. A kallon farko, zaku iya amfani da sunan aikace-aikacen azaman ƙimar, kuma a mafi yawan lokuta wannan zaiyi aiki. Ko da yake halin da ake ciki lokacin da aikace-aikacen ke amfani da Abokan Ciniki da yawa kuma kuna ba su abokin ciniki iri ɗaya.id, yana haifar da gargaɗin mai zuwa:
org.apache.kafka.common.utils.AppInfoParser — Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=kafka.test-0Idan kuna son amfani da JMX a cikin aikace-aikacen tare da Kafka, to wannan na iya zama matsala. Don wannan yanayin, yana da kyau a yi amfani da haɗin sunan aikace-aikacen da, alal misali, sunan jigon azaman ƙimar abokin ciniki.id. Ana iya ganin sakamakon daidaitawar mu a cikin fitarwar umarni ƙungiyoyin kafka-mabukaci daga utilities daga Confluent:
Yanzu bari mu kalli yanayin yanayin isar da saƙon garanti. Kafka Producer yana da siga aks, wanda ke ba ka damar daidaitawa bayan nawa ne yarda da shugaban ƙungiyar ya buƙaci yin la'akari da saƙon da aka rubuta cikin nasara. Wannan siga na iya ɗaukar dabi'u masu zuwa:
- 0 - ba za a yi la'akari da yarda ba.
- 1 shine madaidaicin tsoho, kwafi 1 kawai ake buƙata don ganewa.
- -1 - amincewa daga duk kwafi masu aiki tare ana buƙatar (saitin tari min.insync.replicas).
Daga lissafin da aka lissafa a bayyane yake cewa acks daidai -1 yana ba da garanti mafi ƙarfi cewa ba za a rasa saƙon ba.
Kamar yadda muka sani, tsarin da aka rarraba ba su da aminci. Don karewa daga kurakuran ɗan lokaci, Kafka Producer yana ba da zaɓi sake gwadawa, wanda ke ba ka damar saita yawan ƙoƙarin sake aikawa a ciki bayarwa.lokacin ƙarewa.ms. Tunda ma'aunin sake gwadawa yana da tsohuwar ƙima ta Integer.MAX_VALUE (2147483647), adadin sakewar saƙon za a iya daidaita shi ta canza isarwa kawai.timeout.ms.
Muna tafiya zuwa ga isarwa sau ɗaya daidai
Saitunan da aka jera suna ba Mai samarwa mu damar isar da saƙonni tare da babban garanti. Yanzu bari mu yi magana game da yadda za a tabbatar da cewa kwafin saƙo ɗaya ne kawai aka rubuta zuwa batun Kafka? A cikin mafi sauƙi, don yin wannan, kuna buƙatar saita siga akan Mai samarwa ba da damar.idempotence ga gaskiya. Idempotency yana ba da tabbacin cewa saƙo ɗaya ne kawai aka rubuta zuwa takamaiman yanki na jigo ɗaya. Sharadi don kunna idempotency shine dabi'u acks = duk, sake gwadawa> 0, max.in.flight.requests.per.connection ≤ 5. Idan mai haɓakawa bai bayyana waɗannan sigogi ba, za a saita ƙimar da ke sama ta atomatik.
Lokacin da aka daidaita idempotency, ya zama dole a tabbatar da cewa saƙonni iri ɗaya sun ƙare a cikin ɓangarori ɗaya kowane lokaci. Ana iya yin wannan ta hanyar saita maɓallin partitioner.class da siga zuwa Producer. Bari mu fara da maɓalli. Dole ne ya zama iri ɗaya ga kowane ƙaddamarwa. Ana iya samun wannan cikin sauƙi ta amfani da kowane ID na kasuwanci daga ainihin sakon. Sigar partitioner.class yana da tsohuwar ƙima - . Tare da wannan dabarar rarraba, ta tsohuwa muna aiki kamar haka:
- Idan partition din ya bayyana a sarari lokacin aika saƙon, to muna amfani da shi.
- Idan ba a fayyace ɓangaren ba, amma an ayyana maɓalli, zaɓi ɓangaren ta hanyar hash na maɓallin.
- Idan ba a bayyana bangare da maɓalli ba, zaɓi ɓangarori ɗaya bayan ɗaya (round-robin).
Hakanan, ta amfani da maɓalli da aika mai ƙarfi tare da siga max.in.flight.requests.per.connection = 1 yana ba ku ingantaccen sarrafa saƙon akan Mabukaci. Hakanan yana da kyau a tuna cewa idan an saita ikon samun dama akan tarin ku, to kuna buƙatar haƙƙoƙin yin rubutu da ƙarfi ga wani batu.
Idan ba zato ba tsammani ba ku da ikon aika aika ta maɓalli ko ma'ana a gefen Mai samarwa yana buƙatar kiyaye daidaiton bayanai tsakanin ɓangarori daban-daban, to, ma'amaloli za su zo ga ceto. Bugu da kari, ta yin amfani da ma'amalar sarkar, zaku iya daidaita rikodin a cikin Kafka bisa sharadi, misali, tare da rikodi a cikin bayanan. Don ba da damar aika ma'amala ga Mai samarwa, dole ne ya kasance mai ƙarfi kuma an saita shi ma'amala.id. Idan gungu na Kafka yana da tsarin sarrafa damar shiga, to rikodin ma'amala, kamar rikodin mai ƙarfi, zai buƙaci rubuta izini, wanda za'a iya ba da shi ta hanyar abin rufe fuska ta amfani da ƙimar da aka adana a cikin transactional.id.
A bisa ƙa'ida, kowane kirtani, kamar sunan aikace-aikacen, ana iya amfani da shi azaman mai gano ma'amala. Amma idan kun ƙaddamar da lokuta da yawa na aikace-aikacen iri ɗaya tare da ma'amala iri ɗaya.id, to za a dakatar da misalin farko da aka ƙaddamar da kuskure, tunda Kafka zai ɗauki shi azaman tsarin aljan.
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.Don magance wannan matsalar, muna ƙara ƙaranci ga sunan aikace-aikacen a cikin nau'in sunan mai masauki, wanda muke samu daga masu canjin yanayi.
An saita furodusa, amma ma'amaloli akan Kafka kawai suna sarrafa iyakar saƙon. Ko da kuwa matsayin ma'amala, saƙon nan da nan ya tafi kan batun, amma yana da ƙarin halayen tsarin.
Don hana irin waɗannan saƙon daga karantawa daga mabukaci kafin lokaci, yana buƙatar saita sigogi ware.matakin don karanta_ƙimar da aka sadaukar. Irin wannan mabukaci zai iya karanta saƙonnin da ba na kasuwanci ba kamar da, da kuma saƙonnin ma'amala kawai bayan ƙaddamarwa.
Idan kun saita duk saitunan da aka jera a baya, to kun saita daidai sau ɗaya bayarwa. Taya murna!
Amma akwai ƙarin nuance. Transactional.id, wanda muka saita a sama, shine ainihin ma'anar ma'amala. A kan mai sarrafa ma'amala, ana ƙara lambar jeri zuwa gare ta. Ana ba da mai ganowa ga ma'amala.id.karewa.ms, wanda aka saita akan gungu na Kafka kuma yana da ƙimar tsoho na "kwanaki 7". Idan a wannan lokacin aikace-aikacen bai sami wani sako ba, to lokacin da kuka gwada aika ma'amala na gaba zaku karɓi InvalidPidMappingException. Sannan mai gudanar da mu'amala zai fitar da sabuwar lamba don ciniki na gaba. Koyaya, ana iya rasa saƙon idan ba a kula da InvalidPidMappingException daidai ba.
Maimakon sakamako
Kamar yadda kuke gani, bai isa kawai aika saƙonni zuwa Kafka ba. Kuna buƙatar zaɓar haɗin sigogi kuma ku kasance cikin shiri don yin canje-canje mai sauri. A cikin wannan labarin, na yi ƙoƙarin nuna dalla-dalla daidai saitin isarwa sau ɗaya kuma na bayyana matsaloli da yawa tare da saitunan abokin ciniki.id da ma'amala.id waɗanda muka ci karo da su. A ƙasa akwai taƙaitaccen saitunan Mai samarwa da Mai amfani.
Mai saka:
- acks = duk
- sake dubawa > 0
- ikon.idempotence = gaskiya
- max.in.flight.requests.per.connection ≤ 5 (1 don aikawa cikin tsari)
- transactional.id = ${application-name}-${hostname}
Mabukaci:
- ware.level = karanta_committed
Don rage kurakurai a aikace-aikace na gaba, mun yi namu nannade akan tsarin bazara, inda an riga an saita ƙimar wasu sigogin da aka jera.
Ga wasu abubuwa biyu don nazarin kai:
source: www.habr.com