Hei Habr!
Ini ndinoshanda paTinkoff timu, iyo iri kugadzira yayo yekuzivisa nzvimbo. Ini ndinonyanya kukudziridza muJava ndichishandisa Spring boot uye kugadzirisa akasiyana siyana ehunyanzvi matambudziko anomuka mupurojekiti.
Mazhinji emamicroservices edu anotaurirana asynchronously kuburikidza nemeseji broker. Kare, takashandisa IBM MQ semutengesi, iyo yakanga isisakwanisi kutarisana nemutoro, asi panguva imwechete yakanga ine zvivimbiso zvepamusoro zvekutumira.
Sekutsiva, isu takapihwa Apache Kafka, iyo ine yakakwirira scaling mukana, asi, zvinosuruvarisa, inoda ingangoita yemunhu nzira yekumisikidza akasiyana siyana. Mukuwedzera, iyo kamwechete kamwe chete nzira yekuendesa iyo inoshanda muKafka nekusagadzika haina kubvumira kuchengetedza danho rinodiwa rekuenderana kunze kwebhokisi. Tevere, ini ndichagovera ruzivo rwedu muKafka kumisikidzwa, kunyanya, ini ndichakuudza maitiro ekugadzirisa uye kurarama nekunyatso kuendesa.
Guaranteed delivery and more
Izvo zvigadziriso zvinokurukurwa pazasi zvichabatsira kudzivirira akati wandei matambudziko ane default kubatana marongero. Asi chekutanga ndinoda kutarisisa kune imwe parameter inogonesa debug inogoneka.
Izvi zvichabatsira client.id yeMugadziri uye Mutengi. Pakutanga kutarisa, unogona kushandisa zita rekushandisa seyo kukosha, uye kazhinji izvi zvinoshanda. Kunyangwe iyo mamiriro kana application ichishandisa akati wandei Vatengi uye iwe uchivapa imwechete client.id, inoguma neyambiro inotevera:
org.apache.kafka.common.utils.AppInfoParser β Error registering AppInfo mbean javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=kafka.test-0Kana iwe uchida kushandisa JMX mukushandisa neKafka, saka izvi zvinogona kuve dambudziko. Panyaya iyi, zviri nani kushandisa musanganiswa wezita rekushandisa uye, semuenzaniso, zita remusoro sekuti mutengi.id kukosha. Mhedzisiro yekumisikidzwa kwedu inogona kuoneka mune yekuraira kubuda kafka-vatengi-mapoka kubva kune zvishandiso kubva kuConfluent:
Zvino ngatitarisei mamiriro ekuvimbiswa meseji kutumira. Kafka Producer ane parameter acks, iyo inokutendera kuti ugadzirise mushure mekunge vangani vanobvuma kuti mutungamiri weboka anofanirwa kufunga nezve meseji yakanyorwa zvinobudirira. Iyi parameter inogona kutora zvinotevera zvakakosha:
- 0 - kubvuma hakuzotariswe.
- 1 ndiyo yakasarudzika parameter, chete 1 replica inodiwa kuti ubvume.
- -1 - bvuma kubva kune ese akawiriraniswa replicas inodiwa (kuseta sumbu min.insync.replicas).
Kubva pane zvakanyorwa kukosha zviri pachena kuti acks akaenzana ne -1 inopa vimbiso yakasimba yekuti meseji haizorasikirwe.
Sezvatinoziva tese, masisitimu akagoverwa haana kuvimbika. Kuchengetedza kubva kuzvikanganiso zvenguva pfupi, Kafka Mugadziri anopa sarudzo inoedzazve, iyo inokubvumira kuti uise nhamba yekuedzazve kutumira mukati delivery.timeout.ms. Sezvo inoedzazve parameter iine kukosha kwekutanga kweInteger.MAX_VALUE (2147483647), nhamba yemeseji yekuedza zvakare inogona kugadziriswa nekuchinja chete delivery.timeout.ms.
Isu tiri kuenda kune chaiyo kamwe chete kutumirwa
Iwo akanyorwa marongero anobvumira Mugadziri wedu kuendesa mameseji ane garandi yepamusoro. Ngatitaurei nezve maitiro ekuona kuti kopi imwe chete yemeseji inonyorwa kune Kafka musoro? Muchiitiko chakareruka, kuita izvi, unofanirwa kuseta parameter paProducer gonesa.kusava nesimba kuti chokwadi. Idempotency inovimbisa kuti meseji imwe chete inonyorwa kune yakatarwa chikamu chemusoro mumwe. Iyo yekutanga yekugonesa idempotency ndiyo tsika acks = zvose, edzazve > 0, max.in.flight.requests.per.connection β€ 5. Kana aya ma paramita asina kutaurwa nemugadziri, iwo ari pamusoro apa maitiro anozoiswa otomatiki.
Kana idempotency ikagadziriswa, zvinodikanwa kuve nechokwadi kuti iwo mameseji mamwechete anoguma ari muzvikamu zvakafanana nguva dzese. Izvi zvinogona kuitwa nekuisa partitioner.class kiyi uye parameter kuProducer. Ngatitange nekiyi. Inofanira kunge yakafanana pane imwe neimwe kutumira. Izvi zvinogona kuwanikwa zviri nyore nekushandisa chero ebhizinesi ID kubva kune yekutanga positi. The partitioner.class parameter ine default value - . Neiyi nzira yekugovera, nekusarudzika tinoita seizvi:
- Kana kupatsanura kwakanyatso kutsanangurwa pakutumira meseji, isu tinoishandisa.
- Kana chikamu chisina kutaurwa, asi kiyi inotsanangurwa, sarudza kupatsanurwa nehashi yekiyi.
- Kana chikamu uye kiyi zvisina kutaurwa, sarudza zvikamu chimwe nechimwe (kutenderera-robin).
Zvakare, kushandisa kiyi uye idempotent kutumira ine parameter max.in.flight.requests.per.connection = 1 inokupa yakagadziridzwa meseji kugadzirisa paMutengi. Izvo zvakakoshawo kuyeuka kuti kana kutonga kwekuwana kwakagadziriswa pane yako cluster, saka iwe unozoda kodzero dzekunyora zvisina musoro kune imwe nyaya.
Kana ukangoerekana washaya hunyanzvi hwekutumira nekiyi kana kuti pfungwa padivi reProducer inoda kuchengetedza kuenderana kwedata pakati pezvikamu zvakasiyana, ipapo kutengeserana kuchauya kuzonunura. Mukuwedzera, uchishandisa cheni yekutengeserana, unogona kuwiriranisa rekodhi muKafka, semuenzaniso, nerekodhi mudhatabhesi. Kugonesa kutengeserana kutumira kune Mugadziri, inofanirwa kuve isina simba uye nekuwedzera kuseta transaction.id. Kana Kafka cluster yako iine control control yakagadziriswa, saka rekodhi rekodhi, senge rekodhi risingaite, rinoda mvumo yekunyora, iyo inogona kupihwa nemasiki uchishandisa kukosha kwakachengetwa mutransactional.id.
Zvekare, chero tambo, senge zita rekushandisa, inogona kushandiswa sechiratidzo chekutengeserana. Asi kana iwe ukatangisa akati wandei echishandiso chimwe chete neiyo transactional.id, ipapo yekutanga yakatangwa chiitiko ichamiswa nekanganiso, sezvo Kafka ichaiona se zombie maitiro.
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.Kuti tigadzirise dambudziko iri, isu tinowedzera suffix kune zita rekunyorera muchimiro chezita rezita, iro ratinowana kubva kune zvakatipoteredza zvinosiyana.
Mugadziri anogadziriswa, asi kutengeserana paKafka kunongodzora chiyero chemeseji. Pasinei nemamiriro ekutengeserana, meseji yacho inobva yaenda kumusoro wenyaya, asi ine humwe hunhu hwehurongwa.
Kudzivirira mameseji akadai kubva kuverengwa neMutengi nguva isati yakwana, inoda kuseta parameter kuzviparadzanisa nevamwe.level kuverenga_yakazvipira kukosha. Mutengi akadaro anozokwanisa kuverenga mameseji asiri ekutengeserana sepakutanga, uye mameseji ekutengeserana chete mushure mekuita.
Kana iwe wakaseta ese magadzirirwo akanyorwa kare, saka iwe wakagadzirisa chaizvo kamwe chete kutumirwa. Makorokoto!
Asi pane imwe nuance. Transactional.id, yatakamisa pamusoro, ndiyo prefix yekutengeserana. Pane maneja wekutengesa, nhamba yekutevedzana inowedzerwa kwairi. Chiziviso chakagamuchirwa chinopihwa transactional.id.expiration.ms, iyo inogadziriswa pane Kafka cluster uye ine default value ye "7 days". Kana panguva ino application isati yawana chero meseji, saka paunoyedza iyo inotevera transaction tumira iwe uchagamuchira InvalidPidMappingException. Murongi wekutengeserana anozoburitsa nhamba itsva yekutevedzana kune inotevera kutengeserana. Zvakadaro, meseji inogona kurasika kana InvalidPidMappingException isina kubatwa nemazvo.
Panzvimbo pezvizere
Sezvauri kuona, hazvina kukwana kungotumira mameseji kuKafka. Iwe unofanirwa kusarudza musanganiswa wemaparamita uye gadzirira kuita shanduko nekukurumidza. Muchinyorwa chino, ndakaedza kuratidza zvakadzama iyo chaiyo kamwechete kuburitsa setup uye ndikatsanangura akati wandei matambudziko neclient.id uye transactional.id masisitimu atakasangana nawo. Pazasi pane pfupiso yeMugadziri uye Consumer marongero.
Murimi:
- acks = zvose
- inoedzazve > 0
- enable.idempotence = chokwadi
- max.in.flight.requests.per.connection β€ 5 (1 yekutumira zvakarongeka)
- transactional.id = ${application-name}-${hostname}
Mutengi:
- isolation.level = read_committed
Kuti tideredze zvikanganiso mune ramangwana maapplication, isu takagadzira yedu wrapper pamusoro pekumisikidzwa kwechirimo, uko ukoshi hwemamwe maparamita akanyorwa atoiswa.
Heano mashoma ezvinyorwa zvekuzvidzidzira wega:
Source: www.habr.com