Hai Habr.
Kwanan nan I game da waɗanne sigogi mu a matsayin ƙungiya galibi muke amfani da su don Kafka Producer da Consumer don samun kusanci ga isar da garanti. A cikin wannan labarin ina so in gaya muku yadda muka shirya sake aiwatar da wani taron da aka samu daga Kafka sakamakon rashin samun na ɗan lokaci na tsarin waje.
Aikace-aikace na zamani suna aiki a cikin yanayi mai rikitarwa. Dabarun kasuwanci da aka naɗe a cikin tarin fasahar zamani, yana gudana a cikin hoton Docker wanda mawaƙa kamar Kubernetes ko OpenShift ke gudanarwa, da kuma sadarwa tare da wasu aikace-aikace ko mafita na kasuwanci ta hanyar jerin hanyoyin sadarwa na zahiri da kama-da-wane. A cikin irin wannan yanayi, wani abu na iya rushewa koyaushe, don haka sake sarrafa abubuwan da suka faru idan ɗaya daga cikin tsarin waje ba ya samuwa wani muhimmin sashi ne na tsarin kasuwancin mu.
Yadda abin yake kafin Kafka
Tun da farko a cikin aikin mun yi amfani da IBM MQ don isar da saƙon asynchronous. Idan kowane kuskure ya faru yayin aikin sabis ɗin, za a iya sanya saƙon da aka karɓa a cikin layin mataccen wasiƙa (DLQ) don ƙarin fassarori da hannu. An ƙirƙiri DLQ kusa da layin da ke shigowa, an tura saƙon cikin IBM MQ.
Idan kuskuren ya kasance na ɗan lokaci kuma zamu iya ƙayyade shi (misali, ResourceAccessException akan kiran HTTP ko MongoTimeoutException akan buƙatar MongoDb), to dabarun sake gwadawa zai yi tasiri. Ba tare da la’akari da dabarun reshe na aikace-aikacen ba, an matsar da ainihin saƙon ko dai zuwa layin tsarin don jinkirin aikawa, ko kuma zuwa wani aikace-aikacen daban da aka yi tuntuni don sake aika saƙonni. Wannan ya haɗa da lambar sake aikawa a cikin taken saƙo, wanda ke daura da tazarar jinkiri ko ƙarshen dabarar matakin aikace-aikacen. Idan mun kai ƙarshen dabarun amma tsarin waje har yanzu ba a samu ba, to za a sanya saƙon a cikin DLQ don tantancewa ta hannu.
Neman mafita
, za ku iya samun wadannan . A takaice, an ba da shawarar ƙirƙirar batu don kowane tazarar jinkiri da aiwatar da aikace-aikacen masu amfani a gefe, wanda zai karanta saƙonni tare da jinkirin da ake buƙata.
Duk da yawan adadin tabbataccen sake dubawa, ga alama ba ni da nasara gaba ɗaya. Da farko dai, saboda mai haɓakawa, ban da aiwatar da buƙatun kasuwanci, dole ne ya ciyar da lokaci mai yawa don aiwatar da tsarin da aka bayyana.
Bugu da ƙari, idan an kunna ikon samun dama akan gungu na Kafka, za ku ɗauki ɗan lokaci don ƙirƙirar batutuwa da samar da damar da suka dace. Baya ga wannan, kuna buƙatar zaɓar madaidaicin madaidaicin retention.ms don kowane ɗayan batutuwan sake gwadawa don saƙon su sami lokacin jin daɗi kuma kar su ɓace daga ciki. Dole ne a maimaita aiwatarwa da buƙatar samun dama ga kowane sabis na yanzu ko sabon sabis.
Bari yanzu mu ga irin hanyoyin bazara gabaɗaya da kuma bazara-kafka musamman ke ba mu don sake sarrafa saƙo. Spring-kafka yana da dogaro mai jujjuyawa akan sake gwadawar bazara, wanda ke ba da abstractions don sarrafa BackOffPolicies daban-daban. Wannan kayan aiki ne mai sauƙin sassauƙa, amma babban koma bayansa shine adana saƙonni don sake aikawa a ƙwaƙwalwar aikace-aikacen. Wannan yana nufin sake kunna aikace-aikacen saboda sabuntawa ko kuskuren aiki zai haifar da asarar duk saƙonnin da ke jiran sake sarrafawa. Tun da wannan batu yana da mahimmanci ga tsarin mu, ba mu yi la'akari da shi ba.
spring-kafka kanta yana samar da yawancin aiwatarwa na ContainerAwareErrorHandler, misali , wanda da shi zaku iya sarrafa saƙon daga baya ba tare da canjawa ba idan akwai kuskure. An fara da sigar bazara-kafka 2.3, ya zama mai yiwuwa a saita BackOffPolicy.
Wannan hanyar tana ba da damar sake sarrafa saƙonni don tsira da sake kunna aikace-aikacen, amma har yanzu babu wata hanyar DLQ. Mun zaɓi wannan zaɓi a farkon 2019, da kyakkyawan imani cewa ba za a buƙaci DLQ ba (mun yi sa'a kuma a zahiri ba mu buƙatar shi bayan watanni da yawa na aiki da aikace-aikacen tare da tsarin sake sarrafawa). Kurakurai na ɗan lokaci sun haifar da SeekToCurrentErrorHandler wuta. An buga sauran kurakuran a cikin log ɗin, wanda ya haifar da kashewa, kuma ana ci gaba da sarrafawa tare da saƙo na gaba.
Shawarar ƙarshe
Aiwatar da bisa SeekToCurrentErrorHandler ya sa mu haɓaka tsarin namu don sake aika saƙonni.
Da farko, muna so mu yi amfani da ƙwarewar da ke akwai kuma mu faɗaɗa shi dangane da dabarun aikace-aikacen. Don aikace-aikacen dabaru na layi, zai zama mafi kyau a daina karanta sabbin saƙonni na ɗan gajeren lokaci da dabarun sake gwadawa. Don sauran aikace-aikacen, Ina so in sami maki guda ɗaya wanda zai tilasta dabarun sake gwadawa. Bugu da ƙari, wannan batu ɗaya dole ne ya sami aikin DLQ don hanyoyin biyu.
Dole ne a adana dabarun sake gwadawa kanta a cikin aikace-aikacen, wanda ke da alhakin dawo da tazara ta gaba lokacin da kuskuren ɗan lokaci ya faru.
Dakatar da Mabukaci don Aikace-aikacen Hannun Litattafai
Lokacin aiki tare da spring-kafka, lambar don dakatar da Abokin ciniki na iya yin wani abu kamar haka:
public void pauseListenerContainer(MessageListenerContainer listenerContainer,
Instant retryAt) {
if (nonNull(retryAt) && listenerContainer.isRunning()) {
listenerContainer.stop();
taskScheduler.schedule(() -> listenerContainer.start(), retryAt);
return;
}
// to DLQ
}A cikin misali, sake gwadawa shine lokacin sake kunna MessageListenerContainer idan har yanzu yana gudana. Sake ƙaddamarwa zai faru ne a cikin wani zaren daban da aka ƙaddamar a cikin TaskScheduler, aiwatar da shi kuma ana ba da shi ta bazara.
Muna samun ƙimar sake gwadawa ta hanya mai zuwa:
- Ana duba ƙimar ma'aunin sake kira.
- Dangane da ƙimar ƙima, ana bincika tazarar jinkiri na yanzu a cikin dabarun sake gwadawa. An bayyana dabarun a cikin aikace-aikacen kanta; mun zaɓi tsarin JSON don adana shi.
- Tazarar da aka samo a cikin tsararrun JSON ya ƙunshi adadin daƙiƙai bayan haka ana buƙatar maimaita aiki. Ana ƙara wannan adadin na daƙiƙa zuwa lokacin yanzu don samar da ƙimar sake gwadawaAt.
- Idan ba a sami tazarar ba, to ƙimar sake gwadawa ba ta da amfani kuma za a aika saƙon zuwa DLQ don tantancewa da hannu.
Tare da wannan hanyar, abin da ya rage shine adana adadin maimaita kira ga kowane saƙon da ake sarrafa shi a halin yanzu, misali a ƙwaƙwalwar ajiyar aikace-aikacen. Tsayawa ƙidaya sake gwadawa a ƙwaƙwalwar ajiya ba shi da mahimmanci ga wannan hanya, tun da aikace-aikacen dabaru na linzamin kwamfuta ba zai iya sarrafa sarrafawa gaba ɗaya ba. Ba kamar sake gwadawar bazara ba, sake kunna aikace-aikacen ba zai haifar da asarar duk saƙonni don sake sarrafa su ba, amma kawai zai sake farawa dabarun.
Wannan hanyar tana taimakawa cire nauyin daga tsarin waje, wanda ba zai iya samuwa ba saboda nauyi mai nauyi. A wasu kalmomi, ban da sake sarrafawa, mun sami nasarar aiwatar da tsarin .
A cikin yanayinmu, madaidaicin kuskuren 1 ne kawai, kuma don rage lokacin raguwar tsarin saboda katsewar hanyar sadarwa ta wucin gadi, muna amfani da dabarar sake gwadawa sosai tare da ƙananan tazarar jinkiri. Wannan bazai dace da duk aikace-aikacen rukuni ba, don haka alakar da ke tsakanin madaidaicin kuskure da ƙimar tazara dole ne a zaɓi dangane da halayen tsarin.
Aikace-aikace daban don sarrafa saƙonni daga aikace-aikace tare da dabaru marasa ƙima
Ga misalin lambar da ke aika saƙo zuwa irin wannan aikace-aikacen (Mai Maimaitawa), wanda zai sake aikawa zuwa taken DESTINATION idan lokacin RETRY_AT ya cika:
public <K, V> void retry(ConsumerRecord<K, V> record, String retryToTopic,
Instant retryAt, String counter, String groupId, Exception e) {
Headers headers = ofNullable(record.headers()).orElse(new RecordHeaders());
List<Header> arrayOfHeaders =
new ArrayList<>(Arrays.asList(headers.toArray()));
updateHeader(arrayOfHeaders, GROUP_ID, groupId::getBytes);
updateHeader(arrayOfHeaders, DESTINATION, retryToTopic::getBytes);
updateHeader(arrayOfHeaders, ORIGINAL_PARTITION,
() -> Integer.toString(record.partition()).getBytes());
if (nonNull(retryAt)) {
updateHeader(arrayOfHeaders, COUNTER, counter::getBytes);
updateHeader(arrayOfHeaders, SEND_TO, "retry"::getBytes);
updateHeader(arrayOfHeaders, RETRY_AT, retryAt.toString()::getBytes);
} else {
updateHeader(arrayOfHeaders, REASON,
ExceptionUtils.getStackTrace(e)::getBytes);
updateHeader(arrayOfHeaders, SEND_TO, "backout"::getBytes);
}
ProducerRecord<K, V> messageToSend =
new ProducerRecord<>(retryTopic, null, null, record.key(), record.value(), arrayOfHeaders);
kafkaTemplate.send(messageToSend);
}Misalin yana nuna cewa ana watsa bayanai da yawa a cikin rubutun kai. Ana samun ƙimar RETRY_AT daidai da tsarin sake gwadawa ta wurin Tasha. Baya ga DESTINATION da RETRY_AT mun wuce:
- GROUP_ID, ta inda muke tara saƙonni don bincike na hannu da sauƙaƙe bincike.
- ORIGINAL_PARTITION don ƙoƙarin kiyaye mabukaci iri ɗaya don sake sarrafawa. Wannan siga na iya zama banza, inda za a sami sabon ɓangaren ta amfani da maɓallin rikodin.key() na ainihin saƙon.
- An sabunta ƙimar COUNTER don bin dabarun sake gwadawa.
- SEND_TO akai-akai yana nuna ko an aika saƙon don sake aiwatarwa yayin isa RETRY_AT ko sanya shi cikin DLQ.
- DALILI - dalilin da ya sa aka katse sarrafa saƙon.
Mai sake gwadawa yana adana saƙonni don sake aikawa da kuma tantancewa da hannu a cikin PostgreSQL. Mai ƙidayar lokaci yana fara aikin da zai nemo saƙonni tare da RETRY_AT kuma ya aika da su zuwa ORIGINAL_PARTITION bangaren DESTINATION tare da maɓalli.key().
Da zarar an aika, ana share saƙonni daga PostgreSQL. Fassarar saƙonnin da hannu yana faruwa a cikin UI mai sauƙi wanda ke hulɗa tare da Mai sake gwadawa ta REST API. Babban fasalinsa shine sake aikawa ko share saƙonni daga DLQ, duba bayanan kuskure da neman saƙonni, misali ta sunan kuskure.
Tunda an kunna ikon samun dama akan gungu na mu, ya zama dole a bugu da ƙari don neman samun dama ga batun da Mai sake gwadawa yake sauraro, kuma a ba da damar Mai sake gwadawa ya rubuta zuwa batun DESTINATION. Wannan ba shi da daɗi, amma, ba kamar tsarin tazara ba, muna da cikakken DLQ da UI don sarrafa shi.
Akwai lokuta lokacin da ƙungiyoyin mabukaci daban-daban ke karanta batun mai shigowa, waɗanda aikace-aikacensu ke aiwatar da dabaru daban-daban. Sake sarrafa saƙo ta hanyar Retryer na ɗayan waɗannan aikace-aikacen zai haifar da kwafi akan ɗayan. Don kare kai daga wannan, mun ƙirƙiri wani batu na daban don sake sarrafawa. Mabukaci iri ɗaya na iya karanta batutuwa masu shigowa da sake gwadawa ba tare da wani hani ba.
Ta hanyar tsoho wannan hanyar ba ta samar da aikin mai watsewar kewayawa ba, duk da haka ana iya ƙara shi zuwa aikace-aikacen ta amfani da shi ko sabo , Kunna wuraren da ake kiran sabis na waje zuwa abubuwan da suka dace. Bugu da kari, yana yiwuwa a zabi dabarun don tsari, wanda kuma zai iya zama da amfani. Misali, a cikin spring-cloud-netflix wannan na iya zama tafkin zare ko semaphore.
ƙarshe
Sakamakon haka, muna da aikace-aikacen daban wanda ke ba mu damar maimaita sarrafa saƙon idan babu wani tsarin waje na ɗan lokaci.
Ɗaya daga cikin manyan fa'idodin aikace-aikacen shine cewa ana iya amfani da shi ta tsarin waje da ke gudana akan gungu na Kafka ɗaya, ba tare da gyare-gyare masu mahimmanci a gefen su ba! Irin wannan aikace-aikacen zai buƙaci kawai don isa ga batun sake gwadawa, cike wasu ƴan rubutun Kafka da aika sako zuwa ga Mai sake gwadawa. Babu buƙatar tada wani ƙarin kayan more rayuwa. Kuma don rage adadin saƙonnin da ake turawa daga aikace-aikacen zuwa Maimaitawa da baya, mun gano aikace-aikacen tare da ma'ana ta layi kuma muka sake sarrafa su ta hanyar Tasha.
source: www.habr.com