Kuenderera mberi kweshanduro yebhuku diki:
Kunzwisisa Message Brokers
munyori: Jakub Korab, muparidzi: O'Reilly Media, Inc., zuva rekubudiswa: June 2017, ISBN: 9781492049296.
Chikamu chakamboshandurwa:
CHITSAUKO 3
Kafka
Kafka yakagadziridzwa neLinkedIn kuti itenderere zvimwe zvipimo zvechinyakare meseji broker uye kudzivirira kumisikidza akawanda meseji mabhureki kune akasiyana point-to-point yekudyidzana, inotsanangurwa mubhuku rino pasi pe "Kuwedzera uye kubuda" papeji 28. .Kushandisa zviitiko LinkedIn yakatsamira zvakanyanya pane imwe-nzira yekupinzwa kwehuwandu hwakawanda kwazvo hwe data, senge kudzvanya peji uye kupinda matanda, uku vachiri kubvumira iyo data kuti ishandiswe neakawanda masisitimu pasina kukanganisa kubereka kwevagadziri kana vamwe vatengi. Muchokwadi, chikonzero chiripo Kafka ndechekuwana rudzi rwemameseji ekuvaka anotsanangurwa neUniversal Data Pipeline.
Tichifunga nezvechinangwa chekupedzisira ichi, zvimwe zvinodiwa zvakangomuka. Kafka inofanira:
- Iva nekukurumidza zvakanyanya
- Ipa yakawanda bandwidth paunenge uchishanda nemameseji
- Tsigira Muparidzi-Munyori uye Point-to-Point modhi
- Usanonoke nekuwedzera vatengi. Semuyenzaniso, kuita kwezvose zviri zviviri mutsetse uye musoro wenyaya muActiveMQ zvinodzikisira sezvo nhamba yevatengi painoenda inokura.
- Iva horizontally scalable; kana bhuroka mumwe anorambira mameseji anogona kungoita saizvozvo pahupamhi disk kumhanya, saka zvine musoro kuenda kupfuura imwe chete broker muenzaniso kuti uwedzere mashandiro.
- Deredza kuwana uye kudzoreredza meseji
Kuzadzisa zvese izvi, Kafka yakatora chivakwa chakatsanangura zvakare mabasa uye mabasa evatengi uye mameseji broker. Iyo JMS modhi yakanangana nebroker, uko broker ane basa rekuparadzira mameseji uye vatengi vanongofanira kunetseka nezve kutumira uye kugamuchira meseji. Kafka, kune rumwe rutivi, ndeye mutengi-centric, nemutengi achitora akawanda ezvimiro zvechinyakare bhuroka, sekugovera kwakaringana mameseji akakodzera kune vatengi, mukutsinhana neakanyanya kukurumidza uye scalable broker. Kune vanhu vakashanda nemaitiro echinyakare meseji, kushanda neKafka kunoda shanduko yakakosha yepfungwa.
Iyi dhizaini yeinjiniya yatungamira mukugadzirwa kwechivakwa chekutumira mameseji chinokwanisa kuwedzera kuburikidza nemirairo yakawanda yehukuru zvichienzaniswa neyakajairwa broker. Sezvatichaona, nzira iyi inouya nekutengeserana-offs, zvinoreva kuti Kafka haina kukodzera kune mamwe marudzi emabasa uye software yakaiswa.
Unified Destination Model
Kuti azadzise zvinodiwa zvakatsanangurwa pamusoro, Kafka yakabatanidza kuburitsa-kunyoresa uye kunongedza-kune-meseji pasi pemhando imwe yekuenda - topic. Izvi zvinovhiringa kune vanhu vakashanda nemasisitimu ekutumira mameseji, apo izwi rekuti "musoro" rinoreva nzira yekutepfenyura kubva (kubva pamusoro) kuverenga kusingagoneki. Misoro yeKafka inofanirwa kutorwa semhando yekwakasanganiswa, sekutsanangurwa kwazvinoitwa munhanganyaya yebhuku rino.
Kune kwasara chitsauko chino, kunze kwekunge tanyatso taura neimwe nzira, izwi rekuti "musoro" rinenge richireva dingindira reKafka.
Kuti tinzwisise zvizere mabatiro anoita misoro uye kuti ndezvipi vimbiso dzavanopa, tinoda kutanga tatarisa kuti dzinoitwa sei muKafka.
Chinyorwa chega chega muKafka chine chinyorwa chayo.
Vagadziri vanotumira mameseji kuKafka nyora kune iyi log, uye vatengi vanoverenga kubva kurogi vachishandisa anonongedzera anogara achienderera mberi. Nguva nenguva, Kafka anodzima zvikamu zvekare zvegiyo, angave mameseji ari muzvikamu izvozvo akaverengwa kana kwete. Chikamu chepakati chekugadzirwa kweKafka ndechekuti mutengesi haana basa kana mameseji akaverengwa kana kuti kwete - ndiro basa remutengi.
Mazwi ekuti "log" uye "pointer" haaonekwe mukati . Aya mazwi anozivikanwa zvikuru anoshandiswa pano kubatsira kunzwisisa.
Iyi modhi yakasiyana zvachose neActiveMQ, uko mameseji kubva kumitsara yese anochengetwa mugwaro rimwe chete, uye broker anomaka mameseji sekudzimwa mushure mekunge averengwa.
Ngatimbozamai zvishoma uye titarise iyo chinyorwa chinyorwa mune zvakadzama.
Iyo Kafka log ine zvikamu zvinoverengeka () Kafka inovimbisa kurongeka kwakasimba muchikamu chega chega. Izvi zvinoreva kuti mameseji anonyorerwa kuchikamu mune imwe nhevedzano anozoverengwa nenzira imwechete. Chikamu chega chega chinoitwa senge rolling log faira rine a subset (subset) yemameseji ese anotumirwa kumusoro nevagadziri vayo. Musoro wakagadzirwa une, nekusingaperi, chikamu chimwe chete. Pfungwa yezvikamu ndiyo pfungwa yepakati yeKafka yekuyera yakatwasuka.
Mufananidzo 3-1. Kafka Partitions
Kana mugadziri anotumira meseji kumusoro weKafka, inosarudza kuti ndeipi chikamu chekutumira meseji. Tichatarisa izvi zvakadzama gare gare.
Kuverenga mameseji
Mutengi anoda kuverenga mameseji anotonga ane zita pointer inonzi boka revatengi, iyo inonongedzera offset mameseji muchikamu. An offset inzvimbo yekuwedzera inotangira pa 0 pakutanga kwechikamu. Iri boka revatengi, rinotaurwa muAPI kuburikidza nemushandisi-rinotsanangurwa boka_id, rinoenderana mutengi ane musoro kana system.
Mazhinji masisitimu ekutumira mameseji anoverenga data kubva kwaanosvika achishandisa akawanda zviitiko uye shinda kugadzirisa mameseji akafanana. Saka, panowanzova nezviitiko zvakawanda zvevatengi vanogovana boka rimwechete revatengi.
Dambudziko rekuverenga rinogona kumiririrwa sezvizvi:
- Nyaya ine zvikamu zvakawanda
- Mapoka akawanda evatengi anogona kushandisa musoro panguva imwe chete
- Boka revatengi rinogona kuva nezviitiko zvakawanda zvakasiyana
Iri idambudziko risiri diki kune vakawanda-kune-vazhinji. Kuti tinzwisise kuti Kafka inobata sei hukama pakati pemapoka evatengi, zviitiko zvevatengi, uye zvikamu, ngatitarisei nhevedzano yezvishoma zvishoma zvishoma zvakaoma kuverenga kuverenga.
Vatengi uye mapoka evatengi
Ngatitorei sechinhu chekutanga musoro wenyaya ine chikamu chimwe ().
Mufananidzo 3-2. Mutengi anoverenga kubva kuchikamu
Kana chiitiko chemutengi chikabatana nechayo boka_id kune ino musoro, inopihwa chikamu chekuverenga uye chinogadziriswa mune icho chikamu. Nzvimbo yeiyi offset inogadziriswa mutengi seanongedzo kune ichangoburwa chinzvimbo (itsva meseji) kana chinzvimbo chekare (chekare meseji). Mutengi anokumbira (mavhoti) mameseji kubva pachinyorwa, izvo zvinoita kuti vaverengerwe zvakatevedzana kubva kurogi.
Iyo yekubvisa chinzvimbo inogara ichidzoserwa kuKafka uye inochengetwa semeseji mune yemukati musoro _consumer_offsets. Verenga mameseji haasati adzimwa, kusiyana neanogara achibhuroka, uye mutengi anogona kudzoreredza offset kuti agadzirise zvakare mameseji atoonekwa.
Kana mutengi wechipiri ane musoro akabatana achishandisa boka rakasiyana_id, inobata chechipiri chinongedzo chakazvimiririra chekutanga () Saka, musoro weKafka unoita semutsara pane mutengi mumwe uye seyakajairwa kuburitsa-kunyorera (pub-sub) musoro unonyoreswa nevatengi vakawanda, nerubatsiro rwakawedzerwa rwekuti mameseji ese anochengetwa uye anogona kugadziriswa kakawanda.
Mufananidzo 3-3. Vatengi vaviri mumapoka akasiyana evatengi vanoverenga kubva kune imwechete partition
Vatengi vari muboka revatengi
Kana imwe mutengi chiitiko ichiverenga data kubva pachikamu, ine yakazara kutonga kweinongedzo uye inogadzirisa mameseji sezvakatsanangurwa muchikamu chakapfuura.
Kana zviitiko zvakati zvevatengi zvakabatana neboka rimwechete_id kune imwe nyaya ine chikamu chimwe chete, ipapo muenzaniso wakabatana wekupedzisira uchapihwa kutonga pamusoro peinongedzo uye kubva panguva iyoyo zvichienda mberi ichagamuchira mameseji ese ().
Mufananidzo 3-4. Vatengi vaviri muboka rimwechete revatengi vanoverenga kubva kune imwechete partition
Iyi nzira yekugadzirisa, umo nhamba yezviitiko zvevatengi inodarika nhamba yezvikamu, inogona kufungidzirwa semhando yemutengi akasarudzika. Izvi zvinogona kubatsira kana iwe uchida "active-passive" (kana "hot-warm") kuunganidza kwevatengi vako zviitiko, kunyange hazvo uchimhanyisa vatengi vakawanda zvakafanana ("active-active" kana "hot-hot") inowanzoitika kupfuura. vatengi.
Aya maitiro ekuparadzira meseji anotsanangurwa pamusoro anogona kushamisa zvichienzaniswa nemabatiro anoita mutsara weJMS. Mumuenzaniso uyu, mameseji anotumirwa kumutsara achagoverwa zvakaenzana pakati pevatengi vaviri.
Kazhinji, kana isu tichigadzira akawanda maekisheni evatengi, tinoita izvi kungave kugadzirisa mameseji nenzira yakafanana, kana kuwedzera kumhanya kwekuverenga, kana kuwedzera kugadzikana kwemaitiro ekuverenga. Sezvo mutengi mumwe chete muenzaniso unogona kuverenga data kubva pachikamu panguva, izvi zvinowanikwa sei muKafka?
Imwe nzira yekuita izvi ndeye kushandisa mutengi mumwechete muenzaniso kuverenga mameseji ese uye oapfuudza kune tambo dziva. Nepo nzira iyi ichiwedzera kugadzirwa kwemukati, inowedzera kuomarara kwemutengi logic uye haiite chinhu kuwedzera kusimba kweiyo kuverenga system. Kana imwe kopi yemutengi ichidzika nekuda kwekutadza kwemagetsi kana chiitiko chakafanana, ipapo kubvisa kunomira.
Iyo canonical nzira yekugadzirisa dambudziko iri muKafka ndeye kushandisa bΠzvimwe partitions.
Partitioning
Zvikamu ndiyo nzira huru yekufananidza kuverenga uye kuyera musoro kupfuura bandwidth yeimwe broker muenzaniso. Kuti tinzwisise zviri nani izvi, ngatitarisei mamiriro ezvinhu apo pane musoro une zvikamu zviviri uye mutengi mumwe anonyoresa kune iyi nyaya ().
Mufananidzo 3-5. Mumwe mutengi anoverenga kubva kune akawanda partitions
Mune ino mamiriro, mutengi anopihwa kutonga pamusoro peanongedzo anoenderana neboka rayo_id muzvikamu zviviri uye otanga kuverenga mameseji kubva kune ese ari maviri zvikamu.
Kana mutengi wekuwedzera weboka rimwe chete_id awedzerwa kune ino musoro, Kafka inoisa imwe yezvikamu kubva kune yekutanga kusvika kune yechipiri mutengi. Mushure meizvozvo, chiitiko chega chega chemutengi chinoverenga kubva pane chimwe chikamu chemusoro ().
Kuti uve nechokwadi chekuti mameseji anogadziriswa mukufanana mushinda makumi maviri, unoda kanenge makumi maviri zvikamu. Kana pane zvikamu zvishomanana, iwe uchasara nevatengi vasina chekushanda, sezvakatsanangurwa kare mukukurukurirana kwevatengi vakasarudzika.
Mufananidzo 3-6. Vatengi vaviri vari muboka rimwechete revatengi vanoverenga kubva kuzvikamu zvakasiyana
Ichi chirongwa chinoderedza zvakanyanya kuoma kweKafka broker kana ichienzaniswa nekuparadzirwa kwemeseji inodiwa kuchengetedza mutsara weJMS. Pano iwe haufanirwe kunetseka nezve anotevera mapoinzi:
- Ndeupi mutengi anofanirwa kugashira meseji inotevera, zvichibva pakugovewa kwe-round-robin, huwandu hwazvino hweprefetch buffers, kana mameseji apfuura (semapoka emashoko eJMS).
- Ndeapi mameseji anotumirwa kune vatengi uye kana vachifanira kuendeswa zvakare kana vatadza.
Zvese zvinofanirwa kuitwa nemutengesi weKafka kupfuudza mameseji zvakateerana kumutengi kana wekupedzisira avakumbira.
Nekudaro, izvo zvinodikanwa zvekufananidza kuongorora uye kutumirazve mameseji akakundikana hazviendi - mutoro kwavari unongopfuura kubva kune bhuroka kuenda kumutengi. Izvi zvinoreva kuti ivo vanofanirwa kuverengerwa mukodhi yako.
Kutumira mameseji
Ibasa remugadziri weiyo meseji kusarudza kuti ndeipi chikamu chekutumira meseji. Kuti tinzwisise mashandisiro anoita izvi, tinofanira kutanga tafunga kuti chii chaizvo chatiri kutumira.
Ipo muJMS isu tinoshandisa meseji chimiro chine metadata (misoro uye zvivakwa) uye muviri une mubhadharo (mubhadharo), muKafka iwo meseji. pair "key-value". Iyo meseji payload inotumirwa semutengo. Kiyi, kune rumwe rutivi, inonyanya kushandiswa kugovera uye inofanira kunge iine bhizinesi logic chaiyo kiyikuisa mameseji ane hukama muchikamu chimwe chete.
MuChitsauko 2, takakurukura mamiriro ekubheja epamhepo apo zviitiko zvine hukama zvinoda kugadziriswa kuitira nemutengi mumwechete:
- Iyo account yemushandisi yakagadziriswa.
- Mari inoiswa kuaccount.
- Kubheti kunoitwa kunobvisa mari kubva kuaccount.
Kana chiitiko chega chega chiri meseji yakatumirwa kune musoro, saka kiyi yechisikigo ingave iyo account ID.
Kana meseji inotumirwa uchishandisa Kafka Producer API, inopfuudzwa kune yekugovera basa iyo, yakapihwa meseji uye mamiriro azvino eKafka cluster, inodzosera ID yekugovera iyo meseji inofanira kutumirwa. Ichi chiitiko chinoitwa muJava kuburikidza nePartitioner interface.
Iyi interface inoita seizvi:
interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster);
}Iyo Partitioner kuita inoshandisa iyo yakajairika-chinangwa hashing algorithm pamusoro pekiyi kuona kupatsanurwa, kana kutenderera-robin kana pasina kiyi yataurwa. Iyi default kukosha inoshanda zvakanaka muzviitiko zvakawanda. Zvisinei, mune ramangwana iwe uchada kunyora iwe pachako.
Kunyora yako wega partitioning strategy
Ngatitarisei muenzaniso waunoda kutumira metadata pamwe chete nemeseji payload. Iyo payload mumuenzaniso wedu murairo wekuita dhipoziti kuaccount account. Murairo ndechimwe chinhu chatinoda kuvimbiswa kuti hachigadziriswe pakutapurirana uye tinoda kuve nechokwadi chekuti inovimbika chete yekumusoro system inogona kutanga iyo rairo. Muchiitiko ichi, kutumira uye kugashira masisitimu anobvumirana pakushandiswa kwesaini yekusimbisa meseji.
Mune zvakajairika JMS, isu tinongotsanangura "meseji siginecha" chivakwa uye towedzera kune meseji. Nekudaro, Kafka haitipe nzira yekupfuura metadata, kiyi chete uye kukosha.
Sezvo kukosha kuri mubhadharo wekubhadhara kubhengi uyo kuvimbika kwatinoda kuchengetedza, isu hatina sarudzo kunze kwekutsanangura chimiro che data chekushandisa mukiyi. Tichifunga kuti isu tinoda ID yeakaundi yekugovanisa, sezvo mameseji ese ane chekuita neakaunti anofanirwa kugadziriswa zvakarongeka, tichauya neinotevera JSON chimiro:
{
"signature": "541661622185851c248b41bf0cea7ad0",
"accountId": "10007865234"
}Nekuti kukosha kweiyo siginicha kuchasiyana zvichienderana nemubhadharo, iyo yekusagadzika hashing zano rePartitioner interface haizovimbike mameseji ane hukama. Naizvozvo, isu tichada kunyora isu pachedu zano rinozoparura kiyi iyi uye kugovera iyo accountId kukosha.
Kafka inosanganisira macheki ekuona huwori hwemeseji muchitoro uye ine yakazara seti yezvekuchengetedza maficha. Kunyangwe zvakadaro, zvinodikanwa zveindasitiri, senge zviri pamusoro, dzimwe nguva zvinoonekwa.
Mazano emushandisi ekukamura anofanira kuona kuti meseji dzine hukama dzinopera muchikamu chimwe chete. Kunyange izvi zvichiita sezviri nyore, chinodiwa chinogona kuomeswa nekukosha kwekuraira mameseji ane hukama uye kuti yakagadziriswa sei nhamba yezvikamu mumusoro.
Huwandu hwezvikamu mumusoro hunogona kuchinja nekufamba kwenguva, sezvo hunogona kuwedzerwa kana traffic ikapfuura zvinotarisirwa. Nekudaro, makiyi emeseji anogona kubatanidzwa nechikamu chavakatumirwa pakutanga, zvichireva kuti chidimbu chenyika chinofanira kugovaniswa pakati pezviitiko zvevagadziri.
Chimwe chinhu chekufunga nezvacho kugovaniswa kwemeseji kumativi ese. Kazhinji, makiyi haana kugovaniswa zvakaenzana pameseji, uye hashi mabasa haavimbise kugovaniswa kwakanaka kwemameseji kune diki seti yemakiyi.
Zvakakosha kuziva kuti zvisinei iwe unosarudza kupatsanura mameseji, iyo inoparadzanisa pachayo ingangoda kushandiswa zvakare.
Funga nezvechinodikanwa chekudzokorora data pakati pemasumbu eKafka munzvimbo dzakasiyana dzenzvimbo. Nechinangwa ichi, Kafka inouya neine command line tool inonzi MirrorMaker, iyo inoshandiswa kuverenga mameseji kubva kune rimwe sumbu nekuaendesa kune rimwe.
MirrorMaker inofanirwa kunzwisisa makiyi emusoro wakadzokororwa kuitira kuchengetedza kurongeka pakati pemeseji kana ichidzokorora pakati pemapoka, sezvo huwandu hwezvikamu zvemusoro wenyaya yacho hunogona kunge husina kufanana mumasumbu maviri.
Tsika dzekuganhura nzira hadzina kushoma, sezvo default hashing kana kutenderera robin inoshanda nemazvo mumamiriro mazhinji. Nekudaro, kana iwe uchida vimbiso dzakasimba dzekuodha kana kuda kuburitsa metadata kubva pamibhadharo, ipapo kupatsanura chimwe chinhu chaunofanira kunyatso kutarisisa.
Iyo scalability uye mabhenefiti ekuita kweKafka anobva pakuchinja mamwe mabasa echinyakare broker kumutengi. Muchiitiko ichi, sarudzo inoitwa yekugovera zvingangove zvine hukama pakati pevatengi vakati wandei vanoshanda zvakafanana.
JMS broker inodawo kubata nezvinodiwa. Sezvineiwo, iyo nzira yekutumira mameseji ane hukama kumutengi mumwechete, inoshandiswa kuburikidza neJMS Message Mapoka (mutsauko pane inonamira mutoro wekuenzanisa (SLB) zano), inodawo kuti anotumira amake mameseji ane hukama. Panyaya yeJMS, mutengesi ane basa rekutumira boka iri remashoko ane hukama kune mumwe mutengi kubva kune akawanda, uye kuendesa muridzi weboka kana mutengi akadonha.
Zvibvumirano zvevagadziri
Kupatsanura handicho chinhu chega chekufunga kana uchitumira meseji. Ngatitarisei nzira dzekutumira () dzeClass Producer muJava API:
Future < RecordMetadata > send(ProducerRecord < K, V > record);
Future < RecordMetadata > send(ProducerRecord < K, V > record, Callback callback);Zvinofanira kuonekwa pakarepo kuti nzira mbiri dzinodzoka Future, iyo inoratidza kuti basa rekutumira harina kuitwa pakarepo. Mhedzisiro yacho ndeyekuti meseji (ProducerRecord) inonyorerwa kune inotumira buffer kune yega yega inoshanda partition uye inotumirwa kune bhuroka senge tambo yekumashure muKafka client library. Nepo izvi zvichiita kuti zvinhu zvikurumidze, zvinoreva kuti isina ruzivo application inogona kurasikirwa nemeseji kana maitiro ayo akamiswa.
Senguva dzose, kune nzira yekuita kuti kutumira kwekushanda kuwedzere kuvimbika pamutengo wekuita. Saizi yeiyi buffer inogona kusetwa ku0, uye tambo yekunyorera yekutumira ichamanikidzwa kumirira kudzamara kutumira meseji kune bhuroka kwapera, sezvinotevera:
RecordMetadata metadata = producer.send(record).get();Zvimwe pamusoro pekuverenga mameseji
Kuverenga mameseji kune zvimwe zvinonetsa zvinoda kufungidzira. Kusiyana neJMS API, iyo inogona kumhanya muteereri wemeseji mukupindura meseji, iyo Consumer Kafka chete sarudzo. Ngatinyatsoongororai nzira sarudzo()inoshandiswa kune chinangwa ichi:
ConsumerRecords < K, V > poll(long timeout);Iko kukosha kwekudzoka kweiyo nzira igadziriso yemudziyo ine zvinhu zvakawanda rekodhi yevatengi kubva kune zvingangoita zvikamu zvakawanda. rekodhi yevatengi iyo pachayo chinhu chekubata chekiyi-kukosha pair ine yakabatana metadata, senge kupatsanurwa kwainotorwa.
Sezvakakurukurwa muChitsauko 2, tinofanira kuyeuka zvinoitika kumameseji mushure mekunge abudirira kana kusabudirira, semuenzaniso, kana mutengi asingakwanisi kugadzirisa meseji kana kuti ikabvisa. MuJMS, izvi zvakabatwa nenzira yekubvuma. Iyo bhuroka inodzima meseji yakagadziridzwa zvakabudirira, kana kuendesa zvakare mbishi kana fake meseji (uchifungidzira kuti kutengeserana kwakashandiswa).
Kafka inoshanda zvakasiyana chaizvo. Mameseji haabviswe mubhuroka mushure mekuongorora, uye zvinoitika pakukundikana ibasa reiyo kodhi yekuongorora pachayo.
Sezvatareva, boka revatengi rinosanganiswa neiyo offset murogi. Nzvimbo yelogi ine chekuita neiyi offset inoenderana neinotevera meseji ichapihwa mukupindura sarudzo(). Iyo poindi yenguva iyo iyi yekubvisa inowedzera inosarudza kuverenga.
Kudzokera kumuenzaniso wekuverenga wakambokurukurwa, kugadzirisa meseji kunosanganisira matanho matatu:
- Tora meseji yekuverenga.
- Gadzirisa meseji.
- Simbisa meseji.
Mutengi weKafka anouya nesarudzo yekugadzirisa enable.auto.commit. Iyi inogara ichishandiswa default set, seyakajairwa nemaseting ane izwi rekuti "auto".
Pamberi peKafka 0.10, mutengi anoshandisa iyi sarudzo anotumira iyo yekubvisa meseji yekupedzisira kuverengwa pane inotevera runhare. sarudzo() mushure mekugadzirisa. Izvi zvaireva kuti chero mameseji anga atotorwa anogona kudzokororwa kana mutengi anga atoagadzirisa asi akaparadzwa zvisingatarisirwi asati afona. sarudzo(). Sezvo mutengesi asingachengete chero mamiriro ekuti kangani meseji yakaverengwa, mutengi anotevera anotora iyo meseji haazoziva kuti pane chakaipa chakaitika. Maitiro aya aive emanyepo-transaction. Iyo yekubvisa yaingoitwa chete kana meseji yakagadziriswa zvinobudirira, asi kana mutengi akabvisa, mutengesi anotumira meseji imwechete zvakare kune mumwe mutengi. Maitiro aya aienderana nevimbiso yekutumira meseji "kamwechete".
MuKafka 0.10, kodhi yemutengi yakashandurwa kuitira kuti kuzvipira kukonzereswe nguva nenguva neraibhurari yemutengi, sekugadziriswa. auto.commit.interval.ms. Maitiro aya ari pakati peJMS AUTO_ACKNOWLEDGE neDUPS_OK_ACKNOWLEDGE modhi. Paunenge uchishandisa autocommit, mameseji anogona kuzvipira zvisinei nekuti akagadziriswa - izvi zvinogona kuitika kana mutengi anononoka. Kana mutengi akabvisa, mameseji aizotorwa nemutengi anotevera, achitangira panzvimbo yakazvipira, izvo zvinogona kuguma meseji yarasika. Muchiitiko ichi, Kafka haana kurasikirwa nemashoko, kodhi yekuverenga haina kungoigadzirisa.
Iyi modhi ine vimbiso yakafanana neyeiyo vhezheni 0.9: mameseji anogona kugadziriswa, asi kana ikatadza, iyo yekubvisa inogona kusaitwa, zvichikonzera kuti kutumirwa kupe nekaviri. Iwo mameseji akawanda aunotora paunenge uchiita sarudzo(), zvakanyanya dambudziko iri.
Sezvakakurukurwa mu "Kuverenga Mharidzo kubva kuMutsetse" papeji 21, hapana chinhu chakadai sekutumira meseji kamwe chete muhurongwa hwekutumira mameseji kana nzira dzekutadza dzichitariswa.
MuKafka, kune nzira mbiri dzekuita (kuita) chinobvisa (offset): otomatiki uye nemaoko. Mune ese ari maviri mameseji, mameseji anogona kugadziriswa kakawanda kana meseji yakagadziriswa asi yakundikana isati yaitwa. Iwe unogona zvakare kusarudza kusagadzirisa meseji zvachose kana kuzvipira kwakaitika kumashure uye kodhi yako yakapedzwa isati yagadziriswa (pamwe muKafka 0.9 uye yapfuura).
Iwe unogona kudzora manyorero ekubvisa kuita maitiro muKafka mutengi API nekuisa iyo parameter enable.auto.commit kunyepa uye nekujekesa imwe yenzira dzinotevera:
void commitSync();
void commitAsync();Kana iwe uchida kugadzirisa meseji "kamwe chete", iwe unofanirwa kuita iyo offset nemawoko nayo commitSync()nekuita uyu murairo pakarepo mushure mekugadzirisa mameseji.
Nzira idzi hadzitenderi mameseji kubvumwa asati agadziriswa, asi hapana chaanoita kubvisa kunonoka kwekugadzirisa uku achipa kutaridzika kwekutengeserana. Iko hakuna kutengeserana muKafka. Mutengi haakwanise kuita zvinotevera:
- Gadzirisa meseji yenhema. Vatengi pachavo vanofanirwa kubata kusarudzika kunokonzerwa nedambudziko rekubhadhara uye kubuda kwemashure, sezvo vasingakwanise kuvimba nebroker kuti vatumirezve mameseji.
- Tumira mameseji kune akawanda misoro mune imwe atomu kushanda. Sezvatichaona munguva pfupi iri kutevera, kutonga pamusoro pemisoro yakasiyana uye zvikamu zvinogona kugara pamichina yakasiyana muKafka cluster iyo isingabatanidzi kutengeserana kana yatumirwa. Panguva yekunyora uku, rimwe basa rakaitwa kuti izvi zviitike neKIP-98.
- Batanidza kuverenga meseji kubva pane imwe musoro nekutumira imwe meseji kune imwe musoro. Zvekare, chivakwa cheKafka chinoenderana nemichina yakawanda yakazvimirira inomhanya sebhazi rimwe chete uye hapana kuedza kunoitwa kuvanza izvi. Semuenzaniso, hapana zvikamu zveAPI zvingakubvumidza kuti ubatanidze mutengi ΠΈ Mhizha mukutengeserana. MuJMS, izvi zvinopihwa nechinhu chiitikokubva pazvakasikwa MessageProducers ΠΈ MessageConsumers.
Kana isu tisingakwanise kuvimba nekutengeserana, tingapa sei semantics padyo neiyo inopihwa nemaitiro echinyakare mameseji masisitimu?
Kana paine mukana wekuti kuderedzwa kwemutengi kunogona kuwedzera meseji isati yagadziriswa, senge panguva yekuparara kwevatengi, saka mutengi haana nzira yekuziva kana boka revatengi vake rakapotsa meseji payakapihwa chikamu. Saka rimwe zano nderekudzosera kumashure kune yakare chinzvimbo. Iyo Kafka mutengi API inopa nzira dzinotevera dzeizvi:
void seek(TopicPartition partition, long offset);
void seekToBeginning(Collection < TopicPartition > partitions);
Method kutsvaga () inogona kushandiswa nenzira
offsetsForTimes(Mepu timestampsToSearch) kudzosera kumashure kune imwe nzvimbo pane imwe nguva yakapfuura.
Zviripachena, kushandisa nzira iyi zvinoreva kuti zvine mukana wekuti mamwe mameseji akambogadziriswa achaverengwa nekugadziriswa zvakare. Kuti tidzivise izvi, tinogona kushandisa kuverenga kusingafungi, sekutsanangurwa kwazvino muChitsauko 4, kuti titarise mameseji aimboonekwa uye kubvisa zvakapetwa.
Neimwe nzira, kodhi yako yemutengi inogona kuchengetwa yakapusa, chero bedzi kurasikirwa kwemeseji kana kudzokorora kuchigamuchirwa. Kana tichifunga nezvemashandisirwo ayo Kafka anowanzo shandiswa, sekubata log zviitiko, metrics, kubaya tracking, zvichingodaro, tinonzwisisa kuti kurasikirwa kwemameseji ega ega hakugone kuve nekukanganisa kukuru pamaapplication akatenderedza. Mumamiriro ezvinhu akadaro, iyo default values ββinogamuchirwa zvakakwana. Kune rimwe divi, kana chikumbiro chako chichida kutumira mubhadharo, unofanirwa kunyatso tarisira meseji yega yega. Zvose zvinoenderana nemamiriro ezvinhu.
Kuongorora kwemunhu kunoratidza kuti sezvo kusimba kwemameseji kunowedzera, kukosha kwemeseji yega yega kunodzikira. Mharidzo huru dzinowanzove dzakakosha kana dzichionekwa muchimiro chakaunganidzwa.
Kuwanikwa Kwepamusoro
Maitiro eKafka ekuwanikwa kwepamusoro akasiyana zvakanyanya nemaitiro eActiveMQ. Kafka yakagadzirirwa kutenderedza masango-masumbu uko ese mabroker zviitiko zvinogashira nekugovera mameseji panguva imwe chete.
Kafka cluster ine akawanda mabroker zviitiko anomhanya pamaseva akasiyana. Kafka yakagadzirwa kuti imhanye pane yakajairwa standalone Hardware, uko imwe neimwe node ine yayo yakatsaurwa yekuchengetedza. Iko kushandiswa kwetiweki yakasungirirwa kuchengetedza (SAN) haikurudzirwe nekuti akawanda compute node anogona kukwikwidza kwenguva.Π«e kuchengetedza nguva uye kugadzira kusawirirana.
Kafka ari nguva dzose system. Vazhinji vashandisi veKafka vakakura havana kumbovhara masumbu avo uye software inogara ichivandudza neyakateedzana restart. Izvi zvinowanikwa nekuvimbisa kuenderana neyakapfuura vhezheni yemeseji uye kudyidzana pakati pevatengesi.
Mabroker akabatana kune server cluster , iyo inoshanda seyekugadzirisa data registry uye inoshandiswa kurongedza mabasa emutengesi wega wega. ZooKeeper pachayo ndiyo yakagoverwa sisitimu inopa kuwanikwa kwepamusoro kuburikidza nekudzokorora kweruzivo nekumisikidza quorum.
Mune iyo base kesi, musoro unogadzirwa muKafka cluster ine zvinotevera zvivakwa:
- Nhamba yezvikamu. Sezvakakurukurwa kare, ukoshi chaihwo hwakashandiswa pano hunotsamira pamwero unodiwa wokurava kwakafanana.
- Iyo replication factor (factor) inotara kuti mangani mabroker zviitiko musumbu anofanira kunge aine matanda echikamu ichi.
Ichishandisa ZooKeepers yekubatanidza, Kafka inoedza kugovera zvakaringana zvikamu zvitsva pakati pemabhuroka musumbu. Izvi zvinoitwa nechiitiko chimwe chete chinoita seMutungamiriri.
Panguva yekumhanya pachikamu chemusoro wega wega Mutungamiriri kupa mabasa kune bhuroka mutungamiri (mutungamiri, tenzi, muparidzi) uye vateveri (vateveri, varanda, vari pasi). Iye broker, achiita semutungamiri wechikamu ichi, ane basa rekugamuchira mameseji ese anotumirwa kwaari nevagadziri uye nekugovera mameseji kune vatengi. Kana mameseji achitumirwa kune chikamu chemusoro, anodzokororwa kune ese mabroker node anoita sevateveri vechikamu ichocho. Imwe neimwe node ine matanda echikamu inonzi replica. Mutengesi anogona kuita semutungamiri wezvimwe zvikamu uye semuteveri wevamwe.
Muteveri ane mameseji ese anobatwa nemutungamiri anonzi synchronized replica (chifananidzo chiri muchibvumirano, in-sync replica). Kana bhuroka anoshanda semutungamiri wechikamu akadzikira, chero broker ari kufambidzana kana kuwiriraniswa kune icho chikamu anogona kutora chinzvimbo chemutungamiri. Iro dhizaini inogoneka zvisingaite.
Chikamu chekugadziriswa kwemugadziri ndiyo parameter acks, iyo inosarudza kuti zvingani zvidhori zvinofanirwa kubvuma (kubvuma) kugamuchira meseji fomu rekunyorera risati raenderera nekutumira: 0, 1, kana zvese. Kana yaiswa zvose, zvino kana meseji yagamuchirwa, mutungamiri anotumira simbiso kumashure kumugadziri kana angogamuchira zvisimbiso (kubvuma) kwerekodhi kubva kune akati wandei cues (kusanganisira pachayo) inotsanangurwa nemusoro wenyaya. min.insync.replicas (default 1). Kana iyo meseji isingagone kudzokororwa zvakabudirira, ipapo mugadziri anokanda application kunze (NotEnoughReplicas kana NotEnoughReplicasAfterAppend).
Kugadziriswa kwakajairwa kunogadzira musoro une replication factor ye3 (mutungamiri 1, vateveri 2 pachikamu) uye parameter. min.insync.replicas yakaiswa ku 2. Muchiitiko ichi, cluster ichabvumira mumwe wevatengesi vanotungamirira chikamu chehurukuro kuti vaende pasi pasina kukanganisa kushandiswa kwevatengi.
Izvi zvinotidzosera kune yakatojaira kutengeserana pakati pekuita uye kuvimbika. Kudzokorora kunoitika nekubhadharira imwe nguva yekumirira yekusimbisa (kubvuma) kubva kuvateveri. Kunyangwe, nekuti inomhanya mukuwirirana, kudzokorora kune angangoita matatu node ine mashandiro akafanana maviri (kuregeredza kuwedzera kwekushandisa network bandwidth).
Nekushandisa chirongwa ichi chekudzokorora, Kafka anodzivirira zvine hungwaru kukosha kwekunyora meseji yega yega ku diski nekushanda. sync(). Meseji yega yega inotumirwa nemugadziri inonyorerwa kugogi rekugovera, asi sezvakakurukurwa muChitsauko 2, kunyora kufaira kunotanga kuitwa mubhafa yehurongwa hwekushandisa. Kana meseji iyi ikadzokororwa kune imwe Kafka muenzaniso uye iri mundangariro dzayo, kurasikirwa nemutungamiri hazvirevi kuti meseji yacho yakarasika - inogona kutorwa neyakafanana.
Kuramba kuita oparesheni sync() zvinoreva kuti Kafka inogona kugamuchira mameseji nekukurumidza sezvaanogona kuinyorera kundangariro. Sezvineiwo, iyo yakareba iwe yaunokwanisa kudzivirira kumhanyisa ndangariro kudhisiki, zviri nani. Nechikonzero ichi, hazvina kujairika kuti Kafka vatengesi vapiwe 64 GB kana kupfuura yendangariro. Uku kushandiswa kwendangariro kunoreva kuti imwe Kafka muenzaniso inogona kumhanya zviri nyore nekumhanya zviuru zvenguva nekukurumidza kupfuura yechinyakare meseji broker.
Kafka inogonawo kugadzirwa kuti ishandise kushanda sync() kune meseji mapakeji. Sezvo zvese zviri muKafka zvakatarisana nemapakeji, zvinonyatso shanda nemakesi akawanda ekushandisa uye chishandiso chinobatsira kune vashandisi vanoda vimbiso dzakasimba kwazvo. Kuwanda kwekuita kwakachena kweKafka kunobva kune mameseji anotumirwa kune broker semapaketi uye kuti mameseji aya anoverengwa kubva kumubhuroka mumabhuroki anoteedzana uchishandisa. mashandiro (mashandiro panguva iyo basa rekukopa data kubva kune imwe nzvimbo yekurangarira kuenda kune imwe harina kuitwa). Iyo yekupedzisira ndeyekuita kukuru uye kuwana zviwanikwa uye zvinogoneka chete kuburikidza nekushandiswa kweiyo yepasi regi data chimiro chinotsanangura chirongwa chekuparadzanisa.
Kuita kurinani kwakanyanya kunogoneka muchikwata cheKafka pane neKafka bhuroka imwechete, nekuti zvikamu zvemusoro zvinogona kukwira mumichina yakawanda yakaparadzana.
Migumisiro
Muchitsauko chino, takatarisa kuti Kafka architecture inofungidzira sei hukama pakati pevatengi nemabhuroka kuti vape pombi yakasimba yekutumira mameseji, ine kubuda kwakapetwa kakawanda kupfuura kweakajairika meseji broker. Takakurukura nezvekushanda kwainoshandisa kuita izvi uye takatarisa muchidimbu mavakirwo ezvishandiso zvinopa kuita uku. Muchitsauko chinotevera, tichatarisa matambudziko akajairika ekutumira mameseji anodiwa kugadzirisa uye kukurukura mazano ekugadzirisa nawo. Tichapedzisa chitsauko nekutsanangura matauriro e tekinoroji dzekutumira mameseji zvakazara kuti utarise kukodzera kwavo pamakesi ako ekushandisa.
Chikamu chakamboshandurwa:
Shanduro yaitwa:
Zvichaenderera mberiβ¦
Vashandisi vakanyoresa chete ndivo vanogona kutora chikamu muongororo. , Munogamuchirwa.
Kafka inoshandiswa musangano rako here?
-
kuti
-
kwete
-
Yakamboshandiswa, ikozvino kwete
-
Tinoronga kushandisa
38 vashandisi vakavhota. 8 vashandisi vakaramba.
Source: www.habr.com