Musangano unotevera weHighLoad++ uchaitwa muna Kubvumbi 6 na7, 2020 muSt.
Details uye matikiti . HighLoad ++ Siberia 2019. Horo "Krasnoyarsk". Chikumi 25, 12:00. Izvi uye .
Zvinoitika kuti zvinodiwa zvinoshanda zvinopesana nedzidziso, uko zvinhu zvakakosha zvechigadzirwa chekutengesa hazvifungirwe. Hurukuro iyi inopa maitiro ekusarudza nekubatanidza nzira dzakasiyana dzekugadzira Causal kusawirirana zvikamu zvichibva patsvakiridzo yedzidzo zvichienderana nezvinodiwa zvechigadzirwa chekutengesa. Vateereri vanozodzidza nezve nzira dziripo dzedzidziso dzewachi dzine musoro, kutsamira kwekutevera, kuchengetedzeka kwehurongwa, kuwiriranisa wachi, uye nei MongoDB yakagadzikana pane dzimwe mhinduro.
Mikhail Tyulenev (zvino anonzi MT): - Ini ndichataura nezve Causal consistency - ichi chinhu chatakashanda paMongoDB. Ini ndinoshanda muboka rezvirongwa zvakagoverwa, takazviita makore maviri apfuura.
Mukuita izvi, ndaifanira kujairana netsvakiridzo yakawanda yedzidzo, nekuti chimiro ichi chakadzidzwa chaizvo. Zvakazoitika kuti hapana kana chinyorwa chinoenderana nezvinodiwa mudura rekugadzira nekuda kwezvinodiwa chaizvo izvo zvinogona kunge zviripo mune chero chigadzirwa chekugadzira.
Ini ndichataura pamusoro pekuti isu, sevatengi veOngororo yezvidzidzo, tinogadzirira sei chimwe chinhu kubva mairi chatinogona kuzopa kune vashandisi vedu sedhishi rakagadzirwa-rakakwana uye rakachengeteka kushandisa.
Causal kuenderana. Ngatitsanangurei pfungwa
Kutanga, ini ndinoda kutaura mune zvakajairika kuti chii chinonzi Causal consistency. Kune mavara maviri - Leonard naPenny (TV yakatevedzana "The Big Bang Theory"):
Ngatitii Penny ari kuEurope uye Leonard anoda kumuitira mabiko asingashamisiri. Uye haafunge chero chinhu chiri nani pane kumurasa kubva kushamwari yake, achitumira shamwari dzake dzese update pamusoro pekudya: "Ngatifadze Penny!" (ari muEurope, paanovata, haaoni zvose izvi uye haazvioni, nokuti haapo). Pakupedzisira, anodzima positi iyi, oidzima kubva kuKudyisa uye kudzoreredza kupinda kuitira kuti asaone chero chinhu uye pasave nekunyomba.
Izvi zvese zvakanaka uye zvakanaka, asi ngatifungei kuti sisitimu inogovaniswa uye zvinhu zvakaenda nepamwe. Zvinogona, semuenzaniso, kuti kurambidzwa kwePenny kuwana kwakaitika mushure mekuonekwa uku, kana zviitiko zvisingaenderane nechikonzero nemhedzisiro. Chaizvoizvo, uyu muenzaniso weapo Causal kusawirirana kunodiwa kuitira kuti uite bhizinesi basa (munyaya iyi).
Muchokwadi, izvi zvinhu zvisiri-zvidiki zve database - vashoma vanhu vanovatsigira. Ngatiendei kune mienzaniso.
Consistency Models
Chii chaizvo chinonzi consistency modhi mumadatabase? Aya ndiwo mamwe ezvivimbiso zvinopihwa nehurongwa hwekugovera nezve data iro mutengi anogona kugamuchira uye mukutevedzana kupi.
Mumusimboti, ese mamodhi anoenderana anouya pasi kuti yakafanana sei system yakagoverwa kune sisitimu inomhanya, semuenzaniso, pane imwe node pane laptop. Uye iyi ndiyo yakafanana sisitimu inomhanya pazviuru zvegeo-yakagoverwa "Nodes" kune laptop, umo zvese izvi zvivakwa zvinoitwa otomatiki musimboti.
Naizvozvo, mamodhi anoenderana anoshandiswa chete kune akagoverwa masisitimu. Masisitimu ese aimbovepo uye akashanda pachiyero chakati chechetere haana kusangana nematambudziko akadai. Paive neBuffer Cache imwe, uye zvese zvaigara zvichiverengwa kubva pazviri.
Muenzaniso Wakasimba
Chaizvoizvo, iyo yekutanga modhi yakasimba (kana iyo yekusimuka yekugona mutsara, sezvainowanzo zivikanwa). Uyu ndiwo muenzaniso wekuenderana unova nechokwadi chekuti shanduko yese, kana yangosimbiswa kuti yaitika, inoonekwa kune vese vashandisi vehurongwa.
Izvi zvinogadzira kurongeka kwepasirese kwezviitiko zvese zviri mudhatabhesi. Ichi chivakwa chakasimba chekuenderana, uye kazhinji chinodhura zvakanyanya. Zvisinei, inotsigirwa zvikuru. Inongodhura zvakanyanya uye inononoka - haingowanzo kushandiswa. Uku kunonzi kusimuka kugona.
Pane imwe, yakasimba pfuma inotsigirwa muSpanner - inonzi External Consistency. Tichazotaura nezvazvo gare gare.
Zvinokonzera
Inotevera iCausal, ndizvo chaizvo zvandaitaura nezvazvo. Kune akati wandei mamwe madiki-mazinga pakati Akasimba uye Causal andisingataure nezvawo, asi iwo ese anowira pasi kuCausal. Uyu muenzaniso wakakosha nekuti ndiyo yakasimba pane ese mamodheru, kusimba kwakasimba muhupo hwetiweki kana zvikamu.
Zvikonzero ndeye mamiriro ezvinhu umo zviitiko zvakabatanidzwa nechikonzero-uye-mhedzisiro hukama. Kazhinji ivo vanoonekwa se Verenga yako pane kodzero kubva pakuona kwemutengi. Kana mutengi akacherechedza humwe hunhu, haakwanise kuona hunhu hwaive munguva yakapfuura. Ave kutotanga kuona prefix kuverenga. Zvose zvinouya kuchinhu chimwe chete.
Zvikonzero semuenzaniso wekuenderana ndeyekurongeka kwechikamu chezviitiko pane sevha, umo zviitiko kubva kune vese vatengi zvinocherechedzwa mukutevedzana kwakafanana. Muchiitiko ichi, Leonard naPenny.
Sezvineiwo
Yechitatu modhi ndeye Eventual Consistency. Izvi ndizvo zvinotsigirwa nemasisitimu ese akagoverwa, iyo diki modhi inoita zvine musoro zvachose. Zvinoreva zvinotevera: kana isu tine shanduko mune data, pane imwe nguva ivo vanofanana.
Panguva yakadai haatauri chero chinhu, kana zvisina kudaro aizoshanduka kuva External Consistency - inenge iri nyaya yakasiyana zvachose. Zvakangodaro, iyi ndiyo yakakurumbira modhi, yakajairika. Nekutadza, vese vashandisi veakagoverwa masisitimu vanoshandisa Eventual Consistency.
Ndinoda kupa mimwe mienzaniso yekuenzanisa:
Miseve iyi inorevei?
- Kunonoka. Sezvo kusimba kwesimba kunowedzera, inova yakakura nekuda kwezvikonzero zviri pachena: iwe unofanirwa kuita mamwe marekodhi, kuwana simbiso kubva kune ese mauto uye node dzinotora chikamu musumbu kuti data yatovepo. Saizvozvo, Pakupedzisira Consistency ine mhinduro inokurumidza, nekuti ipapo, sekutonga, iwe unogona kutozviisa mundangariro uye izvi zvichave zvakakwana.
- Kuwanika. Kana isu tikanzwisisa izvi sekugona kwehurongwa hwekupindura pamberi pe network breaks, partitions, kana imwe mhando yekutadza, kukanganisa kushivirira kunowedzera sezvo kusachinja-chinja modhi kunoderera, sezvo zvakatikwanira isu kuti muenzi mumwe anogara uye panguva imwe chete. nguva inogadzira imwe data. Pakupedzisira Consistency haivimbisi chero chinhu nezve data zvachose - inogona kuve chero chinhu.
- Anomalies. Panguva imwecheteyo, hongu, nhamba yeanomalies inowedzera. MuKusimba Kwakasimba ivo vanenge vasingafanire kunge varipo zvachose, asi muEkupedzisira Consistency vanogona kuve chero chinhu. Mubvunzo unomuka: sei vanhu vachisarudza Eventual Consistency kana iine anomalies? Mhinduro ndeyokuti Eventual Consistency modhi dzinoshanda uye anomalies aripo, semuenzaniso, munguva pfupi; zvinokwanisika kushandisa wizard kuverenga uye yakawanda kana zvishoma kuverenga inowirirana data; Zvinowanzogoneka kushandisa mienzaniso yakasimba yekuenderana. Mukuita izvi zvinoshanda, uye kazhinji nhamba yeanomalies inogumira munguva.
CAP theorem
Paunoona mazwi anoenderana, kuwanikwa - chii chinouya mupfungwa dzako? Ndizvozvo - CAP theorem! Iye zvino ndinoda kubvisa nhema ... Handisi ini - ndiMartin Kleppmann, akanyora chinyorwa chinoshamisa, bhuku rakanaka.
Iyo CAP theorem musimboti wakagadzirwa muma2000s kuti Consistency, Kuwanikwa, Zvikamu: tora chero maviri, uye haugone kusarudza matatu. Yaiva imwe nheyo. Yakaratidzwa se theorem makore mashoma gare gare naGilbert naLynch. Zvino izvi zvakatanga kushandiswa semantra - masisitimu akatanga kukamurwa kuva CA, CP, AP uye zvichingodaro.
Iyi theorem yakanyatso kupupurirwa kune zvinotevera nyaya ... Chekutanga, Kuwanikwa kwaionekwa kusiri sekukosha kunoenderera kubva kune zero kusvika kumazana (0 - iyo system "yakafa", 100 - inopindura nekukurumidza; isu tajaira kufunga nezvazvo nenzira iyoyo) , asi sechinhu che algorithm , iyo inovimbisa kuti pakuita kwayo kwese inodzorera data.
Hapana kana izwi nezve nguva yekupindura zvachose! Pane algorithm inodzosa data mushure memakore zana - inoshamisa inowanika algorithm, inova chikamu cheCAP theorem.
Chechipiri: theorem yakaratidza shanduko muhunhu hwekiyi imwechete, zvisinei nekuti shanduko idzi dzinogoneka. Izvi zvinoreva kuti muchokwadi ivo havasati vashandiswa, nekuti iwo mamodheru akasiyana Ekupedzisira Consistency, Strong Consistency (pamwe).
Zvese izvi ndezvei? Zvakare, iyo CAP theorem mune chaizvo iyo fomu iyo yakaratidzwa haishande uye haiwanzo kushandiswa. Muchimiro che theoretical, neimwe nzira inomisa zvese. Zvinobuda imwe nheyo iyo intuitively yakarurama, asi kazhinji haina kuratidzwa.
Causal consistency ndiyo yakasimba muenzaniso
Zviri kuitika izvozvi ndezvekuti iwe unogona kuwana ese matatu zvinhu: Consistency, Kuwanikwa uchishandisa Partitions. Kunyanya, Causal consistency ndiyo yakasimba kuenderana modhi, iyo ichiri kushanda pamberi peZvikamu (kuputsa mune network). Ndosaka ichifadza zvikuru, ndosaka takachitora.
Chekutanga, inorerutsa basa revagadziri vekushandisa. Kunyanya, kuvapo kwerutsigiro rwakakura kubva kune sevha: kana zvinyorwa zvese zvinoitika mukati memumwe mutengi zvinovimbiswa kusvika mukutevedzana kwakafanana pane mumwe mutengi. Chechipiri, inoshingirira zvikamu.
MongoDB Yemukati Kitchen
Tichiyeuka kuti kwave kudya kwemasikati, tinotamira kukicheni. Ini ndichakuudza nezve system modhi, iyo, iyo MongoDB ndeye avo vari kunzwa nezve database yakadai kekutanga.
MongoDB (inozonzi "MongoDB") igadziriro yakagoverwa inotsigira kuyera kwakatwasuka, ndiko kuti, sharding; uye mukati meshadhi yega yega inotsigirawo data redundancy, ndiko kuti, kudzokorora.
Sharding muMongoDB (kwete dhatabhesi rehukama) inoita otomatiki kuenzanisa, ndiko kuti, yega yega yezvinyorwa (kana "tafura" maererano nehukama data) inokamurwa kuita zvidimbu, uye sevha inozvifambisa pakati pemashadhi.
Iyo Query Router, iyo inogovera zvikumbiro, kune mutengi imwe mutengi iyo inoshanda kuburikidza nayo. Inotoziva kuti kupi uye ndeipi data iripo uye inotungamira zvese zvikumbiro kune chaiyo shard.
Imwe pfungwa yakakosha: MongoDB ishe imwe chete. Pane Puraimari imwe - inogona kutora marekodhi anotsigira makiyi ayo anayo. Iwe haugone kuita Multi-master kunyora.
Isu takaburitsa 4.2 - zvinhu zvitsva zvinonakidza zvakaonekwa ipapo. Kunyanya, vakaisa Lucene - kutsvaga - iyo inogoneka java yakananga muMongo, uye ipapo zvakave zvichiita kutsvaga kuburikidza neLucene, zvakafanana neElastica.
Uye vakagadzira chigadzirwa chitsva - Charts, inowanikwawo paAtlas (Mongo's yega Cloud). Ivo vane Yemahara Tier - unogona kutamba uchitenderedza nayo. Ini ndaifarira maChati - data visualization, intuitive kwazvo.
Ingredients Causal kuenderana
Ndakaverenga nezve 230 zvinyorwa zvakaburitswa pamusoro penyaya iyi - kubva kuna Leslie Lampert. Zvino kubva mundangariro dzangu ndichasvitsa kwamuri zvimwe zvikamu zvezvinhu izvi.
Zvose zvakatanga nechinyorwa chaLeslie Lampert, chakanyorwa muma1970. Sezvauri kuona, kumwe kutsvagisa nezvenyaya iyi kuchiri kuenderera. Ikozvino Causal consistency iri kusangana nekufarira maererano nekuvandudzwa kwemasystem akagoverwa.
Zvibvumirano
Ndezvipi zvirambidzo zviripo? Iyi ndiyo imwe yemapoinzi makuru, nekuti zvirambidzo zvinoiswa nehurongwa hwekugadzira zvakasiyana kwazvo nezvirambidzo zviripo muzvinyorwa zvedzidzo. Vanowanzova vakanyatsogadzirwa.
- Chekutanga, "MongoDB" ishe mumwechete, sezvandambotaura (izvi zvinorerutsa zvakanyanya).
- Isu tinotenda kuti iyo system inofanirwa kutsigira nezve 10 zviuru shards. Hatigone kuita chero sarudzo dzezvivakwa zvinonyatso dzikamisa kukosha uku.
- Tine gore, asi isu tinofungidzira kuti munhu anofanirwa kunge achiri nemukana kana akarodha mabhinari, anomhanyisa palaptop yake, uye zvese zvinoshanda zvakanaka.
- Isu tinofungidzira chimwe chinhu icho Tsvagurudzo isingawanzo kufungidzira: vatengi vekunze vanogona kuita chero chavanoda. MongoDB ndiyo yakavhurika sosi. Saizvozvo, vatengi vanogona kuve vakangwara uye vakatsamwa - vanogona kuda kuputsa zvese. Isu tinofungidzira kuti Byzantine Feilors inogona kutanga.
- Kune vatengi vekunze vari kunze kweiyo perimeter, pane yakakosha kudzikisira: kana chimiro ichi chakaremara, saka hapana kuderedzwa kwekuita kunofanirwa kucherechedzwa.
- Imwe pfungwa inowanzo kupesana nedzidzo: kuenderana kweshanduro dzakapfuura uye dzeramangwana. Vatyairi vekare vanofanirwa kutsigira zvitsva, uye dhatabhesi inofanirwa kutsigira vatyairi vekare.
Kazhinji, izvi zvose zvinoisa zvirambidzo.
Causal consistency components
Ini zvino ndichataura nezve zvimwe zvezvikamu. Kana isu tikafunga Causal kuwirirana mune zvakajairika, isu tinogona kusarudza zvidhinha. Isu takasarudza kubva kumabasa eimwe bhuroka: Dependency Tracking, kusarudza wachi, mawachi aya anogona kuwiriraniswa sei, uye nemabatiro atinoita chengetedzo - iyi iratidziro yakaoma yezvandichataura nezvazvo:
Full Dependency Tracking
Nei zvichidiwa? Kuitira kuti kana data ichidzokororwa, rekodhi yega yega, shanduko yega yega ine ruzivo rwekuti ishanduko ipi inoenderana nayo. Shanduko yekutanga uye yekusaziva ndeye kana meseji yega yega ine rekodhi ine ruzivo nezve yapfuura mameseji:
Mumuenzaniso uyu, nhamba iri mumabhuraketi akamonana ndiyo nhamba dzerekodhi. Dzimwe nguva aya marekodhi ane kukosha anototamiswa ari ese, dzimwe nguva mamwe mavhezheni anotamiswa. Chinokosha ndechekuti shanduko yega yega ine ruzivo nezve yapfuura (zviri pachena inotakura zvese izvi mukati mayo).
Nei takasarudza kusashandisa nzira iyi (kutevera kuzere)? Zviripachena, nekuti iyi nzira haigoneki: chero shanduko kune social network inotsamira pane zvese zvakambochinja kune iyo social network, kutamisa, toti, Facebook kana VKontakte mune yega yega update. Zvakadaro, kune kutsvagisa kwakawanda paFull Dependency Tracking - aya ndiwo pre-social network; kune mamwe mamiriro anoshanda chaizvo.
Zvakajeka Dependency Tracking
Inotevera yakanyanya kuganhurirwa. Kuendeswa kweruzivo kunotariswawo pano, asi chete izvo zvinonyatsoenderana. Izvo zvinoenderana nei, sekutonga, zvinotemerwa neKushandisa. Kana data ichidzokororwa, mubvunzo unongodzosa mhinduro kana zvakambomira zvakagutswa, kureva, zvichiratidzwa. Ichi ndicho musimboti wekuti Causal kuenderana kunoshanda sei.
Anoona kuti rekodhi 5 zvinoenderana nemarekodhi 1, 2, 3, 4 - nekudaro, anomirira mutengi asati awana shanduko dzakaitwa nesarudzo yekuwana yaPenny, apo shanduko dzese dzakapfuura dzatopfuura nedhatabhesi.
Izvi hazvitikodzeri isu, nekuti pachine ruzivo rwakawandisa, uye zvinononoka zvinhu. Pane imwe nzira ...
Lamport Clock
Vachembera chaizvo. Lamport Clock zvinoreva kuti izvi zvinotsamira zvakapetwa kuita scalar basa, iro rinonzi Lamport Clock.
A scalar function ndiyo imwe abstract number. Inowanzonzi nguva ine musoro. Nechiitiko chimwe nechimwe, counter iyi inowedzera. Counter, iyo parizvino inozivikanwa kune maitiro, inotumira meseji yega yega. Zviripachena kuti maitiro anogona kunge asiri kuenderana, anogona kuve nenguva dzakasiyana zvachose. Zvakadaro, sisitimu yacho inoenzanisa wachi nekutumira mameseji. Chii chinoitika munyaya iyi?
Ini ndakapatsanura iyo shard hombe nepakati kuti ijekese: Shamwari dzinogona kugara mune imwe node, iyo ine chidimbu chemuunganidzwa, uye Feed inogona kugara mune imwe node, iyo ine chidimbu chemuunganidzwa uyu. Zvakajeka here kuti vangabuda sei mumutsara? Chekutanga Feed ichati: "Replicated", uyezve Shamwari. Kana iyo sisitimu isingape imwe rudzi rwekuvimbisa kuti Iyo Feed haizoratidzirwe kudzamara Shamwari dzinotsamira muKuunganidzwa kweShamwari dzaunzwa zvakare, isu tichava chaizvo nemamiriro ezvinhu andataura.
Iwe unoona kuti iyo counter nguva paFeed zvine musoro inowedzera sei:
Saka chinhu chikuru cheiyi Lamport Clock uye Causal kuwirirana (inotsanangurwa kuburikidza neLamport Clock) ndeichi: kana isu tine Zviitiko A uye B, uye Chiitiko B chinoenderana neChiitiko A *, zvino zvinotevera kuti LogicalTime yeChiitiko A ishoma pane. LogicalTime kubva Chiitiko B.
* Dzimwe nguva vanotaurawo kuti A akamboitika B asati avapo, ndiko kuti, A akamboitika B asati avapo - uhwu hukama hwakati hunorayira chikamu chose chezviitiko zvakaitika mukuzara.
Zvakapesana hazvina kururama. Ichi ndicho chimwe chezvakashata zvakanyanya zveLamport Clock - chikamu chekurongeka. Pane pfungwa yezviitiko zvenguva imwe chete, ndiko kuti, zviitiko umo (A zvisati zvaitika B) kana (A zvisati zvaitika B). Muenzaniso ungave wekuwedzera kwaLeonard kune mumwe munhu seshamwari (kwete kunyangwe Leonard, asi Sheldon, semuenzaniso).
Ichi ndicho chivakwa chinowanzo shandiswa kana uchishanda neLamport wachi: ivo vanotarisa chaizvo basa uye kubva pane izvi vanogumisa kuti pamwe zviitiko izvi zvinoenderana. Nekuti imwe nzira ndeyechokwadi: kana LogicalTime A iri pasi peLogicalTime B, saka B haigone kuitika A asati; uye kana zvakawanda, zvichida.
Vector Clock
Iko kunzwisisika kusimudzira kweLamport wachi ndiyo Vector Clock. Vanosiyana pakuti imwe neimwe node iri pano ine wachi yayo yakaparadzana, uye inofambiswa sevheji.
Muchiitiko ichi, iwe unoona kuti zeroth indekisi ye vector ine basa reKudyisa, uye yekutanga index yevector ndeye Shamwari (imwe neimwe yeiyi node). Uye zvino ivo vachawedzera: zero index ye "Feed" inowedzera pakunyora - 1, 2, 3:
Sei Vector Clock iri nani? Nekuti ivo vanokutendera iwe kuti uone kuti ndezvipi zviitiko zviri panguva imwe chete uye pazvinoitika panzvimbo dzakasiyana. Izvi zvakanyanya kukosha kune sharding system seMongoDB. Nekudaro, isu hatina kusarudza izvi, kunyangwe chiri chinhu chinoshamisa, uye chinoshanda zvakanaka, uye chingangoenderana nesu ...
Kana isu tiine zviuru gumi shards, hatigone kutamisa zviuru gumi zvezvikamu, kunyangwe tikazvimanikidza kana kuuya nechimwe chinhu - iyo payload icharamba yakapetwa kakati wandei pane vhoriyamu yeiyi vector yese. Naizvozvo, tichirumanisa mwoyo yedu nemazino, takasiya iyi nzira ndokupfuurira kune imwe.
Spanner TrueTime. Atomic wachi
Ndakati pane nyaya yaSpanner. Ichi chinhu chinotonhorera, kubva muzana ramakore rechiXNUMX: wachi dzeatomu, GPS kuwiriranisa.
Ipfungwa yei? "Spanner" iGoogle system ichangobva kuwanikwa kuvanhu (vakawedzera SQL kwairi). Imwe neimwe kutengeserana ikoko ine imwe nguva chitambi. Sezvo nguva ichienderana*, chiitiko chega chega chinogona kupihwa nguva yakatarwa - wachi dzeatomu dzine nguva yekumirira, mushure mezvo imwe nguva inovimbiswa "kuitika".
Saka, nekungonyorera kune dhatabhesi uye kumirira imwe nguva yenguva, iyo Serializability yechiitiko inovimbiswa otomatiki. Vane yakasimba Consistency modhi inogona kufungidzira musimboti - iri External Consistency.
* Iri ndiro dambudziko guru neLampart wachi - hadzina kumbobvira dzakaenderana pane akagoverwa masisitimu. Ivo vanogona kupatsanuka; kunyangwe neNTP, havasati vanyatsoshanda. "Spanner" ine wachi yeatomu uye kuwiriranisa, zvinoita kunge, microseconds.
Nei tisina kusarudza? Isu hatifungi kuti vashandisi vedu vane yakavakirwa-mukati atomu wachi. Pazvinozoonekwa, zvichivakwa mulaptop yega yega, pachave neimwe mhando yemhando yepamusoro inotonhorera GPS kuwiriranisa - ipapo hongu... Asi parizvino iyo yakanyanya kunaka inogoneka iAmazon, Base Stations - yevanofarira... Saka takashandisa mamwe wachi. .
Hybrid Clock
Izvi ndizvo chaizvo zvinobatika muMongoDB kana uchiona Causal kuenderana. Ko ivo vakasanganiswa sei? Hybrid kukosha kwe scalar, asi ine zvikamu zviviri:
- Yekutanga ndiyo Unix epoch (mangani masekonzi apfuura kubva "kutanga kwenyika yekombuta").
- Yechipiri ndeyekuwedzera, zvakare 32-bit isina kusaina int.
Ndizvo zvose, chaizvoizvo. Pane nzira iyi: chikamu chinokonzera nguva chinopindirana newachi nguva dzose; pese panoitika gadziriso, chikamu ichi chinowiriraniswa newachi uye zvinozoitika kuti nguva inogara yakawanda kana kushoma, uye kuwedzera kunobvumira iwe kusiyanisa pakati pezviitiko zvakaitika panguva imwe chete panguva.
Sei izvi zvakakosha kuMongoDB? Nekuti inokutendera iwe kuti ugadzire imwe mhando yekuchengetera maresitorendi pane imwe nguva nenguva, ndiko kuti, chiitiko chacho chinonongedzwa nenguva. Izvi zvakakosha kana zvimwe zviitiko zvichidikanwa; Kune dhatabhesi, zviitiko shanduko mudhatabhesi yakaitika pane dzimwe nguva nenguva.
Ini ndichakuudza chikonzero chakakosha chete kwauri (ndapota, usaudza chero munhu)! Isu takaita izvi nekuti izvi ndizvo zvakarongeka, indexed data inoita senge muMongoDB OpLog. OpLog idhizaini yedata iyo ine zvachose shanduko dzese mudhatabhesi: ivo vanotanga vaenda kuOpLog, vozoiswa kuKuzvichengeta pachayo mune iyo kesi kana iri zuva rakadzokororwa kana shard.
Ichi ndicho chaiva chikonzero chikuru. Zvakadaro, kune zvakare zvinoshanda zvinodiwa pakugadzira dhatabhesi, izvo zvinoreva kuti inofanira kunge iri nyore - diki kodhi, sezvishoma zvinhu zvakaputsika sezvinobvira zvinoda kunyorwazve uye kuyedzwa. Icho chokwadi chekuti maolog edu akaiswa indexed newachi yakasanganiswa akabatsira zvakanyanya uye akatibvumira kuita sarudzo chaiyo. Yakabhadhara chaizvo uye neimwe nzira yakashanda nemashiripiti pane yekutanga prototype. Kwakanaka kwazvo!
Kuwiriranisa wachi
Kune akati wandei nzira dzekubatanidza dzinotsanangurwa muzvinyorwa zvesainzi. Ndiri kutaura nezve synchronization kana tine mashadhi maviri akasiyana. Kana pane imwe replica set, hapana chikonzero chekufananidza: iyi i "tenzi mumwechete"; isu tine OpLog, mairi shanduko dzese dzinowira - mune iyi kesi, zvese zvakatove zvakateerana zvakarongerwa mu "Oplog" pachayo. Asi kana isu tine mashadhi maviri akasiyana, kuwiriranisa nguva kwakakosha pano. Apa ndipo pakabatsira mavheti wachi zvakanyanya! Asi isu hatina.
Yechipiri yakakodzera - iyi ndiyo "Heartbeats". Zvinokwanisika kuchinjanisa mamwe masaini anoitika imwe neimwe unit yenguva. Asi Heartbeats inononoka, hatigone kupa latency kumutengi wedu.
Nguva yechokwadi, chokwadika, chinhu chinoshamisa. Asi, zvakare, iyi ingangodaro iri ramangwana ... Kunyange zvazvo inogona kutoitwa muAtlas, pane kare inokurumidza "Amazon" nguva synchronizers. Asi hazvizowanikwe kune wese munhu.
Makuhwa ndipo apo mameseji ese anosanganisira nguva. Izvi zvinenge zvatinoshandisa. Yese meseji pakati penodhi, mutyairi, data node router, zvachose zvese zveMongoDB imwe mhando yechinhu, chikamu chedhatabhesi chine wachi inomhanya. Ivo vane chirevo chenguva yakasanganiswa kwese kwese, inofambiswa. 64 zvishoma? Izvi zvinobvumira, izvi zvinogoneka.
Zvose zvinoshanda sei pamwe chete?
Pano ndiri kutarisa imwe replica set kuti iite nyore. Kune Primary neSecondary. Sekondari inoita kudzokorora uye haigari yakanyatso wiriraniswa nePrimary.
Kuiswa kunoitika mu "Primery" ine imwe nguva kukosha. Kuiswa uku kunowedzera nhamba yemukati ne11, kana iri iyo yakanyanya. Kana kuti ichatarisa kukosha kwewachi uye sync kune wachi kana mawachi ari makuru. Izvi zvinokubvumira kuronga nenguva.
Mushure mokunge aita kurekodha, nguva inokosha inoitika. Wachi iri mu "MongoDB" uye inowedzerwa chete kana ichinyorwa ku "Oplog". Ichi ndicho chiitiko chinoshandura mamiriro ehurongwa. Muzvinyorwa zvese zvekare, chiitiko chinoonekwa sekunge meseji ichirova node: meseji yasvika, zvinoreva kuti system yachinja mamiriro ayo.
Izvi zvinokonzerwa nekuti panguva yetsvakurudzo hazvina kunyatsojeka kuti shoko iri richadudzirwa sei. Isu tinoziva zvechokwadi kuti kana isina kuratidzwa mu "Oplog", saka haizodudzirwa chero nzira, uye shanduko yemamiriro ehurongwa ingopinda mu "Oplog". Izvi zvinorerutsira zvese kwatiri: modhi inorerutsa, uye inotitendera kuironga mukati meimwe replica set, uye zvimwe zvakawanda zvinobatsira.
Kukosha kwakatonyorerwa ku "Oplog" kunodzoserwa - tinoziva kuti "Oplog" yatova nehukoshi uhu, uye nguva yayo ndeye 12. Zvino, toti, kuverenga kunotanga kubva kune imwe node (Secondary), uye inotumira afterClusterTime mu. shoko. Anoti: "Ndinoda zvose zvakaitika mushure me12 kana panguva yegumi nemaviri" (ona mufananidzo uri pamusoro).
Izvi ndizvo zvinonzi Causal a consistent (CAT). Pane pfungwa yakadaro mudzidziso yokuti ichi chimwe chikamu chenguva, chinopindirana pachacho. Muchiitiko ichi, tinogona kutaura kuti iyi ndiyo mamiriro ehurongwa hwakaonekwa panguva ye12.
Ikozvino hapachina chinhu pano, nekuti iyi mhando inotevedzera mamiriro kana iwe uchida iyo Sekondari kuti idzokorore data kubva kuPuraimari. Anomirira ... Uye ikozvino data rasvika - anodzosera izvi zvakakosha kumashure.
Ndizvo chaizvo kuti zvose zvinoshanda sei. Kunenge.
Zvinorevei kuti "zvinenge"? Ngatifungei kuti pane mumwe munhu akaverenga uye akanzwisisa kuti izvi zvinoshanda sei. Ndakaona kuti pese ClusterTime painoitika, inogadzirisa wachi yemukati, uyezve iyo inotevera yekupinda inowedzera neimwe. Basa iri rinotora mitsara makumi maviri. Ngatitii munhu uyu anotumira nhamba huru ye20-bit, kubvisa imwe.
Sei "minus one"? Nekuti wachi yemukati ichatsiviwa muhukoshi uhu (zviri pachena, ichi ndicho chikuru chinogoneka uye chikuru kupfuura nguva yazvino), ipapo yekupinda ichaitika mu "Oplog", uye wachi ichawedzerwa neimwe yuniti - uye ichatove. kuve yakakosha kukosha (kungori mayuniti ese, hapana kumwe kwaunoenda), unsaint ints).
Zviri pachena kuti mushure meizvi hurongwa hunova husingasviki zvachose kune chero chinhu. Inogona chete kubudiswa uye kucheneswa - yakawanda yebasa remaoko. Kuwanikwa kwakazara:
Uyezve, kana izvi zvikadzokororwa kune imwe nzvimbo, ipapo sumbu rose rinongodonha. Mamiriro ezvinhu asingagamuchirwi zvachose ayo chero munhu anogona kuronga nekukurumidza uye nyore! Naizvozvo, takafunga iyi nguva seimwe yeakanyanya kukosha. Nzira yekudzivirira sei?
Nzira yedu ndeyekusaina clusterTime
Aya ndiwo mafambisirwo azvinoitwa mumeseji (pamberi pemavara eblue). Asi isu takatangawo kugadzira siginecha (mavara ebhuruu):
Siginecha inogadzirwa nekiyi inochengetwa mukati medhatabhesi, mukati meiyo yakachengeteka perimeter; pachayo inogadzirwa uye yakagadziridzwa (vashandisi havaone chero chinhu nezvazvo). Hashi inogadzirwa, uye meseji yega yega inosainwa kana yagadzirwa uye yakasimbiswa kana yagamuchirwa.
Mubvunzo ungangodaro unomuka mupfungwa dzevanhu: "Izvi zvinononoka sei zvinhu?" Ndakakuudza kuti inofanira kushanda nekukasira, kunyanya kana ichi chisipo.
Zvinorevei kushandisa Causal consistency munyaya iyi? Uku ndiko kuratidza afterClusterTime parameter. Pasina izvi, zvinongopfuura kukosha zvakadaro. Makuhwa, kutanga kubva mushanduro 3.6, anogara achishanda.
Kana isu tikasiya chizvarwa chinogara chichisaina, chinononoka dhizaini kunyangwe pasina chimiro, icho chisingaenderane nemaitiro edu uye zvatinoda. Saka takaitei?
Zviite nekukurumidza!
Icho chinhu chiri nyore, asi chinyengeri chinonakidza - ndichachigovera, zvichida mumwe munhu achafarira.
Tine hashi inochengeta data rakasainwa. Yese data inoenda kuburikidza necache. Iyo cache haina kusaina iyo yakatarwa nguva, asi iyo Range. Kana imwe kukosha yasvika, isu tinogadzira Range, mask kunze ekupedzisira gumi nematanhatu mabits, uye isu tinosaina kukosha uku:
Nekugamuchira siginecha yakadaro, isu tinomhanyisa sisitimu (zvishoma) 65 zviuru nguva. Inoshanda zvakanaka: patakaita zviyedzo, nguva yakadzikira nezviuru gumi nguva patakange tave nekutevedzana kwekuvandudza. Zviri pachena kuti kana vakapesana, izvi hazvishande. Asi muzviitiko zvakawanda zvinoshanda. Iko kusanganiswa kweRange siginecha pamwe chete nesaini yakagadzirisa dambudziko rekuchengetedza.
Tadzidzei?
Zvidzidzo zvatakawana kubva pane izvi:
- Tinofanira kuverenga zvinyorwa, nyaya, zvinyorwa, nokuti tine zvinhu zvakawanda zvinonakidza. Kana isu tichishanda pane chimwe chimiro (kunyanya ikozvino, patakaita kutengeserana, nezvimwewo), tinofanira kuverenga nekunzwisisa. Zvinotora nguva, asi zvinobatsira chaizvo nekuti zvinojekesa patiri. Hatina kuita setichaunza chero chinhu chitsva - takangotora zvigadzirwa.
Kazhinji, pane imwe mutsauko mukufunga kana paine musangano wezvidzidzo (Sigmon, semuenzaniso) - munhu wese anotarisa pane zvitsva. Chii chitsva nezve algorithm yedu? Hapana chinhu chitsva pano. Hutsva huri munzira yatinosanganisa nzira dziripo pamwechete. Naizvozvo, chinhu chekutanga kuverenga classics, kutanga naLampart.
- Mukugadzira, zvinodiwa zvakasiyana zvachose. Ndine chokwadi chekuti vazhinji venyu havana kutarisana ne "spherical" dhatabhesi mune abstract vacuum, asi nezvakajairika, zvinhu chaizvo zvine matambudziko nekuwanikwa, latency uye kukanganisa kushivirira.
- Chinhu chekupedzisira ndechekuti taifanira kutarisa pfungwa dzakasiyana uye nekusanganisa akati wandei akasiyana zvinyorwa mune imwe nzira, pamwechete. Pfungwa yekusaina, semuenzaniso, yakauya kubva kune imwe nyaya yaifunga nezvePaxos protocol, iyo yevasiri-Byzantine Vakundikana iri mukati memvumo protocol, yeByzantine avo - kunze kwemvumo protocol ... Kazhinji, izvi ndizvo chaizvo zvatinoita. apedza kuita.
Hapana chitsva zvachose pano! Asi pakarepo patakavhenganisa zvose pamwe chete ... Zvakafanana nekutaura kuti Olivier saladi recipe haina maturo, nokuti mazai, mayonnaise uye makomkomere zvakatogadzirwa ... Zviri pamusoro penyaya imwe chete.
Ndichapedzisa neizvi. Ndatenda!
Mibvunzo yako
Mubvunzo unobva kuvateereri (zvino uchanzi B): - Ndatenda, Mikhail, neshumo! Musoro wenguva unonakidza. Uri kushandisa Makuhwa. Vakataura kuti munhu wese ane nguva yake, munhu wese anoziva nguva yake. Sezvandinozvinzwisisa, tine mutyairi - panogona kuva nevatengi vakawanda vane vatyairi, query-planners zvakare, shards zvakare ... Uye iyo system inodzika sei kana tikangoerekana taita mutsauko: mumwe munhu anosarudza kuti ndeye miniti mberi, mumwe munhu kweminiti kumashure? Tichasvika kupi?
MT: - Mubvunzo wakanaka zvechokwadi! Ndaida kutaura zvemashavi chete. Kana ini ndichinzwisisa mubvunzo nenzira kwayo, tine mamiriro anotevera: kune shard 1 uye shard 2, kuverenga kunoitika kubva kune aya maviri shards - vane mutsauko, havadyidzani nemumwe, nekuti nguva yavanoziva yakasiyana. kunyanya nguva yavanenge varipo mumaplogs.
Ngatitii shard 1 yakaita miriyoni zvinyorwa, shard 2 hapana chaakaita, uye chikumbiro chakasvika kune maviri shards. Uye yekutanga ine afterClusterTime inopfuura miriyoni. Mumamiriro ezvinhu akadaro, sezvandakatsanangura, shard 2 haizombopinduri zvachose.
Mu: - Ndaida kuziva kuti vanowiriranisa sei uye vanosarudza imwe nguva ine musoro?
MT: - Zviri nyore kwazvo kuwiriranisa. Shard, kana afterClusterTime ichisvika kwaari uye akasawana nguva mu "Oplog", inotanga isina kubvumidzwa. Ndiko kuti, anosimudza nguva yake nemaoko ake kune kukosha uku. Izvi zvinoreva kuti haina zviitiko zvinoenderana nechikumbiro ichi. Anogadzira chiitiko ichi zvisirizvo uye nokudaro anova Causal Consistent.
Mu: - Ko kana mushure meizvi zvimwe zviitiko zvikauya kwaari izvo zvakarasika kumwe kunhu mumambure?
MT: - Shard yakagadzirwa nenzira yekuti havazouye zvakare, sezvo iri tenzi mumwechete. Kana iye atonyora, saka havauye, asi vachauya gare gare. Izvo hazvigone kuitika kuti chimwe chinhu chinonamira kumwe kunhu, ipapo iye haanyore, uye ipapo zviitiko izvi zvinosvika - uye iyo Causal kusawirirana kwakaputsika. Kana akasanyora, ngavauye vese vanotevera (achavamirira).
Mu: - Ndine mibvunzo yakati wandei maererano nemitsara. Causal consistency inofungidzira kuti pane chaiyo mutsara wezviito zvinoda kuitwa. Chii chinoitika kana imwe yemapakeji edu ikanyangarika? Heinoi yechigumi, yechigumi neimwe... yechigumi nembiri yanyangarika, uye vamwe vose vakamirira kuti zviitike. Uye kamwe kamwe motokari yedu yakafa, hapana chatinogona kuita. Pane hurefu hwakanyanya here hwemutsara unoungana usati waitwa? Kukundikana kupi kunouraya kunoitika kana imwe nyika ikarasika? Uyezve, kana tikanyora pasi kuti kune imwe nyika yapfuura, saka tinofanira kutanga kubva pairi? Asi havana kumusundira kure!
MT: - Zvakare mubvunzo wakanaka! Tiri kuitei? MongoDB ine pfungwa yekuti quorum inonyora, quorum inoverengwa. Muzviitiko zvipi meseji inogona kurasika? Kana kunyora kusiri koramu kana kuverenga kusiri quorum (rumwe rudzi rwemarara runogona kunamatira).
Nezve Causal kusachinja-chinja, bvunzo huru yekuyedza yakaitwa, mhedzisiro yacho yaive yekuti kana ichinyorwa uye kuverenga isiri-quorum, kutyorwa kweCausal kusachinja-chinja kunoitika. Ndizvo zvaunotaura!
Zano redu: shandisa kanenge quorum kuverenga kana uchishandisa Causal kuenderana. Muchiitiko ichi, hapana chicharasika, kunyange kana quorum rekodhi yakarasika ... Iyi ndiyo orthogonal mamiriro ezvinhu: kana mushandisi asingadi kuti data irasike, anoda kushandisa quorum rekodhi. Causal consistency haivimbisi kusimba. Kugara kunovimbiswa nekudzokorora uye muchina wakabatana nekudzokorora.
Mu: - Patinogadzira chiitiko chinotiitira sharding (kwete tenzi, asi muranda, zvichiteerana), inovimba neUnix nguva yemuchina wayo kana panguva ye "tenzi"; Inowiriranisa kekutanga here kana nguva nenguva?
MT: - Ndichajekesa ikozvino. Shard (i.e. horizontal partition) - panogara paine Primary ipapo. Uye shard inogona kuva ne "tenzi" uye panogona kuva nemifananidzo. Asi shard inogara ichitsigira kurekodha, nekuti inofanirwa kutsigira imwe domain (iyo shard ine Primary).
Mu: - Saka zvese zvinoenderana ne "tenzi"? Nguva yatenzi inoshandiswa nguva dzose here?
MT: - Ehe. Iwe unogona kutaura nenzira yekufananidzira: wachi iri kukwenya kana yekupinda mu "tenzi", mu "Oplog" inoitika.
Mu: - Tine mutengi anobatanidza uye haadi kuziva chero chinhu nezve nguva?
MT: - Iwe haufanirwe kuziva chero chinhu zvachose! Kana tikataura nezve mashandiro azvinoita pamutengi: kana mutengi achida kushandisa Causal kuenderana, anofanirwa kuvhura chikamu. Iye zvino zvese zvavepo: kutengeserana muchikamu, uye tora kodzero ... Chikamu ndiko kurongeka kwezviitiko zvine musoro zvichiitika nemutengi.
Kana akavhura chikamu ichi uye otaura ipapo kuti anoda Causal kuenderana (kana iyo musangano ichitsigira Causal kuenderana nekukasira), zvese zvinoshanda zvoga. Mutyairi anoyeuka nguva iyi uye anoiwedzera kana agamuchira meseji itsva. Inorangarira kuti mhinduro yakadzoswa neyakapfuura kubva kuseva yakadzosa data. Chikumbiro chinotevera chine afterCluster("nguva yakakura kupfuura iyi").
Mutengi haafanire kuziva zvachose chero chinhu! Izvi hazvina kunyatsojeka kwaari. Kana vanhu vakashandisa zvinhu izvi, chii chavanogona kuita? Chekutanga, iwe unogona kuverenga zvakachengetedzeka sekondari: unogona kunyorera kuPuraimari uye uverenge kubva kunzvimbo dzakadzokororwa sekondari uye uve nechokwadi chekuti inoshanda. Panguva imwecheteyo, zvikamu zvakanyorwa paPuraimari zvinogona kutoendeswa kuSekondari, kureva kuti haugone kushandisa chikamu chimwe chete, asi akati wandei.
Mu: -Chitsva chitsva cheCompute sainzi - CRDT (Kusawirirana-isina Replicated Data Types) mhando dzedata - ine hukama zvakanyanya nemusoro wekupedzisira kuenderana. Wakambofunga kubatanidza aya marudzi e data mudhatabhesi uye chii chaungataure nezvazvo?
MT: - Mubvunzo wakanaka! CRDT ine musoro pakunyora kupokana: muMongoDB, single master.
Mu: - Ndine mubvunzo kubva kune devops. Munyika chaiyo, kune akadai maJesuitical mamiriro apo Kukundikana kweByzantine kunoitika, uye vanhu vakaipa mukati meiyo yakadzivirirwa perimita vanotanga kupinza muprotocol, kutumira mapakeji ehunyanzvi nenzira yakakosha?
MT: -Vanhu vakaipa mukati memupendero vakaita sebhiza reTrojan! Vanhu vakaipa mukati memuganhu vanogona kuita zvinhu zvakawanda zvakaipa.
Mu: - Zviri pachena kuti kusiya, zvishoma kutaura, gomba muvhavha iyo iwe unogona kuisa zoo yenzou uye kuputsa boka rose zvachose ... Zvichatora nguva yekudzorera nemaoko ... Izvi, kuti uzviise zvinyoro, ndizvo zvakaipa. Kune rumwe rutivi, izvi zvinonakidza: muhupenyu chaihwo, mukuita, pane mamiriro ezvinhu apo zvakasikwa zvakafanana kurwisa kwemukati kunoitika?
MT: - Sezvo ini ndisingawanzo kusangana nekutyorwa kwekuchengetedza muhupenyu chaihwo, handikwanise kutaura kana zvikaitika. Asi kana tichitaura nezvehuzivi hwebudiriro, tinofunga sezvizvi: tine perimeter inopa vakomana vanoita chengetedzo - iyi inhare, madziro; uye mukati memupendero unogona kuita chero chaunoda. Zviri pachena kuti kune vashandisi vane kugona kuona chete, uye kune vashandisi vane kugona kudzima dhairekitori.
Zvichienderana nekodzero, kukuvadzwa kunoitwa nevashandisi kunogona kuita mbeva, kana kuti nzou. Zviri pachena kuti mushandisi ane kodzero dzakakwana anogona kuita chero chinhu zvachose. Mushandisi ane kodzero shoma anogona kukonzera kukuvadza kushoma. Kunyanya, haigoni kuputsa hurongwa.
Mu: - Munharaunda yakachengetedzwa, mumwe munhu akaedza kugadzira zvibvumirano zvisingatarisirwi zvevhavha kuitira kuti aparadze zvachose sevha, uye kana iwe uine rombo rakanaka, boka rose ... Inombowana "yakanaka" here?
MT: "Handisati ndambonzwa nezvezvinhu zvakadaro." Icho chokwadi chekuti iwe unogona kukanganisa server nenzira iyi hachisi chakavanzika. Kukundikana mukati, kuva kubva kuprotocol, kuva mushandisi ane mvumo anogona kunyora chimwe chinhu chakadai mumashoko ... Kutaura zvazviri, hazvibviri, nokuti zvicharamba zvichisimbiswa. Zvinogoneka kudzima kuvimbiswa uku kune vashandisi vasingazvide - saka ndiro dambudziko ravo; ivo, zvishoma, vakaparadza madziro pachavo uye iwe unogona kusundira nzou imomo, iyo ichatsika-tsika ... Asi kazhinji, iwe unogona kupfeka semugadziri wekugadzirisa, uya uye ubudise kunze!
Mu: β Thanks nokuda mushumo. Sergey (Yandex). Kune nguva dzose muMong iyo inodzika nhamba yenhengo dzekuvhota muReplica Set, uye izvi zvinogara zvakaenzana ne7 (zvinomwe). Sei izvi zviri zvenguva dzose? Sei iyi isiri imwe mhando yeparameter?
MT: -Tine Replica Sets ine makumi mana node. Kunogara kune ruzhinji. Handizivi kuti ndeipi version...
Mu: - MuReplica Set unogona kumhanyisa nhengo dzisiri dzekuvhota, asi kune huwandu hwenhengo dzekuvhota 7. Tingapona sei kuvharwa munyaya iyi kana Replica Set yedu yakapararira munzvimbo nhatu dze data? Imwe data data inogona kudzima nyore, uye mumwe muchina unogona kudonha.
MT: - Izvi zvatove zvishoma kudarika chikamu chemushumo. Uyu mubvunzo mukuru. Pamwe ndinogona kuzokuudza nezvazvo gare gare.
Dzimwe ads π
Ndinokutendai nekugara nesu. Unoda zvinyorwa zvedu here? Unoda kuona zvimwe zvinonakidza zvemukati? Titsigire nekuisa odha kana kukurudzira kushamwari, , yakasarudzika analogue yekupinda-level maseva, iyo yakagadzirwa nesu kuti iwe: (inowanikwa neRAID1 uye RAID10, kusvika ku24 cores uye kusvika ku40GB DDR4).
Dell R730xd 2 nguva yakachipa muEquinix Tier IV data center muAmsterdam? Chete pano muNetherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - kubva pamadhora makumi mapfumbamwe nemapfumbamwe! Verenga nezve
Source: www.habr.com