Netflix ndiye mutungamiri mumusika weInternet terevhizheni - kambani yakagadzira uye iri kushingaira kugadzira chikamu ichi. Netflix inozivikanwa kwete chete nekabhuku kayo kakakura kemamuvhi uye TV akateedzana anowanikwa kubva kunenge kona yese yepasi uye chero mudziyo une chiratidziro, asiwo neayo akavimbika masisitimu uye yakasarudzika tsika yeinjiniya.
Muenzaniso wakajeka weiyo nzira yeNetflix yekugadzira uye kutsigira yakaoma masisitimu yakaunzwa kuDevOops 2019 -Director weBudiriro paNetflix. Akapedza kudzidza kuFaculty yeComputational Mathematics uye Masvomhu yeNizhny Novgorod State University. Lobachevsky, Sergey mumwe wevanjiniya vekutanga muOpen Connect - CDN timu paNetflix. Akavaka masisitimu ekutarisa uye kuongorora vhidhiyo data, akatanga sevhisi yakakurumbira yekuongorora Internet yekubatanidza kumhanya FAST.com, uye kwemakore mashoma apfuura anga achishanda mukugadzirisa zvikumbiro zveInternet kuitira kuti Netflix application ishande nekukurumidza sezvinobvira kune vashandisi.
Chirevo chakagamuchira wongororo dzakanakisa kubva kuvatori vechikamu mumusangano, uye takakugadzirira iwe chinyorwa chinyorwa.
Mumushumo wake, Sergei akataura zvakadzama
- nezve izvo zvinokanganisa kunonoka kweInternet zvikumbiro pakati pemutengi uye server;
- sei kuderedza kunonoka uku;
- maitiro ekugadzira, kuchengetedza uye kutarisa maitiro ekukanganisa-anoshivirira masisitimu;
- maitiro ekuwana mhedzisiro munguva pfupi, uye nenjodzi shoma kune bhizinesi;
- maitiro ekuongorora zvabuda uye kudzidza kubva mukukanganisa.
Mhinduro dzemibvunzo iyi dzinodiwa kwete chete neavo vanoshanda mumakambani makuru.
Misimboti uye matekiniki akaunzwa anofanirwa kuzivikanwa uye kuitwa nemunhu wese anogadzira uye anotsigira zvigadzirwa zveInternet.
Inotevera irondedzero kubva pamaonero emukurukuri.
Kukosha kwekumhanya kweinternet
Iko kumhanya kweInternet zvikumbiro kune hukama zvakananga nebhizinesi. Funga nezve indasitiri yekutenga: Amazon muna 2009 kuti kunonoka kwe100ms kunoguma nekurasikirwa kwe1% yekutengesa.
Kune akawanda uye akawanda nharembozha, inoteverwa nenharembozha uye maapplication. Kana peji rako richitora nguva yakareba kupfuura masekonzi matatu kurodha, uri kurasikirwa nehafu yevashandisi vako. WITH Google inofunga nezvekukurumidza kurodha kwepeji rako mumibairo yekutsvaga: iyo inokurumidza peji, inokwirisa chinzvimbo chayo muGoogle.
Yekubatanidza kumhanya kwakakosha zvakare mumasangano emari uko latency yakakosha. Muna 2015, Hibernia Networks tambo yemamiriyoni mazana mana emadhora pakati peNew York neLondon kudzikisa latency pakati pemaguta ne400ms. Fungidzira $6 miriyoni ye 66 ms yekudzikisa latency!
Maererano ne , kumhanya kwekubatanidza pamusoro pe5 Mbit / s haichakanganisa zvakananga kumhanyisa kumhanya kweiyo webhusaiti webhusaiti. Nekudaro, pane hukama hwemutsara pakati pekubatanidza latency uye peji yekurodha kumhanya:
Nekudaro, Netflix haisi yakajairika chigadzirwa. Iko kukanganisa kwe latency uye kumhanya pamushandisi inzvimbo inoshanda yekuongorora uye kusimudzira. Iko kune kurodha kwekushandisa uye kusarudzwa kwemukati zvinoenderana ne latency, asi kurodha static zvinhu uye kutenderera zvinoenderanawo nekumhanya kwekubatanidza. Kuongorora nekugadzirisa zvinhu zvakakosha zvinopesvedzera ruzivo rwemushandisi inzvimbo inoshanda yekusimudzira yezvikwata zvakati wandei paNetflix. Chimwe chezvinangwa ndechekudzikisa latency yezvikumbiro pakati peNetflix zvishandiso uye cloud infrastructure.
Mumushumo isu tichatarisa zvakanyanya pakudzikisa latency tichishandisa muenzaniso weNetflix infrastructure. Ngatifungei kubva pane inoshanda yekuona maitiro ekuita maitiro ekugadzira, kusimudzira uye kushanda kweakaomesesa akagoverwa masisitimu uye kushandisa nguva pakuvandudza nemhedzisiro, pane kuongorora matambudziko ekushanda uye kuparara.
Mukati meNetflix
Zviuru zvezvishandiso zvakasiyana zvinotsigira Netflix apps. Iwo anogadzirwa nezvikwata zvina zvakasiyana, izvo zvinogadzira zvakaparadzana shanduro yemutengi yeApple, iOS, TV uye web browser. Uye isu tinopedza simba rakawanda mukuvandudza uye kugadzirisa munhu ruzivo rwemushandisi. Kuti tiite izvi, tinomhanyisa mazana ebvunzo dzeA/B dzakafanana.
Personalization inotsigirwa nemazana emamicroservices ari mugore reAWS, ichipa yakasarudzika mushandisi data, query dispatch, telemetry, Big Data uye Encoding. Traffic visualization inoita seizvi:
Kuruboshwe ndiko kwekupinda, uyezve traffic inogovaniswa pakati pemazana akati wandei ma microservices anotsigirwa neakasiyana backend zvikwata.
Chimwe chinhu chakakosha chezvivakwa zvedu ndeye Open Connect CDN, iyo inopa static zvemukati kune yekupedzisira mushandisi - mavhidhiyo, mifananidzo, kodhi yemutengi, nezvimwe. Iyo CDN iri pamaseva etsika (OCA - Vhura Batanidza Appliance). Mukati mune madhiraivha eSSD uye HDD madhiraivha anomhanyisa FreeBSD, ine NGINX uye seti yemasevhisi. Isu tinogadzira uye nekunatsiridza hardware uye software zvikamu kuitira kuti sevha yeCDN yakadaro itumire data rakawanda sezvinobvira kune vashandisi.
Iwo "madziro" emaseva aya paInternet traffic exchange point (Internet eXchange - IX) inoita seizvi:
Internet Exchange inopa kugona kwevanopa masevhisi eInternet uye vanopa zvemukati "kubatanidza" kune mumwe nemumwe kuti vachinjane zvakananga data paInternet. Kune angangoita 70-80 Internet Exchange mapoinzi kutenderera pasirese uko maseva edu akaiswa, uye isu takazvimiririra tinoisa nekuachengeta:
Pamusoro pezvo, isu tinopawo maseva zvakananga kune vanopa Internet, ivo vanoisa munetiweki yavo, vachivandudza iyo yenzvimbo yeNetflix traffic uye mhando yekushambadzira yevashandisi:
Seti yemasevhisi eAWS ine basa rekutumira zvikumbiro zvevhidhiyo kubva kune vatengi kuenda kumaseva eCDN, pamwe nekugadzirisa maseva ivo pachavo - kugadzirisa zvirimo, kodhi yepurogiramu, zvigadziriso, nezvimwe. Kune yekupedzisira, isu takavakawo backbone network inobatanidza maseva muInternet Exchange points neAWS. Iyo backbone network inetiweki yepasi rose yefiber optic tambo uye ma router atinogona kugadzira nekugadzirisa zvichienderana nezvatinoda.
By , yedu CDN zvivakwa zvinounza ingangoita β yepasi rose traffic yeInternet panguva dzepamusoro uye β yetraffic muNorth America, uko Netflix yanga yakatenderedza kwenguva refu. Nhamba dzinokatyamadza, asi kwandiri chimwe chezvakanakisa kubudirira ndechekuti iyo CDN system yese inogadzirwa uye inochengetwa nechikwata chevasingasviki zana nemakumi mashanu.
Pakutanga, iyo CDN hurongwa hwakagadzirirwa kuendesa data data. Nekudaro, nekufamba kwenguva takaona kuti isu tinogona zvakare kuishandisa kukwiridzira zvikumbiro zvine simba kubva kune vatengi vari mugore reAWS.
Nezve kukurumidza kweInternet
Nhasi, Netflix ine 3 AWS matunhu, uye latency yezvikumbiro kune gore zvinoenderana nekuti mutengi ari kure sei kubva kunharaunda iri pedyo. Panguva imwecheteyo, isu tine akawanda maCDN maseva anoshandiswa kuendesa static content. Pane imwe nzira yekushandisa iyi gadziriro yekumhanyisa mibvunzo ine simba? Nekudaro, zvinosuruvarisa, hazvigoneke kuchengetedza zvikumbiro izvi - maAPIs akasarudzika uye mhedzisiro yega yega.
Ngatigadzirei proxy paCDN server uye titange kutumira traffic kuburikidza nayo. Zvichakurumidza here?
Materiel
Ngatirangarirei kuti network protocol inoshanda sei. Nhasi, traffic yakawanda paInternet inoshandisa HTTPs, zvinoenderana neiyo yakaderera layer protocol TCP neTLS. Kuti mutengi abatanidze kune sevha, inobata ruoko, uye kumisikidza yakachengeteka yekubatanidza, mutengi anofanirwa kuchinjana mameseji nesevha katatu uye imwezve nguva yekuendesa data. Iine latency parwendo rwekutenderera (RTT) ye100 ms, zvingatora isu 400 ms kuti tigamuchire data rekutanga:
Kana tikaisa zvitupa paCDN server, ipapo nguva yekubata maoko pakati pemutengi uye sevha inogona kuderedzwa zvakanyanya kana CDN iri pedyo. Ngatifungei kuti latency kune CDN server ndeye 30ms. Ipapo zvinotora 220 ms kugamuchira yekutanga bhiti:
Asi zvazvakanakira hazvigumire ipapo. Kana kubatanidza kwave kusimbiswa, TCP inowedzera hwindo rekusangana (huwandu hwemashoko aanogona kuendesa pamusoro pekubatana ikoko kwakafanana). Kana data packet rikarasika, saka mashandisirwo ekare eTCP protocol (seTCP New Reno) inoderedza yakavhurika "hwindo" nehafu. Kukura kwehwindo rekusangana, uye nekukurumidza kwekupora kwayo kubva pakurasikirwa zvakare kunoenderana nekunonoka (RTT) kune server. Kana kubatana uku kuchingoenda kune sevha yeCDN, kupora uku kuchakurumidza. Panguva imwecheteyo, kurasikirwa kwepaketi chinhu chakajairwa, kunyanya kune isina waya network.
Internet bandwidth inogona kuderedzwa, kunyanya panguva dzepamusoro-soro, nekuda kwetraffic kubva kune vashandisi, izvo zvinogona kutungamirira kumigwagwa. Nekudaro, hapana nzira paInternet yekupa pekutanga kune zvimwe zvikumbiro pane zvimwe. Semuenzaniso, ipa pamberi kune zvidiki uye latency-sensitive zvikumbiro pane "inorema" data hova inoremedza network. Zvisinei, kwatiri, kuva netiweki yedu yemusana inotibvumira kuita izvi pane chikamu chenzira yekukumbira - pakati peCDN negore, uye tinogona kuzvigadzirisa zvizere. Iwe unogona kuve nechokwadi chekuti madiki uye latency-sensitive mapaketi anoiswa pamberi, uye yakakura data inoyerera inoenda zvishoma gare gare. Iyo iri pedyo neCDN kune mutengi, iyo inowedzera kushanda zvakanaka.
Application level protocol (OSI Level 7) zvakare ine mhedzisiro pane latency. Mitemo mitsva yakadai seHTTP/2 inokwenenzvera kuita kwezvikumbiro zvakafanana. Nekudaro, isu tine vatengi veNetflix vane zvishandiso zvekare zvisingatsigire maprotocol matsva. Havasi vese vatengi vanogona kuvandudzwa kana kugadziridzwa zvakakwana. Panguva imwecheteyo, pakati peCDN proxy uye gore pane kutonga kwakazara uye kugona kushandisa mitsva, yakakwana mapuroteni uye marongero. Chikamu chisingashande chine maprotocol ekare chinongoshanda pakati pemutengi neCDN server. Uyezve, isu tinogona kuita zvikumbiro zvemultiplex pane yakatosimbiswa kubatana pakati peCDN negore, tichivandudza mashandisirwo ekubatanidza padanho reTCP:
Tinopima
Kunyangwe ichokwadi chekuti dzidziso inovimbisa kuvandudzwa, isu hatimhanyire nekukurumidza kuvhura iyo system mukugadzira. Pane kudaro, tinofanira kutanga taratidza kuti pfungwa yacho ichashanda mukuita. Kuti uite izvi unofanirwa kupindura mibvunzo yakawanda:
- Speed: proxy ichakurumidza here?
- Kuvimbika: Ichaputika kazhinji here?
- Complexity: nzira yekubatanidza nemaapplication?
- mari: Inodhura marii kuendesa zvimwe zvivakwa?
Ngationgororei zvakadzama maitiro edu ekuongorora pfungwa yekutanga. Vamwe vose vanobatwa nenzira yakafanana.
Kuongorora kumhanya kwezvikumbiro, isu tinoda kuwana data kune vese vashandisi, pasina kushandisa nguva yakawanda mukusimudzira uye pasina kutyora kugadzirwa. Pane nzira dzakawanda dzeizvi:
- RUM, kana kungoita chikumbiro kuyerwa. Isu tinoyera nguva yekuitwa yezvikumbiro zvazvino kubva kune vashandisi uye tinoona kuzara kwemushandisi kufukidzwa. Izvo zvakashata ndezvekuti chiratidzo hachina kugadzikana zvakanyanya nekuda kwezvinhu zvakawanda, semuenzaniso, nekuda kwehukuru hwakasiyana hwekukumbira, nguva yekugadzira pane server uye mutengi. Uye zvakare, haugone kuyedza gadziriso nyowani pasina mhedzisiro mukugadzira.
- Laboratory bvunzo. Masevha akakosha uye zvivakwa zvinotevedzera vatengi. Nekubatsira kwavo tinoita miedzo inodiwa. Nenzira iyi tinowana kutonga kwakazara pamusoro pemigumisiro yekuyera uye chiratidzo chakajeka. Asi hapana kuvharwa kwakazara kwemidziyo uye nzvimbo dzevashandisi (kunyanya nesevhisi yepasirese uye rutsigiro rwezviuru zvemhando dzemidziyo).
Ungabatanidza sei zvakanakira nzira mbiri?
Chikwata chedu chawana mhinduro. Isu takanyora chidimbu chidiki chekodhi - sampuli - yatakavaka mukushandisa kwedu. Probes inotitendera kuti tiite inodzorwa zvizere network bvunzo kubva kumidziyo yedu. Inoshanda seizvi:
- Nguva pfupi mushure mekurodha application uye nekupedza basa rekutanga, tinomhanyisa maprobes edu.
- Mutengi anoita chikumbiro kune sevha uye anogamuchira "recipe" yebvunzo. Iyo resipi rondedzero ye URLs uko chikumbiro cheHTTP(s) chinoda kuitwa. Uye zvakare, iyo resipi inogadzirisa zvikumbiro paramita: kunonoka pakati pezvikumbiro, huwandu hwe data yakakumbirwa, HTTP (s) misoro, nezvimwe. Panguva imwecheteyo, isu tinogona kuyedza akati wandei akasiyana mabikirwo akafanana - kana tichikumbira gadziriso, isu tinongoona kuti ndeipi resipi yekuburitsa.
- Iyo probe yekuvhura nguva inosarudzwa kuitira kuti isapesane nekushanda kwekushandisa network zviwanikwa pamutengi. Chaizvoizvo, nguva inosarudzwa kana mutengi asiri kushanda.
- Mushure mekugamuchira resipi, mutengi anoita zvikumbiro kune yega yega maURL, zvakafanana. Chikumbiro kune imwe neimwe yekero inogona kudzokororwa - iyo inonzi. "pulses". Pane yekutanga pulse, tinoyera kuti zvakatora nguva yakareba sei kumisa yekubatanidza uye kurodha data. Pane yechipiri pulse, tinoyera nguva inotora kurodha data pamusoro peiyo yakatosimbiswa yekubatanidza. Pamberi peyechitatu, tinogona kuseta kunonoka uye kuyera kumhanya kwekutanga kubatanidzwazve, nezvimwe.
Panguva yekuyedzwa, tinoyera ese ma parameter anogona kuwanikwa nemudziyo:
- DNS nguva yekukumbira;
- TCP yekubatanidza kuseta nguva;
- TLS yekubatanidza kuseta nguva;
- nguva yekugamuchira yekutanga byte yedata;
- nguva yose yekurodha;
- status result code.
- Mushure mekunge mapurusi ese apera, sampuli inotakura zviyero zvese zvekuongorora.
Iwo mapoinzi akakosha anonyanya kutsamira pane logic pane mutengi, kugadzirisa data pane server uye kuyerwa kwezvikumbiro zvakafanana. Nekudaro, isu tinokwanisa kutsaura uye kuyedza kupesvedzera kwezvinhu zvakasiyana zvinokanganisa kuita kwemubvunzo, kusiyanisa mukati meresipi imwe chete, uye kuwana mhinduro kubva kune chaivo vatengi.
Izvi zvivakwa zvakaratidza kubatsira kune zvinopfuura kungobvunza mashandiro ekuita. Parizvino tine 14 mabikirwo anoshanda, anopfuura 6000 samples pasekondi, achigamuchira data kubva kumativi ese epasi uye yakazara mudziyo kufukidzwa. Kana Netflix yakatenga sevhisi yakafanana kubva kune wechitatu bato, yaizodhura mamirioni emadhora pagore, nekuvhara kwakanyanya.
Kuedza dzidziso mukuita: prototype
Nehurongwa hwakadaro, takakwanisa kuongorora kushanda kweCDN proxies pakukumbira latency. Iye zvino unoda:
- gadzira proxy prototype;
- isa prototype paCDN;
- tarisa maitiro ekutungamira vatengi kune proxy pane chaiyo CDN server;
- Enzanisa kuita kune zvikumbiro muAWS pasina proxy.
Basa nderekuongorora kushanda kwemhinduro yakarongwa nekukurumidza sezvinobvira. Isu takasarudza Go kuti tiite prototype nekuda kwekuwanikwa kwemaraibhurari akanaka etiweki. Pane yega yega CDN server, isu takaisa prototype proxy seyakamira bhinari kudzikisa kutsamira uye kurerutsa kubatanidza. Mukutanga kuita, takashandisa zvakajairwa zvikamu zvakanyanya uye zvidiki zvigadziriso zveHTTP/2 yekubatanidza kubatanidza uye kukumbira kuwanda.
Kuenzanisa pakati pematunhu eAWS, takashandisa geographic DNS dhatabhesi, imwecheteyo inoshandiswa kuenzanisa vatengi. Kusarudza sevha yeCDN yemutengi, tinoshandisa TCP Anycast yemaseva muInternet Exchange (IX). Mune iyi sarudzo, tinoshandisa imwe kero yeIP kune ese maCDN maseva, uye mutengi anozotungamirwa kune CDN server ine shoma nhamba yeIP hops. Mune maseva eCDN akaiswa neInternet vanopa (ISPs), isu hatina simba pamusoro peiyo router kugadzirisa TCP Anycast, saka tinoshandisa. , iyo inotungamirira vatengi kune vanopa Indaneti yemavhidhiyo ekutenderera.
Saka, isu tine marudzi matatu enzira dzekukumbira: kune gore kuburikidza neInternet yakavhurika, kuburikidza neCDN server muIX, kana kuburikidza neCDN server iri pane Internet provider. Chinangwa chedu ndechekunzwisisa kuti ndeipi nzira iri nani, uye chii chinobatsira cheproxy, zvichienzaniswa nematumire anoita zvikumbiro kukugadzira. Kuti tiite izvi, tinoshandisa sampling system sezvinotevera:
Imwe neimwe yenzira inova chinangwa chakasiyana, uye tinotarisa panguva yatakawana. Kuti tiongororwe, tinosanganisa zvabuda muboka rimwechete (sarudza nguva yakanaka pakati peIX neISP proxies), tozvienzanisa nenguva yekukumbira kugore pasina mumiriri:
Sezvauri kuona, mhedzisiro yakasanganiswa - kazhinji proxy inopa kukurumidza kukurumidza, asi kunewo nhamba yakakwana yevatengi avo mamiriro acho achawedzera zvakanyanya.
Nekuda kweizvozvo, takaita zvinhu zvakakosha zvakati:
- Isu takaongorora kuita kunotarisirwa kuita kwezvikumbiro kubva kune vatengi kune gore kuburikidza neCDN proxy.
- Takagamuchira data kubva kune vatengi chaivo, kubva kumarudzi ese emidziyo.
- Takaona kuti dzidziso yacho yakanga isiri 100% yakasimbiswa uye yekutanga kupa neCDN proxy yaisazoshanda kwatiri.
- Hatina kutora njodzi - isu hatina kushandura masisitimu ekugadzirwa kwevatengi.
- Hapana chakatyoka.
Prototype 2.0
Saka, dzokera kubhodhi rekudhirowa uye dzokorora maitiro zvakare zvakare.
Pfungwa ndeyekuti pane kushandisa 100% proxy, isu tichaona nzira inokurumidza kumutengi wega wega, uye isu tichatumira zvikumbiro ipapo - ndiko kuti, tichaita izvo zvinonzi mutengi steering.
Nzira yekushandisa sei izvi? Hatigone kushandisa pfungwa padivi reseva, nekuti... Chinangwa ndechekubatanidza kune iyi server. Panofanira kuva neimwe nzira yekuita izvi pane mutengi. Uye zvakanaka, ita izvi nehuwandu hushoma hwepfungwa dzakaoma, kuitira kuti usagadzirise nyaya yekubatanidza nehuwandu hukuru hwemapuratifomu evatengi.
Mhinduro ndeyokushandisa DNS. Muchiitiko chedu, isu tine yedu DNS zvivakwa, uye isu tinokwanisa kumisikidza domain zone iyo maseva edu achave ane masimba. Inoshanda seizvi:
- Mutengi anokumbira kuDNS server achishandisa mugamuchiri, semuenzaniso api.netflix.xom.
- Chikumbiro chinosvika kune yedu DNS server
- Iyo DNS server inoziva kuti ndeipi nzira inokurumidza kumutengi uyu uye inoburitsa inoenderana IP kero.
Mhinduro ine imwezve kuoma: vane masimba DNS vanopa havaone IP kero yemutengi uye vanogona kungoverenga kero yeIP yeanodzokororwa anogadzirisa ayo mutengi anoshandisa.
Nekuda kweizvozvo, mugadziri wedu ane chiremera anofanira kuita sarudzo kwete yemutengi wega, asi yeboka revatengi zvinoenderana neanodzokorora kugadzirisa.
Kuti tigadzirise, tinoshandisa masamples mamwe chetewo, batanidza kuyerwa kwemhedzisiro kubva kune vatengi kune yega yega inodzokororwa inogadzirisa uye tosarudza kwekutumira boka iri - mumiriri kuburikidza neIX uchishandisa TCP Anycast, kuburikidza neISP proxy, kana zvakananga kune gore.
Isu tinowana iyo inotevera system:
Iyo inokonzeresa DNS inotungamira modhi inobvumidza iwe kutungamira vatengi zvichienderana nenhoroondo yekutarisa kumhanya kwekubatanidza kubva kune vatengi kune gore.
Zvekare, mubvunzo ndewekuti nzira iyi ichashanda sei? Kupindura, isu zvakare tinoshandisa yedu probe system. Naizvozvo, isu tinogadzirisa iyo presenter kumisikidza, apo chimwe chezvinangwa zvinotevera gwara kubva kuDNS kutungamira, imwe inoenda yakananga kune gore (ikozvino kugadzirwa).
Nekuda kweizvozvo, tinofananidza zvabuda uye tinowana ongororo yekubudirira:
Somugumisiro, takadzidza zvinhu zvakati zvinokosha:
- Isu takaongorora kuita kunotarisirwa kuita kwezvikumbiro kubva kune vatengi kune gore vachishandisa DNS Steering.
- Takagamuchira data kubva kune vatengi chaivo, kubva kumarudzi ese emidziyo.
- Kushanda kweiyo pfungwa yakarongwa kwakaratidza.
- Hatina kutora njodzi - isu hatina kushandura masisitimu ekugadzirwa kwevatengi.
- Hapana chakatyoka.
Zvino nezve chikamu chakaoma - tinochivhura mukugadzira
Chikamu chiri nyore chapera - kune prototype inoshanda. Ikozvino chikamu chakaoma chiri kuburitsa mhinduro kune ese traffic yeNetflix, ichiendesa kune 150 miriyoni vashandisi, zviuru zvemidziyo, mazana emamicroservices, uye inogara ichichinja chigadzirwa uye zvivakwa. Masevha eNetflix anogashira mamirioni ezvikumbiro pasekondi, uye zviri nyore kutyora sevhisi nekusaita hanya. Panguva imwecheteyo, isu tinoda dynamically nzira traffic kuburikidza nezviuru zveCDN maseva paInternet, apo chimwe chinhu chinoshanduka uye chinoputika nguva dzose uye panguva isina kukodzera.
Uye nezvese izvi, timu ine 3 mainjiniya ane basa rekusimudzira, kutumira uye kutsigirwa kuzere kwehurongwa.
Nokudaro, ticharamba tichitaura nezvekuzorora uye kurara kwakanaka.
Nzira yekuenderera mberi nekusimudzira uye kusapedza nguva yako yese parutsigiro? Maitiro edu akavakirwa pamisimboti mitatu:
- Isu tinoderedza chiyero chinogona kuparara (blast radius).
- Tiri kugadzirira zvinoshamisa - isu tinotarisira kuti chimwe chinhu chichaputsika, kunyangwe kuyedzwa uye ruzivo rwemunhu.
- Kushatisa kwakanaka - kana chimwe chinhu chikasashanda nemazvo, chinofanira kugadziriswa otomatiki, kunyangwe zvisiri nenzira inoshanda.
Zvakazoitika kuti kwatiri, nenzira iyi yedambudziko, tinogona kuwana mhinduro iri nyore uye inoshanda uye nekurerutsa rutsigiro rwehurongwa. Takaona kuti tinogona kuwedzera chidimbu chekodhi kumutengi uye kutarisa kukanganisa kwekukumbira network kunokonzerwa nezvinetso zvekubatanidza. Kana pane kukanganisa kwetiweki, tinoita kudzoka zvakananga kune gore. Iyi mhinduro haidi kushanda nesimba kune zvikwata zvevatengi, asi inoderedza zvakanyanya njodzi yekuparara kusingatarisirwe uye zvinoshamisa kwatiri.
Ehe, kunyangwe nekudonha, isu zvakadaro tinotevera chirango chakajeka panguva yekuvandudza:
- Muenzaniso bvunzo.
- A / B kuyedza kana Canaries.
- Progressive rollout.
Nemasampuli, nzira yacho yakatsanangurwa - shanduko dzinotanga kuedzwa uchishandisa resipi yakasarudzika.
Pakuyedzwa kwecanary, isu tinofanirwa kuwana anofananidzwa mapairi emaseva atinogona kuenzanisa kuti sisitimu inoshanda sei pamberi uye mushure mekuchinja. Kuti tiite izvi, kubva kumasaiti edu mazhinji eCDN, tinosarudza mapairi emaseva anogamuchira traffic yakafanana:
Ipapo isu tinoisa iyo kuvaka pamwe nekuchinja paCanary server. Kuti tiongorore mhedzisiro, isu tinomhanyisa sisitimu inofananidza ingangoita 100-150 metrics nemuenzaniso weKudzora maseva:
Kana kuongorora kweCanary kwakabudirira, saka tinoisunungura zvishoma nezvishoma, mumafungu. Isu hatigadzirise maseva pane yega saiti panguva imwe chete - kurasikirwa nesaiti yese nekuda kwematambudziko kune zvakanyanya kukosha pasevhisi yevashandisi pane kurasikirwa nenhamba yakafanana yemaseva munzvimbo dzakasiyana.
Kazhinji, kushanda uye kuchengeteka kweiyi nzira kunoenderana nehuwandu uye hutano hwema metrics akaunganidzwa. Yedu yemubvunzo yekumhanyisa system, isu tinounganidza metrics kubva kune zvese zvinogoneka zvikamu:
- kubva kune vatengi - nhamba yezvikamu uye zvikumbiro, fallback rates;
- proxy - nhamba yenhamba uye nguva yekukumbira;
- DNS - nhamba uye mhedzisiro yezvikumbiro;
- cloud edge - nhamba uye nguva yekugadzirisa zvikumbiro mugore.
Zvese izvi zvinounganidzwa kuita pombi imwe chete, uye, zvichienderana nezvinodiwa, isu tinosarudza kuti ndeapi metrics ekutumira kune chaiyo-nguva analytics, uye ndeipi kuElasticsearch kana Big Data kune yakadzama diagnostics.
Tinocherechedza
Mune yedu kesi, tiri kuita shanduko munzira yakaoma yekukumbira pakati pemutengi neserver. Panguva imwecheteyo, nhamba yezvikamu zvakasiyana pamutengi, pane sevha, uye munzira kuburikidza neInternet yakakura. Shanduko pamutengi uye sevha dzinoitika nguva dzose - panguva yebasa remakumi ezvikwata uye shanduko yechisikigo mune ecosystem. Isu tiri pakati - pakuongorora matambudziko, pane mukana wakanaka wekuti tichabatanidzwa. Naizvozvo, isu tinofanirwa kunyatsonzwisisa nzira yekutsanangura, kuunganidza uye kuongorora metrics kukurumidza kuparadzanisa matambudziko.
Sezvineiwo, kuwana kuzere kune ese marudzi emetrics uye mafirita munguva chaiyo. Asi kune akawanda metrics, saka mubvunzo wemutengo unomuka. Muchiitiko chedu, tinoparadzanisa metrics uye maturusi ekuvandudza sezvinotevera:
Kuona uye kuongorora matambudziko tinoshandisa yedu yakavhurika sosi chaiyo-nguva system ΠΈ - yekuona. Iyo inochengeta akaunganidzwa metrics mundangariro, inovimbika uye inobatanidza neyeyambiro system. Nekuda kwenzvimbo uye kuongororwa, isu tinokwanisa kuwana matanda kubva kuElasticsearch uye Kibana. Kuongorora nhamba uye modhi, isu tinoshandisa hombe data uye kuona muTableau.
Zvinoratidzika kuti nzira iyi yakaoma zvikuru kushanda nayo. Nekudaro, nekuronga metrics uye zvishandiso zvine hierarchic, isu tinokwanisa kukurumidza kuongorora dambudziko, toona rudzi rwedambudziko, tobva tadzika pasi mune yakadzama metrics. Kazhinji, tinopedza anenge 1-2 maminetsi kuti tione kwainobva kuparara. Mushure meizvi, tinoshanda neboka chairo pakuongorora - kubva pamakumi emaminitsi kusvika kumaawa akati wandei.
Kunyangwe kuongororwa kukaitwa nekukurumidza, hatidi kuti izvi zviitike kazhinji. Sezvineiwo, tinongogashira yambiro yakakosha kana paine zvakakosha pasevhisi. Kune yedu query acceleration system, isu tine 2 chete chenjedzo dzinozozivisa:
- Client Fallback percentage - kuongorora maitiro evatengi;
- muzana Probe zvikanganiso - kugadzikana data yetiweki zvikamu.
Aya machenjedzo akakosha anotarisisa kana sisitimu iri kushanda kune ruzhinji rwevashandisi. Isu tinotarisa kuti vangani vatengi vakashandisa fallback kana vakatadza kuwana chikumbiro chekuwedzera. Isu tinoavhareji isingasviki 1 yakakosha yambiro pasvondo, kunyangwe paine toni yekuchinja kuri kuitika muhurongwa. Sei izvi zvakatikwanira?
- Pane mutengi anodzokera kumashure kana proxy yedu isingashande.
- Pane automatic steering system inopindura matambudziko.
Mamwe mashoko pamusoro pekupedzisira. Yedu yekuyedza sisitimu, uye iyo sisitimu yekuzvisarudzira otomatiki nzira yezvikumbiro kubva kumutengi kuenda kune gore, inotitendera kuti tigadzirise otomatiki mamwe matambudziko.
Ngatidzokei kune yedu sampuli kumisikidza uye matatu nzira zvikamu. Mukuwedzera pakurodha nguva, tinogona kutarisa chokwadi chekuendesa pachako. Kana zvaisaita kurodha data, saka nekutarisa mhedzisiro munzira dzakasiyana tinogona kuona kupi uye chii chakaputsika, uye kana isu tichigona kuzvigadzirisa nekuchinja nzira yekukumbira.
mienzaniso:
Iyi nzira inogona kuve yega. Isanganise muhurongwa hwekutungamirira. Uye dzidzisa kuti ipindure kune mashandiro uye matambudziko ekuvimbika. Kana chimwe chinhu chikatanga kutsemuka, ita kana paine sarudzo iri nani. Panguva imwecheteyo, kukurumidza kuita hakusi kutsoropodza, nekuda kwekudzokera kumashure kune vatengi.
Saka, misimboti yekutsigira system inogona kuumbwa seinotevera:
- kuderedza chiyero chekuputsika;
- kuunganidza metrics;
- Isu tinozvigadzirisa zvinoparara kana tichikwanisa;
- kana zvisingakwanisi, tinokuzivisa;
- Tiri kushanda pama dashboards uye triage tool kuti tipindure nekukurumidza.
Zvidzidzo Zvakadzidzwa
Hazvitore nguva yakawanda kunyora prototype. Muchiitiko chedu, yakanga yakagadzirira mushure memwedzi mina. Nayo takagamuchira metrics matsva, uye mwedzi gumi kubva pakatanga budiriro takagamuchira yekutanga kugadzira traffic. Ipapo basa rinonetesa uye rakaoma kwazvo rakatanga: zvishoma nezvishoma gadzira uye kuyera sisitimu, fambisa iyo huru traffic uye dzidza kubva mukukanganisa. Nekudaro, iyi inoshanda maitiro haizove mutsara - kunyangwe zvese kuedza, zvese hazvigone kufanotaurwa. Izvo zvakanyanya kushanda kukurumidza kudzokorora uye kupindura kune nyowani data.
Kubva pane zvakaitika kwatiri, tinogona kukurudzira zvinotevera:
- Usavimbe neruzivo rwako.
Yedu intuition yakatitadza nguva dzose, kunyangwe ruzivo rwakakura rwenhengo dzechikwata chedu. Semuenzaniso, isu takafanotaura zvisizvo iyo inotarisirwa kukurumidza kubva pakushandisa CDN proxy, kana maitiro eTCP Anycast.
- Wana data kubva mukugadzira.
Izvo zvakakosha kuwana mukana kune kanenge kadiki diki data rekugadzira nekukurumidza sezvinobvira. Zvinenge zvisingaite kuwana huwandu hwemakesi akasiyana, zvigadziriso, uye zvigadziriso mumamiriro erabhoritari. Kukurumidza kuwana mibairo kuchakubvumidza iwe kukurumidza kudzidza pamusoro pezvingangoitika matambudziko uye uzvifungidzire muhurongwa hwekuvaka.
- Usateedzere mazano evamwe vanhu uye mhedzisiro - unganidza yako wega data.
Tevedzera misimboti yekuunganidza nekuongorora data, asi usabvuma neupofu zvakabuda uye zvirevo zvevamwe vanhu. Iwe chete ndiwe unogona kuziva chaizvo izvo zvinoshanda kune vashandisi vako. Masisitimu ako nevatengi vako vanogona kunge vakasiyana zvakanyanya kubva kune mamwe makambani. Neraki, maturusi ekuongorora ave kuwanikwa uye ari nyore kushandisa. Mhedzisiro yaunowana inogona kunge isiri iyo Netflix, Facebook, Akamai nemamwe makambani anoti. Muchiitiko chedu, kushanda kweTLS, HTTP2 kana nhamba pane zvikumbiro zveDNS zvakasiyana nemigumisiro ye Facebook, Uber, Akamai - nokuti tine michina yakasiyana, vatengi uye data inoyerera.
- Usatevedzera mafashoni zvisina basa uye ongorora kushanda.
Tanga nyore. Zviri nani kugadzira yakapusa sisitimu yekushanda munguva pfupi pane kupedza nguva yakawanda uchigadzira zvinhu zvausingade. Gadzirisa mabasa nematambudziko ane basa zvichienderana nezviyero zvako nemhedzisiro.
- Gadzirira maapplication matsva.
Sezvo zvakaoma kufanotaura matambudziko ose, zvakaoma kufanotaura zvikomborero uye mashandisirwo mberi. Tora cue kubva kutanga - kugona kwavo kuchinjika kune vatengi mamiriro. Muchiitiko chako, unogona kuwana matambudziko matsva nemhinduro dzawo. Muchirongwa chedu, tinoisa chinangwa chekudzikisa kukumbira latency. Nekudaro, panguva yekuongorora uye nhaurirano, takaona kuti isu tinogona zvakare kushandisa maseva eproxy:
- kuenzanisa traffic mumatunhu eAWS uye kuderedza mutengo;
- kutevedzera CDN kugadzikana;
- kugadzirisa DNS;
- kugadzirisa TLS/TCP.
mhedziso
Mumushumo, ndakatsanangura kuti Netflix inogadzirisa sei dambudziko rekumhanyisa zvikumbiro zveInternet pakati pevatengi uye gore. Matorero atinoita data tichishandisa sampling system pane vatengi, uye toshandisa iyo yakaunganidzwa nhoroondo data kufambisa zvikumbiro zvekugadzira kubva kune vatengi kuburikidza nenzira inokurumidza paInternet. Mashandisiro atinoita misimboti yetiweki protocol, yedu CDN zvivakwa, backbone network, uye DNS maseva kuita basa iri.
Nekudaro, mhinduro yedu ingori muenzaniso wekuti isu paNetflix takaita sei system yakadai. Chii chakashanda kwatiri. Chikamu chakashandiswa chemushumo wangu kwauri ndiwo misimboti yekusimudzira nerutsigiro yatinotevera nekuwana mibairo yakanaka.
Mhinduro yedu kune dambudziko inogona kunge isingakodzeri iwe. Nekudaro, iyo dzidziso uye misimboti yekugadzira inoramba iripo, kunyangwe iwe usina yako wega CDN zvivakwa, kana kana ichisiyana zvakanyanya neyedu.
Kukosha kwekumhanya kwezvikumbiro zvebhizinesi zvakare kunoramba kwakakosha. Uye kunyangwe sevhisi yakapusa iwe unofanirwa kuita sarudzo: pakati pevanopa gore, server nzvimbo, CDN uye DNS vanopa. Sarudzo yako ichapesvedzera kushanda kwemibvunzo yeInternet kune vatengi vako. Uye zvakakosha kuti iwe uyere uye unzwisise pesvedzero iyi.
Tanga nemhinduro dzakareruka, funga nezvekuti unoshandura sei chigadzirwa. Dzidza paunenge uchienda uye uvandudze sisitimu yakavakirwa pane data kubva kune vatengi vako, zvivakwa zvako, uye bhizinesi rako. Funga nezve mukana wekuputsika kusingatarisirwi panguva yekugadzira maitiro. Uye ipapo iwe unogona kukurumidzira maitiro ako ekuvandudza, kuvandudza kugadzirisa kwekugadzirisa, kudzivirira husina basa mutoro wekutsigira uye kurara murunyararo.
Gore rino mu online format. Unogona kubvunza mibvunzo kune mumwe wemadzibaba eDevOps, John Willis pachake!
Source: www.habr.com