I-Netflix ingumholi emakethe yethelevishini ye-inthanethi - inkampani eyakha futhi ithuthukisa le ngxenye ngenkuthalo. INetflix ayaziwa kuphela ngekhathalogi yayo ebanzi yamamuvi nochungechunge lwe-TV olutholakala cishe kuwo wonke amagumbi omhlaba nanoma iyiphi idivayisi enesibonisi, kodwa futhi nengqalasizinda yayo ethembekile nesiko lobunjiniyela eliyingqayizivele.
Isibonelo esicacile sendlela ye-Netflix yokuthuthukisa nokusekela amasistimu ayinkimbinkimbi yethulwe ku-Devoops 2019 - Umqondisi Wezokuthuthukiswa kwaNetflix. Uthole iziqu ze-Faculty of Computational Mathematics and Mathematics e-Nizhny Novgorod State University. U-Lobachevsky, u-Sergey ongomunye wonjiniyela bokuqala ku-Open Connect - ithimba le-CDN e-Netflix. Wakha amasistimu okuqapha nokuhlaziya idatha yevidiyo, wethula insizakalo edumile yokuhlola isivinini sokuxhuma ku-inthanethi i-FAST.com, futhi eminyakeni embalwa edlule ubesebenza ekuthuthukiseni izicelo ze-inthanethi ukuze uhlelo lokusebenza lweNetflix lusebenze ngokushesha ngangokunokwenzeka kubasebenzisi.
Umbiko uthole ukubuyekezwa okuhle kakhulu okuvela kubahlanganyeli benkomfa, futhi sikulungiselele inguqulo yombhalo.
Embikweni wakhe, uSergei wakhuluma ngokuningiliziwe
- mayelana nokuthi yini ethinta ukubambezeleka kwezicelo ze-inthanethi phakathi kweklayenti neseva;
- indlela yokunciphisa lokhu kubambezeleka;
- indlela yokuklama, ukugcina nokuqapha amasistimu abekezelela amaphutha;
- indlela yokuzuza imiphumela ngesikhathi esifushane, futhi ngengozi encane ebhizinisini;
- indlela yokuhlaziya imiphumela futhi ufunde emaphutheni.
Izimpendulo zale mibuzo azidingeki kuphela yilabo abasebenza ezinkampanini ezinkulu.
Izimiso namasu ethulwe kufanele kwaziwe futhi kwenziwe yiwo wonke umuntu othuthukisa futhi asekele imikhiqizo ye-inthanethi.
Okulandelayo ukulandisa ngokombono wesikhulumi.
Ukubaluleka kwesivinini se-inthanethi
Ijubane lezicelo ze-inthanethi lihlobene ngqo nebhizinisi. Cabanga ngemboni yezitolo: I-Amazon ngo-2009 ukuthi ukubambezeleka okungu-100ms kuholela ekulahlekelweni okungu-1% kokuthengisiwe.
Ayanda amadivaysi eselula, alandelwa izingosi zeselula nezinhlelo zokusebenza. Uma ikhasi lakho lithatha isikhathi esingaphezu kwamasekhondi angu-3 ukulayisha, ulahlekelwa cishe uhhafu wabasebenzisi bakho. NGE I-Google ibheka isivinini sokulayisha sekhasi lakho emiphumeleni yosesho: lapho ikhasi lishesha khona, liphakama isikhundla salo ku-Google.
Isivinini sokuxhuma sibalulekile nasezikhungweni zezezimali lapho ukubambezeleka kubaluleke kakhulu. Ngo-2015, i-Hibernia Networks ikhebula eliyizigidi ezingama-$400 phakathi kweNew York neLondon ukunciphisa ukubambezeleka phakathi kwamadolobha ngo-6ms. Cabanga ngezigidi ezingu-66 zamaRandi nge-1 ms yokunciphisa ukubambezeleka!
Ngokusho , isivinini sokuxhuma esingaphezu kuka-5 Mbit/s asisathinti ngokuqondile isivinini sokulayisha sewebhusayithi evamile. Nokho, kunobudlelwano bomugqa phakathi kokubambezeleka kokuxhumana nesivinini sokulayisha ikhasi:
Kodwa-ke, iNetflix akuwona umkhiqizo ojwayelekile. Umthelela wokubambezeleka nesivinini kumsebenzisi uyindawo esebenzayo yokuhlaziya nokuthuthukiswa. Kukhona ukulayishwa kohlelo lokusebenza nokukhetha kokuqukethwe okuncike ekubambezelekeni, kodwa ukulayisha izici ezimile nokusakaza-bukhoma nakho kuncike esivinini sokuxhuma. Ukuhlaziya nokwenza kahle izici ezibalulekile ezithonya ulwazi lomsebenzisi kuyindawo esebenzayo yokuthuthukiswa kwamaqembu amaningana ku-Netflix. Enye yezinhloso ukunciphisa ukubambezeleka kwezicelo phakathi kwamadivayisi weNetflix nengqalasizinda yamafu.
Embikweni sizogxila ngokukhethekile ekunciphiseni ukubambezeleka sisebenzisa isibonelo sengqalasizinda ye-Netflix. Ake sicabangele ngokombono ongokoqobo ukuthi singabhekana kanjani nezinqubo zokuklama, ukuthuthukiswa kanye nokusebenza kwezinhlelo eziyinkimbinkimbi ezisabalalisiwe futhi sichithe isikhathi emisha kanye nemiphumela, kunokuxilonga izinkinga zokusebenza nokuqhekeka.
Ngaphakathi kweNetflix
Izinkulungwane zamadivayisi ahlukene asekela izinhlelo zokusebenza ze-Netflix. Athuthukiswa amaqembu amane ahlukene, enza izinguqulo ezihlukene zeklayenti le-Android, iOS, TV kanye neziphequluli zewebhu. Futhi sisebenzisa umzamo omkhulu ekuthuthukiseni nasekwenzeni ulwazi lomsebenzisi lube ngokwakho. Ukwenza lokhu, sisebenzisa amakhulukhulu esivivinyo se-A/B ngokuhambisana.
Ukwenza kube ngokwakho kusekelwa amakhulukhulu ezinsiza ezincane efwini le-AWS, ezihlinzeka ngedatha yomuntu siqu yomsebenzisi, ukuthunyelwa kwemibuzo, i-telemetry, Idatha Enkulu kanye Nombhalo Wekhodi. Ukubuka kwethrafikhi kubukeka kanje:
Kwesokunxele indawo yokungena, bese ithrafikhi isatshalaliswa phakathi kwamakhulu amaningana ama-microservices asekelwa amaqembu ahlukene angemuva.
Enye ingxenye ebalulekile yengqalasizinda yethu i-Open Connect CDN, eletha okuqukethwe okumile kumsebenzisi wokugcina - amavidiyo, izithombe, ikhodi yeklayenti, njll. I-CDN itholakala kumaseva angokwezifiso (OCA - Open Connect Appliance). Ngaphakathi kukhona amadrayivu e-SSD kanye ne-HDD asebenzisa i-FreeBSD elungiselelwe, ene-NGINX kanye nesethi yezinsizakalo. Siklama futhi sithuthukise izingxenye zehadiwe nesofthiwe ukuze iseva enjalo ye-CDN ikwazi ukuthumela idatha eningi ngangokunokwenzeka kubasebenzisi.
βUdongaβ lwalawa maseva endaweni yokushintshiselana nge-inthanethi (Internet eXchange - IX) lubukeka kanje:
I-Internet Exchange inikeza ikhono labahlinzeki besevisi ye-inthanethi nabahlinzeki bokuqukethwe ukuthi "baxhumane" ukuze bashintshisane ngokuqondile ngedatha ku-inthanethi. Cishe kunamaphuzu angama-70-80 we-Internet Exchange emhlabeni jikelele lapho amaseva ethu efakwe khona, futhi siyawafaka ngokuzimela futhi siyawanakekela:
Ngaphezu kwalokho, siphinde sinikeze amaseva ngqo kubahlinzeki be-inthanethi, abafaka kunethiwekhi yabo, ukuthuthukisa ukwenziwa kwasendaweni kwethrafikhi ye-Netflix kanye nekhwalithi yokusakaza kubasebenzisi:
Isethi yezinsizakalo ze-AWS inesibopho sokuthumela izicelo zevidiyo ezivela kumakhasimende kuya kumaseva e-CDN, kanye nokumisa amaseva ngokwawo - ukubuyekeza okuqukethwe, ikhodi yohlelo, izilungiselelo, njll. Okokugcina, futhi sakhe inethiwekhi yomgogodla exhuma amaseva kumaphoyinti e-Internet Exchange nge-AWS. Inethiwekhi yomgogodla iyinethiwekhi yomhlaba wonke yezintambo ze-fiber optic namarutha esingawaklama futhi siwalungiselele ngokusekelwe ezidingweni zethu.
Ngu , ingqalasizinda yethu ye-CDN iletha cishe β wethrafikhi ye-inthanethi emhlabeni ngamahora aphakeme kanye β wethrafikhi eNyakatho Melika, lapho i-Netflix isibe khona isikhathi eside kakhulu. Izinombolo ezihlaba umxhwele, kodwa kimina enye yezimpumelelo ezimangalisayo ukuthi lonke uhlelo lwe-CDN luthuthukiswa futhi lugcinwe ithimba labantu abangaphansi kuka-150.
Ekuqaleni, ingqalasizinda ye-CDN yayiklanyelwe ukuletha idatha yevidiyo. Kodwa-ke, ngokuhamba kwesikhathi sabona ukuthi singaphinde siyisebenzise ukukhulisa izicelo eziguqukayo ezivela kumakhasimende efwini le-AWS.
Mayelana nokusheshisa i-inthanethi
Namuhla, i-Netflix inezifunda ezingu-3 ze-AWS, futhi ukubambezeleka kwezicelo efwini kuzoncika ekutheni ikhasimende likude kangakanani nesifunda esiseduze. Ngesikhathi esifanayo, sinamaseva amaningi e-CDN asetshenziselwa ukuletha okuqukethwe okumile. Ingabe ikhona indlela yokusebenzisa lolu hlaka ukusheshisa imibuzo eguquguqukayo? Kodwa-ke, ngeshwa, akunakwenzeka ukugcina lezi zicelo - ama-API enziwe aqondene nawe futhi umphumela ngamunye uhlukile.
Masenze ummeleli kuseva ye-CDN futhi siqale ukuthumela ithrafikhi ngayo. Ingabe izoshesha?
Izinto zokwakha
Masikhumbule ukuthi amaphrothokholi enethiwekhi asebenza kanjani. Namuhla, ithrafikhi eningi ku-inthanethi isebenzisa ama-HTTP, okuncike kumaphrothokholi wesendlalelo esiphansi i-TCP ne-TLS. Ukuze iklayenti lixhume kuseva, libamba isandla, futhi ukuze lisungule uxhumano oluphephile, iklayenti lidinga ukushintshanisa imilayezo neseva izikhathi ezintathu futhi okungenani isikhathi esisodwa sokudlulisa idatha. Ukubambezeleka ngohambo ngalunye lokuya nokubuya (RTT) okungu-100 ms, kuzosithatha u-400 ms ukuthola ingxenye yokuqala yedatha:
Uma sibeka izitifiketi kuseva ye-CDN, khona-ke isikhathi sokuxhawula phakathi kweklayenti neseva singancishiswa kakhulu uma i-CDN iseduze. Ake sicabange ukuthi ukubambezeleka kweseva ye-CDN kungu-30ms. Bese kuzothatha i-220 ms ukuthola ibhithi lokuqala:
Kodwa izinzuzo azigcini lapho. Uma uxhumano selusunguliwe, i-TCP inyusa iwindi lokuminyana (inani lolwazi engalidlulisela kulokho kuxhumana ngokuhambisana). Uma iphakethe ledatha lilahleka, ukusetshenziswa okungajwayelekile kwephrothokholi ye-TCP (njenge-TCP New Reno) kunciphisa βiwindiβ elivuliwe ngohhafu. Ukukhula kwewindi lokuminyana, kanye nesivinini sokululama kwalo ekulahlekeni futhi kuncike ekubambezelekeni (RTT) kuseva. Uma lokhu kuxhumana kufinyelela kuphela kuseva ye-CDN, lokhu kuthola kuzoshesha. Ngesikhathi esifanayo, ukulahlekelwa kwephakethe kuyinto evamile, ikakhulukazi kumanethiwekhi angenantambo.
Umkhawulokudonsa we-inthanethi ungancishiswa, ikakhulukazi phakathi namahora aphakeme, ngenxa yethrafikhi evela kubasebenzisi, okungaholela esiminyaminyeni sezimoto. Nokho, ayikho indlela ku-inthanethi yokubeka kuqala ezinye izicelo kunezinye. Isibonelo, nika kuqala izicelo ezincane nezizwela ukubambezeleka ngaphezu kokusakaza kwedatha "esindayo" elayisha inethiwekhi. Kodwa-ke, esimweni sethu, ukuba nenethiwekhi yethu yomgogodla kusivumela ukuthi senze lokhu engxenyeni yendlela yokucela - phakathi kwe-CDN nefu, futhi singayilungisa ngokugcwele. Ungaqinisekisa ukuthi amaphakethe amancane nazwelayo ukubambezeleka abekwa kuqala, futhi ukugeleza kwedatha okukhulu kuhamba ngokuhamba kwesikhathi. Lapho i-CDN isondelana neklayenti, kulapho kusebenza kahle kakhulu.
Imithetho yezinga lohlelo lokusebenza (OSI Level 7) nayo inomthelela ekubambezelekeni. Izivumelwano ezintsha ezifana ne-HTTP/2 zithuthukisa ukusebenza kwezicelo ezifanayo. Kodwa-ke, sinamakhasimende eNetflix anamadivayisi amadala angawasekeli amaphrothokholi amasha. Akuwona wonke amaklayenti angabuyekezwa noma alungiselelwe kahle. Ngesikhathi esifanayo, phakathi kommeleli we-CDN nefu kukhona ukulawula okuphelele kanye nekhono lokusebenzisa amaphrothokholi amasha, alungile kanye nezilungiselelo. Ingxenye engasebenzi enezivumelwano ezindala izosebenza kuphela phakathi kweklayenti neseva ye-CDN. Ngaphezu kwalokho, singenza izicelo ze-multiplex ekuxhumekeni osekusunguliwe phakathi kwe-CDN namafu, sithuthukise ukusetshenziswa koxhumano ezingeni le-TCP:
Siyalinganisa
Naphezu kweqiniso lokuthi ithiyori ithembisa ukuthuthuka, asisheshi ngokushesha ukwethula uhlelo ekukhiqizeni. Kunalokho, kufanele siqale sifakazele ukuthi umbono uzosebenza ngokusebenza. Ukuze wenze lokhu udinga ukuphendula imibuzo eminingana:
- Isivinini: ingabe ummeleli uzoshesha?
- Ukuthembeka: Ingabe izophuka kaningi?
- Ukuncintisana: ungahlanganisa kanjani nezinhlelo zokusebenza?
- izindleko: Kubiza malini ukuhambisa ingqalasizinda eyengeziwe?
Ake sicabangele ngokuningiliziwe indlela yethu yokuhlola iphuzu lokuqala. Abanye baphathwa ngendlela efanayo.
Ukuze sihlaziye isivinini sezicelo, sifuna ukuthola idatha yabo bonke abasebenzisi, ngaphandle kokuchitha isikhathi esiningi ekuthuthukisweni nangaphandle kokuphula ukukhiqizwa. Kunezindlela eziningana zalokhu:
- I-RUM, noma isilinganiso sesicelo sokwenziwa. Sikala isikhathi sokwenziwa kwezicelo zamanje ezivela kubasebenzisi futhi siqinisekisa ukufakwa okugcwele komsebenzisi. Okubi ukuthi isignali ayizinzile kakhulu ngenxa yezici eziningi, isibonelo, ngenxa yobukhulu bezicelo ezahlukene, isikhathi sokucubungula kuseva kanye neklayenti. Ngaphezu kwalokho, awukwazi ukuhlola ukucushwa okusha ngaphandle komphumela ekukhiqizeni.
- Ukuhlolwa kwaselabhorethri. Amaseva akhethekile nengqalasizinda ezilingisa amakhasimende. Ngosizo lwabo senza izivivinyo ezidingekayo. Ngale ndlela sithola ukulawula okugcwele phezu kwemiphumela yokulinganisa kanye nesignali ecacile. Kodwa akukho ukufakwa okuphelele kwamadivayisi nezindawo zabasebenzisi (ikakhulukazi ngesevisi yomhlaba wonke nosekelo lwezinkulungwane zamamodeli wedivayisi).
Ungazihlanganisa kanjani izinzuzo zazo zombili izindlela?
Ithimba lethu lithole isixazululo. Sibhale ucezu oluncane lwekhodi - isampula - esilakhe kuhlelo lwethu lokusebenza. Ama-Probes asivumela ukuthi senze izivivinyo zenethiwekhi ezilawulwa ngokugcwele kusukela kumadivayisi ethu. Isebenza kanje:
- Ngokushesha ngemva kokulayisha uhlelo lokusebenza nokuqeda umsebenzi wokuqala, siqhuba ama-probe ethu.
- Iklayenti lenza isicelo kuseva futhi lithola βiresiphiβ yokuhlolwa. Iresiphi iwuhlu lwama-URL okudingeka kwenziwe kuwo isicelo se-HTTP. Ngaphezu kwalokho, iresiphi ilungiselela imingcele yesicelo: ukubambezeleka phakathi kwezicelo, inani ledatha eceliwe, izihloko ze-HTTP, njll. Ngesikhathi esifanayo, singahlola izindlela zokupheka eziningana ezihlukene ngokuhambisana - lapho sicela ukucushwa, sinquma ngokungahleliwe ukuthi iyiphi iresiphi okufanele siyikhiphe.
- Isikhathi sokuqalisa uphenyo sikhethiwe ukuze singangqubuzani nokusetshenziswa okusebenzayo kwezinsiza zenethiwekhi kuklayenti. Empeleni, isikhathi sikhethwa lapho iklayenti lingasebenzi.
- Ngemva kokuthola iresiphi, iklayenti lenza izicelo ku-URL ngayinye, ngokuhambisana. Isicelo kumakheli ngalinye singaphindwa - okuthiwa. "ama-pulses". Ku-pulse yokuqala, sikala ukuthi kuthathe isikhathi esingakanani ukusungula uxhumano nokulanda idatha. Ku-pulse yesibili, sikala isikhathi esisithathayo ukulayisha idatha ngoxhumano oseluvele lusunguliwe. Ngaphambi kweyesithathu, singasetha ukubambezeleka futhi silinganise isivinini sokusungula ukuxhuma kabusha, njll.
Ngesikhathi sokuhlolwa, sikala wonke amapharamitha idivayisi engawathola:
- Isikhathi sesicelo se-DNS;
- Isikhathi sokumisa uxhumano lwe-TCP;
- Isikhathi sokumisa uxhumano lwe-TLS;
- isikhathi sokuthola ibhayithi yokuqala yedatha;
- isikhathi sokulayisha esiphelele;
- ikhodi yomphumela wesimo.
- Ngemuva kokuthi wonke ama-pulses eseqedile, isampula ilayisha zonke izilinganiso ukuze zihlaziywe.
Amaphuzu abalulekile ancike kancane kungqondongqondo kuklayenti, ukucutshungulwa kwedatha kuseva kanye nokukalwa kwezicelo ezifanayo. Ngakho-ke, siyakwazi ukuhlukanisa nokuhlola umthelela wezinto ezihlukahlukene ezithinta ukusebenza kombuzo, sizihlukanise phakathi kweresiphi eyodwa, futhi sithole imiphumela kumakhasimende angempela.
Le ngqalasizinda ibonakale iwusizo ekuhlaziyeni ukusebenza kwemibuzo. Njengamanje sinamaresiphi angu-14 asebenzayo, amasampula angaphezu kuka-6000 ngomzuzwana, athola idatha evela kuwo wonke amagumbi omhlaba kanye nokufakwa okugcwele kwedivayisi. Uma iNetflix ithenge insizakalo efanayo kumuntu wesithathu, izobiza izigidi zamaRandi ngonyaka, nekhava elibi kakhulu.
Ithiyori yokuhlola ekusebenzeni: i-prototype
Ngesistimu enjalo, sikwazile ukuhlola ukusebenza kahle kwama-proxies e-CDN ekubambezelekeni kwesicelo. Manje udinga:
- dala i-prototype ye-proxy;
- beka i-prototype ku-CDN;
- thola indlela yokuqondisa amaklayenti kummeleli kuseva ethile ye-CDN;
- Qhathanisa ukusebenza nezicelo ku-AWS ngaphandle kommeleli.
Umsebenzi uwukuhlola ukusebenza kwesixazululo esihlongozwayo ngokushesha ngangokunokwenzeka. Sikhethe i-Go ukuze isebenzise i-prototype ngenxa yokutholakala kwemitapo yolwazi emihle yokuxhumana. Kuseva ngayinye ye-CDN, sifake ummeleli we-prototype njengonambambili omile ukuze sinciphise ukuncika futhi senze ukuhlanganisa kube lula. Ekusetshenzisweni kokuqala, sisebenzise izingxenye ezijwayelekile ngangokunokwenzeka kanye nezinguqulo ezincane zokuhlanganisa i-HTTP/2 nokucela ukuphindaphinda.
Ukuze sibhalansise phakathi kwezifunda ze-AWS, sisebenzise imininingo egciniwe ye-DNS, yona efanayo esetshenziselwa ukulinganisela amaklayenti. Ukukhethela iklayenti iseva ye-CDN, sisebenzisa i-TCP Anycast kumaseva aku-Internet Exchange (IX). Kule nketho, sisebenzisa ikheli le-IP elilodwa kuwo wonke amaseva e-CDN, futhi iklayenti lizoqondiswa kuseva ye-CDN enenombolo encane ye-IP hops. Kumaseva e-CDN afakwe abahlinzeki be-inthanethi (ama-ISP), asinakho ukulawula umzila ukuze simise i-TCP Anycast, ngakho-ke sisebenzisa. , eqondisa amakhasimende kubahlinzeki be-inthanethi ukuze basakaze ividiyo.
Ngakho-ke, sinezinhlobo ezintathu zezindlela zokucela: ukuya emafini nge-inthanethi evulekile, ngeseva ye-CDN ku-IX, noma ngeseva ye-CDN etholakala kumhlinzeki we-inthanethi. Umgomo wethu uwukuqonda ukuthi iyiphi indlela engcono, futhi iyini inzuzo yommeleli, uma kuqhathaniswa nendlela izicelo ezithunyelwa ngayo ekukhiqizeni. Ukwenza lokhu, sisebenzisa isistimu yesampula kanje:
Umzila ngamunye uba yithagethi ehlukile, futhi sibheka isikhathi esisitholile. Ukuze sihlaziywe, sihlanganisa imiphumela yommeleli ibe yiqembu elilodwa (khetha isikhathi esingcono kakhulu phakathi kwama-proxies we-IX ne-ISP), bese uwaqhathanisa nesikhathi sezicelo kumafu ngaphandle kommeleli:
Njengoba ubona, imiphumela yayixubene - ezimweni eziningi ummeleli unikeza isivinini esihle, kodwa kukhona nenani elanele lamakhasimende lapho isimo sizoba sibi kakhulu.
Ngenxa yalokho, senze izinto ezimbalwa ezibalulekile:
- Sihlole ukusebenza okulindelekile kwezicelo ezisuka kumakhasimende kuya efwini sisebenzisa ummeleli we-CDN.
- Sithole idatha evela kumakhasimende wangempela, kuzo zonke izinhlobo zamadivayisi.
- Sibonile ukuthi ithiyori ayizange iqinisekiswe ngo-100% futhi ukunikezwa kokuqala ngommeleli we-CDN ngeke kusisebenzele.
- Asizange sizibeke engcupheni - asizange sishintshe ukulungiselelwa kokukhiqiza kwamaklayenti.
- Akukho okwakuphukile.
I-Prototype 2.0
Ngakho-ke, buyela ebhodini lokudweba bese uphinda inqubo futhi.
Umqondo uwukuthi esikhundleni sokusebenzisa i-proxy engu-100%, sizonquma indlela esheshayo yeklayenti ngalinye, futhi sizothumela izicelo lapho - okungukuthi, sizokwenza lokho okubizwa ngokuthi i-client steering.
Ungakusebenzisa kanjani lokhu? Asikwazi ukusebenzisa ingqondo ohlangothini lweseva, ngoba... Umgomo uwukuxhuma kule seva. Kudingeka kube nendlela ethile yokwenza lokhu kuklayenti. Futhi ngokufanelekile, yenza lokhu ngenani elincane le-logic eyinkimbinkimbi, ukuze ungaxazululi indaba yokuhlanganiswa nenani elikhulu lamapulatifomu amaklayenti.
Impendulo iwukusebenzisa i-DNS. Esimweni sethu, sinengqalasizinda yethu ye-DNS, futhi singasetha indawo yesizinda lapho amaseva ethu azoba negunya. Isebenza kanje:
- Iklayenti lenza isicelo kuseva ye-DNS lisebenzisa umsingathi, isibonelo api.netflix.xom.
- Isicelo sifika kuseva yethu ye-DNS
- Iseva ye-DNS iyazi ukuthi iyiphi indlela eshesha kakhulu kuleli klayenti futhi ikhipha ikheli le-IP elihambisanayo.
Isixazululo sinobunzima obungeziwe: abahlinzeki be-DNS abanegunya abaliboni ikheli lasesizindeni se-inthanethi leklayenti futhi bangafunda kuphela ikheli lasesizindeni se-inthanethi lesixazululi esiphindaphindayo elisetshenziswa iklayenti.
Ngenxa yalokho, umxazululi wethu onegunya kufanele enze isinqumo hhayi seklayenti ngalinye, kodwa seqembu lamaklayenti ngokusekelwe kusixazululi esiphindaphindayo.
Ukuze sixazulule, sisebenzisa amasampuli afanayo, sihlanganise imiphumela yokulinganisa evela kumakhasimende kusixazululi ngasinye esiphindaphindayo futhi sinqume ukuthi silithumela kuphi leli qembu lazo - ummeleli nge-IX sisebenzisa i-TCP Anycast, ngommeleli we-ISP, noma ngokuqondile emafini.
Sithola isistimu elandelayo:
Imodeli yokuqondisa ye-DNS ewumphumela ikuvumela ukuthi uqondise amaklayenti ngokusekelwe ekuqapheliseni okungokomlando kwejubane lokuxhuma kusuka kumakhasimende kuya efwini.
Futhi, umbuzo uwukuthi le ndlela izosebenza ngempumelelo kangakanani? Ukuze siphendule, sisebenzisa futhi uhlelo lwethu lokuhlola. Ngakho-ke, silungiselela ukucushwa komethuli, lapho enye yezinhloso ilandela isiqondiso esivela ku-DNS steering, enye iya ngqo efwini (ukukhiqizwa kwamanje).
Njengomphumela, siqhathanisa imiphumela futhi sithole ukuhlolwa kokusebenza kahle:
Ngenxa yalokho, safunda izinto ezimbalwa ezibalulekile:
- Sihlole ukusebenza okulindelekile kwezicelo ezivela kumakhasimende kuya efwini sisebenzisa i-DNS Steering.
- Sithole idatha evela kumakhasimende wangempela, kuzo zonke izinhlobo zamadivayisi.
- Ukusebenza komqondo ohlongozwayo kufakazelwe.
- Asizange sizibeke engcupheni - asizange sishintshe ukulungiselelwa kokukhiqiza kwamaklayenti.
- Akukho okwakuphukile.
Manje mayelana nengxenye enzima - sethula ekukhiqizeni
Ingxenye elula manje isiphelile - kune-prototype esebenzayo. Manje ingxenye enzima yethula isixazululo sayo yonke i-traffic ye-Netflix, ithumela kubasebenzisi abayizigidi eziyi-150, izinkulungwane zamadivayisi, amakhulukhulu ama-microservices, kanye nomkhiqizo nengqalasizinda eshintsha njalo. Amaseva e-Netflix athola izigidi zezicelo ngomzuzwana, futhi kulula ukuphula isevisi ngesenzo sokunganaki. Ngesikhathi esifanayo, sifuna ukuhambisa ithrafikhi ngamandla ngezinkulungwane zamaseva e-CDN ku-inthanethi, lapho okuthile kushintsha futhi kunqamuke njalo nangesikhathi esingafanelekile.
Futhi ngakho konke lokhu, ithimba linonjiniyela abangu-3 abanomthwalo wemfanelo wokuthuthukisa, ukuthunyelwa nokusekelwa ngokugcwele kohlelo.
Ngakho-ke, sizoqhubeka nokukhuluma ngokulala okuphumula nokunempilo.
Ungaqhubeka kanjani nentuthuko futhi ungachithi sonke isikhathi sakho ekusekelweni? Indlela yethu isuselwe ezimisweni ezi-3:
- Sehlisa isikali esingaba khona sokuqhekeka (i-blast radius).
- Silungiselela izimanga - silindele ukuthi okuthile kuzophuka, naphezu kokuhlolwa kanye nolwazi lomuntu siqu.
- Ukucekelwa phansi okuhle - uma okuthile kungasebenzi kahle, kufanele kulungiswe ngokuzenzakalelayo, noma kungenjalo ngendlela esebenza kahle kakhulu.
Kwavela ukuthi kithi, ngale ndlela yokubhekana nenkinga, singathola isisombululo esilula nesisebenzayo futhi senze lula kakhulu ukusekelwa kwesistimu. Sibonile ukuthi singakwazi ukwengeza ucezu oluncane lwekhodi kuklayenti futhi siqaphe amaphutha esicelo senethiwekhi abangelwa izinkinga zokuxhuma. Esimeni samaphutha enethiwekhi, senza ukubuyela emuva ngqo efwini. Lesi sixazululo asidingi umzamo obalulekile wamathimba amaklayenti, kodwa sinciphisa kakhulu ingcuphe yokuphuka okungalindelekile nezimangaliso kithi.
Yiqiniso, naphezu kokuhlehla, nokho silandela isiyalo esicacile ngesikhathi sokuthuthukiswa:
- Ukuhlolwa kwesampula.
- Ukuhlolwa kwe-A/B noma amaCanaries.
- Ukukhishwa okuqhubekayo.
Ngamasampula, indlela ichaziwe - izinguquko zihlolwa kuqala kusetshenziswa iresiphi eyenziwe ngokwezifiso.
Ukuze kuhlolwe i-canary, sidinga ukuthola amapheya afanayo wamaseva esingaqhathanisa kuwo ukuthi isistimu isebenza kanjani ngaphambi nangemuva kwezinguquko. Ukwenza lokhu, kumasayithi ethu amaningi e-CDN, sikhetha amapheya amaseva athola ithrafikhi efanayo:
Bese sifaka ukwakhiwa ngezinguquko kuseva ye-Canary. Ukuze sihlole imiphumela, sisebenzisa isistimu eqhathanisa cishe ama-metrics angu-100-150 nesampuli yamaseva okulawula:
Uma ukuhlolwa kwe-Canary kuphumelela, khona-ke siyikhulula kancane kancane, ngamagagasi. Asiwabuyekezi amaseva kusayithi ngayinye ngesikhathi esisodwa - ukulahlekelwa isayithi lonke ngenxa yezinkinga kunomthelela obaluleke kakhulu kusevisi kubasebenzisi kunokulahlekelwa inombolo efanayo yamaseva ezindaweni ezahlukene.
Ngokuvamile, ukusebenza kahle nokuphepha kwale ndlela kuncike enanini nekhwalithi yamamethrikhi aqoqiwe. Kusistimu yethu yokusheshisa imibuzo, siqoqa amamethrikhi kuzo zonke izingxenye ezingaba khona:
- kusuka kumakhasimende - inani lezikhathi nezicelo, amanani okubuyela emuva;
- ummeleli - izibalo zenombolo nesikhathi sezicelo;
- I-DNS - inombolo nemiphumela yezicelo;
- cloud edge - inombolo nesikhathi sokucubungula izicelo emafini.
Konke lokhu kuqoqwe kube yipayipi elilodwa, futhi, kuye ngezidingo, sinquma ukuthi imaphi amamethrikhi esizowathumela kuzibalo zesikhathi sangempela, nokuthi yiziphi ku-Elasticsearch noma i-Big Data ukuze uthole ukuxilonga okuningiliziwe.
Siqapha
Esimweni sethu, senza izinguquko endleleni ebalulekile yezicelo phakathi kweklayenti neseva. Ngesikhathi esifanayo, inani lezingxenye ezihlukene kuklayenti, kuseva, nasendleleni ye-inthanethi likhulu kakhulu. Izinguquko kuklayenti neseva zenzeka njalo - phakathi nomsebenzi wamaqembu amaningi kanye nezinguquko zemvelo ku-ecosystem. Siphakathi nendawo - uma sixilonga izinkinga, maningi amathuba okuthi sizobandakanyeka. Ngakho-ke, sidinga ukuqonda ngokucacile ukuthi singawachaza kanjani, siqoqe futhi sihlaziye amamethrikhi ukuze sihlukanise ngokushesha izinkinga.
Ngokufanelekile, ukufinyelela okugcwele kuzo zonke izinhlobo zamamethrikhi nezihlungi ngesikhathi sangempela. Kodwa kunamamethrikhi amaningi, ngakho-ke umbuzo wezindleko uyavela. Esimweni sethu, sihlukanisa amamethrikhi namathuluzi okuthuthukisa ngale ndlela elandelayo:
Ukuze sithole futhi sihlole izinkinga sisebenzisa isistimu yethu yesikhathi sangempela yomthombo ovulekile ΠΈ - ukuze ubone ngeso lengqondo. Igcina ama-metrics ahlanganisiwe enkumbulweni, ithembekile futhi ihlanganisa nesistimu yokuxwayisa. Ngokwenziwa kwasendaweni nokuxilonga, siyakwazi ukufinyelela amalogi asuka ku-Elasticsearch nase-Kibana. Ukuhlaziya izibalo nokumodela, sisebenzisa idatha enkulu kanye nokuboniswa ku-Tableau.
Kubonakala sengathi le ndlela inzima kakhulu ukusebenza nayo. Nokho, ngokuhlela amamethrikhi namathuluzi ngokohlelo, singakwazi ukuhlaziya inkinga ngokushesha, sinqume uhlobo lwenkinga, bese singena kumamethrikhi anemininingwane. Ngokuvamile, sichitha cishe amaminithi angu-1-2 ukukhomba umthombo wokuphazamiseka. Ngemva kwalokhu, sisebenza nethimba elithile ekuxilongeni - kusukela emashumini emizuzu kuya emahoreni ambalwa.
Noma ngabe ukuxilonga kwenziwa ngokushesha, asifuni ukuthi lokhu kwenzeke njalo. Ngokufanelekile, sizothola kuphela isexwayiso esibucayi uma kuba nomthelela omkhulu kusevisi. Kuhlelo lwethu lokusheshisa imibuzo, sinezaziso ezi-2 kuphela ezizokwazisa:
- Iphesenti le-Client Fallback - ukuhlolwa kokuziphatha kwekhasimende;
- iphesenti Amaphutha e-Probe - idatha yokuzinza yezingxenye zenethiwekhi.
Lezi zexwayiso ezibucayi ziqapha ukuthi ingabe isistimu isebenzela iningi labasebenzisi. Sibheka ukuthi mangaki amaklayenti asebenzise ukubuyela emuva uma engakwazi ukuthola ukusheshiswa kwesicelo. Silinganisa ngaphansi kwesexwayiso esibalulekile esingu-1 ngeviki, nakuba kukhona izinguquko eziningi ezenzeka ohlelweni. Kungani lokhu kusanele kithi?
- Kukhona ukubuyisela emuva iklayenti uma ummeleli wethu engasebenzi.
- Kukhona i-automatic steering system ephendula izinkinga.
Imininingwane eyengeziwe mayelana nokwakamuva. Isistimu yethu yesilingo, kanye nesistimu yokunquma ngokuzenzakalelayo indlela efanele yezicelo ezisuka kuklayenti eziya efwini, kusivumela ukuthi sibhekane ngokuzenzekelayo nezinkinga ezithile.
Ake sibuyele ekucushweni kwethu okuyisampula nasezigabeni ezingu-3 zendlela. Ngaphezu kokulayisha isikhathi, singabheka iqiniso lokulethwa ngokwalo. Uma kwakungenakwenzeka ukulayisha idatha, khona-ke ngokubheka imiphumela emigwaqweni ehlukene singakwazi ukunquma ukuthi kuphi nokuthi yini ephukile, nokuthi singakwazi yini ukuyilungisa ngokuzenzakalelayo ngokushintsha indlela yokucela.
izibonelo:
Le nqubo ingenziwa ngokuzenzakalelayo. Ifake ohlelweni lokuqondisa. Futhi uyifundise ukuphendula ezinkingeni zokusebenza nokuthembeka. Uma okuthile kuqala ukuphuka, sabela uma kukhona inketho engcono. Ngesikhathi esifanayo, ukusabela okusheshayo akubalulekile, ngenxa yokubuyela emuva kumakhasimende.
Ngakho-ke, izimiso zokusekelwa kwesistimu zingakhiwa ngale ndlela elandelayo:
- ukunciphisa izinga lokuqhekeka;
- ukuqoqa amamethrikhi;
- Sizilungisa ngokuzenzakalelayo ukuphuka uma sikwazi;
- uma ingakwazi, siyakwazisa;
- Sisebenza kumadeshibhodi kanye nesethi yamathuluzi e-triage ukuze siphendule ngokushesha.
Izifundo Ezifundiwe
Akuthathi isikhathi esiningi ukubhala i-prototype. Esimeni sethu, yayilungile ngemva kwezinyanga ezi-4. Ngayo sithole amamethrikhi amasha, futhi ezinyangeni eziyi-10 ngemuva kokuqala kwentuthuko sathola ithrafikhi yokuqala yokukhiqiza. Khona-ke umsebenzi oyisicefe futhi onzima kakhulu waqala: kancane kancane khiqiza futhi ulinganise uhlelo, uthuthele ithrafikhi enkulu futhi ufunde emaphutheni. Kodwa-ke, le nqubo ephumelelayo ngeke ihambisane - naphezu kwayo yonke imizamo, yonke into ayikwazi ukubikezelwa. Kusebenza kakhulu ukuphindaphinda ngokushesha futhi uphendule kudatha entsha.
Ngokusekelwe kokuhlangenwe nakho kwethu, singancoma okulandelayo:
- Ungayethembi intuition yakho.
Ingqondo yethu isihluleke njalo, naphezu kokuhlangenwe nakho okuningi kwamalungu eqembu lethu. Isibonelo, sibikezele ngokungalungile ukusheshisa okulindelekile ekusebenziseni ummeleli we-CDN, noma ukuziphatha kwe-TCP Anycast.
- Thola idatha kusuka ekukhiqizeni.
Kubalulekile ukuthola ukufinyelela okungenani inani elincane ledatha yokukhiqiza ngokushesha ngangokunokwenzeka. Cishe akunakwenzeka ukuthola inani lamacala ahlukile, ukucushwa, nezilungiselelo ezimeni zaselabhorethri. Ukufinyelela okusheshayo emiphumeleni kuzokuvumela ukuthi ufunde ngokushesha ngezinkinga ezingaba khona futhi uzicabangele ekwakhiweni kwesistimu.
- Ungalandeli iseluleko sabanye abantu nemiphumela - qoqa idatha yakho.
Landela izimiso zokuqoqa nokuhlaziya idatha, kodwa ungamukeli ngokungaboni imiphumela nezitatimende zabanye abantu. Uwena kuphela ongazi kahle ukuthi yini esebenzela abasebenzisi bakho. Amasistimu akho namakhasimende akho angase ahluke kakhulu kwezinye izinkampani. Ngenhlanhla, amathuluzi okuhlaziya aseyatholakala futhi kulula ukuwasebenzisa. Imiphumela oyitholayo ingase ingabi yilokho okufunwa yi-Netflix, Facebook, Akamai nezinye izinkampani. Esimeni sethu, ukusebenza kwe-TLS, HTTP2 noma izibalo zezicelo ze-DNS kuyahluka emiphumeleni ye-Facebook, Uber, Akamai - ngoba sinamadivayisi ahlukene, amakhasimende kanye nokugeleza kwedatha.
- Ungalandeli izitayela zemfashini ngokungadingekile futhi uhlole ukusebenza kahle.
Qala elula. Kungcono ukwenza uhlelo lokusebenza olulula ngesikhathi esifushane kunokuchitha isikhathi esiningi uthuthukisa izingxenye ongazidingi. Xazulula imisebenzi nezinkinga ezibalulekile ngokusekelwe kuzilinganiso zakho nemiphumela.
- Lungela izinhlelo zokusebenza ezintsha.
Njengoba nje kunzima ukubikezela zonke izinkinga, kunzima ukubikezela izinzuzo nezicelo kusengaphambili. Thatha isiqephu kwabasaqalayo - ikhono labo lokuzivumelanisa nezimo zekhasimende. Endabeni yakho, ungathola izinkinga ezintsha nezixazululo zazo. Kuphrojekthi yethu, sizibekela umgomo wokunciphisa ukubambezeleka kwesicelo. Kodwa-ke, ngesikhathi sokuhlaziya nezingxoxo, sabona ukuthi singasebenzisa amaseva abambayo:
- ukulinganisa ithrafikhi kuzo zonke izifunda ze-AWS nokunciphisa izindleko;
- ukwenza imodeli yokuzinza kwe-CDN;
- ukumisa i-DNS;
- ukuze ulungiselele i-TLS/TCP.
isiphetho
Embikweni, ngichaze ukuthi iNetflix iyixazulula kanjani inkinga yokusheshisa izicelo ze-inthanethi phakathi kwamakhasimende namafu. Siqoqa kanjani idatha sisebenzisa isistimu yokusampula kumakhasimende, futhi sisebenzise idatha yomlando eqoqiwe ukuhambisa izicelo zokukhiqiza ezivela kumakhasimende ngendlela eshesha kakhulu ku-inthanethi. Indlela esisebenzisa ngayo imigomo yezivumelwano zenethiwekhi, ingqalasizinda yethu ye-CDN, inethiwekhi yomgogodla, namaseva e-DNS ukuze sifeze lo msebenzi.
Kodwa-ke, isisombululo sethu siyisibonelo sokuthi thina kwaNetflix silusebenzise kanjani uhlelo olunjalo. Okusisebenzele. Ingxenye esetshenzisiwe yombiko wami kuwena yimigomo yentuthuko nokwesekwa esiyilandelayo futhi sizuze imiphumela emihle.
Isixazululo sethu senkinga singase singakufaneli. Nokho, ithiyori kanye nezimiso zokuklama zihlala, ngisho noma ungenayo ingqalasizinda ye-CDN yakho, noma uma ihluke kakhulu kweyethu.
Ukubaluleka kwejubane lezicelo zebhizinisi nakho kuhlala kubalulekile. Futhi ngisho nangesevisi elula udinga ukukhetha: phakathi kwabahlinzeki bamafu, indawo yeseva, abahlinzeki be-CDN kanye ne-DNS. Okukhethayo kuzoba nomthelela ekusebenzeni kwemibuzo ye-inthanethi kumakhasimende akho. Futhi kubalulekile kuwe ukuthi ulinganise futhi uqonde leli thonya.
Qala ngezixazululo ezilula, khathalela ukuthi uwushintsha kanjani umkhiqizo. Funda njengoba uhamba futhi uthuthukise isistimu ngokusekelwe kudatha evela kumakhasimende akho, ingqalasizinda yakho, nebhizinisi lakho. Cabanga ngamathuba okuwohloka okungalindelekile phakathi nenqubo yokuklama. Futhi-ke ungakwazi ukusheshisa inqubo yakho yokuthuthukisa, uthuthukise ukusebenza kahle kwesixazululo, ugweme umthwalo ongadingekile wokusekela futhi ulale ngokuthula.
Lo nyaka ngefomethi ye-inthanethi. Ungabuza imibuzo komunye wobaba be-DevOps, uJohn Willis ngokwakhe!
Source: www.habr.com