Hlo Habr!
Nyob rau hauv 2017, thaum lub sij hawm Dub Friday, lub load tau nce los ntawm yuav luag ib thiab ib nrab lub sij hawm, thiab peb cov servers nyob ntawm lawv txwv. Xyoo dhau los, tus naj npawb ntawm cov neeg siv khoom tau nce siab, thiab nws tau pom tseeb tias tsis muaj kev npaj ua ntej, lub platform tsuas yog tsis tuaj yeem tiv taus cov khoom ntawm 2018.
Peb tau teem lub hom phiaj xav tau tshaj plaws: peb xav npaj siab rau txhua yam, txawm tias muaj zog tshaj plaws, nce kev ua haujlwm thiab pib tsim cov peev txheej tshiab ua ntej thoob plaws hauv lub xyoo.
Peb CTO Andrey Chizh () qhia seb peb npaj li cas rau Dub Friday 2018, peb tau ntsuas dab tsi kom tsis txhob ntog, thiab, tau kawg, cov txiaj ntsig ntawm kev npaj ceev faj.
Hnub no kuv xav tham txog kev npaj rau Dub Friday 2018. Vim li cas tam sim no, thaum feem ntau ntawm cov muag khoom loj tom qab peb? Peb pib npaj ib xyoos ua ntej cov xwm txheej loj, thiab dhau los ntawm kev sim thiab qhov yuam kev peb pom qhov kev daws teeb meem zoo. Peb xav kom koj saib xyuas lub caij kub ua ntej thiab tiv thaiv kev dag ntxias uas tuaj yeem tshwm sim thaum lub sijhawm tsis zoo tshaj plaws.
Cov khoom siv yuav muaj txiaj ntsig zoo rau txhua tus neeg uas xav nyem qhov nyiaj siab tshaj plaws los ntawm cov khoom lag luam zoo li no, vim Cov txheej txheem ntawm qhov teeb meem tsis yog qis dua rau sab kev lag luam ntawm no.
Nta ntawm kev khiav tsheb ntawm kev muag khoom loj
Raws li kev ntseeg nrov, Dub Friday tsis yog tsuas yog ib hnub hauv ib xyoos, tab sis yuav luag tag nrho lub lim tiam: thawj qhov kev txo nqi tuaj txog 7-8 hnub ua ntej muag. Lub vev xaib kev lag luam pib loj hlob zoo thoob plaws lub asthiv, nce mus txog nws qhov siab tshaj plaws rau hnub Friday thiab poob qis heev rau hnub Saturday mus rau lub khw cov qib tsis tu ncua.
Qhov no yog ib qho tseem ceeb uas yuav tau xav txog: cov khw muag khoom hauv online tau dhau los ua qhov tshwj xeeb tshaj yog rau txhua qhov "slowdowns" hauv qhov system. Tsis tas li ntawd, peb cov ntawv xov xwm email kuj tau ntsib qhov nce ntxiv ntawm cov ntawv xa tawm.
Nws yog lub tswv yim tseem ceeb rau peb mus txog Dub Friday yam tsis muaj kev sib tsoo, vim tias ... Qhov tseem ceeb tshaj plaws ua haujlwm ntawm cov vev xaib thiab cov ntawv xov xwm hauv khw yog nyob ntawm kev ua haujlwm ntawm lub platform, uas yog:
- Taug qab thiab tshaj tawm cov lus pom zoo ntawm cov khoom,
- Tshaj tawm cov ntaub ntawv ntsig txog (piv txwv li, cov duab ntawm kev tsim cov ntawv pom zoo, xws li cov xub, logos, icons thiab lwm yam khoom pom),
- Muab cov duab ntawm qhov yuav tsum tau muaj (rau cov hom phiaj no peb muaj "ImageResizer" - lub subsystem uas rub tawm cov duab los ntawm lub khw server, compresses rau qhov loj me thiab, los ntawm caching servers, tsim cov duab ntawm qhov xav tau loj rau txhua yam khoom hauv txhua qhov kev pom zoo thaiv).
Qhov tseeb, thaum Dub Friday 2019, cov khoom thauj ntawm cov kev pabcuam tau nce 40%, piv txwv li. tus naj npawb ntawm cov xwm txheej uas Lub Chaw Muag Khoom Rocket tau taug qab thiab cov txheej txheem ntawm cov chaw muag khoom hauv online tau nce los ntawm 5 mus rau 8 txhiab thov ib ob. Vim tias peb tab tom npaj rau kev thauj khoom hnyav dua, peb muaj txoj sia nyob zoo li no yooj yim.
Kev cob qhia dav dav
Dub Friday yog lub sijhawm tsis khoom rau txhua lub khw muag khoom thiab e-lag luam tshwj xeeb. Tus naj npawb ntawm cov neeg siv thiab lawv cov haujlwm nyob rau lub sijhawm no tau nce zuj zus, yog li peb, raws li ib txwm muaj, npaj txhij txhua rau lub sijhawm tsis khoom no. Cia peb ntxiv ntawm no qhov tseeb tias peb muaj ntau lub khw muag khoom online txuas nrog tsis yog hauv Russia nkaus xwb, tab sis kuj nyob hauv Europe, qhov twg qhov kev zoo siab yog ntau dua, thiab peb tau txais qib siab dua li Brazilian series. Yuav ua li cas yuav tsum tau ua kom tag nrho cov loads?
Ua haujlwm nrog servers
Ua ntej, nws yog ib qho tsim nyog yuav tsum paub seb peb xav tau dab tsi los ua kom lub zog server. Twb tau lub Yim Hli, peb tau pib xaj cov servers tshiab tshwj xeeb rau Dub Friday - tag nrho peb tau ntxiv 10 lub tshuab ntxiv. Thaum lub Kaum Ib Hlis lawv tau tag nrho hauv kev sib ntaus sib tua.
Nyob rau tib lub sijhawm, qee lub tshuab tsim tau rov nruab rau siv raws li Daim Ntawv Thov servers. Peb tam sim ntawd npaj lawv los siv cov haujlwm sib txawv: ob qho tib si rau kev tawm tswv yim thiab rau ImageResizer kev pabcuam, yog li ntawd, nyob ntawm seb hom kev thauj khoom, txhua tus tuaj yeem siv rau ib qho ntawm cov haujlwm no. Hauv hom ib txwm muaj, Daim Ntawv Thov thiab ImageResizer servers tau hais meej meej cov haujlwm: cov lus pom zoo yav dhau los, tom kawg muab cov duab rau cov ntawv thiab cov lus pom zoo ntawm cov khw hauv online. Hauv kev npaj rau Dub Friday, nws tau txiav txim siab ua txhua lub hom phiaj ob lub hom phiaj txhawm rau txhawm rau sib npaug ntawm lawv nyob ntawm seb hom rub tawm.
Tom qab ntawd peb ntxiv ob lub servers loj rau Kafka (Apache Kafka) thiab tau txais ib pawg ntawm 5 lub tshuab muaj zog. Hmoov tsis zoo, txhua yam tsis mus raws li qhov peb xav tau: thaum lub sijhawm cov ntaub ntawv synchronization txheej txheem, ob lub tshuab tshiab tau tuav tag nrho qhov dav ntawm lub network channel, thiab peb yuav tsum ceev nrooj xyuas seb yuav ua li cas thiaj li ua tiav cov txheej txheem ntxiv sai thiab nyab xeeb rau lub tag nrho cov infrastructure. Txhawm rau daws qhov teeb meem no, peb cov thawj coj yuav tsum tau ua siab loj txi lawv cov hnub so.
Ua haujlwm nrog cov ntaub ntawv
Ntxiv rau cov servers, peb tau txiav txim siab los txhim kho cov ntaub ntawv kom ua kom lub nra hnyav thiab ib kauj ruam loj rau peb yog kev txhais cov ntaub ntawv zoo li qub. Tag nrho cov ntaub ntawv zoo li qub uas tau tuav yav dhau los ntawm cov servers tau tsiv mus rau S3 + Cloudfront. Peb tau xav ua qhov no tau ntev, txij li lub sijhawm thauj khoom ntawm lub server tau ze rau qhov txwv qhov tseem ceeb, thiab tam sim no lub sijhawm zoo tau tshwm sim.
Ib lub lis piam ua ntej Dub Friday, peb tau nce cov duab caching lub sij hawm mus rau 3 hnub, yog li ntawd yog ImageResizer poob lawm, yav tas los cached dluab yuav tsum tau retrieve los ntawm lub cdn. Nws kuj txo cov load ntawm peb cov servers, txij li cov duab khaws cia ntev dua, qhov tsawg dua peb yuav tsum siv cov peev txheej ntawm kev hloov pauv.
Thiab qhov kawg tab sis tsis tsawg kawg: 5 hnub ua ntej Hnub Dub Friday, qhov kev ncua sij hawm tau tshaj tawm ntawm kev xa tawm ntawm ib qho kev ua haujlwm tshiab, nrog rau kev ua haujlwm nrog kev tsim kho vaj tse - tag nrho cov kev saib xyuas yog tsom rau kev tiv thaiv kev nce ntxiv.
Cov phiaj xwm los teb rau cov xwm txheej nyuaj
Txawm hais tias qhov kev npaj tau zoo npaum li cas, fakaps yeej ib txwm ua tau. Thiab peb tau tsim 3 cov phiaj xwm teb rau cov xwm txheej tseem ceeb:
- txo load,
- cuam ββtshuam qee qhov kev pabcuam,
- ua tiav kev kaw haujlwm.
Plan A: Txo cov load. Yuav tsum tau qhib yog tias, vim muaj kev nce siab hauv kev thauj khoom, peb cov servers mus dhau lub sijhawm teb tau txais. Hauv qhov no, peb tau npaj cov txheej txheem rau maj mam txo cov load los ntawm kev hloov ib feem ntawm kev khiav mus rau Amazon servers, uas tsuas yog teb rau txhua qhov kev thov nrog "200 OK" thiab muab cov lus teb khoob. Peb nkag siab tias qhov no yog qhov kev puas tsuaj ntawm cov kev pabcuam zoo, tab sis qhov kev xaiv ntawm qhov tseeb tias qhov kev pabcuam tsis ua haujlwm tag nrho lossis tsis qhia cov lus pom zoo rau kwv yees li 10% ntawm kev khiav tsheb yog pom tseeb.
Plan B: Disable services. Implied ib nrab degradation ntawm kev pabcuam. Piv txwv li, txo qhov ceev ntawm kev xam tus kheej cov lus pom zoo txhawm rau txhawm rau tshem tawm qee cov ntaub ntawv thiab kev sib txuas lus. Hauv hom ib txwm, cov lus pom zoo suav nrog hauv lub sijhawm tiag tiag, tsim kom muaj qhov sib txawv ntawm lub khw hauv online rau txhua tus qhua, tab sis nyob rau hauv cov xwm txheej ntawm kev nce load, txo qhov ceev tso cai rau lwm cov kev pabcuam tseem ceeb txuas ntxiv mus.
Plan C: nyob rau hauv rooj plaub ntawm Armageddon. Yog tias ua tiav qhov system tsis ua haujlwm, peb tau npaj ib txoj kev npaj uas yuav tso cai rau peb kom tsis txhob cuam tshuam nrog peb cov neeg siv khoom. Cov neeg yuav khoom hauv khw yuav tsuas tsis pom cov lus pom zoo; qhov ua tau zoo ntawm lub khw hauv online yuav tsis raug kev txom nyem txhua txoj kev. Ua li no, peb yuav tau rov pib dua peb cov ntaub ntawv sib koom ua ke kom cov neeg siv tshiab yuav tsum tsis txhob cuam tshuam nrog cov kev pabcuam. Ntawd yog, peb yuav lov tes taw peb cov lej taug qab tseem ceeb, cov kev pabcuam yuav tsum tsis txhob sau cov ntaub ntawv thiab suav cov lus pom zoo, thiab tus neeg siv tsuas pom nplooj ntawv yam tsis muaj kev pom zoo. Rau txhua tus neeg uas yav tas los tau txais cov ntaub ntawv sib koom ua ke, peb tau muab cov kev xaiv ntawm kev hloov cov ntaub ntawv DNS rau Amazon thiab 200 OK stub.
Cov txiaj ntsim tau los
Peb tuav tag nrho cov load txawm tias tsis tas yuav siv cov tshuab tsim ntxiv. Thiab ua tsaug rau kev npaj ua ntej, peb tsis xav tau ib qho ntawm cov phiaj xwm teb tau tsim. Tab sis tag nrho cov hauj lwm ua tiav yog invaluable kev uas yuav pab tau peb tiv nrog cov feem ntau npaj txhij txog thiab loj influxes ntawm lub tsheb.
Raws li xyoo 2017, kev thauj khoom ntawm cov kev pabcuam tau nce 40%, thiab cov neeg siv khoom hauv khw hauv online tau nce 60% ntawm Dub Friday. Txhua qhov teeb meem thiab kev ua yuam kev tau tshwm sim thaum lub sijhawm npaj, uas tau cawm peb thiab peb cov neeg siv khoom los ntawm cov xwm txheej tsis tau pom dua.
Koj ua li cas nrog Black Friday? Koj npaj li cas rau kev thauj khoom hnyav?
Tau qhov twg los: www.hab.com