Failover: perfectionism thiab ... tub nkeeg ua rau peb

Thaum lub caij ntuj sov, ob qho tib si kev yuav khoom thiab kev siv zog ntawm kev hloov pauv hauv cov txheej txheem ntawm cov phiaj xwm hauv web ib txwm txo qis, Tus Thawj Tub Ceev Xwm Qhia rau peb. Tsuas yog vim tias txawm tias cov kws tshaj lij IT qee zaum mus so. Thiab CTO ib yam nkaus. Nws yog txhua yam nyuaj rau cov uas tseem nyob hauv chaw ua haujlwm, tab sis qhov no tsis yog lub ntsiab lus tam sim no: tej zaum yog vim li cas lub caij ntuj sov yog lub sijhawm zoo tshaj plaws los maj mam xav txog cov phiaj xwm tshwj xeeb uas twb muaj lawm thiab teeb tsa lub phiaj xwm los txhim kho nws. Thiab qhov kev paub ntawm Yegor Andreev los ntawm AdminDivision, uas nws tau hais txog ntawm lub rooj sab laj Hnub ua haujlwm.

Muaj ob peb qhov pitfalls koj tuaj yeem poob rau hauv thaum tsim cov chaw thaub qab. Thiab nws yog tsis yooj yim sua kom ntes tau nyob rau hauv lawv. Thiab dab tsi ua rau peb nyob rau hauv tag nrho cov no, xws li nyob rau hauv ntau lwm yam, yog perfectionism thiab ... laziness. Peb tab tom sim ua txhua yam, txhua yam, txhua yam zoo kawg nkaus, tab sis peb tsis tas yuav ua kom zoo kawg nkaus! Koj tsuas yog yuav tsum ua qee yam, tab sis ua kom raug, ua kom tiav lawv kom lawv ua haujlwm zoo.

Failover tsis yog qee yam kev lom zem, kev lom zem; qhov no yog ib yam uas yuav tsum tau ua raws nraim ib yam - txo downtime kom cov kev pab cuam, lub tuam txhab, poob nyiaj tsawg. Thiab nyob rau hauv txhua txoj kev tshwj tseg, kuv xav xav hauv cov ntsiab lus hauv qab no: cov nyiaj nyob qhov twg?

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Thawj cuab: thaum peb tsim cov tshuab loj, txhim khu kev qha thiab koom nrog kev rov ua dua tshiab, peb txo cov xwm txheej. Qhov no yog ib tug txaus ntshai misconception. Thaum peb koom nrog kev rov ua dua, peb yuav ua rau muaj kev sib tsoo ntau ntxiv. Thiab yog tias peb ua txhua yam zoo, tom qab ntawd sib sau ua ke peb yuav txo qis sijhawm. Yuav muaj kev sib tsoo ntau dua, tab sis lawv yuav tshwm sim ntawm tus nqi qis dua. Reservation yog dab tsi? - Qhov no yog qhov teeb meem ntawm qhov system. Txhua qhov teeb meem tsis zoo: peb muaj ntau lub cog, ntau lub zog, hauv ib lo lus, ntau lub ntsiab lus - thiab, yog li ntawd, muaj feem ntau dua ntawm kev tawg. Thiab lawv yeej yuav tawg. Thiab lawv yuav tawg ntau zaus. Ib qho piv txwv yooj yim: cia peb hais tias peb muaj lub vev xaib nrog PHP thiab MySQL. Thiab nws urgently yuav tsum tau tshwj tseg.

Shtosh (c) Peb coj qhov chaw thib ob, tsim kom muaj qhov zoo ib yam... Qhov nyuaj ua ob zaug zoo dua - peb muaj ob qhov chaw. Peb kuj yob tawm ib qho kev xav rau kev xa cov ntaub ntawv los ntawm ib lub xaib mus rau lwm qhov - uas yog, cov ntaub ntawv rov ua dua, luam cov ntaub ntawv zoo li qub, thiab lwm yam. Yog li, cov ntaub ntawv rov ua dua yog feem ntau nyuaj, thiab yog li ntawd, tag nrho cov complexity ntawm lub system yuav tsis yog 2, tab sis 3, 5, 10 lub sij hawm ntau dua.

Thib ob ntxiab: Thaum peb tsim cov txheej txheem loj heev, peb xav txog qhov peb xav tau thaum kawg. Voila: peb xav kom tau txais kev ntseeg siab heev uas ua haujlwm yam tsis muaj lub sijhawm poob qis, hloov mus rau ib nrab thib ob (lossis zoo dua, tam sim ntawd), thiab peb pib ua kev npau suav los muaj tseeb. Tab sis kuj tseem muaj qhov nuance ntawm no: lub sijhawm luv luv ntawm qhov xav tau hloov pauv, qhov nyuaj ntawm qhov system logic ua. Qhov ntau peb yuav tsum ua qhov logic no, ntau zaus lub kaw lus yuav tawg. Thiab koj tuaj yeem xaus rau hauv qhov xwm txheej tsis zoo: peb tab tom sim nrog tag nrho peb lub peev xwm los txo qis sijhawm, tab sis qhov tseeb peb tab tom ua txhua yam nyuaj dua, thiab thaum muaj qee yam tsis zoo, lub sijhawm poob yuav ntev dua. Ntawm no koj feem ntau ntes koj tus kheej xav: zoo ... nws yuav zoo dua tsis ua kom tau txais kev tshwj tseg. Nws yuav zoo dua yog tias nws ua haujlwm ib leeg thiab nrog kev nkag siab tsis zoo.

Koj tuaj yeem sib ntaus li cas? Peb yuav tsum tsum tsis txhob dag rau peb tus kheej, tsis txhob thuam peb tus kheej tias peb yuav tsim lub dav hlau ntawm no tam sim no, tab sis nkag siab txog qhov project yuav dag tau ntev npaum li cas. Thiab rau lub sijhawm siab tshaj plaws no, peb yuav xaiv txoj hauv kev uas peb yuav siv los ua kom muaj kev ntseeg siab ntawm peb lub cev.

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Nws yog lub sij hawm rau "dab neeg ntawm w" ... ntawm lub neej, tau kawg.

Piv txwv naj npawb ib

Xav txog lub vev xaib ua lag luam rau Pipe Rolling Plant No. 1 hauv nroog N. Nws hais tias nyob rau hauv cov ntawv loj - PIPE ROLLING PLANT No. 1. Hauv qab no yog cov lus hais: "Peb cov kav dej yog cov kav dej sib npaug hauv N." Thiab hauv qab no yog CEO tus xov tooj thiab nws lub npe. Peb nkag siab tias koj yuav tsum tau txais kev tshwj tseg - qhov no yog qhov tseem ceeb heev! Cia peb pib paub seb nws muaj dab tsi. Html-statics - uas yog, ob peb daim duab uas tus thawj tswj hwm, qhov tseeb, tab tom tham txog qee yam kev sib tham tom ntej ntawm lub rooj hauv chav da dej nrog nws tus khub. Peb pib xav txog downtime. Nws los txog rau hauv siab: koj yuav tsum tau pw ntawm tsib feeb, tsis muaj ntxiv lawm. Thiab tom qab ntawd cov lus nug tshwm sim: pes tsawg qhov muag tau muaj los ntawm qhov chaw ntawm peb feem ntau? Ntau npaum li cas - ntau npaum li cas? "zero" txhais li cas? Thiab qhov ntawd txhais tau tias: vim hais tias tus thawj coj tau ua tag nrho plaub kev lag luam xyoo tas los ntawm tib lub rooj, nrog tib cov neeg uas lawv mus rau lub tsev da dej thiab zaum ntawm lub rooj. Thiab peb nkag siab tias txawm tias qhov chaw zaum rau ib hnub, tsis muaj dab tsi txaus ntshai yuav tshwm sim.

Raws li cov lus qhia, muaj ib hnub los tsa zaj dab neeg no. Cia peb pib xav txog lub tswv yim redundancy. Thiab peb xaiv qhov zoo tshaj plaws redundancy tswvyim rau qhov piv txwv no: peb tsis siv redundancy. Tag nrho cov khoom no tuaj yeem raug tsa los ntawm ib tus admin hauv ib nrab ib teev nrog cov pa luam yeeb. Nruab ib lub web server, ntxiv cov ntaub ntawv - yog nws. Nws yuav ua haujlwm. Koj tsis tas yuav saib xyuas ib yam dab tsi, koj tsis tas yuav them nyiaj tshwj xeeb rau ib yam dab tsi. Ntawd yog, qhov xaus ntawm qhov piv txwv tus naj npawb ib yog qhov pom tseeb: cov kev pabcuam uas tsis tas yuav tsum tau khaws tseg tsis tas yuav khaws cia.

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Piv txwv li ob

Lub tuam txhab blog: tshwj xeeb tshaj yog cov neeg kawm sau xov xwm nyob rau ntawd, peb tau koom nrog xws li thiab xws li ib qho kev nthuav qhia, tab sis peb tso lwm yam khoom tshiab, thiab lwm yam. Cia peb hais tias qhov no yog tus qauv PHP nrog WordPress, cov ntaub ntawv me me thiab me ntsis zoo li qub. Tau kawg, nws los txog rau hauv siab dua tias koj yuav tsum tsis muaj xwm txheej pw tsaug zog - "tsis pub dhau tsib feeb!" Qhov ntawd yog tag nrho. Tab sis cia peb xav ntxiv. Qhov blog no ua dab tsi? Cov neeg tuaj ntawm Yandex, los ntawm Google raws li qee cov lus nug, organically. Zoo heev. Kev muag khoom puas muaj dab tsi ua nrog nws? Epiphany: tsis yog tiag. Kev tshaj tawm tsheb mus rau qhov chaw tseem ceeb, uas yog nyob rau ntawm lub tshuab sib txawv. Cia peb pib xav txog lub tswv yim booking. Nyob rau hauv txoj kev zoo, nws yuav tsum tau tsa nyob rau hauv ob peb teev, thiab nws yuav zoo los npaj rau qhov no. Nws yuav tsim nyog coj lub tshuab los ntawm lwm qhov chaw khaws ntaub ntawv, yob ib puag ncig mus rau nws, uas yog, lub vev xaib server, PHP, WordPress, MySQL, thiab cia nws nyob ntawd. Thaum lub sijhawm peb nkag siab tias txhua yam tawg, peb yuav tsum ua ob yam - yob tawm lub pob tseg mysql 50 meters, nws yuav ya mus rau hauv ib feeb, thiab yob tawm qee cov duab los ntawm cov thaub qab ntawd. Qhov no kuj tsis muaj rau Vajtswv paub ntev npaum li cas. Yog li ntawd, nyob rau hauv ib nrab ib teev, tag nrho cov khoom nce. Tsis muaj replication, los yog Vajtswv zam txim rau kuv, tsis siv neeg tsis siv neeg. Xaus: dab tsi peb tuaj yeem yob tawm sai sai ntawm qhov thaub qab tsis tas yuav rov qab.

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Piv txwv li peb, nyuaj dua

Lub khw hauv online. PhP nrog qhib lub siab yog me ntsis tweaked, mysql nrog lub hauv paus ruaj khov. Ntau qhov zoo li qub (tom qab tag nrho, lub khw hauv online muaj cov duab HD zoo nkauj thiab txhua yam khoom), Redis rau kev sib kho thiab Elasticsearch rau kev tshawb nrhiav. Peb pib xav txog downtime. Thiab ntawm no, tau kawg, nws pom tseeb tias lub khw hauv online tsis tuaj yeem pw ib puag ncig tsis mob rau ib hnub. Tom qab tag nrho, qhov ntev nws dag, nyiaj ntau peb poob. Nws tsim nyog ua kom nrawm. Ntau npaum li cas? Kuv xav tias yog peb pw ib teev ces tsis muaj leej twg yuav vwm. Yog lawm, peb yuav poob ib yam dab tsi, tab sis yog tias peb pib ua haujlwm hnyav, nws tsuas yog mob hnyav dua. Peb txhais ib lub tswv yim ntawm downtime tso cai rau ib teev.

Yuav ua li cas tag nrho cov no tau tseg? Koj xav tau lub tsheb nyob rau hauv txhua rooj plaub: ib teev ntawm lub sij hawm yog heev me ntsis. Mysql: ntawm no peb twb xav tau replication, nyob replication, vim hais tias nyob rau hauv ib teev 100 GB feem ntau yuav tsis muab ntxiv rau cov pov tseg. Statics, duab: dua, hauv ib teev 500 GB yuav tsis muaj sijhawm ntxiv. Yog li ntawd, nws yog qhov zoo dua los luam cov duab tam sim ntawd. Redis: qhov no yog qhov uas cov khoom tau nthuav. Hauv Redis, kev sib tham tau khaws cia - peb tsis tuaj yeem nqa nws thiab faus nws. Vim tias qhov no yuav tsis zoo heev: txhua tus neeg siv yuav raug rho tawm, lawv cov pob tawb yuav raug muab pov tseg, thiab lwm yam. Tib neeg yuav raug yuam kom rov sau lawv lub npe siv thiab tus password, thiab ntau tus neeg yuav tawg thiab tsis ua tiav qhov kev yuav khoom. Ntxiv dua thiab, kev hloov pauv yuav poob. Ntawm qhov tod tes, Redis yog ncaj qha mus-rau-hnub, nrog cov neeg siv nkag mus dhau los tej zaum tsis xav tau. Thiab qhov kev pom zoo zoo yog coj Redis thiab rov qab los ntawm kev thaub qab nag hmo, lossis, yog tias koj ua nws txhua teev, txij li ib teev dhau los. Hmoov zoo, rov kho nws los ntawm kev thaub qab txhais tau tias luam ib cov ntaub ntawv. Thiab zaj dab neeg nthuav tshaj plaws yog Elasticsearch. Leej twg puas tau khaws MySQL replication? Leej twg puas tau khaws Elasticsearch replication? Thiab rau leej twg tau ua haujlwm ib txwm ua tom qab? Qhov kuv txhais tau li cas yog tias peb pom ib qho chaw hauv peb lub cev. Nws zoo li yuav pab tau - tab sis nws yog complex.
Complex nyob rau hauv lub siab hais tias peb cov phooj ywg engineers tsis muaj kev paub ua hauj lwm nrog nws. Los yog muaj kev tsis zoo. Los yog peb nkag siab tias qhov no tseem yog ib qho tshiab siv tshuab nrog nuances lossis rawness. Peb xav tias... Damn, elastic kuj noj qab nyob zoo, nws kuj siv sij hawm ntev los kho nws los ntawm thaub qab, kuv yuav ua li cas? Peb nkag siab tias elastic hauv peb rooj plaub yog siv rau kev tshawb nrhiav. Peb lub khw hauv online muag li cas? Peb mus rau cov neeg lag luam thiab nug seb cov neeg tuaj ntawm qhov twg. Lawv teb: "90% los ntawm Yandex Market tuaj ncaj qha rau daim npav khoom." Thiab txawm lawv yuav los yog lawv tsis. Yog li, kev tshawb nrhiav yog xav tau los ntawm 10% ntawm cov neeg siv. Thiab ua kom elastic replication, tshwj xeeb tshaj yog ntawm cov chaw sib txawv cov ntaub ntawv nyob rau hauv ib cheeb tsam sib txawv, yeej muaj ntau nuances. Qhov twg tawm? Peb muab elastic los ntawm qhov chaw tshwj tseg thiab tsis ua dab tsi nrog nws. Yog tias qhov teeb meem rub tawm, peb yuav zaum tsa nws ib hnub, tab sis qhov no tsis paub tseeb. Qhov tseeb, qhov xaus yog tib yam, ntxiv lossis rho tawm: peb, dua, tsis txhob khaws cov kev pabcuam uas tsis cuam tshuam txog nyiaj. Txhawm rau khaws daim duab yooj yim dua.

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Piv txwv li plaub, txawm nyuaj dua

Integrator: muag paj, hu rau tsheb tavxij, muag khoom, feem ntau, txhua yam. Ib qho tseem ceeb uas ua haujlwm 24/7 rau ntau tus neeg siv. Nrog rau tag nrho-fledged nthuav pawg, qhov twg muaj kev nthuav hauv paus, kev daws teeb meem, siab load, thiab tseem ceeb tshaj plaws, nws mob siab rau pw rau ntau tshaj 5 feeb. Tsis tas li thiab tsis ntau vim tib neeg yuav tsis yuav, tab sis vim tib neeg yuav pom tias qhov khoom no tsis ua haujlwm, lawv yuav chim siab thiab tsis tuaj yeem rov qab los.

OK. Tsib feeb. Peb yuav ua li cas txog qhov no? Nyob rau hauv cov ntaub ntawv no, peb, zoo li cov neeg laus, siv tag nrho cov nyiaj los tsim ib tug tiag tiag backup site, nrog replication ntawm txhua yam, thiab tej zaum txawm automate hloov mus rau qhov chaw no ntau npaum li sai tau. Thiab ntxiv rau qhov no, koj yuav tsum nco ntsoov ua ib qho tseem ceeb: ua tau, sau cov kev hloov pauv. Cov kev cai, txawm tias koj muaj txhua yam automated, tuaj yeem yooj yim heev. Los ntawm cov series "khiav xws li thiab xws li ib tug ansible tsab ntawv", "nias xws li thiab xws li ib tug checkbox nyob rau hauv txoj kev 53" thiab hais txog - tab sis qhov no yuav tsum yog ib co ntawm cov npe ntawm cov kev ua.

Thiab txhua yam zoo li meej. Kev hloov pauv hloov pauv yog ib txoj haujlwm tsis tseem ceeb, lossis nws yuav hloov nws tus kheej. Rewriting lub npe sau npe hauv DNS yog los ntawm tib lub koob. Qhov teeb meem yog tias thaum xws li ib txoj haujlwm ua tsis tiav, kev ceeb ntshai pib, thiab txawm tias qhov muaj zog tshaj plaws, cov tswj hwm muaj hwj chim tuaj yeem raug rau nws. Yog tsis muaj cov lus qhia meej "qhib lub davhlau ya nyob twg, tuaj ntawm no, peb tus neeg rau zaub mov qhov chaw nyob tseem zoo li no," nws nyuaj rau ua tau raws li 5-feeb lub sijhawm txwv rau kev rov ua haujlwm. Zoo, ntxiv rau, thaum peb siv cov kev cai no, nws yooj yim los sau qee qhov kev hloov pauv hauv cov txheej txheem, piv txwv li, thiab hloov cov kev cai raws li.
Zoo, yog hais tias lub reservation system yog heev complex thiab nyob rau hauv tej lub sij hawm peb ua yuam kev, ces peb yuav rhuav tshem peb backup site, thiab nyob rau hauv tas li ntawd mus tig cov ntaub ntawv mus rau hauv lub taub dag ntawm ob qhov chaw - qhov no yuav tu siab tag nrho.

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Piv txwv li tsib, ua tiav hardcore

Kev pabcuam thoob ntiaj teb nrog ntau pua lab tus neeg siv thoob ntiaj teb. Txhua lub sij hawm ib cheeb tsam muaj, siab load ntawm siab tshaj plaws ceev, koj yuav pw tsis tau tag nrho. Ib feeb - thiab nws yuav tu siab. Yuav ua li cas? Reserve, dua, raws li tag nrho cov kev pab cuam. Peb tau ua txhua yam kuv tau tham txog hauv qhov piv txwv dhau los, thiab me ntsis ntxiv. Lub ntiaj teb zoo tagnrho, thiab peb cov kev tsim kho vaj tse yog raws li txhua lub ntsiab lus ntawm IaaC devops. Ntawd yog, txhua yam nyob hauv git, thiab koj tsuas yog nias lub pob.

Dab tsi yog ploj lawm? Ib - kev ua si. Nws yog tsis yooj yim sua yam tsis muaj lawv. Nws zoo nkaus li tias txhua yam zoo nrog peb, feem ntau peb muaj txhua yam tswj hwm. Peb nias lub pob, txhua yam tshwm sim. Txawm hais tias qhov no yog li ntawd - thiab peb nkag siab tias nws tsis tshwm sim li no - peb lub kaw lus cuam tshuam nrog qee lub tshuab. Piv txwv li, qhov no yog dns los ntawm txoj kev 53, s3 cia, kev koom ua ke nrog qee qhov api. Peb yuav tsis tuaj yeem pom txhua yam hauv qhov kev sim ntsuas no. Thiab kom txog thaum peb rub qhov hloov pauv, peb yuav tsis paub tias nws yuav ua haujlwm lossis tsis ua haujlwm.

Failover: perfectionism thiab ... tub nkeeg ua rau peb

Qhov ntawd yog txhua yam. Tsis txhob ua tub nkeeg lossis overdo nws. Thiab tej zaum yuav muaj sijhawm nrog koj!

Tau qhov twg los: www.hab.com

Ntxiv ib saib