Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architecturalLOST los ntawm sophiagworld

Kab lus no muaj qee cov qauv siv los pab cov engineers ua haujlwm nrog cov kev pabcuam loj uas tau nkag los ntawm ntau lab tus neeg siv. 

Nyob rau hauv tus sau qhov kev paub, qhov no tsis yog ib daim ntawv teev tag nrho, tab sis qhov tseeb siv tau advi ua. Yog li, cia peb pib.

Txhais nrog kev txhawb nqa Mail.ru Huab Solutions.

Thawj qib

Cov kev ntsuas tau teev tseg hauv qab no kuj yooj yim rau kev siv tab sis muaj kev cuam tshuam loj. Yog tias koj tsis tau sim lawv ua ntej, koj yuav xav tsis thoob ntawm qhov kev txhim kho tseem ceeb.

Infrastructure li code

Thawj feem ntawm cov lus qhia yog siv cov infrastructure raws li cov cai. Qhov no txhais tau hais tias koj yuav tsum muaj txoj hauv kev los siv tag nrho cov txheej txheem. Nws suab nyuaj, tab sis peb tab tom tham txog cov cai hauv qab no:

Kev xa tawm ntawm 100 lub tshuab virtual

  • nrog Ubuntu
  • 2 GB RAM txhua
  • lawv yuav muaj cov cai hauv qab no
  • nrog cov kev txwv no

Koj tuaj yeem taug qab cov kev hloov pauv rau koj qhov kev tsim kho vaj tse thiab rov qab mus sai sai rau lawv siv kev tswj hwm version.

Tus neeg niaj hnub hauv kuv hais tias koj tuaj yeem siv Kubernetes / Docker ua txhua yam saum toj no, thiab nws yog.

Tsis tas li ntawd, koj tuaj yeem muab automation siv kws ua zaub mov, Puppet lossis Terraform.

Kev sib xyaw ua ke tsis tu ncua thiab kev xa khoom

Yuav kom tsim tau ib qho kev pab cuam scalable, nws yog ib qho tseem ceeb uas yuav tsum muaj kev tsim thiab sim cov kav dej rau txhua qhov kev thov rub. Txawm hais tias qhov kev sim no yooj yim heev, nws yuav tsawg kawg kom paub meej tias cov cai koj xa tuaj sau ua ke.

Txhua zaus ntawm theem no koj teb cov lus nug: kuv lub rooj sib txoos puas yuav suav thiab xeem dhau, puas siv tau? Qhov no yuav zoo li qhov qis qis, tab sis nws daws tau ntau yam teeb meem.

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Tsis muaj dab tsi zoo nkauj dua li pom cov zuam no

Rau qhov thev naus laus zis no koj tuaj yeem ntsuas Github, CircleCI lossis Jenkins.

Load Balancers

Yog li, peb xav khiav lub load balancer rau redirect tsheb thiab xyuas kom meej sib npaug load ntawm tag nrho cov nodes los yog kev pab cuam txuas ntxiv nyob rau hauv cov ntaub ntawv ntawm tsis ua hauj lwm:

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Lub load balancer feem ntau ua haujlwm zoo ntawm kev faib tsheb. Qhov kev xyaum zoo tshaj plaws yog kev sib npaug kom tsis txhob muaj ib qho kev ua tsis tiav.

Feem ntau, load balancers tau teeb tsa hauv huab koj siv.

RayID, correlation ID lossis UUID rau kev thov

Koj puas tau ntsib ib daim ntawv thov yuam kev nrog cov lus zoo li no: "Ib yam dab tsi mus tsis ncaj ncees lawm. Txuag tus ID no thiab xa mus rau peb pab neeg txhawb nqa"?

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Ib tus cim tshwj xeeb, tus ID sib raug zoo, RayID, lossis ib qho ntawm cov kev hloov pauv, yog tus cim tshwj xeeb uas tso cai rau koj taug qab qhov kev thov thoob plaws hauv nws lub neej. Qhov no tso cai rau koj taug qab txoj kev thov tag nrho hauv cov cav.

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Tus neeg siv ua qhov kev thov rau system A, ces A hu B, uas hu rau C, khaws cia hauv X, thiab tom qab ntawd qhov kev thov rov qab mus rau A

Yog tias koj yuav tsum tau txuas mus rau lub tshuab virtual thiab sim taug qab txoj kev thov (thiab manually sib cuam tshuam cov hu xov tooj), koj yuav vwm. Muaj tus cim tshwj xeeb ua rau lub neej yooj yim dua. Qhov no yog ib qho yooj yim tshaj plaws uas koj tuaj yeem ua kom txuag tau sijhawm thaum koj qhov kev pabcuam loj tuaj.

Qib Nruab Nrab

Cov lus qhia ntawm no yog qhov nyuaj dua li yav dhau los, tab sis cov cuab yeej tsim nyog ua rau txoj haujlwm yooj yim dua, muab kev rov qab los ntawm kev nqis peev txawm rau cov tuam txhab me thiab nruab nrab.

Centralized txiav

Nrog koj zoo siab! Koj tau siv 100 lub tshuab virtual. Hnub tom qab, CEO tuaj thiab yws txog qhov yuam kev uas nws tau txais thaum sim cov kev pabcuam. Nws qhia txog tus lej ID uas peb tau tham txog saum toj no, tab sis koj yuav tsum tau saib los ntawm cov cav ntawm 100 lub tshuab kom pom qhov uas ua rau muaj kev sib tsoo. Thiab nws yuav tsum tau pom ua ntej tag kis kev nthuav qhia.

Txawm hais tias qhov no zoo li kev lom zem taug txuj kev nyuaj, nws yog qhov zoo tshaj kom paub tseeb tias koj muaj peev xwm tshawb nrhiav tag nrho cov ntawv xov xwm hauv ib qho chaw. Kuv daws qhov teeb meem ntawm centralizing cav siv lub built-in functionality ntawm ELK pawg: nws txhawb searchable log sau. Qhov no yuav pab daws qhov teeb meem ntawm kev nrhiav ib phau ntawv tshwj xeeb. Raws li qhov nyiaj tau los, koj tuaj yeem tsim cov kab kos thiab lwm yam kev lom zem zoo li ntawd.

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
ELK pawg ua haujlwm

Cov neeg saib xyuas

Tam sim no tias koj qhov kev pabcuam tau nce thiab ua haujlwm, koj yuav tsum ua kom nws ua haujlwm zoo. Txoj kev zoo tshaj los ua qhov no yog khiav ob peb tus neeg sawv cev, uas ua hauj lwm nyob rau hauv parallel thiab xyuas tias nws ua hauj lwm thiab cov hauj lwm yooj yim yog ua.

Hauv qhov no koj tuaj yeem tshawb xyuas qhov ntawd qhov kev ua haujlwm khiav zoo li zoo thiab ua haujlwm zoo.

Rau cov haujlwm me thiab nruab nrab, kuv pom zoo kom Postman saib xyuas thiab sau cov ntaub ntawv APIs. Tab sis feem ntau, koj tsuas yog xav kom paub tseeb tias koj muaj txoj hauv kev paub thaum twg muaj kev puas tsuaj tshwm sim thiab raug ceeb toom raws sijhawm.

Autoscaling nyob ntawm load

Nws yooj yim heev. Yog tias koj muaj VM kev pabcuam thov thiab nws nce mus txog 80% kev siv nco, koj tuaj yeem nce nws cov peev txheej lossis ntxiv VMs ntxiv rau pawg. Tsis siv neeg ua haujlwm ntawm cov haujlwm no yog qhov zoo heev rau kev hloov pauv hluav taws xob hauv qab load. Tab sis koj yuav tsum tau ceev faj txog cov nyiaj koj siv nyiaj ntau npaum li cas thiab teeb tsa cov kev txwv tsim nyog.

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Nrog rau feem ntau cov kev pabcuam huab, koj tuaj yeem teeb tsa nws rau autoscale siv ntau cov servers lossis cov servers muaj zog dua.

Kev sim system

Ib txoj hauv kev zoo kom muaj kev nyab xeeb yob tawm qhov hloov tshiab yog tuaj yeem sim qee yam rau 1% ntawm cov neeg siv rau ib teev. Koj tau, tau kawg, pom cov mechanisms hauv kev nqis tes ua. Piv txwv li, Facebook qhia qee qhov ntawm cov neeg tuaj saib sib txawv xim lossis hloov cov font loj kom pom tias cov neeg siv pom cov kev hloov pauv li cas. Qhov no hu ua A/B testing.

Txawm tso tawm qhov tshiab tuaj yeem pib ua qhov kev sim thiab tom qab ntawd txiav txim siab yuav tso nws li cas. Koj kuj tau txais lub peev xwm los "nco" lossis hloov qhov kev teeb tsa ntawm yoov raws li kev ua haujlwm uas ua rau muaj kev puas tsuaj hauv koj qhov kev pabcuam.

Qib siab heev

Nov yog cov lus qhia uas nyuaj heev rau kev siv. Tej zaum koj yuav xav tau kev pab me ntsis ntxiv, yog li lub tuam txhab me lossis nruab nrab yuav muaj lub sijhawm nyuaj los tswj qhov no.

Blue-ntsuab deployments

Qhov no yog qhov kuv hu ua "Erlang" txoj kev nthuav tawm. Erlang tau siv dav thaum cov tuam txhab xov tooj tshwm sim. Softswitches tau pib siv los xa xov tooj. Lub hom phiaj tseem ceeb ntawm software ntawm cov keyboards no yog kom tsis txhob poob hu thaum hloov kho qhov system. Erlang muaj txoj hauv kev zoo ntawm kev thauj khoom module tshiab yam tsis muaj kev cuam tshuam yav dhau los.

Cov kauj ruam no yog nyob ntawm qhov muaj cov load balancer. Cia peb xav txog tias koj muaj version N ntawm koj lub software, thiab tom qab ntawd koj xav xa cov version N + 1. 

Ces peb ua tau cia li nres qhov kev pab cuam thiab yob tawm lub tom ntej version ntawm lub sij hawm uas ua hauj lwm rau koj cov neeg siv thiab tau txais ib co downtime. Tab sis xav tias koj muaj tiag tiag nruj SLA tej yam kev mob. Yog li, SLA 99,99% txhais tau tias koj tuaj yeem mus offline tsuas los ntawm 52 feeb hauv ib xyoos.

Yog tias koj xav ua tiav cov ntsuas no, koj xav tau ob qho kev xa tawm tib lub sijhawm: 

  • tus uas yog tam sim no (N);
  • tom ntej no version (N+1). 

Koj qhia tus load balancer redirect ib feem pua ​​ntawm cov tsheb mus rau lub tshiab version (N + 1) thaum koj nquag saib xyuas rau regressions.

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Ntawm no peb muaj ntsuab N xa tawm uas ua haujlwm zoo. Peb tab tom sim txav mus rau qhov txuas ntxiv ntawm qhov kev xa tawm no

Ua ntej peb xa ib qho kev sim me me kom pom tias peb qhov N + 1 xa mus ua haujlwm nrog cov tsheb me me:

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Thaum kawg, peb muaj cov txheej txheem ntawm kev txheeb xyuas qhov tseeb uas peb tau khiav mus txog thaum peb ua tiav. Yog koj heev heev ceev faj, koj tseem tuaj yeem txuag koj N xa mus mus ib txhis rau kev rov qab sai thaum muaj kev rov qab tsis zoo:

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Yog tias koj xav mus rau qib siab dua, cia txhua yam hauv xiav-ntsuab xa mus ua haujlwm tau txais.

Anomaly nrhiav kom tau thiab tsis siv neeg mitigation

Muab hais tias koj muaj centralized txiav thiab sau cov cav zoo, koj tuaj yeem teem lub hom phiaj siab dua. Piv txwv li, proactively kwv yees ua tsis tiav. Kev ua haujlwm tau taug qab ntawm cov saib thiab hauv cov cav thiab ntau cov duab kos tau tsim - thiab koj tuaj yeem kwv yees ua ntej dab tsi yuav mus tsis ncaj ncees lawm:

Yuav ua li cas kom tsaug zog zoo thaum koj muaj kev pabcuam huab: cov lus qhia yooj yim architectural
Thaum kuaj pom muaj qhov tsis txaus ntseeg, koj pib tshuaj xyuas qee qhov kev qhia uas qhov kev pabcuam muab. Piv txwv li, qhov nce siab hauv CPU load yuav qhia tau tias lub hard drive tsis ua haujlwm, thaum qhov kev thov nce siab yuav qhia tau tias koj yuav tsum tau nce. Cov ntaub ntawv txheeb cais zoo li no tso cai rau koj los ua qhov kev pabcuam ua haujlwm.

Nrog rau cov kev nkag siab no, koj tuaj yeem teev nyob rau hauv txhua qhov ntev thiab ua haujlwm thiab hloov pauv cov yam ntxwv ntawm cov tshuab, databases, kev sib txuas thiab lwm yam kev pabcuam.

Yog tag nrho!

Cov npe ntawm qhov tseem ceeb no yuav cawm koj ntau yam teeb meem yog tias koj tab tom nrhiav kev pabcuam huab.

Tus sau thawj tsab xov xwm caw cov neeg nyeem tawm lawv cov lus thiab hloov pauv. Cov kab lus tau muab faib ua qhov qhib, rub cov lus thov los ntawm tus sau lees txais ntawm Github.

Lwm yam uas yuav tsum tau nyeem ntawm lub ncauj lus:

  1. Mus thiab CPU caches
  2. Kubernetes nyob rau hauv lub siab ntawm piracy nrog ib tug qauv rau kev siv
  3. Peb channel ncig Kubernetes hauv Telegram

Tau qhov twg los: www.hab.com

Ntxiv ib saib