Txhua yam zoo!
Kuv lub npe yog Nikita, Kuv yog tus thawj coj ntawm pab pawg Cian engineering. Ib qho ntawm kuv lub luag haujlwm ntawm lub tuam txhab yog txo cov xwm txheej cuam tshuam txog kev tsim kho vaj tse hauv kev tsim khoom mus rau xoom.
Dab tsi yuav tau tham hauv qab no ua rau peb mob siab heev, thiab lub hom phiaj ntawm tsab xov xwm no yog los tiv thaiv lwm tus neeg los ntawm kev rov ua dua peb qhov yuam kev lossis tsawg kawg txo lawv qhov cuam tshuam.
Npaj
Ib lub sij hawm ntev dhau los, thaum Cian muaj cov monoliths, thiab tsis muaj cov lus qhia ntawm microservices, peb tau ntsuas qhov muaj peev xwm los ntawm kev txheeb xyuas 3-5 nplooj ntawv.
Lawv teb - txhua yam zoo, yog tias lawv tsis teb rau lub sijhawm ntev - ceeb toom. Ntev npaum li cas lawv yuav tsum tau tawm haujlwm rau nws kom pom tias qhov xwm txheej tau txiav txim siab los ntawm cov neeg hauv cov rooj sib tham. Ib pab neeg ua haujlwm ib txwm koom nrog hauv kev tshawb nrhiav qhov xwm txheej. Thaum qhov kev tshawb nrhiav tiav lawm, lawv tau sau ib tsab ntawv tshaj tawm - ib hom ntawv ceeb toom los ntawm email hauv hom: dab tsi tshwm sim, nws nyob ntev npaum li cas, peb tau ua dab tsi tam sim no, peb yuav ua dab tsi yav tom ntej.
Cov nplooj ntawv tseem ceeb ntawm lub xaib lossis qhov peb nkag siab tias peb tau ntaus hauv qab
Txhawm rau kom nkag siab qhov tseem ceeb ntawm qhov yuam kev, peb tau txheeb xyuas cov nplooj ntawv tseem ceeb tshaj plaws rau kev ua lag luam. Siv lawv, peb suav cov naj npawb ntawm kev thov ua tiav / ua tsis tiav thiab sijhawm sijhawm. Qhov no yog li cas peb ntsuas uptime.
Cia peb hais tias peb pom tias muaj ntau lub ntsiab lus tseem ceeb ntawm lub xaib uas yog lub luag haujlwm rau cov kev pabcuam tseem ceeb - tshawb nrhiav thiab xa tawm tshaj tawm. Yog tias tus naj npawb ntawm kev thov ua tsis tau ntau dua 1%, qhov no yog qhov teeb meem tseem ceeb. Yog hais tias nyob rau hauv 15 feeb thaum lub sij hawm tseem ceeb tus nqi yuam kev tshaj 0,1%, ces qhov no kuj yog ib qho teeb meem tseem ceeb. Cov txheej txheem no suav nrog feem ntau ntawm cov xwm txheej; tus so yog dhau qhov kev txwv ntawm kab lus no.
Cov xwm txheej zoo tshaj plaws Cian
Yog li, peb tau kawm paub txiav txim siab qhov tseeb tias muaj xwm txheej tshwm sim.
Tam sim no txhua qhov xwm txheej tau piav qhia meej thiab xav txog hauv Jira epic. Los ntawm txoj kev: rau qhov no peb pib ib qhov project cais, hu ua nws FAIL - tsuas yog epics tuaj yeem tsim hauv nws.
Yog tias koj sau tag nrho cov kev ua tsis tiav hauv ob peb xyoos dhau los, cov thawj coj yog:
- mssql ntsig txog xwm txheej;
- xwm txheej tshwm sim los ntawm lwm yam xwm txheej;
- admin yuam kev.
Cia peb saib hauv kev nthuav dav ntxiv ntawm kev ua yuam kev ntawm cov thawj coj, nrog rau qee qhov kev ua tsis tiav zoo.
Qhov thib tsib - "Tshaj cov khoom hauv kev txiav txim hauv DNS"
Nws yog cua daj cua dub hnub Tuesday. Peb txiav txim siab rov qab txiav txim hauv DNS pawg.
Kuv xav hloov cov DNS servers sab hauv los ntawm khi rau powerdns, faib cais cov servers rau qhov no, qhov twg tsis muaj dab tsi tshwj tsis yog DNS.
Peb tau tso ib lub DNS server rau hauv txhua qhov chaw ntawm peb DCs, thiab lub sijhawm los hloov chaw los ntawm khi rau powerdns thiab hloov cov txheej txheem rau cov servers tshiab.
Nyob rau hauv nruab nrab ntawm qhov kev txav mus los, ntawm tag nrho cov servers uas tau teev nyob rau hauv lub zos caching khi rau tag nrho cov servers, tsuas yog ib tug nyob, uas yog nyob rau hauv cov ntaub ntawv chaw nyob rau hauv St. Qhov DC no tau pib tshaj tawm tias tsis yog qhov tseem ceeb rau peb, tab sis tam sim ntawd dhau los ua ib qho kev ua tsis tiav.
Nws yog lub sijhawm no ntawm kev hloov chaw uas cov kwj dej ntawm Moscow thiab St. Petersburg poob. Peb tau raug tso tseg tsis muaj DNS rau tsib feeb thiab tau rov qab thaum tus tswv tsev kho qhov teeb meem.
Cov lus xaus:
Yog tias yav dhau los peb tsis saib xyuas lwm yam hauv lub sijhawm npaj ua haujlwm, tam sim no lawv kuj tau suav nrog hauv cov npe ntawm qhov peb tab tom npaj rau. Thiab tam sim no peb siv zog los xyuas kom meej tias tag nrho cov Cheebtsam tau tshwj tseg n-2, thiab thaum lub sijhawm ua haujlwm peb tuaj yeem txo qib no mus rau n-1.
- Thaum kos cov phiaj xwm nqis tes ua, kos cov ntsiab lus uas qhov kev pabcuam tuaj yeem ua tsis tiav, thiab xav txog qhov xwm txheej uas txhua yam mus "los ntawm qhov phem mus rau qhov phem" ua ntej.
- Tshaj tawm sab hauv DNS servers thoob plaws thaj chaw sib txawv / cov chaw khaws ntaub ntawv / racks / switches / inputs.
- Ntawm txhua tus neeg rau zaub mov, nruab ib lub zos caching DNS server, uas redirects thov mus rau lub ntsiab DNS servers, thiab yog hais tias nws yog tsis muaj, nws yuav teb los ntawm lub cache.
Plaub qhov chaw - "Muab tej yam nyob rau hauv Nginx"
Ib hnub zoo, peb pab neeg tau txiav txim siab tias "peb tau muaj txaus ntawm qhov no," thiab cov txheej txheem ntawm refactoring nginx configs pib. Lub hom phiaj tseem ceeb yog coj cov configs mus rau tus qauv intuitive. Yav dhau los, txhua yam yog "keeb kwm tsim" thiab tsis muaj logic. Tam sim no txhua tus server_name tau raug tsiv mus rau ib cov ntaub ntawv ntawm tib lub npe thiab tag nrho cov configs tau muab faib rau hauv folders. Los ntawm txoj kev, lub config muaj 253949 kab lossis 7836520 cim thiab yuav luag 7 megabytes. Sab saum toj theem ntawm cov qauv:
Nginx qauv
├── access
│ ├── allow.list
...
│ └── whitelist.conf
├── geobase
│ ├── exclude.conf
...
│ └── geo_ip_to_region_id.conf
├── geodb
│ ├── GeoIP.dat
│ ├── GeoIP2-Country.mmdb
│ └── GeoLiteCity.dat
├── inc
│ ├── error.inc
...
│ └── proxy.inc
├── lists.d
│ ├── bot.conf
...
│ ├── dynamic
│ └── geo.conf
├── lua
│ ├── cookie.lua
│ ├── log
│ │ └── log.lua
│ ├── logics
│ │ ├── include.lua
│ │ ├── ...
│ │ └── utils.lua
│ └── prom
│ ├── stats.lua
│ └── stats_prometheus.lua
├── map.d
│ ├── access.conf
│ ├── ..
│ └── zones.conf
├── nginx.conf
├── robots.txt
├── server.d
│ ├── cian.ru
│ │ ├── cian.ru.conf
│ │ ├── ...
│ │ └── my.cian.ru.conf
├── service.d
│ ├── ...
│ └── status.conf
└── upstream.d
├── cian-mcs.conf
├── ...
└── wafserver.confNws tau zoo dua, tab sis nyob rau hauv tus txheej txheem ntawm renaming thiab faib configs, ib txhia ntawm lawv muaj qhov tsis ncaj ncees lawm extension thiab tsis tau suav nrog hauv *.conf cov lus qhia. Yog li ntawd, qee tus tswv tsev tsis muaj thiab xa rov qab 301 mus rau nplooj ntawv tseem ceeb. Vim qhov tseeb tias cov lus teb tsis yog 5xx/4xx, qhov no tsis tau pom tam sim ntawd, tab sis tsuas yog thaum sawv ntxov. Tom qab ntawd, peb pib sau cov ntawv xeem los kuaj xyuas cov khoom siv hauv tsev.
Cov lus xaus:
- Txhim kho koj cov kev teeb tsa kom raug (tsis yog nginx nkaus xwb) thiab xav txog cov qauv ntawm qhov pib ntawm qhov project. Txoj kev no koj yuav ua rau lawv nkag siab ntau dua rau pab pawg, uas nyob rau hauv lem yuav txo TTM.
- Sau cov ntawv xeem rau qee qhov kev tsim kho vaj tse. Piv txwv li: xyuas tias tag nrho cov server_names tseem ceeb muab cov xwm txheej raug + lub cev teb. Nws yuav txaus tsuas yog muaj ob peb tsab ntawv ntawm tes uas kuaj xyuas cov haujlwm tseem ceeb ntawm cov khoom siv, yog li tsis txhob hnov qab frantically thaum 3 teev sawv ntxov dab tsi ntxiv yuav tsum tau kuaj xyuas.
Qhov thib peb - "Dab tsi tshwm sim ntawm qhov chaw hauv Cassandra"
Cov ntaub ntawv tau loj hlob tsis tu ncua, thiab txhua yam zoo mus txog rau lub sijhawm thaum kho cov chaw loj loj pib ua tsis tiav hauv Cassandra pawg, vim tias kev cog lus tsis tuaj yeem ua haujlwm rau lawv.
Ib hnub cua daj cua dub cov pawg yuav luag dhau los ua taub dag, uas yog:
- muaj li ntawm 20% ntawm tag nrho qhov chaw seem nyob rau hauv pawg;
- Nws yog tsis yooj yim sua kom tag nrho cov nodes, vim hais tias kev ntxuav tsis mus tom qab ntxiv ib tug ntawm vim tsis muaj qhov chaw ntawm lub partitions;
- productivity maj mam poob vim compaction tsis ua hauj lwm;
- Cov pawg nyob hauv hom xwm txheej ceev.
Tawm - peb ntxiv 5 qhov ntxiv yam tsis muaj kev ntxuav, tom qab ntawd peb tau pib tshem lawv tawm ntawm pawg thiab rov nkag rau lawv, zoo li cov pob khoob uas tau khiav tawm ntawm qhov chaw. Ntau lub sijhawm tau siv ntau dua li peb xav tau. Muaj kev pheej hmoo ntawm ib feem lossis ua tiav qhov tsis muaj ntawm pawg.
Cov lus xaus:
- Ntawm tag nrho cov cassandra servers, tsis pub ntau tshaj 60% ntawm qhov chaw ntawm txhua qhov kev faib yuav tsum tau nyob.
- Lawv yuav tsum tau loaded ntawm tsis ntau tshaj 50% cpu.
- Koj yuav tsum tsis txhob hnov qab txog kev npaj muaj peev xwm thiab yuav tsum xav txog nws los ntawm txhua yam, raws li nws qhov tshwj xeeb.
- Qhov ntau cov nodes hauv pawg, qhov zoo dua. Cov servers uas muaj me me ntawm cov ntaub ntawv yog overloaded sai, thiab xws li ib pawg yog yooj yim rau revive.
Qhov chaw thib ob - "Cov ntaub ntawv ploj ntawm tus neeg saib xyuas tus nqi tseem ceeb"
Rau kev tshawb nrhiav kev pabcuam, peb, zoo li ntau, siv consul. Tab sis peb kuj siv nws qhov tseem ceeb-tus nqi rau xiav-ntsuab layout ntawm monolith. Nws khaws cov ntaub ntawv hais txog cov dej ntws tsis muaj zog thiab tsis ua haujlwm, uas hloov chaw thaum xa mus. Rau lub hom phiaj no, kev xa tawm kev pabcuam tau sau uas cuam tshuam nrog KV. Qee lub sij hawm, cov ntaub ntawv los ntawm KV ploj lawm. Rov qab los ntawm kev nco, tab sis nrog ntau qhov yuam kev. Yog li ntawd, thaum lub sij hawm upload, lub load ntawm upstreams tau faib unevenly, thiab peb tau txais ntau 502 yuam kev vim lub backends overloaded ntawm lub CPU. Raws li qhov tshwm sim, peb tau tsiv los ntawm consul KV mus rau postgres, los ntawm qhov uas nws tsis yooj yim li tshem tawm lawv.
Cov lus xaus:
- Cov kev pabcuam uas tsis muaj kev tso cai yuav tsum tsis muaj cov ntaub ntawv tseem ceeb rau kev ua haujlwm ntawm lub xaib. Piv txwv li, yog tias koj tsis muaj kev tso cai hauv ES, nws yuav zoo dua rau kev tsis kam nkag mus rau theem network los ntawm txhua qhov chaw uas nws tsis xav tau, tawm tsuas yog qhov tsim nyog, thiab tseem teeb tsa action.destructive_requires_name: tseeb.
- Xyaum koj thaub qab thiab rov qab mechanism ua ntej. Piv txwv li, ua ib tsab ntawv ua ntej (piv txwv li, hauv python) uas tuaj yeem thaub qab thiab kho dua.
Thawj qhov chaw - "Captain Unobvious"
Qee lub sij hawm, peb pom qhov tsis sib xws ntawm kev thauj khoom ntawm nginx upstreams nyob rau hauv rooj plaub uas muaj 10+ servers hauv backend. Vim lub fact tias round-robin xa cov lus thov los ntawm 1st mus rau lub kawg ntws nyob rau hauv kev txiav txim, thiab txhua nginx reload pib dua, thawj upstreams ib txwm tau txais kev thov ntau tshaj li qhov so. Yog li ntawd, lawv ua hauj lwm qeeb thiab tag nrho cov chaw raug kev txom nyem. Qhov no tau dhau los ua qhov pom tseeb vim tias cov tsheb khiav tau nce ntxiv. Tsuas yog hloov kho nginx los pab kom tsis txhob ua haujlwm - peb yuav tsum rov ua dua ib pawg ntawm lua code uas tsis tau tawm ntawm version 1.15 (thaum lub sijhawm ntawd). Peb yuav tsum tau kho peb nginx 1.14.2, qhia txog kev txhawb nqa rau nws. Qhov no daws qhov teeb meem. Kab laum no yeej "Captain Non-Obviousness" qeb.
Cov lus xaus:
Nws yog qhov nthuav heev thiab zoo siab los tshawb txog cov kab no).
- Npaj koj txoj kev saib xyuas kom nws pab koj nrhiav tau cov kev hloov pauv sai sai. Piv txwv li, koj tuaj yeem siv ELK los saib xyuas rps ntawm txhua qhov backend ntawm txhua qhov dej ntws, saib xyuas lawv lub sijhawm teb los ntawm qhov pom ntawm nginx. Hauv qhov no, qhov no tau pab peb txheeb xyuas qhov teeb meem.
Yog li ntawd, feem ntau ntawm kev ua tsis tiav tuaj yeem raug zam nrog txoj hauv kev zoo dua rau qhov koj tau ua. Peb yuav tsum nco ntsoov Murphy txoj cai: Txhua yam uas yuav mus tsis ncaj ncees lawm yuav mus tsis ncaj ncees lawm, thiab tsim cov khoom raws li nws.
Tau qhov twg los: www.hab.com