Top fakapov Cyan

Shuwiro yakanakisa! 

Ini ndinonzi Nikita, ndiri mutungamiri wechikwata cheCian engineering timu. Rimwe remabasa angu kukambani kudzikisa huwandu hwezviitiko zvine chekuita nekugadzirwa kwezvivakwa kusvika zero.
Izvo zvichakurukurwa pasi apa zvakatiunzira kurwadziwa kwakanyanya, uye chinangwa chechinyorwa chino ndechekudzivirira vamwe vanhu kuti vadzokorore kukanganisa kwedu kana kuti kuderedza maitiro avo. 

Preamble

Kare kare, apo Cian yaisanganisira monoliths, uye pakanga pasati pave nemazano emamicroservices, takayera kuwanikwa kwechishandiso nekutarisa 3-5 mapeji. 

Vanopindura - zvese zvakanaka, kana vakasapindura kwenguva yakareba - kungwarira. Inguva yakareba sei yavaifanira kubva kubasa kuti zvionekwe sechiitiko zvakasarudzwa nevanhu mumisangano. Chikwata chemainjiniya chaigara chichibatanidzwa mukuferefeta kwechiitiko ichi. Pakapera kuferefetwa, vakanyora postmortem - rudzi rwemushumo neemail mufomati: chii chakaitika, chakatora nguva yakareba sei, zvatakaita panguva ino, zvatichaita mune ramangwana. 

Mapeji makuru esaiti kana kuti tinonzwisisa sei kuti tarova pasi

 
Kuti neimwe nzira tinzwisise kukosha kwechikanganiso, isu takaona iwo akanyanya kukosha saiti mapeji ekushanda kwebhizinesi. Kuzvishandisa, tinoverenga nhamba yezvikumbiro zvakabudirira / zvisina kubudirira uye nguva yekubuda. Aya ndiwo maitiro atinoyera uptime. 

Ngatitii takaona kuti kune akati wandei akakosha-akakosha zvikamu zvesaiti izvo zvine basa resevhisi huru - kutsvaga nekuendesa zviziviso. Kana nhamba yezvikumbiro zvinokundikana zvinopfuura 1%, ichi chiitiko chakakosha. Kana mukati memaminitsi gumi nemashanu panguva yekutanga kukanganisa kunopfuura 15%, saka ichi chinotorwawo sechiitiko chakakosha. Aya maitiro anofukidza mazhinji ezviitiko; mamwe ese ari kunze kwechikamu chechinyorwa ichi.

Zviitiko zvakanakisa zveCyan

Saka, isu takadzidza zvechokwadi kuona chokwadi chekuti chiitiko chakaitika. 

Zvino chiitiko chese chinotsanangurwa zvakadzama uye chinoratidzwa muJira epic. Nenzira: nekuda kweizvi takatanga chirongwa chakasiyana, chakadana kuti FAIL - chete epics inogona kugadzirwa mairi. 

Kana iwe ukaunganidza zvese zvakakundikana mumakore mashoma apfuura, vatungamiriri ndeiyi: 

• mssql inoenderana zviitiko;
• zviitiko zvinokonzerwa nekunze kwezvinhu;
• admin zvikanganiso.

Ngatitarisei zvakadzama pane kukanganisa kwevatariri, pamwe nekumwe kukundikana kunonakidza.

Nzvimbo yechishanu - "Kuisa zvinhu muDNS"

Yakanga iri chipiri chedutu. Isu takasarudza kudzoreredza kurongeka muDNS cluster. 

Ini ndaida kuendesa mukati meDNS maseva kubva pakusunga kuenda kune powerdns, kugovera akaparadzana zvachose maseva eizvi, uko pasina chinhu kunze kweDNS. 

Isu takaisa imwe sevha yeDNS munzvimbo yega yega yeDCs edu, uye nguva yakasvika yekufambisa nzvimbo kubva pakusunga kuenda kumagetsi uye kushandura zvivakwa kune maseva matsva. 

Pakati pekufamba, kune ese mavhavha akatsanangurwa munharaunda caching anosunga pamaseva ese, imwe chete yakasara, iyo yaive munzvimbo yedata muSt. Iyi DC yakatanga kuziviswa seasina kutsoropodza isu, asi yakaerekana yave chinhu chimwe chekutadza.
Pakanga pari panguva iyi yokutamiswa apo mugero waiva pakati peMoscow neSt. Petersburg wakadonha. Isu takanyatsosiiwa tisina DNS kwemaminetsi mashanu ndokusimuka apo mugadziri akagadzirisa dambudziko. 

Mhedziso:

Kana kare isu takaregeredza zvinhu zvekunze panguva yekugadzirira basa, ikozvino ivo vanosanganisirwa mune rondedzero yezvatiri kugadzirira. Uye zvino tinoedza kuona kuti zvikamu zvose zvakachengetwa n-2, uye panguva yebasa tinogona kuderedza iyi nhanho kusvika n-1.

• Paunenge uchigadzira hurongwa hwekuita, tara pfungwa dzingakundikana sevhisi, uye funga nezve chiitiko apo zvese zvakabva “kubva pakuipa zvichienda pakuipisisa” pachine nguva.
• Govera mukati meDNS maseva munzvimbo dzakasiyana-siyana / nzvimbo dzedhata / racks / switch / inputs.
• Pane yega yega sevha, isa yemunharaunda caching DNS server, iyo inodzosera zvikumbiro kumasevha makuru eDNS, uye kana isingawanikwe, inopindura kubva kune cache. 

Nzvimbo yechina - "Kuisa zvinhu muNginx"

Rimwe zuva rakanaka, timu yedu yakafunga kuti "takawana zvakakwana neizvi," uye maitiro ekugadzirisa nginx configs akatanga. Chinangwa chikuru ndechekuunza ma configs kune intuitive structure. Kare, zvese zvaive "zvakasimbiswa" uye hazvina kutakura chero pfungwa. Iye zvino yega yega server_name yakaendeswa kune faira rezita rimwechete uye ese magadzirirwo akagoverwa mumaforodha. Nenzira, iyo config ine 253949 mitsetse kana 7836520 mavara uye inotora anenge 7 megabytes. Top level of structure: 

Nginx chimiro

├── access
│   ├── allow.list
...
│   └── whitelist.conf
├── geobase
│   ├── exclude.conf
...
│   └── geo_ip_to_region_id.conf
├── geodb
│   ├── GeoIP.dat
│   ├── GeoIP2-Country.mmdb
│   └── GeoLiteCity.dat
├── inc
│   ├── error.inc
...
│   └── proxy.inc
├── lists.d
│   ├── bot.conf
...
│   ├── dynamic
│   └── geo.conf
├── lua
│   ├── cookie.lua
│   ├── log
│   │   └── log.lua
│   ├── logics
│   │   ├── include.lua
│   │   ├── ...
│   │   └── utils.lua
│   └── prom
│       ├── stats.lua
│       └── stats_prometheus.lua
├── map.d
│   ├── access.conf
│   ├── .. 
│   └── zones.conf
├── nginx.conf
├── robots.txt
├── server.d
│   ├── cian.ru
│   │   ├── cian.ru.conf
│   │   ├── ...
│   │   └── my.cian.ru.conf
├── service.d
│   ├── ...
│   └── status.conf
└── upstream.d
    ├── cian-mcs.conf
    ├── ...
    └── wafserver.conf

Yakava nani zvikuru, asi mukugadzirisa kutumidza uye kugovera configs, mamwe acho aiva nekuwedzera kusina kunaka uye haana kubatanidzwa mune zvinosanganisira * .conf murairo. Nekuda kweizvozvo, vamwe vanogamuchira vakashaya uye vakadzosera 301 kupeji huru. Pamusana pekuti kodhi yekupindura yakanga isiri 5xx / 4xx, izvi hazvina kuonekwa pakarepo, asi mangwanani chete. Mushure meizvozvo, takatanga kunyora bvunzo dzekutarisa zvikamu zvezvivakwa.

Mhedziso: 

• Gadzira zvigadziriso zvako nenzira kwayo (kwete chete nginx) uye funga kuburikidza nechimiro padanho rekutanga reprojekiti. Nenzira iyi iwe uchaita kuti vawedzere kunzwisiswa kune timu, iyo inozoderedza TTM.
• Nyora bvunzo dzezvimwe zvikamu zvezvivakwa. Semuenzaniso: kutarisa kuti ese makiyi server_names anopa iyo chaiyo mamiriro + mhinduro muviri. Zvichave zvakakwana kungova nezvinyorwa zvishoma paruoko zvinotarisa mabasa ekutanga echikamu, kuti urege kurangarira kurangarira na3 am chii chimwe chinoda kuongororwa. 

Nzvimbo yechitatu - "Pakarepo pakapera nzvimbo muCassandra"

Iyo data yakakura zvishoma nezvishoma, uye zvese zvaive zvakanaka kusvika panguva iyo kugadziriswa kwenzvimbo dzakakura dzakatanga kutadza muCassandra cluster, nekuti compaction yaisagona kushanda pavari. 

Rimwe zuva kune dutu sumbu rakapotsa rashanduka kuita nhanga, kureva:

• paive neinenge 20% yenzvimbo yese yakasara musumbu;
• Hazvibviri kuwedzera zvizere nodes, nokuti kuchenesa hakupfuuri mushure mekuwedzera node nekuda kwekushaya nzvimbo pane zvikamu;
• kugadzirwa kunoderera zvishoma nezvishoma sezvo compaction isingashande; 
• Iyo cluster iri mu emergency mode.

Buda - takawedzera 5 mamwe ma node pasina kucheneswa, mushure mezvo takatanga kuabvisa zvakarongeka kubva musumbu uye pindazve mukati mawo, senge isina chinhu isina nzvimbo. Nguva yakawanda yakapedzwa kupfuura yataizoda. Paive nenjodzi yekusavapo kwechikamu kana kusawanikwa kwechikwata. 

Mhedziso:

• Pane ese cassandra maseva, hapana inopfuura 60% yenzvimbo pane yega yega chikamu inofanira kugarwa. 
• Dzinofanirwa kutakurwa pasina kupfuura 50% cpu.
• Iwe haufanirwe kukanganwa nezve kugona kuronga uye unofanirwa kuifungisisa kune imwe neimwe chikamu, zvichibva pane yayo chaiyo.
• Iyo yakawanda nodes musumbu, zviri nani. Masevha ane hudiki hwe data anoremerwa nekukurumidza, uye sumbu rakadaro riri nyore kumutsidzira. 

Nzvimbo yechipiri - "Data rakanyangarika kubva kune consul kiyi-kukosha kuchengetedza"

Kuti tiwane sevhisi, isu, sevakawanda, tinoshandisa consul. Asi isu zvakare tinoshandisa yayo kiyi-kukosha kune yebhuruu-girinhi marongero eiyo monolith. Iyo inochengetedza ruzivo nezve inoshanda uye isingashande kumusoro kwemvura, iyo inoshandura nzvimbo panguva yekutumirwa. Nechinangwa ichi, sevhisi yekutumira yakanyorwa yakabatana neKV. Pane imwe nguva, data kubva kuKV yakanyangarika. Kudzoserwa kubva mundangariro, asi nehuwandu hwezvikanganiso. Nekuda kweizvozvo, panguva yekurodha, mutoro uri kumusoro wakagovaniswa zvisina kuenzana, uye takagamuchira zvikanganiso zvakawanda zve502 nekuda kwekumashure kuri kuremerwa paCPU. Nekuda kweizvozvo, takatama kubva ku consul KV kuenda ku postgres, kubva uko kusiri nyore kuvabvisa.  

Mhedziso:

• Masevhisi asina chero mvumo haafaniri kunge ane data yakakosha pakushanda kwesaiti. Semuenzaniso, kana iwe usina mvumo muES, zvingava zviri nani kuramba kuwana kune network level kubva kwese uko kusingadiwi, siya chete izvo zvinodiwa, uye zvakare kuisa chiito.destructive_requires_name: chokwadi.
• Dzidzira yako backup uye kudzoreredza maitiro pamberi. Semuenzaniso, ita script pachine nguva (somuenzaniso, mupython) inogona kuchengetedza nekudzorera.

Nzvimbo yekutanga - "Captain Unobvious" 

Pane imwe nguva, takaona kusaenzana kwekugoverwa kwemutoro pane nginx kumusoro kwemvura mune zviitiko apo pakanga paine 10+ maseva kumashure. Nekuda kwekuti round-robin yakatumira zvikumbiro kubva ku1 kusvika kune yekupedzisira kumusoro kwerukova zvakarongeka, uye imwe neimwe nginx reload yakatanga pamusoro, yekutanga kumusoro kwerwizi yaigara ichiwana zvikumbiro kupfuura vamwe.Nekuda kweizvozvo, vakashanda zvishoma uye saiti yese yakatambura. Izvi zvakava pachena zvakanyanya sezvo huwandu hwemotokari hwakawedzera. Kungogadzirisa nginx kuti igone kugonesa haina kushanda - isu tinofanirwa kudzoreredza boka relua kodhi isina kubva pane vhezheni 1.15 (panguva iyoyo). Taifanira kubatanidza nginx 1.14.2 yedu, tichiunza tsigiro isina kurongeka mairi. Izvi zvakagadzirisa dambudziko racho. Iyi bug inohwina chikamu che "Captain Non-Obviousness".

Mhedziso:

Zvainakidza uye zvainakidza kuongorora iyi bug). 

• Ronga kutarisa kwako kuitira kuti zvikubatsire kuwana kuchinja kwakadaro nekukurumidza. Semuenzaniso, unogona kushandisa ELK kutarisa rps pane imwe neimwe kumashure kweimwe neimwe kumusoro, kutarisa nguva yavo yekupindura kubva pakuona kwe nginx. Muchiitiko ichi, izvi zvakatibatsira kuziva dambudziko. 

Nekuda kweizvozvo, kukundikana kwakawanda kungadai kwakadziviswa nemaitiro akangwarira kune zvawaiita. Tinofanira kugara tichiyeuka mutemo waMurphy: Chinhu chipi nechipi chinogona kuipa chinoenda chakaipa, uye kuvaka zvikamu zvichibva pazviri. 

Source: www.habr.com