Top fakapov Cyan

Zabwino zonse! 

Dzina langa ndine Nikita, ndine mtsogoleri wa timu ya Cian engineering. Limodzi mwaudindo wanga pakampani ndikuchepetsa kuchuluka kwa zochitika zokhudzana ndi zomangamanga pakupanga mpaka ziro.
Zomwe zidzakambidwe pansipa zidatibweretsera zowawa zambiri, ndipo cholinga cha nkhaniyi ndikuletsa anthu ena kubwereza zolakwa zathu kapena kuchepetsa mphamvu zawo. 

Yambani

Kalekale, pamene Cian inali ndi monoliths, ndipo panalibe zizindikiro za microservices, tinayesa kupezeka kwa gwero poyang'ana masamba 3-5. 

Amayankha - zonse zili bwino, ngati sayankha kwa nthawi yayitali - tcheru. Nthawi yomwe adakhala osagwira ntchito kuti awoneke ngati chochitika adasankhidwa ndi anthu pamisonkhano. Gulu la mainjiniya nthawi zonse linkachita nawo kafukufuku wa zomwe zinachitika. Kafukufukuyo atamalizidwa, adalemba postmortem - mtundu wa lipoti ndi imelo mu mawonekedwe: zomwe zidachitika, zidakhala nthawi yayitali bwanji, zomwe tidachita pakadali pano, zomwe tidzachita m'tsogolomu. 

Masamba akuluakulu a tsambalo kapena momwe timamvetsetsa kuti tagunda pansi

 
Kuti timvetsetse momwe cholakwikacho chiliri patsogolo, tazindikira masamba ofunikira kwambiri pazantchito zamabizinesi. Kugwiritsa ntchito, timawerengera kuchuluka kwa zopempha zopambana / zosachita bwino komanso nthawi yomaliza. Umu ndi momwe timayezera nthawi. 

Tiyerekeze kuti tapeza kuti pali zigawo zingapo zofunika kwambiri zatsambali zomwe zimayang'anira ntchito yayikulu - kusaka ndi kutumiza zotsatsa. Ngati kuchuluka kwa zopempha zomwe zikulephera kupitirira 1%, ichi ndi chochitika chovuta. Ngati mkati mwa mphindi 15 pa nthawi yachiwongoladzanja chiwongoladzanja chikuposa 0,1%, ndiye kuti izi zimaonedwanso kuti ndizofunika kwambiri. Izi zikukhudza zochitika zambiri; zina zonse sizingaganizidwe ndi nkhaniyi.

Zochitika zabwino kwambiri Cian

Chifukwa chake, taphunziradi kudziwa kuti chochitika chinachitika. 

Tsopano chochitika chilichonse chikufotokozedwa mwatsatanetsatane ndikuwonetseredwa mu epic ya Jira. Mwa njira: chifukwa cha ichi tinayambitsa ntchito yosiyana, yotchedwa FAIL - ma epic okha angapangidwe mmenemo. 

Mukasonkhanitsa zolephera zonse zaka zingapo zapitazi, atsogoleri ndi: 

• zochitika zokhudzana ndi mssql;
• zochitika zoyambitsidwa ndi zinthu zakunja;
• zolakwika za admin.

Tiyeni tiwone mwatsatanetsatane zolakwa za oyang'anira, komanso zolephera zina zosangalatsa.

Malo achisanu - "Kukhazikitsa zinthu mu DNS"

Linali Lachiwiri lamphepo yamkuntho. Tinaganiza zobwezeretsa dongosolo mu gulu la DNS. 

Ndinkafuna kusamutsa ma seva amkati a DNS kuchokera ku bind kupita ku powerdns, ndikugawa ma seva osiyana kwambiri pa izi, pomwe palibe chilichonse kupatula DNS. 

Tinayika seva imodzi ya DNS m'malo aliwonse a ma DC athu, ndipo nthawi inafika yosuntha madera kuchoka ku bind kupita ku powerdns ndikusintha zomangamanga kukhala ma seva atsopano. 

Pakati pa kusuntha, pa ma seva onse omwe adatchulidwa mu caching m'deralo amamangiriza ma seva onse, imodzi yokha inatsalira, yomwe inali mu data center ku St. DC uyu poyamba adalengezedwa ngati wosadzudzula kwa ife, koma mwadzidzidzi adangolephera.
Inali nthawi imene anasamuka pamene ngalande yapakati pa Moscow ndi St. Petersburg inagwa. Tidasiyidwa opanda DNS kwa mphindi zisanu ndikuyambiranso pomwe wolandirayo adakonza vutolo. 

Zotsatira:

Ngati kale tidanyalanyaza zinthu zakunja pokonzekera ntchito, tsopano zikuphatikizidwanso pamndandanda wazomwe tikukonzekera. Ndipo tsopano timayesetsa kuonetsetsa kuti zigawo zonse zasungidwa n-2, ndipo pa ntchito tikhoza kuchepetsa mlingo uwu n-1.

• Polemba ndondomeko yoti muchitepo, chongani mfundo zomwe ntchitoyo ingalephereke, ndipo ganizirani zomwe zinachitika pamene zonse "zinayambira poyipa kupita zoyipa" pasadakhale.
• Gawani ma seva amkati a DNS m'malo osiyanasiyana / malo osungiramo data / zoyikapo / masiwichi / zolowetsa.
• Pa seva iliyonse, yikani seva ya DNS yosungiramo malo, yomwe imatumizanso zopempha ku ma seva akuluakulu a DNS, ndipo ngati sichipezeka, idzayankha kuchokera ku cache. 

Malo achinayi - "Kukhazikitsa zinthu mu Nginx"

Tsiku lina labwino, gulu lathu lidaganiza kuti "takhala nazo zokwanira," ndipo njira yokonzanso ma configs a nginx idayamba. Cholinga chachikulu ndikubweretsa ma configs kuti apangidwe mwachilengedwe. M'mbuyomu, zonse "zidakhazikitsidwa kale" ndipo zinalibe zomveka. Tsopano seva_name iliyonse yasunthidwa ku fayilo ya dzina lomwelo ndipo ma configs onse adagawidwa m'mafoda. Mwa njira, kasinthidwe ali ndi mizere 253949 kapena zilembo 7836520 ndipo amatenga pafupifupi 7 megabytes. Mulingo wapamwamba wamapangidwe: 

Nginx structure

├── access
│   ├── allow.list
...
│   └── whitelist.conf
├── geobase
│   ├── exclude.conf
...
│   └── geo_ip_to_region_id.conf
├── geodb
│   ├── GeoIP.dat
│   ├── GeoIP2-Country.mmdb
│   └── GeoLiteCity.dat
├── inc
│   ├── error.inc
...
│   └── proxy.inc
├── lists.d
│   ├── bot.conf
...
│   ├── dynamic
│   └── geo.conf
├── lua
│   ├── cookie.lua
│   ├── log
│   │   └── log.lua
│   ├── logics
│   │   ├── include.lua
│   │   ├── ...
│   │   └── utils.lua
│   └── prom
│       ├── stats.lua
│       └── stats_prometheus.lua
├── map.d
│   ├── access.conf
│   ├── .. 
│   └── zones.conf
├── nginx.conf
├── robots.txt
├── server.d
│   ├── cian.ru
│   │   ├── cian.ru.conf
│   │   ├── ...
│   │   └── my.cian.ru.conf
├── service.d
│   ├── ...
│   └── status.conf
└── upstream.d
    ├── cian-mcs.conf
    ├── ...
    └── wafserver.conf

Zinakhala zabwino kwambiri, koma posinthanso ndikugawa ma configs, ena mwa iwo anali ndi zowonjezera zolakwika ndipo sanaphatikizidwe mu ndondomeko ya * .conf. Zotsatira zake, ena olandila adasowa ndipo adabweza 301 patsamba lalikulu. Chifukwa chakuti nambala yoyankhira sinali 5xx/4xx, izi sizinazindikiridwe nthawi yomweyo, koma m'mawa mokha. Pambuyo pake, tinayamba kulemba mayesero kuti tiwone zigawo za zomangamanga.

Zotsatira: 

• Pangani ma configs anu molondola (osati nginx) ndipo ganizirani momwe polojekitiyi ikuyambira. Mwanjira iyi mudzawapangitsa kukhala omveka bwino kwa gulu, zomwe zidzachepetsa TTM.
• Lembani mayeso a zigawo zina za zomangamanga. Mwachitsanzo: kuyang'ana kuti ma key server_names amapereka malo olondola + thupi loyankhira. Zidzakhala zokwanira kungokhala ndi zolemba zochepa zomwe zimayang'ana zofunikira za gawoli, kuti musakumbukire mopupuluma pa 3 koloko zomwe ziyenera kufufuzidwa. 

Malo achitatu - "Mwadzidzidzi malo adatha ku Cassandra"

Detayo inakula pang'onopang'ono, ndipo zonse zinali bwino mpaka pamene kukonzanso kwa malo akuluakulu kunayamba kulephera m'gulu la Cassandra, chifukwa compaction sichitha kugwira ntchito pa iwo. 

Tsiku lina mkuntho, tsango lidatsala pang'ono kusanduka dzungu, lomwe ndi:

• panali pafupifupi 20% ya malo onse omwe anatsalira mgulu;
• Ndizosatheka kuwonjezera ma node mokwanira, chifukwa kuyeretsa sikudutsa pambuyo powonjezera node chifukwa chosowa malo pazigawo;
• zokolola zimatsika pang'onopang'ono ngati kuphatikizika sikugwira ntchito; 
• Gululi lili munjira yadzidzidzi.

Tulukani - tidawonjezera ma node ena 5 popanda kuyeretsa, kenako tidayamba kuwachotsa mwadongosolo m'gululi ndikulowanso, ngati ma node opanda kanthu omwe adatha. Nthawi yochuluka kwambiri kuposa momwe timafunira. Panali chiwopsezo cha kusapezeka kwa tsankho kapena kwathunthu kwa gululo. 

Zotsatira:

• Pa ma seva onse a cassandra, osapitilira 60% ya malo pagawo lililonse sayenera kukhala. 
• Ayenera kudzazidwa osapitirira 50% CPU.
• Musaiwale za kukonza luso ndipo muyenera kuganizira mozama pa gawo lililonse, kutengera zomwe zimafunikira.
• Ma node ambiri m'gulu, ndi bwino. Ma seva omwe ali ndi data yochepa amadzaza mwachangu, ndipo gulu lotere ndilosavuta kutsitsimutsa. 

Malo achiwiri - "Deta idasowa kuchokera ku consul key-value storage"

Kuti tipeze chithandizo, ife, monga ambiri, timagwiritsa ntchito consul. Koma timagwiritsanso ntchito chinsinsi chake pakupanga mawonekedwe amtundu wa buluu wa monolith. Imasunga zidziwitso za mitsinje yogwira komanso yosagwira ntchito, yomwe imasintha malo panthawi yotumizidwa. Pachifukwa ichi, ntchito yotumizira inalembedwa yomwe imagwirizana ndi KV. Panthawi ina, deta ya KV inasowa. Kubwezeretsedwa kuchokera pamtima, koma ndi zolakwika zingapo. Chotsatira chake, pakukweza, katundu pamitsinjeyo adagawidwa mosagwirizana, ndipo tinalandira zolakwika zambiri za 502 chifukwa cha backends zodzaza pa CPU. Zotsatira zake, tidachoka ku consul KV kupita ku postgres, komwe sikulinso kosavuta kuwachotsa.  

Zotsatira:

• Ntchito popanda chilolezo siziyenera kukhala ndi deta yofunika kwambiri pakugwiritsa ntchito tsambalo. Mwachitsanzo, ngati mulibe chilolezo mu ES, zingakhale bwino kukana kupeza pa intaneti mlingo kuchokera kulikonse kumene sikofunikira, kusiya zofunika zokha, komanso kukhazikitsa action.destructive_requires_name: zoona.
• Yesetsani njira yanu yosungira ndi kubwezeretsa pasadakhale. Mwachitsanzo, pangani script pasadakhale (mwachitsanzo, mu Python) yomwe imatha kusunga ndikubwezeretsa.

Malo oyamba - "Captain Unobvious" 

Panthawi ina, tidawona kugawanika kosagwirizana kwa katundu pa nginx kumtunda pamene panali ma seva 10+ kumbuyo. Chifukwa chakuti robin yozungulira idatumiza zopempha kuchokera ku 1st kupita kumtunda womaliza, ndipo nginx iliyonse imayambanso kuyambiranso, oyambira okwera amalandila zopempha zambiri kuposa ena onse.Chotsatira chake, adagwira ntchito pang'onopang'ono ndipo malo onse adavutika. Izi zinayamba kuonekera kwambiri pamene kuchuluka kwa magalimoto kumawonjezeka. Kungosintha nginx kuti mutsegule mwachisawawa sikunagwire ntchito - tifunika kukonzanso gulu la lua code lomwe silinachoke pa mtundu 1.15 (panthawiyo). Tinayenera kuyika nginx 1.14.2 yathu, ndikuyambitsa chithandizo chachisawawa mmenemo. Izi zinathetsa vutolo. Vutoli limapambana gulu la "Captain Non-Obviousness".

Zotsatira:

Zinali zosangalatsa komanso zosangalatsa kufufuza cholakwikacho). 

• Konzani zowunikira zanu kuti zikuthandizeni kupeza kusinthasintha koteroko mwachangu. Mwachitsanzo, mutha kugwiritsa ntchito ELK kuyang'anira ma rps kumbuyo kwamtundu uliwonse kumtunda, kuwunika nthawi yawo yoyankhira kuchokera pakuwona kwa nginx. Pamenepa, izi zidatithandiza kuzindikira vuto. 

Chotsatira chake, zolephera zambiri zikanapewedwa ndi njira yosamala kwambiri ndi zomwe mukuchita. Tiyenera kukumbukira nthawi zonse lamulo la Murphy: Chilichonse chomwe chitha kuwonongeka chidzalakwika, ndi kumanga zigawo zochokera izo. 

Source: www.habr.com