Okuphezulu fakapov Cyan

Ngikufisela okuhle! 

Igama lami nginguNikita, ngingumholi weqembu leqembu lonjiniyela lakwaCian. Omunye umsebenzi wami enkampanini ukwehlisa isibalo sezigameko ezihlobene nengqalasizinda ekukhiqizeni zibe ziro.
Okuzoxoxwa ngakho ngezansi kwasilethela ubuhlungu obuningi, futhi inhloso yalesi sihloko ukuvimbela abanye abantu ukuba baphinde amaphutha ethu noma okungenani banciphise umthelela wabo. 

Isingeniso

Esikhathini eside esidlule, lapho i-Cian yayihlanganisa ama-monoliths, futhi kungakabi khona ukusikisela kwama-microservices, silinganise ukutholakala kwensiza ngokuhlola amakhasi angu-3-5. 

Bayaphendula - konke kuhamba kahle, uma bengaphenduli isikhathi eside - baxwayise. Ukuthi kwadingeka baphume isikhathi esingakanani emsebenzini ukuze kuthathwe njengesigameko kwanqunywa ngabantu emihlanganweni. Ithimba lonjiniyela belihlale libambe iqhaza ophenyweni lwalesi sigameko. Lapho uphenyo seluqediwe, babhala i-postmortem - uhlobo lombiko nge-imeyili ngefomethi: kwenzekeni, kwathatha isikhathi esingakanani, esikwenzile okwamanje, esizokwenza esikhathini esizayo. 

Amakhasi amakhulu esayithi noma ukuthi siqonda kanjani ukuthi sishaye phansi

 
Ukuze ngandlela thize siqonde ukubaluleka kwephutha, sihlonze amakhasi esayithi abaluleke kakhulu ekusebenzeni kwebhizinisi. Ukuzisebenzisa, sibala inani lezicelo eziyimpumelelo/ezingaphumelelanga kanye nezikhathi zokuvala. Lena yindlela esikala ngayo isikhathi sokuphumula. 

Ake sithi sithole ukuthi kunenombolo yezingxenye ezibaluleke kakhulu zesayithi ezibhekele insizakalo enkulu - ukusesha nokuthumela izikhangiso. Uma inani lezicelo ezingaphumeleli lidlula u-1%, lesi yisigameko esibucayi. Uma phakathi nemizuzu engu-15 ngesikhathi sokuqala izinga lephutha lidlula u-0,1%, khona-ke lokhu kuthathwa njengesigameko esibucayi. Lezi zindlela zibandakanya izigameko eziningi; ezinye zingaphezu kobubanzi balesi sihloko.

Izehlakalo ezinhle kakhulu Cian

Ngakho-ke, sifunde nakanjani ukuthola iqiniso lokuthi isigameko senzeka. 

Manje sonke isigameko sichazwa ngokuningiliziwe futhi sibonakala ku-Jira epic. Ngendlela: salokhu saqala iphrojekthi ehlukile, ebizwa ngokuthi FAIL - ama-epics kuphela angadalwa kuwo. 

Uma uqoqa konke ukwehluleka kule minyaka embalwa edlule, abaholi yilaba: 

• izigameko ezihlobene ne-mssql;
• izehlakalo ezibangelwa izici zangaphandle;
• amaphutha admin.

Ake sibheke ngokuningiliziwe amaphutha abaphathi, kanye nokunye ukwehluleka okuthakazelisayo.

Indawo yesihlanu - "Ukubeka izinto ngokuhlelekile ku-DNS"

BekuyiLwesibili elinesivunguvungu. Sinqume ukubuyisela ukuhleleka kuqoqo le-DNS. 

Bengifuna ukudlulisa amaseva e-DNS angaphakathi ukusuka ku-bind kuya ku-powerdns, nginikeze amaseva ahluke ngokuphelele kulokhu, lapho kungekho lutho ngaphandle kwe-DNS. 

Sibeke iseva eyodwa ye-DNS endaweni ngayinye yama-DC ethu, futhi kwafika isikhathi sokuhambisa izindawo zisuka ku-bind ziye ku-powerdns futhi sishintshe ingqalasizinda iye kumaseva amasha. 

Phakathi nokuhamba, kuzo zonke iziphakeli ezichazwe ku-caching yendawo zibophezela kuwo wonke amaseva, kwasala eyodwa kuphela, eyayisesikhungweni sedatha eSt. Le DC yaqale yamenyezelwa njengengasigxeka kakhulu, kodwa yaphenduka iphuzu elilodwa lokwehluleka.
Kwakungale nkathi yokufuduka lapho kwawa khona umsele ophakathi kweMoscow neSt. Empeleni sishiywe ngaphandle kwe-DNS imizuzu emihlanu futhi sabuyela emuva lapho umsingathi elungisa inkinga. 

Iziphetho:

Uma ngaphambili singazinaki izici zangaphandle ngesikhathi sokulungiselela umsebenzi, manje nazo zifakwe ohlwini lwalokho esikulungiselelayo. Futhi manje silwela ukuqinisekisa ukuthi zonke izingxenye zigcinwe n-2, futhi phakathi nomsebenzi singakwazi ukwehlisa leli zinga ku-n-1.

• Lapho udweba uhlelo lokusebenza, maka amaphuzu lapho isevisi ingase yehluleke khona, bese ucabanga ngesimo lapho yonke into yahamba “isuka kokubi iye kokubi kakhulu” kusengaphambili.
• Sabalalisa amaseva angaphakathi e-DNS ezindaweni ezahlukene/ezikhungweni zedatha/amarack/amaswishi/okokufaka.
• Kuseva ngayinye, faka iseva yendawo ye-DNS yenqolobane, eqondisa kabusha izicelo kumaseva amakhulu e-DNS, futhi uma ingatholakali, izophendula isuka kunqolobane. 

Indawo yesine - “Ukubeka izinto ngohlelo ku-Nginx”

Ngolunye usuku oluhle, ithimba lethu lanquma ukuthi "sekwanele ngalokhu," futhi inqubo yokulungisa kabusha izilungiselelo ze-nginx yaqala. Umgomo oyinhloko ukuletha okulungiselelwe esakhiweni esinembile. Ngaphambilini, yonke into "yasungulwa ngokomlando" futhi ingenawo umqondo. Manje i-server_name ngayinye ihanjiswe efayeleni elinegama elifanayo futhi zonke izilungiselelo zisakazwe kumafolda. Ngendlela, ukulungiselelwa kuqukethe imigqa engu-253949 noma izinhlamvu ezingu-7836520 futhi kuthatha cishe ama-megabytes angu-7. Izinga eliphezulu lesakhiwo: 

Isakhiwo se-Nginx

├── access
│   ├── allow.list
...
│   └── whitelist.conf
├── geobase
│   ├── exclude.conf
...
│   └── geo_ip_to_region_id.conf
├── geodb
│   ├── GeoIP.dat
│   ├── GeoIP2-Country.mmdb
│   └── GeoLiteCity.dat
├── inc
│   ├── error.inc
...
│   └── proxy.inc
├── lists.d
│   ├── bot.conf
...
│   ├── dynamic
│   └── geo.conf
├── lua
│   ├── cookie.lua
│   ├── log
│   │   └── log.lua
│   ├── logics
│   │   ├── include.lua
│   │   ├── ...
│   │   └── utils.lua
│   └── prom
│       ├── stats.lua
│       └── stats_prometheus.lua
├── map.d
│   ├── access.conf
│   ├── .. 
│   └── zones.conf
├── nginx.conf
├── robots.txt
├── server.d
│   ├── cian.ru
│   │   ├── cian.ru.conf
│   │   ├── ...
│   │   └── my.cian.ru.conf
├── service.d
│   ├── ...
│   └── status.conf
└── upstream.d
    ├── cian-mcs.conf
    ├── ...
    └── wafserver.conf

Kube ngcono kakhulu, kodwa kunqubo yokuqamba kabusha nokusabalalisa izilungiselelo, ezinye zazo zibe nesandiso esingalungile futhi azizange zifakwe kumyalelo ohlanganisayo *.conf. Ngenxa yalokho, abanye ababungazi abazange batholakale futhi babuyisele angu-301 ekhasini eliyinhloko. Ngenxa yokuthi ikhodi yokuphendula yayingeyona i-5xx/4xx, lokhu akuzange kuqashelwe ngokushesha, kodwa ekuseni kuphela. Ngemuva kwalokho, saqala ukubhala izivivinyo zokuhlola izingxenye zengqalasizinda.

Iziphetho: 

• Yakha izilungiselelo zakho ngendlela efanele (hhayi nje i-nginx) futhi ucabange ngesakhiwo ekuqaleni kwephrojekthi. Ngale ndlela uzobenza baqonde kangcono eqenjini, okuzokwehlisa i-TTM.
• Bhala izivivinyo zezingxenye ezithile zengqalasizinda. Isibonelo: ukuhlola ukuthi wonke ama-server_names angukhiye anikeza isimo esilungile + umzimba wokuphendula. Kuyokwanela ukuba nemibhalo embalwa nje esandleni ehlola imisebenzi eyisisekelo yengxenye, ukuze ungakhumbuli ngo-3 ekuseni ukuthi yini enye okudingeka ihlolwe. 

Indawo yesithathu - "Kungazelelwe kuphele isikhala eCassandra"

Idatha yakhula kancane kancane, futhi yonke into yayihamba kahle kwaze kwaba yilapho ukulungiswa kwezindawo ezinkulu ze-cassandra kwaqala ukwehluleka ku-cluster ye-Cassandra, ngoba ukuhlanganisa akukwazanga ukusebenza kuzo. 

Ngolunye usuku olunesiphepho iqoqo lacishe laphenduka ithanga, okungukuthi:

• kwakukhona cishe u-20% wendawo yonke esele kuqoqo;
• Akunakwenzeka ukwengeza ama-node ngokugcwele, ngoba ukuhlanza akudluli ngemuva kokwengeza i-node ngenxa yokuntuleka kwesikhala kuma-partitions;
• ukukhiqiza kuyehla kancane kancane njengoba ukucinana kungasebenzi; 
• Iqoqo likumodi ephuthumayo.

Phuma - sengeze amanye ama-node angu-5 ngaphandle kokuhlanza, ngemva kwalokho saqala ukuwakhipha ngokuhlelekile ku-cluster futhi siwafake kabusha, njengama-node angenalutho ayephelelwe isikhala. Kwachithwa isikhathi esiningi kakhulu kunalokho ebesingathanda. Kube nobungozi bokungatholakali kwengxenye noma okuphelele kweqoqo. 

Iziphetho:

• Kuwo wonke amaseva e-cassandra, akufanele kuthathwe isikhala esingaphezu kuka-60% endaweni ngayinye. 
• Kufanele zilayishwe ngokungeqi ku-50% cpu.
• Akufanele ukhohlwe mayelana nokuhlelwa kwamandla futhi udinga ukucabangisisa ngengxenye ngayinye, ngokusekelwe kulokho okucacisiwe.
• Amanodi amaningi kuqoqo, ngcono. Amaseva aqukethe inani elincane ledatha alayishwa ngokushesha kakhulu, futhi iqoqo elinjalo kulula ukulivuselela. 

Indawo yesibili - "Idatha inyamalala ku-consul key-value storage"

Ukuze uthole isevisi, thina, njengabaningi, sisebenzisa i-consul. Kodwa futhi sisebenzisa ukhiye-value yayo yokwakheka okuluhlaza okwesibhakabhaka kwe-monolith. Igcina ulwazi mayelana nemifula esebenzayo nengasebenzi, eshintsha izindawo phakathi nokuthunyelwa. Ngale njongo, isevisi yokuthumela yabhalwa esebenzisana ne-KV. Ngesinye isikhathi, idatha evela ku-KV yanyamalala. Kubuyiselwe kusuka kumemori, kodwa ngamaphutha ambalwa. Njengomphumela, ngesikhathi sokulayisha, umthwalo ophuma phezulu usabalaliswe ngokungalingani, futhi sithole amaphutha amaningi angu-502 ngenxa yokuthi ingemuva lilayishwe kakhulu ku-CPU. Ngenxa yalokho, sisuke ku-consul KV saya kuma-postgres, lapho kungaselula khona ukuwasusa.  

Iziphetho:

• Amasevisi ngaphandle kokugunyazwa akufanele abe nedatha ebalulekile ekusebenzeni kwesayithi. Isibonelo, uma ungenakho ukugunyazwa ku-ES, kungaba ngcono ukwenqaba ukufinyelela ezingeni lenethiwekhi ukusuka yonke indawo lapho kungadingeki khona, shiya kuphela okudingekayo, futhi usethe isenzo.destructive_requires_name: true.
• Prakthiza indlela yakho yokusekelayo nokuthola kusenesikhathi. Isibonelo, yenza iskripthi kusengaphambili (isibonelo, ku-python) esingakwazi ukwenza isipele futhi sibuyisele.

Indawo yokuqala - "UCaptain Unobvious" 

Ngesinye isikhathi, saqaphela ukusatshalaliswa okungalingani komthwalo ku-nginx phezulu ezimeni lapho kwakukhona amaseva angu-10+ ku-backend. Ngenxa yokuthi i-round-robin ithumele izicelo ukusuka kowoku-1 ukuya kowokugcina okhuphukayo ngohlelo, futhi ukulayisha kabusha kwe-nginx kwaqala phansi, ama-upstreams okuqala ahlala ethola izicelo eziningi kunabanye. Ngenxa yalokho, basebenze kancane futhi isayithi lonke lahlupheka. Lokhu kwaqala ukubonakala njengoba inani lezimoto likhula. Ukumane ubuyekeze i-nginx ukuze unike amandla okungahleliwe akusebenzanga - sidinga ukwenza kabusha inqwaba yekhodi ye-lua engazange isuke kunguqulo 1.15 (ngaleso sikhathi). Bekufanele sipeshe i-nginx 1.14.2 yethu, sethula ukwesekwa okungahleliwe kuyo. Lokhu kwayixazulula inkinga. Lesi siphazamisi siwina isigaba esithi "UCaptain Okungabonakali".

Iziphetho:

Bekujabulisa kakhulu futhi kujabulisa ukuhlola lesi siphazamisi). 

• Hlela ukuqapha kwakho ukuze kukusize uthole ukushintshashintsha okunjalo ngokushesha. Isibonelo, ungasebenzisa i-ELK ukuqapha ama-rps ku-backend ngayinye yomfula ngamunye okhuphukayo, uqaphe isikhathi sabo sokuphendula kusukela endaweni yokubuka ye-nginx. Kulokhu, lokhu kwasisiza ukuba sibone inkinga. 

Ngenxa yalokho, ukwehluleka okuningi bekungagwenywa ngendlela ecophelelayo kulokho obukwenza. Kufanele sihlale sikhumbula umthetho kaMurphy: Noma yini engahamba kahle izokonakala, futhi wakhe izingxenye ezisekelwe kuyo. 

Source: www.habr.com