Sanibonani nonke, igama lami ngingu-Alexander, ngisebenza kwa-CIAN njengonjiniyela futhi ngibambe iqhaza ekuphathweni kwesistimu kanye ne-automation yezinqubo zengqalasizinda. Emazwaneni kwesinye sezindatshana ezedlule, sicelwe ukuthi sisho ukuthi siyitholaphi i-4 TB yezingodo ngosuku nokuthi senzani ngayo. Yebo, sinamalogi amaningi, futhi iqoqo lengqalasizinda elihlukile lenziwe ukuze licubungulwe, elisivumela ukuthi sixazulule ngokushesha izinkinga. Kulesi sihloko ngizokhuluma ngokuthi siyivumelanise kanjani phakathi nonyaka ukuze isebenze ngokugeleza okukhula njalo kwedatha.
Siqale kuphi?
Kule minyaka embalwa edlule, umthwalo ku-cian.ru ukhule ngokushesha okukhulu, futhi ngekota yesithathu ka-2018, ukugcwala kwezinsiza kufinyelele kubasebenzisi abahlukile abayizigidi eziyi-11.2 ngenyanga. Ngaleso sikhathi, ngezikhathi ezibucayi salahlekelwa kufika ku-40% wezingodo, yingakho singakwazi ukubhekana nezigameko ngokushesha futhi sachitha isikhathi esiningi nomzamo wokuzixazulula. Futhi ngokuvamile asikwazanga ukuthola imbangela yenkinga, futhi yayizovela ngemva kwesikhathi esithile. Kwakuyisihogo futhi kufanele kwenziwe okuthile ngakho.
Ngaleso sikhathi, sasebenzisa iqoqo lama-data node angu-10 ngenguqulo ye-ElasticSearch 5.5.2 enezilungiselelo ezijwayelekile zenkomba yokugcina amalogi. Yethulwa ngaphezu konyaka odlule njengesixazululo esithandwayo nesithengekayo: khona-ke ukugeleza kwamalogi kwakungekukhulu kakhulu, kwakungekho phuzu lokuza nokucushwa okungezona okujwayelekile.
Ukucutshungulwa kwamalogi angenayo kunikezwe i-Logstash kumachweba ahlukene kubaxhumanisi abahlanu be-ElasticSearch. Inkomba eyodwa, kungakhathaliseki ukuthi ingakanani, yayinamashadi amahlanu. Kwahlelwa ukuzungezisa ngehora nosuku, ngenxa yalokho, kwavela amashadi amasha angaba ngu-100 kuqoqo njalo ngehora. Nakuba zazingekho izingodo eziningi, iqoqo libhekane kahle futhi akekho owanaka izilungiselelo zalo.
Izinselele zokukhula ngokushesha
Umthamo wezingodo ezikhiqiziwe wakhula ngokushesha okukhulu, njengoba izinqubo ezimbili zanqwabelana. Ngakolunye uhlangothi, inani labasebenzisi besevisi likhule. Ngakolunye uhlangothi, saqala ukushintshela ekwakhiweni kwe-microservice, sabona ama-monoliths ethu amadala ku-C # kanye ne-Python. Ishumi nambili lamasevisi amancanyana amasha athathe indawo yezingxenye ze-monolith akhiqize amalogi engeziwe eqoqo lengqalasizinda.
Kwakuwukukhula okwasiholela endaweni lapho iqoqo lagcina lingalawuleki. Lapho izingodo ziqala ukufika ngenani lemilayezo eyizinkulungwane ezingu-20 ngomzuzwana, ukujikeleza okungenamsebenzi okuvamile kwandisa inani lama-shards kuya ku-6 ayizinkulungwane, futhi kwakukhona ama-shards angaphezu kuka-600 nge-node ngayinye.
Lokhu kwaholela ezinkingeni ngokunikezwa kwe-RAM, futhi lapho i-node iphahlazeka, wonke ama-shards aqala ukuhamba ngesikhathi esisodwa, aphindaphinda ithrafikhi futhi alayishe amanye ama-node, okwenza kube nzima ukubhala idatha ku-cluster. Futhi ngalesi sikhathi sasishiywe ngaphandle kwezingodo. Futhi uma kube nenkinga ngeseva, silahlekelwe ngu-1/10 weqoqo. Inani elikhulu lezinkomba ezincane lengeze ubunkimbinkimbi.
Ngaphandle kwamalogi, asizange siziqonde izizathu zesigameko futhi singaphinda sinyathele erekeni elifanayo futhi, futhi ngokombono wethimba lethu lokhu kwakungamukeleki, ngoba zonke izindlela zethu zokusebenza zenzelwe ukwenza okuphambene - ungalokothi uphinde. izinkinga ezifanayo. Ukuze senze lokhu, besidinga umthamo ogcwele wamalogi kanye nokulethwa kwawo cishe ngesikhathi sangempela, njengoba ithimba lonjiniyela abasemsebenzini liqaphe izexwayiso hhayi kuphela kumamethrikhi, kodwa nakumalogi. Ukuze uqonde ubukhulu benkinga, ngaleso sikhathi isamba sevolumu yamalogi sasicishe sibe ngu-2 TB ngosuku.
Sibeka umgomo wokuqeda ngokuphelele ukulahlekelwa kwamalogi futhi sinciphise isikhathi sokulethwa kwabo ku-ELK cluster kuze kufike kumaminithi angu-15 ngesikhathi se-force majeure (kamuva sathembela kulesi sibalo njenge-KPI yangaphakathi).
Indlela entsha yokuzungezisa kanye namanodi ashisayo afudumele
Siqale ukuguqulwa kweqoqo ngokubuyekeza inguqulo ye-ElasticSearch ukusuka ku-5.5.2 ukuya ku-6.4.3. Nakulokhu iqoqo lethu lenguqulo engu-5 lifile, futhi sanquma ukulivala futhi silibuyekeze ngokuphelele - azikabi bikho izingodo. Ngakho-ke senze lolu shintsho emahoreni ambalwa nje.
Uguquko olukhulu kakhulu kulesi sigaba kwaba ukuqaliswa kwe-Apache Kafka ezindaweni ezintathu ezinomxhumanisi njengesilondolozi esimaphakathi. Umthengisi womlayezo usisindise ekulahlekelweni amalogi phakathi nezinkinga nge-ElasticSearch. Ngesikhathi esifanayo, sengeze ama-node angu-2 ku-cluster futhi sishintshela esakhiweni esishisayo esishisayo esinama-node amathathu "ashisayo" atholakala kuma-rack ahlukene esikhungweni sedatha. Siqondise kabusha amalogi kubo sisebenzisa imaski okungafanele ilahleke ngaphansi kwanoma yiziphi izimo - i-nginx, kanye namalogi wephutha lohlelo lokusebenza. Amalogi amancane athunyelwa kuma-node asele - ukulungisa iphutha, isexwayiso, njll., futhi ngemva kwamahora angu-24, izingodo "ezibalulekile" ezivela kuma-node "ashisayo" zidluliselwe.
Ukuze singakhulisi inani lezinkomba ezincane, sishintshile ukusuka ekuzungezweni kwesikhathi ukuya kumshini wokugoqa. Kube nolwazi oluningi ezinkundleni ukuthi ukuzungezisa ngosayizi wenkomba akuthembeki kakhulu, ngakho-ke sinqume ukusebenzisa ukuzungezisa ngenani lamadokhumenti kunkomba. Sihlaziye inkomba ngayinye futhi sarekhoda inombolo yamadokhumenti okuthi ngemva kwalokho ukuzungezisa kusebenze. Ngakho-ke, sesifinyelele usayizi olungile we-shard - akukho ngaphezu kwe-50 GB.
Ukuthuthukisa iqoqo
Nokho, asikaqedi ngokuphelele izinkinga. Ngeshwa, izinkomba ezincane zisavela: azizange zifinyelele ivolumu eshiwo, azizange zijikelezwe, futhi zisuswe ngokuhlanzwa komhlaba wonke kwezinkomba ezindala kunezinsuku ezintathu, njengoba sisuse ukuzungezisa ngedethi. Lokhu kuholele ekulahlekeni kwedatha ngenxa yokuthi inkomba evela kuqoqo yanyamalala ngokuphelele, futhi umzamo wokubhalela inkomba engekho wephule umqondo womlondolozi esasiwusebenzisela ukuphatha. I-Alias ββyokubhala yaguqulwa yaba inkomba futhi yaphula ingqondo ye-rollover, yabangela ukukhula okungalawuleki kwezinye izinkomba kufika ku-600 GB.
Isibonelo, ngokulungiselelwa kokuzungezisa:
Ρurator-elk-rollover.yaml
---
actions:
1:
action: rollover
options:
name: "nginx_write"
conditions:
max_docs: 100000000
2:
action: rollover
options:
name: "python_error_write"
conditions:
max_docs: 10000000
Uma bekungekho isibizo se-rollover, kwenzeke iphutha:
ERROR alias "nginx_write" not found.
ERROR Failed to complete action: rollover. <type 'exceptions.ValueError'>: Unable to perform index rollover with alias "nginx_write".
Sishiye isixazululo sale nkinga ngokuphindaphindiwe futhi sathatha enye inkinga: sishintshele kumqondo wokudonsa we-Logstash, ocubungula amalogi angenayo (ukususa ulwazi olungadingekile nokunothisa). Siyibeke endaweni ye-docker, esiyisungula nge-docker-compose, futhi sabeka i-logstash-exporter lapho, ethumela amamethrikhi ku-Prometheus ukuze kuqashwe ukusebenza kokusakaza kwelogi. Ngale ndlela sizinike ithuba lokushintsha ngokushelela inani lezenzakalo ze-logstash ezinomthwalo wokucubungula uhlobo ngalunye lwelogi.
Ngenkathi sithuthukisa iqoqo, ithrafikhi ye-cian.ru ikhuphuke yaba abasebenzisi abahlukile abayizigidi eziyi-12,8 ngenyanga. Ngenxa yalokho, kwavela ukuthi ukuguqulwa kwethu kwaba kancane ngemuva kwezinguquko ekukhiqizeni, futhi sasibhekene neqiniso lokuthi ama-node "afudumele" awakwazanga ukubhekana nomthwalo futhi anciphisa konke ukulethwa kwamalogi. Sithole idatha "eshisayo" ngaphandle kokwehluleka, kodwa kudingeke ukuthi singenelele ekulethweni kokunye futhi senze i-rollover mathupha ukuze sisabalalise izinkomba ngokulinganayo.
Ngesikhathi esifanayo, ukukala nokushintsha izilungiselelo zezenzakalo ze-logstash kuqoqo kwaba nzima yiqiniso lokuthi kwakuyi-docker-compose yendawo, futhi zonke izenzo zenziwa ngesandla (ukwengeza iziphetho ezintsha, kwakudingeka ukuthi udlule kukho konke ngesandla. amaseva futhi wenze i-docker-compose up -d yonke indawo).
Ukusabalalisa kabusha kwelogi
NgoSeptemba walo nyaka, sisasika i-monolith, umthwalo ku-cluster wawukhula, futhi ukugeleza kwamalogi kwakusondela emilayezo eyizinkulungwane ezingu-30 ngomzuzwana.
Siqale ukuphindaphinda okulandelayo ngesibuyekezo sehadiwe. Sisuke kubaxhumanisi abahlanu saya kwabathathu, sashintsha ama-data node sawina ngokwemali nendawo yokugcina. Kumanodi sisebenzisa ukucushwa okubili:
- Kumanodi βashisayoβ: E3-1270 v6 / 960Gb SSD / 32 Gb x 3 x 2 (3 kwe-Hot1 no-3 kwe-Hot2).
- Kumanodi βafudumeleβ: E3-1230 v6 / 4Tb SSD / 32 Gb x 4.
Kulokhu kuphindaphinda, sihambise inkomba enamalogi okufinyelela ama-microservices, athatha indawo efanayo namalogi angaphambili we-nginx, eqenjini lesibili lamanodi amathathu "ashisayo". Manje sigcina idatha kuma-node "ashisayo" amahora angu-20, bese siyidlulisela kumanodi "afudumele" kuwo wonke amalogi.
Sixazulule inkinga yezinkomba ezincane ezinyamalalayo ngokulungisa kabusha ukuzungezisa kwazo. Manje izinkomba ziphenduka njalo emahoreni angama-23 kunoma yikuphi, ngisho noma kunedatha encane lapho. Lokhu kwandisa kancane inani lama-shards (kwakukhona okungaba ngu-800), kodwa kusukela ekubukeni kokusebenza kweqoqo kuyabekezeleleka.
Ngenxa yalokho, kwakukhona ama-node ayisithupha "ashisayo" futhi amane kuphela "afudumele" eqenjini. Lokhu kubangela ukubambezeleka okuncane ezicelweni ngezikhathi ezinde, kodwa ukwandisa inani lamanodi esikhathini esizayo kuzoxazulula le nkinga.
Lokhu kuphindaphinda kuphinde kwalungisa inkinga yokushoda kokukalwa kwe-semi-automatic. Ukwenza lokhu, sikhiphe iqoqo lengqalasizinda ye-Nomad - efana naleyo esesivele siyifakile ekukhiqizeni. Okwamanje, inani le-Logstash alishintshi ngokuzenzakalelayo kuye ngomthwalo, kodwa sizofika kulokhu.
Izinhlelo zekusasa
Izikali zokucushwa ezisetshenzisiwe zilinganisa kahle, futhi manje sigcina idatha engu-13,3 TB - wonke amalogi izinsuku ezingu-4, okudingekayo ekuhlaziyweni okuphuthumayo kwezixwayiso. Siguqula amanye amalogi abe amamethrikhi, esiwangeza ku-Graphite. Ukuze senze umsebenzi wonjiniyela ube lula, sinamamethrikhi eqoqo lengqalasizinda kanye nemibhalo yokulungisa okuzenzakalelayo kwezinkinga ezivamile. Ngemva kokwandisa inani lama-data node, ahlelelwe unyaka ozayo, sizoshintshela ekugcinweni kwedatha kusukela ezinsukwini ezi-4 kuye kweziyi-7. Lokhu kuzokwanela emsebenzini wokusebenza, njengoba sihlala sizama ukuphenya izigameko ngokushesha ngangokunokwenzeka, futhi uphenyo lwesikhathi eside kunedatha ye-telemetry.
Ngo-Okthoba 2019, ithrafikhi ku-cian.ru yayisivele ikhule yaba abasebenzisi abahlukile abayizigidi eziyi-15,3 ngenyanga. Lokhu kwaba uvivinyo olungathi sΓna lwesixazululo sezakhiwo sokuletha izingodo.
Manje silungiselela ukubuyekeza i-ElasticSearch ibe yinguqulo yesi-7. Nokho, ngenxa yalokhu kuzodingeka sibuyekeze ukuklanywa kwezinkomba eziningi ku-ElasticSearch, njengoba zasuka enguqulweni engu-5.5 futhi zamenyezelwa njengezihoxisiwe enguqulweni yesi-6 (azikho nje enguqulweni. 7). Lokhu kusho ukuthi phakathi nenqubo yokuvuselela nakanjani kuzoba khona uhlobo oluthile lwe-force majeure, oluzosishiya ngaphandle kwamalogi ngenkathi inkinga ixazululwa. Kunguqulo yesi-7, siyibheke ngabomvu i-Kibana ene-interface ethuthukisiwe nezihlungi ezintsha.
Sifinyelele umgomo wethu oyinhloko: siyekile ukulahlekelwa amalogi futhi sanciphisa isikhathi sokungasebenzi seqoqo lengqalasizinda sisuka ezingozini ezingu-2-3 ngesonto saya emahoreni ambalwa omsebenzi wokulungisa ngenyanga. Wonke lo msebenzi ekukhiqizeni cishe awubonakali. Kodwa-ke, manje singakwazi ukunquma kahle ukuthi kwenzekani ngenkonzo yethu, singakwenza ngokushesha ngemodi ethule futhi singakhathazeki ukuthi izingodo zizolahleka. Ngokuvamile, sinelisekile, sijabule futhi silungiselela ukuxhaphaza okusha, esizokhuluma ngakho kamuva.
Source: www.habr.com