Mhoroi mese, zita rangu ndiAlexander, ndinoshanda kuCIAN seinjiniya uye ndinobatanidzwa mukutonga kwehurongwa uye otomatiki yezvivakwa. Mune zvakataurwa kune chimwe chezvinyorwa zvakapfuura, takabvunzwa kuti titaure kwatinowana 4 TB yematanda pazuva uye zvatinoita navo. Hongu, isu tine matanda akawanda, uye yakaparadzana masumbu ezvivakwa akagadzirwa kuti avagadzirise, izvo zvinotibvumira kukurumidza kugadzirisa matambudziko. Muchinyorwa chino ini ndichataura pamusoro pekuti takaigadzirisa sei mukati megore kuti ishande nekuyerera kuri kuramba kuchikura kwedata.
Takatangira papi?
Mumakore mashoma apfuura, mutoro uri pa cian.ru wakakura nekukurumidza, uye nechikamu chechitatu chegore ra2018, traffic traffic yakasvika 11.2 miriyoni vashandisi vakasiyana pamwedzi. Panguva iyoyo, panguva dzakaoma takarasikirwa kusvika ku40% yematanda, ndicho chikonzero isu taisakwanisa kukurumidza kubata nezviitiko uye takapedza nguva yakawanda nekuedza kuzvigadzirisa. Isu takatadzawo kuwana chikonzero chedambudziko, uye raizodzoka mushure menguva yakati. Rakanga riri gehena uye chimwe chinhu chaifanira kuitwa pamusoro pacho.
Panguva iyoyo, takashandisa cluster ye10 data nodes neElasticSearch version 5.5.2 ine standard index settings kuchengetedza matanda. Yakaunzwa kupfuura gore rapfuura seyakakurumbira uye inokwanisika mhinduro: ipapo kuyerera kwematanda kwakanga kusiri kukuru, pakanga pasina chikonzero chekuuya neasina-standard magadzirirwo.
Kugadziriswa kwematanda anouya kwakapihwa neLogstash pamachiteshi akasiyana pane mashanu ElasticSearch varongi. Imwe indekisi, pasinei nehukuru, yaiva nezvidimbu zvishanu. Kutenderera kweawa uye kwezuva nezuva kwakarongwa, semugumisiro, anenge 100 shards itsva akaonekwa musumbu awa imwe neimwe. Kunyange pakanga pasina matanda akawanda, sumbu racho rakabata zvakanaka uye hapana akateerera marongero aro.
Matambudziko ekukura nokukurumidza
Huwandu hwematanda akagadzirwa akakura nekukurumidza, sezvo maitiro maviri akadhumana. Kune rumwe rutivi, nhamba yevashandisi vebasa racho yakakura. Kune rimwe divi, takatanga kushingaira kushandura kune microservice architecture, tichiona mamonoliths edu ekare muC # nePython. Mazhinji gumi nemaviri mamicroservices matsva akatsiva zvikamu zve monolith akagadzira zvakanyanya matanda eiyo sumbu rezvivakwa.
Kwakanga kuri kuyera kwakatisvitsa padanho rekuti sumbu racho rive risingatarisike. Apo matanda akatanga kusvika pamwero we20 zviuru zvemashoko pasekondi, nguva dzose kutenderera kusina maturo kwakawedzera nhamba ye shards kusvika ku6 zviuru, uye paiva neanopfuura 600 shards per node.
Izvi zvakakonzera matambudziko nekugoverwa kwe RAM, uye apo node yakaputsika, shards dzose dzakatanga kufamba panguva imwe chete, dzichiwedzera traffic uye kurodha dzimwe node, izvo zvakaita kuti zvisave nyore kunyora data kune cluster. Uye panguva iyi takasara tisina matanda. Uye kana paine dambudziko neserver, isu takarasikirwa ne1/10 yesumbu. Nhamba huru yediki indexes yakawedzera kuoma.
Pasina matanda, isu hatina kunzwisisa zvikonzero zvechiitiko ichi uye taigona kukurumidza kana kuti gare gare kutsika pane imwechete reki zvakare, uye mumafungiro echikwata chedu izvi zvaisagamuchirwa, sezvo maitiro edu ese akagadzirirwa kuita zvakapesana - usambodzokorora. matambudziko mamwe chete. Kuti tiite izvi, taida huwandu hwakazara hwematanda uye kuendesa kwavo munguva chaiyo, sezvo boka revainjiniya vari pabasa rakatarisisa chenjedzo kwete chete kubva kumetrics, asiwo kubva mumatanda. Kuti tinzwisise kukura kwedambudziko, panguva iyoyo huwandu hwematanda hwaive hunosvika 2 TB pazuva.
Isu takaisa chinangwa chekubvisa zvachose kurasikirwa kwematanda uye kuderedza nguva yekuendesa kwavo kune ELK cluster kusvika kuhuwandu hwemaminitsi e15 panguva yeforce majeure (isu gare gare takavimba nemufananidzo uyu seKPI yemukati).
New kutenderera michina uye inopisa-inodziya node
Takatanga kushandura kwechikwata nekuvandudza shanduro yeElasticSearch kubva pa5.5.2 kusvika ku6.4.3. Zvakare vhezheni yedu 5 cluster yakafa, uye isu takafunga kuidzima nekuigadzirisa zvachose - hapasati pasisina matanda. Saka takaita shanduko iyi mumaawa mashoma.
Shanduko huru-yakakura pane ino nhanho yaive yekuitwa kweApache Kafka pamanodhi matatu ane murongi senge yepakati buffer. Iyo meseji broker yakatiponesa kubva pakurasikirwa nematanda panguva yematambudziko neElasticSearch. Panguva imwecheteyo, takawedzera 2 nodes kune sumbu uye takashandura kune inopisa-inodziya dhizaini ine nhatu "inopisa" node dziri mumaraki akasiyana munzvimbo yedata. Isu takadzosera matanda kwavari tichishandisa mask iyo isingafanirwe kurasika chero mamiriro ezvinhu - nginx, pamwe nekushandisa kukanganisa matanda. Madiki matanda akatumirwa kune akasara node - debug, yambiro, nezvimwewo, uye mushure memaawa makumi maviri nemana, "akakosha" matanda kubva "anopisa" node akatamiswa.
Kuti tisawedzera nhamba yezvinyorwa zviduku, takashandura kubva panguva yekutenderera kusvika kune rollover mechanism. Paive neruzivo rwakawanda pamaforamu ekuti kutenderera nehukuru hweindex hakuna kuvimbika, saka takasarudza kushandisa kutenderera nenhamba yemagwaro mune index. Takaongorora imwe neimwe index uye takanyora nhamba yezvinyorwa mushure mokunge kutenderera kunofanira kushanda. Nekudaro, isu tasvika kune yakaringana shard saizi - kwete kupfuura 50 GB.
Cluster optimization
Zvisinei, hatina kubvisa zvachose matambudziko. Nehurombo, madiki madiki achiri kuoneka: haana kusvika kuvhoriyamu yakataurwa, haana kutenderedzwa, uye akabviswa nekucheneswa kwepasi rose kwema indexes akare kupfuura mazuva matatu, sezvo isu takabvisa kutenderera nezuva. Izvi zvakakonzera kurasikirwa kwedata nekuda kwekuti index kubva musumbu yakanyangarika zvachose, uye kuedza kunyorera kune isiripo indekisi yakaputsa pfungwa yemuchengeti wataishandisa pakutarisira. Alias ββyekunyora yakashandurwa kuita index uye yakatyora iyo rollover logic, zvichikonzera kusadzora kukura kwemamwe ma index kusvika ku600 GB.
Semuenzaniso, kune kutenderera config:
Ρurator-elk-rollover.yaml
---
actions:
1:
action: rollover
options:
name: "nginx_write"
conditions:
max_docs: 100000000
2:
action: rollover
options:
name: "python_error_write"
conditions:
max_docs: 10000000
Kana pakanga pasina rollover alias, kukanganisa kwakaitika:
ERROR alias "nginx_write" not found.
ERROR Failed to complete action: rollover. <type 'exceptions.ValueError'>: Unable to perform index rollover with alias "nginx_write".
Isu takasiya mhinduro yedambudziko iri kune inotevera iteration uye takatora imwe nyaya: isu takachinjira kune yekudhonza logic yeLogstash, iyo inogadzirisa matanda anouya (kubvisa ruzivo rusina basa uye kupfumisa). Isu takaiisa mudocker, yatinovhura kuburikidza ne-docker-compose, uye takaisawo logstash-exporter ipapo, iyo inotumira metrics kuPrometheus yekutarisa kushanda kweiyo log rwizi. Nenzira iyi takazvipa isu mukana wekuchinja zvakanaka huwandu hwelogstash zviitiko zvine chekuita nekugadzirisa yega yega yegi.
Tichiri kuvandudza cluster, traffic ye cian.ru yakawedzera kusvika 12,8 miriyoni akasiyana vashandisi pamwedzi. Somugumisiro, zvakazoitika kuti kushandurwa kwedu kwaiva zvishoma shure kwekuchinja kwekugadzirwa, uye takanga takatarisana nechokwadi chokuti "node" dzinodziya dzaisagona kutarisana nemutoro uye kuderedza kuendeswa kwematanda ose. Takagamuchira data "inopisa" pasina kukundikana, asi taifanira kupindira mukuendeswa kwevamwe uye kuita rollover yebhuku kuitira kuti tigovane kugovera zvinyorwa.
Panguva imwecheteyo, kuyera uye kushandura marongero ezviitiko zvelogstash musumbu kwakaomeswa nekuti yaive yemuno docker-compose, uye zvese zviito zvakaitwa nemaoko (kuwedzera magumo matsva, zvaive zvakafanira kuti uende nemaoko zvese. maseva uye ita docker-compose up -d kwese kwese).
Log redistribution
MunaSeptember wegore rino, takanga tichiri kutema monolith, mutoro uri pasumbu wakanga uchiwedzera, uye kuyerera kwematanda kwaiswedera kune 30 zviuru zvemashoko pasekondi.
Takatanga iyo inotevera iteration nehardware update. Isu takachinja kubva kuvarongi vashanu kuenda kune vatatu, takatsiva data node uye takakunda maererano nemari nenzvimbo yekuchengetedza. Kune node tinoshandisa maviri magadzirirwo:
- Kune "zvinopisa" node: E3-1270 v6 / 960Gb SSD / 32 Gb x 3 x 2 (3 yeHot1 uye 3 yeHot2).
- Kune "anodziya" node: E3-1230 v6 / 4Tb SSD / 32 Gb x 4.
Pakudzokororwa uku, takafambisa indekisi ine matanda ekuwana e microservices, iyo inotora nzvimbo yakafanana neyepamberi-mutsara nginx matanda, kune rechipiri boka rematatu "anopisa" node. Isu ikozvino tinochengetedza data pa "hot" node kwemaawa makumi maviri, uye tozoendesa kune "inodziya" node kune mamwe matanda.
Isu takagadzirisa dambudziko remadiki madiki anonyangarika nekugadzirisazve kutenderera kwawo. Iye zvino ma indexes anotenderedzwa maawa makumi maviri nematatu ega ega chero zvakadaro, kunyangwe paine data shoma ipapo. Izvi zvishoma zvakawedzera nhamba ye shards (paiva ne23 yavo), asi kubva pakuona kwekuita kwechikwata kunoshivirira.
Nekuda kweizvozvo, pakanga paine matanhatu "anopisa" uye mana chete "anodziya" node musumbu. Izvi zvinokonzeresa kunonoka kudiki pane zvikumbiro kwenguva yakareba, asi kuwedzera nhamba yemanodhi mune ramangwana kuchagadzirisa dambudziko iri.
Iyi iteration yakagadzirisawo dambudziko rekushaikwa kwehafu-otomatiki scaling. Kuti tiite izvi, takaisa chivakwa Nomad cluster - yakafanana neyatakatoisa mukugadzira. Parizvino, chiyero cheLogstash hachingoshanduki zvichienderana nemutoro, asi isu tichauya kune izvi.
Zvirongwa zvemangwana
Zviyero zvekugadzirisa zvakashandiswa zvakakwana, uye ikozvino tinochengeta 13,3 TB yedata - matanda ose kwemazuva mana, izvo zvinodiwa pakuongororwa kwekukurumidzira kwezviziviso. Isu tinoshandura mamwe ematanda kuita metrics, yatinowedzera kuGraphite. Kuita kuti basa revainjiniya rive nyore, isu tine metrics yezvivakwa cluster uye zvinyorwa zvehafu-otomatiki kugadzirisa kwezvinetso zvakajairika. Mushure mekuwedzera nhamba yedata node, iyo yakarongerwa gore rinouya, isu tichachinja kune data kuchengetedza kubva ku4 kusvika ku4 mazuva. Izvi zvichave zvakakwana kune basa rekushanda, sezvo isu tinogara tichiedza kuongorora zviitiko nekukurumidza sezvinobvira, uye nekuferefeta kwenguva refu kune telemetry data.
Muna Gumiguru 2019, traffic kune cian.ru yakanga yatokura kusvika 15,3 miriyoni akasiyana vashandisi pamwedzi. Uyu wakave muyedzo wakakomba wemhinduro yekuvaka yekuunza matanda.
Iye zvino tiri kugadzirira kugadzirisa ElasticSearch kuvhezheni 7. Zvisineyi, nekuda kweizvi tichafanirwa kuvandudza mepu yezvinyorwa zvakawanda muElasticSearch, sezvo vakatama kubva muvhezheni 5.5 uye vakanzi vakarambwa muvhezheni 6 (haapo muvhezheni. 7). Izvi zvinoreva kuti panguva yekuvandudza pachave paine imwe mhando yeforce majeure, iyo inotisiya tisina matanda apo nyaya yacho inogadziriswa. Yevhezheni 7, isu tiri kunyanya kutarisira Kibana ine yakagadziridzwa interface uye mafirita matsva.
Takazadzisa chinangwa chedu chikuru: takamira kurasikirwa nematanda uye takaderedza nguva yekuderera kwechikwata chezvivakwa kubva pakuparara kwe2-3 pavhiki kusvika kumaawa mashomanana ekugadzirisa basa pamwedzi. Zvose izvi basa mukugadzira zvinenge zvisingaoneki. Zvisinei, iye zvino tinogona kuziva chaizvo zviri kuitika nebasa redu, tinogona kukurumidza kuzviita nenzira yakanyarara uye tisanetseka kuti matanda acharasika. Kazhinji, isu tinogutsikana, tinofara uye kugadzirira zvitsva zvitsva, zvatichataura nezvazvo gare gare.
Source: www.habr.com