Unyaka ka-2019, futhi asikabi naso isixazululo esijwayelekile sokuhlanganisa amalogi ku-Kubernetes. Kulesi sihloko, singathanda, sisebenzisa izibonelo zokwenziwa kwangempela, ukwabelana ngosesho lwethu, izinkinga esihlangabezane nazo kanye nezixazululo zazo.
Nokho, okokuqala, ngizobhukha ukuthi amakhasimende ahlukene aqonda izinto ezihluke kakhulu ngokuqoqa izingodo:
- othile ufuna ukubona izingodo zokuphepha nokuhlola;
- umuntu - ukugawulwa okuphakathi kwayo yonke ingqalasizinda;
- futhi kwabanye, kwanele ukuqoqa izingodo zohlelo lokusebenza kuphela, ngaphandle, isibonelo, abalinganisi.
Ngezansi kunqunyelwe ngezansi mayelana nokuthi sisebenzise kanjani “uhlu lwezifiso” kanye nezinkinga esihlangabezane nazo.
Ithiyori: mayelana namathuluzi okugawula
Isendlalelo sezingxenye zesistimu yokugawula
Ukugawulwa kwemithi sekuhambe ibanga elide, ngenxa yokuthi iziphi izindlela zokuqoqa nokuhlaziya izingodo esezakhiwe, okuyizona esizisebenzisayo namuhla. Emuva ngawo-1950s, i-Fortran yethula i-analogue yokusakazwa kokufakwayo okujwayelekile/okuphumayo, okwasiza umhleli ukuthi alungise uhlelo lwakhe. Lawa kwakungamalogi okuqala ekhompyutha enza ukuphila kwaba lula kubahleli balezo zikhathi. Namuhla sibona kubo ingxenye yokuqala yesistimu yokugawula - umthombo noma “umkhiqizi” wamalogi.
Isayensi yekhompiyutha ayizange ime: amanethiwekhi amakhompiyutha avela, amaqoqo okuqala ... Izinhlelo eziyinkimbinkimbi ezihlanganisa amakhompiyutha amaningana zaqala ukusebenza. Manje abaphathi besistimu baphoqeleka ukuthi baqoqe izingodo emishinini eminingana, futhi ezimweni ezikhethekile bangangeza imilayezo ye-OS kernel uma kwenzeka bedinga ukuphenya ukwehluleka kwesistimu. Ukuchaza amasistimu wokuqoqwa kwamalogi amaphakathi, ekuqaleni kwawo-2000 yashicilelwa , eyenza iremote_syslog ifane. Yavela kanje enye ingxenye ebalulekile: umqoqi welogi kanye nesitoreji sabo.
Ngokukhula kwevolumu yamalogi kanye nokwethulwa okubanzi kobuchwepheshe bewebhu, kwaphakama umbuzo wokuthi yiziphi izingodo ezidinga ukuboniswa kalula kubasebenzisi. Amathuluzi e-console alula (awk/sed/grep) athathelwe indawo athuthuke kakhulu log ababukeli - ingxenye yesithathu.
Ngenxa yokwanda komthamo wezingodo, okunye kwacaca: izingodo ziyadingeka, kodwa hhayi zonke. Futhi izingodo ezahlukene zidinga amazinga ahlukene okulondoloza: amanye angalahleka ngosuku, kuyilapho amanye adinga ukugcinwa iminyaka engu-5. Ngakho-ke, ingxenye yokuhlunga nokugeleza kwedatha yengezwe ohlelweni lokugawula - ake siyibize isihlungi.
Isitoreji siphinde senza umehluko omkhulu: ukusuka kumafayela avamile kuya kusizindalwazi esihlobene, bese kuya kusitoreji esigxile kumadokhumenti (ngokwesibonelo, i-Elasticsearch). Ngakho isitoreji sahlukaniswa nomqoqi.
Ekugcineni, wona kanye umqondo welogi ukhule waba ohlotsheni lochungechunge lwemicimbi esifuna ukuyigcinela umlando. Noma kunalokho, uma kwenzeka udinga ukwenza uphenyo noma udwebe umbiko wokuhlaziya...
Ngenxa yalokho, esikhathini esifushane uma kuqhathaniswa, ukuqoqwa kwelogi kuye kwathuthukiswa kwaba uhlelo olungaphansi olubalulekile, olungabizwa ngokufanelekile ngesinye sezigatshana kokuthi Idatha Enkulu.
Uma ngesinye isikhathi ukuphrinta okujwayelekile bekunganela “uhlelo lokugawula,” manje isimo sesishintshe kakhulu.
Kubernetes kanye nezingodo
Lapho uKubernetes efika engqalasizinda, inkinga ekhona kakade yokuqoqa izingodo ayizange idlule nayo. Ngandlela thize, kwaba buhlungu nakakhulu: ukuphatha inkundla yengqalasizinda akuzange kube lula nje kuphela, kodwa futhi kube nzima ngesikhathi esifanayo. Izinsizakalo eziningi ezindala seziqalile ukuthuthela kuma-microservices. Ngomongo wamalogi, lokhu kubonakala enanini elikhulayo lemithombo yamalogi, umjikelezo wabo wempilo okhethekile, kanye nesidingo sokulandelela ubudlelwano bazo zonke izingxenye zesistimu ngamalogi...
Uma ngibheka phambili, ngingasho ukuthi manje, ngeshwa, ayikho inketho yokugawulwa kwemithi ejwayelekile ye-Kubernetes engaqhathaniswa kahle nabanye bonke. Izikimu ezidume kakhulu emphakathini yilezi ezilandelayo:
- umuntu eqaqa isitaki I-EFK (Elasticsearch, Fluentd, Kibana);
- othile uzama okusanda kukhishwa noma ukusetshenziswa ;
- I-US (futhi mhlawumbe hhayi thina kuphela?..) Ngeneliseke kakhulu ngokuthuthuka kwami - ...
Njengomthetho, sisebenzisa lezi zinqwaba ezilandelayo kumaqoqo e-K8s (ngezisombululo ezizisingathile):
- ;
- .
Nokho, ngeke ngigxile emiyalweni yokufakwa nokucushwa kwazo. Esikhundleni salokho, ngizogxila ekushiyekeni kwabo kanye neziphetho zomhlaba wonke mayelana nesimo ngamalogi ngokujwayelekile.
Zilolonge ngezingodo ku-K8s
“Izigodo zansuku zonke”, bangaki kini?..
Ukuqoqwa kwamalogi endaweni eyodwa kungqalasizinda enkulu kudinga izinsiza ezinkulu, ezizosetshenziselwa ukuqoqa, ukugcinwa kanye nokucubungula izingodo. Ngesikhathi sokusebenza kwamaphrojekthi ahlukahlukene, sasibhekene nezidingo ezahlukahlukene kanye nezinkinga zokusebenza ezivela kuzo.
Ake sizame i-ClickHouse
Ake sibheke isitoreji esimaphakathi kuphrojekthi enohlelo lokusebenza olukhiqiza izingodo ngokukhutheleyo: imigqa engaphezu kuka-5000 ngomzuzwana. Ake siqale ukusebenza ngezingodo zakhe, sizingeze ku-ClickHouse.
Ngokushesha nje lapho kudingeka isikhathi sangempela esiphezulu, iseva engu-4-core ene-ClickHouse izovele ilayishwe ngokweqile kusistimu engaphansi yediski:
Lolu hlobo lokulayisha kungenxa yokuthi sizama ukubhala ku-ClickHouse ngokushesha okukhulu. Futhi i-database isabela kulokhu ngokwanda komthwalo wediski, okungabangela amaphutha alandelayo:
DB::Exception: Too many parts (300). Merges are processing significantly slower than inserts
Point wukuthi ku-ClickHouse (ziqukethe idatha yelogi) zinobunzima bazo ngesikhathi sokusebenza kokubhala. Idatha efakwe kuzo ikhiqiza ukwahlukanisa kwesikhashana, okube sekuhlanganiswe nethebula eliyinhloko. Ngenxa yalokho, ukuqoshwa kuvela kudingekile kakhulu kudiski, futhi kungaphansi komkhawulo esithole isaziso ngawo ngenhla: akukho izingxenye ezingaphansi kuka-1 ezingahlanganiswa ngomzuzwana ongu-300 (eqinisweni, lokhu ukufakwa okungu-300 ngomzuzwana).
Ukuze ugweme lokhu kuziphatha, ngezingcezu ezinkulu ngangokunokwenzeka futhi hhayi isikhathi esingaphezu kwesi-1 njalo ngemizuzwana emi-2. Kodwa-ke, ukubhala ngokuqhuma okukhulu kuphakamisa ukuthi kufanele sibhale kancane ku-ClickHouse. Lokhu, futhi, kungaholela ekuchichimeni kwe-buffer nokulahlekelwa izingodo. Isixazululo ukukhulisa isigcinalwazi se-Fluentd, kodwa-ke ukusetshenziswa kwememori kuzokwanda.
Ukubhala: Esinye isici esiyinkinga sesixazululo sethu nge-ClickHouse sasihlobene neqiniso lokuthi ukwahlukanisa esimweni sethu (loghouse) kwenziwa ngamatafula angaphandle axhunyiwe. . Lokhu kuholela eqinisweni lokuthi lapho kwenziwa isampula ngezikhathi ezinkulu, i-RAM eningi iyadingeka, njengoba i-metatable iphinda iphindaphindeka kuzo zonke izingxenye - ngisho nalezo ngokusobala ezingenayo idatha edingekayo. Kodwa-ke, manje le ndlela kungathiwa ayisebenzi ngokuphephile ezinguqulweni zamanje ze-ClickHouse (c ).
Ngenxa yalokho, kuyacaca ukuthi akuwona wonke amaphrojekthi anezinsiza ezanele zokuqoqa izingodo ngesikhathi sangempela ku-ClickHouse (ngokunembile, ukusatshalaliswa kwawo ngeke kufane). Ngaphezu kwalokho, uzodinga ukusebenzisa ibhethri, esizobuyela kuyo kamuva. Icala elichazwe ngenhla lingokoqobo. Futhi ngaleso sikhathi asikwazanga ukunikeza isisombululo esinokwethenjelwa nesizinzile esingafanela ikhasimende futhi sisivumele ukuthi siqoqe izingodo ngokubambezeleka okuncane...
Kuthiwani nge-Elasticsearch?
I-Elasticsearch yaziwa ngokuphatha imisebenzi enzima. Ake sizame kuphrojekthi efanayo. Manje umthwalo ubukeka kanje:
I-Elasticsearch ikwazile ukugaya ukusakazwa kwedatha, nokho, ukubhala amavolumu anjalo kuyo kusebenzisa kakhulu i-CPU. Lokhu kunqunywa ngokuhlela iqoqo. Ngobuchwepheshe, lokhu akuyona inkinga, kodwa kuvele ukuthi ukusebenzisa uhlelo lokuqoqwa kwelogi sesivele sisebenzisa cishe ama-cores angu-8 futhi sinengxenye eyengeziwe elayishwe kakhulu ohlelweni...
Ngezansi: le nketho ingathethelelwa, kodwa kuphela uma iphrojekthi inkulu futhi abaphathi bayo belungele ukusebenzisa izinsiza ezibalulekile ohlelweni lokugawula oluphakathi nendawo.
Bese kuphakama umbuzo wemvelo:
Yiziphi izingodo ezidingekayo ngempela?
Ake sizame ukuguqula indlela ngokwayo: izingodo kufanele ngesikhathi esisodwa zibe nolwazi futhi zingambozi ngamunye umcimbi ohlelweni.
Ake sithi sinesitolo se-inthanethi esiphumelelayo. Yiziphi izingodo ezibalulekile? Ukuqoqa ulwazi oluningi ngangokunokwenzeka, isibonelo, kusuka esangweni lokukhokha, kuwumqondo omuhle. Kodwa akuwona wonke amalogi avela kusevisi yokusika izithombe kukhathalogi yomkhiqizo abalulekile kithi: amaphutha kuphela nokuqapha okuthuthukisiwe kwanele (isibonelo, iphesenti lamaphutha angu-500 akhiqizwa yile ngxenye).
Ngakho sesifinyelele esiphethweni sokuthi ukugawulwa kwemithi endaweni eyodwa akuthetheleleki ngaso sonke isikhathi. Kaningi iklayenti lifuna ukuqoqa wonke amalogi endaweni eyodwa, nakuba eqinisweni, kulo lonke lolu logi, kudingeka kuphela u-5% onemibandela wemilayezo ebalulekile ebhizinisini:
- Ngezinye izikhathi kwanele ukumisa, ukusho, ubukhulu kuphela belogi yesitsha kanye nomqoqi wephutha (isibonelo, i-Sentry).
- Isaziso sephutha kanye nelogi enkulu yendawo ngokwayo ingase ibe ngokwanele ukuphenya izehlakalo.
- Sibe namaphrojekthi ahlobene nokuhlola okusebenzayo kuphela namasistimu okuqoqa amaphutha. Umthuthukisi akazange adinge izingodo njengoba enjalo - babone yonke into kusukela ekulandeleni amaphutha.
Umfanekiso wempilo
Enye indaba ingaba isibonelo esihle. Sithole isicelo esivela eqenjini lezokuphepha lelinye lamakhasimende ethu elase livele lisebenzisa isixazululo sezentengiso esakhiwa kudala ngaphambi kokwethulwa kwe-Kubernetes.
Bekudingeka “ukwenza abangani” bohlelo oluphakathi nendawo lokuqoqwa kwelogi ngenzwa yokutholwa kwezinkinga zenkampani - QRadar. Lolu hlelo lungathola amalogi ngephrothokholi ye-syslog futhi iwathole ku-FTP. Nokho, akwenzekanga ngokushesha ukuyihlanganisa ne-remote_syslog plugin ukuze ifundeke kahle (njengoba kwavela, ). Izinkinga ngokusetha i-QRadar kuvele ukuthi zisohlangothini lwethimba lezokuphepha leklayenti.
Ngenxa yalokho, ingxenye yamalogi abalulekile ebhizinisi yalayishwa ku-FTP QRadar, futhi enye ingxenye yaqondiswa kabusha nge-syslog ekude ngokuqondile ukusuka kumanodi. Kulokhu saze sabhala - mhlawumbe kuzosiza umuntu axazulule inkinga efanayo ... Ngenxa yohlelo oluvelayo, iklayenti ngokwalo lithole futhi lahlaziya izingodo ezibucayi (esebenzisa amathuluzi ayithandayo), futhi sakwazi ukunciphisa izindleko zesistimu yokugawula, ukulondoloza kuphela ngenyanga edlule.
Esinye isibonelo sikhombisa lokho okungafanele ukwenze. Elinye lamakhasimende ethu ukuze licutshungulwe ngamunye imicimbi evela kumsebenzisi, eyenziwe nge-multiline okukhiphayo okungahlelekile ulwazi ku-log. Njengoba ungase uqagele, izingodo ezinjalo bezingalungile kakhulu ukuze uzifunde futhi uzigcine.
Imibandela yezingodo
Izibonelo ezinjalo ziholela esiphethweni sokuthi ngaphezu kokukhetha uhlelo lokuqoqwa kwelogi, udinga futhi aklame izingodo ngokwazo! Yiziphi izidingo lapha?
- Amalogi kufanele abe ngefomethi efundeka umshini (isibonelo, i-JSON).
- Amalogi kufanele ahlangane futhi abe namandla okushintsha izinga lokugawulwa kwemithi ukuze kulungiswe izinkinga ezingaba khona. Ngesikhathi esifanayo, ezindaweni zokukhiqiza kufanele usebenzise amasistimu anezinga lokugawula njenge Isexwayiso noma Iphutha.
- Amalogi kufanele ajwayeleke, okungukuthi, entweni yelogi, yonke imigqa kufanele ibe nohlobo lwensimu efanayo.
Amalogi angahlelekile angaholela ezinkingeni zokulayisha amalogi endaweni yokugcina futhi amise ngokuphelele ukucubungula kwawo. Njengomfanekiso, nasi isibonelo esinephutha 400, abaningi abaye bahlangabezana nalo kumalogi ahlakaniphile:
2019-10-29 13:10:43 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch"
Iphutha lisho ukuthi uthumela inkambu uhlobo lwayo olungazinzile kunkomba enemephu esenziwe ngomumo. Isibonelo esilula yinkambu kulogi ye-nginx enoguquko $upstream_status. Ingaqukatha inombolo noma iyunithi yezinhlamvu. Ngokwesibonelo:
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "17ee8a579e833b5ab9843a0aca10b941", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staffs/265.png", "protocol": "HTTP/1.1", "status": "200", "body_size": "906", "referrer": "https://example.com/staff", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.001", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "127.0.0.1:9000", "upstream_status": "200", "upstream_response_length": "906", "location": "staff"}
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "47fe42807f2a7d8d5467511d7d553a1b", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staff", "protocol": "HTTP/1.1", "status": "200", "body_size": "2984", "referrer": "-", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.010", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "10.100.0.10:9000, 10.100.0.11:9000", "upstream_status": "404, 200", "upstream_response_length": "0, 2984", "location": "staff"}
Amalogi abonisa ukuthi iseva engu-10.100.0.10 iphendule ngephutha elingu-404 futhi isicelo sithunyelwe kwesinye isitoreji sokuqukethwe. Ngenxa yalokho, inani lamalogi libe kanje:
"upstream_response_time": "0.001, 0.007"
Lesi simo sivamile kangangokuthi sifanelwe ngisho nokuhlukaniswa .
Kuthiwani ngokwethembeka?
Kunezikhathi lapho zonke izingodo ngaphandle kokukhetha zibalulekile. Futhi ngalokhu, izikimu ezijwayelekile zokuqoqwa kwelogi zama-K8 ahlongozwayo/okuxoxwe ngawo ngenhla anezinkinga.
Isibonelo, abakhuluma kahle abakwazi ukuqoqa amalogi ezitsheni zesikhashana. Kwenye yamaphrojekthi ethu, isitsha sokuthutha sesizindalwazi siphile isikhathi esingaphansi kwamasekhondi angu-4 sabe sesisuswa - ngokuya kwesichasiselo esihambisanayo:
"helm.sh/hook-delete-policy": hook-succeeded
Ngenxa yalokhu, ilogu yokusebenzisa ukuthutha ayizange ifakwe kusitoreji. Ipolitiki ingasiza kulokhu. before-hook-creation.
Esinye isibonelo ukujikeleza kwelogi ye-Docker. Ake sithi kukhona uhlelo lokusebenza olubhalela izingodo ngenkuthalo. Ngaphansi kwezimo ezijwayelekile, siyakwazi ukucubungula wonke amalogi, kodwa ngokushesha nje lapho inkinga ivela - isibonelo, njengoba kuchazwe ngenhla ngefomethi engalungile - ukucubungula kuyayeka, futhi i-Docker izungeza ifayela. Umphumela uba ukuthi amalogi abalulekile ebhizinisi angase alahleke.
Yingakho kubalulekile ukuhlukanisa imifudlana yelogi, ishumeka ukuthumela ezibaluleke kakhulu ngqo kuhlelo lokusebenza ukuze kuqinisekiswe ukuphepha kwazo. Ngaphezu kwalokho, ngeke kube yinto engafaneleki ukudala ezinye "i-accumulator" yamalogi, engasinda ekungatholakalini kwesitoreji esifushane kuyilapho ilondoloza imilayezo ebalulekile.
Okokugcina, akumelwe sikukhohlwe lokho Kubalulekile ukuqapha noma iyiphi isistimu engaphansi ngendlela efanele. Uma kungenjalo, kulula ukuwela esimweni lapho umuntu onekhono esesimweni CrashLoopBackOff futhi ayithumeli lutho, futhi lokhu kuthembisa ukulahlekelwa kolwazi olubalulekile.
okutholakele
Kulesi sihloko, asibheki izixazululo ze-SaaS njenge-Datadog. Izinkinga eziningi ezichazwe lapha sezixazululwe ngendlela eyodwa noma enye izinkampani zezentengiselwano ezigxile ekuqoqeni izingodo, kodwa akuwona wonke umuntu ongasebenzisa i-SaaS ngezizathu ezihlukahlukene. (okuyinhloko yizindleko nokuhambisana ne-152-FZ).
Iqoqo lelogi elibekwe endaweni eyodwa ekuqaleni libukeka njengomsebenzi olula, kodwa akuwona neze. Kubalulekile ukukhumbula ukuthi:
- Izingxenye ezibalulekile kuphela ezidinga ukungena ngokuningiliziwe, kuyilapho ukuqapha nokuqoqwa kwamaphutha kungalungiselelwa amanye amasistimu.
- Amalogi ekukhiqizweni kufanele agcinwe emincane ukuze angangezi umthwalo ongadingekile.
- Amalogi kufanele afundeke ngomshini, ajwayeleke, futhi abe nefomethi eqinile.
- Amalogi abucayi ngempela kufanele athunyelwe ngokusakaza okuhlukile, okufanele kuhlukaniswe kwabayinhloko.
- Kuyafaneleka ukucabangela i-accumulator yelogi, engakusindisa ekuqhumeni komthwalo ophezulu futhi wenze umthwalo kusitoreji ufanane kakhulu.
Le mithetho elula, uma isetshenziswa yonke indawo, ingavumela amasekhethi achazwe ngenhla ukuthi asebenze - nakuba engenazo izingxenye ezibalulekile (ibhethri). Uma unganamatheli ezimisweni ezinjalo, umsebenzi uzokuholela kalula kanye nengqalasizinda kwenye ingxenye egcwele kakhulu (futhi ngesikhathi esifanayo engasebenzi) yohlelo.
PS
Funda futhi kubhulogi yethu:
- «";
- «";
- «".
Source: www.habr.com