Kungowama-2019, kwaye asikabi naso isisombululo esiqhelekileyo sokudityaniswa kwelogi eKubernetes. Kweli nqaku, singathanda, sisebenzisa imizekelo evela kuqheliselo lokwenyani, ukwabelana ngokhangelo lwethu, iingxaki esidibene nazo kunye nezisombululo zazo.
Nangona kunjalo, okokuqala, ndiza kwenza ugcino lokuba abathengi abohlukeneyo baqonde izinto ezahlukeneyo kakhulu ngokuqokelela izingodo:
- umntu ufuna ukubona ukhuseleko kunye nezigodo zokuphicothwa;
- umntu - ukugawulwa okuphakathi kwendawo yonke;
- kwaye kwabanye, kwanele ukuqokelela kuphela iilogi zesicelo, ngaphandle, umzekelo, abalinganisi.
Apha ngezantsi kukho isisikwa esingezantsi malunga nendlela esiphumeze ngayo "uluhlu lweminqweno" kunye nokuba bubuphi ubunzima esidibene nabo.
Ithiyori: malunga nezixhobo zokugawulwa kwemithi
Imvelaphi kumacandelo enkqubo yokuloga
Ukugawulwa kwemithi kumde kakhulu, ngenxa yokuba zeziphi iindlela zokuqokelela kunye nokuhlalutya iinkuni eziye zaphuhliswa, nto leyo esiyisebenzisayo namhlanje. Emuva kwiminyaka yee-1950, iFortran yazisa i-analogue yegalelo eliqhelekileyo / imijelo yemveliso, eyanceda umdwelisi ukuba alungise inkqubo yakhe. Ezi yayiziinqobo zekhompyutha zokuqala ezenza ubomi bube lula kubadwelisi benkqubo baloo maxesha. Namhlanje sibona kubo icandelo lokuqala lenkqubo yokugawulwa kwemithi - umthombo okanye "umvelisi" weelog.
Isayensi yekhompyutha ayizange ime ngxi: iinethiwekhi zekhompyutheni zavela, amaqela okuqala ... Iinkqubo eziyinkimbinkimbi ezibandakanya iikhomputha ezininzi zaqala ukusebenza. Ngoku abalawuli benkqubo banyanzeliswa ukuba baqokelele iingodo koomatshini abaninzi, kwaye kwiimeko ezikhethekileyo banokongeza imiyalezo ye-OS kernel xa befuna ukuphanda ukusilela kwenkqubo. Ukuchaza iinkqubo zokuqokelelwa kweelog ezingundoqo, ekuqaleni koo-2000 zapapashwa , eyenze iremote_syslog. Le yindlela elinye icandelo elibalulekileyo elivele ngayo: umqokeleli welog kunye nokugcinwa kwazo.
Ngokunyuka komthamo weelogi kunye nokuqaliswa ngokubanzi kobuchwepheshe bewebhu, kwavela umbuzo wokuba zeziphi iilogi ezifuna ukuboniswa ngokulula kubasebenzisi. Izixhobo ezilula zekhonsoli (awk/sed/grep) zithatyathelwβ indawo zezihambele phambili ababukeli belog - icandelo lesithathu.
Ngenxa yokunyuka komthamo weelogi, enye into yacaca: iilogi ziyafuneka, kodwa azikho zonke. Kwaye iilogi ezahlukeneyo zifuna amanqanaba ahlukeneyo okugcinwa: ezinye zingalahleka ngosuku, ngelixa ezinye zifuna ukugcinwa iminyaka emi-5. Ke, icandelo lokucoca kunye nokuhamba kwedatha yongezwa kwinkqubo yokugawulwa kwemithi - masiyibize isihluzo.
Ugcino luphinde lwenze umtsi omkhulu: ukusuka kwiifayile eziqhelekileyo ukuya kugcino-lwazi olunxulumeneyo, kwaye emva koko kugcino olujolise kuxwebhu (umzekelo, i-Elasticsearch). Ngoko ukugcinwa kwahlulwa kumqokeleli.
Ekugqibeleni, eyona ngcamango yelog iye yanda yaya kuhlobo lweziganeko ezingabonakaliyo esifuna ukuzigcinela imbali. Okanye, ukuba ufuna ukwenza uphando okanye wenze ingxelo yohlalutyo ...
Ngenxa yoko, ngexesha elifutshane, ukuqokelela kwelogi kuye kwaphuhliswa kwi-subsystem ebalulekileyo, enokuthiwa ngokufanelekileyo enye yamacandelwana kwiDatha enkulu.
Ukuba iiprinta eziqhelekileyo zinokwanela "inkqubo yokugawulwa kwemithi," ngoku imeko itshintshe kakhulu.
Kubernetes kunye nezigodo
Xa uKubernetes efika kwiziseko zophuhliso, ingxaki esele ikhona yokuqokelela iilogi ayizange idlule nayo. Ngandlela-thile, kwaba buhlungu ngakumbi: ukulawula iqonga leziseko zophuhliso akuzange kube lula nje kuphela, kodwa kwakunzima kwangaxeshanye. Iinkonzo ezininzi zakudala ziqalise ukufudukela kwiinkonzo ezincinci. Kumxholo weelogi, oku kubonakaliswa kwinani elikhulayo lemithombo yelogi, umjikelezo wabo wobomi obukhethekileyo, kunye nesidingo sokulandelela ubudlelwane bawo onke amacandelo enkqubo ngokusebenzisa izingodo ...
Ndijonge phambili, ndingatsho ukuba ngoku, ngelishwa, akukho ndlela yokugawulwa kwemithi esemgangathweni ye-Kubernetes enokuthelekisa ngokufanelekileyo nabanye bonke. Ezona zicwangciso zidumileyo ekuhlaleni zezi zilandelayo:
- umntu ukhulula istaki EFK (Elasticsearch, Fluentd, Kibana);
- umntu uzama esanda kukhutshwa okanye usebenzisa ;
- thina (kwaye mhlawumbi ingesithi kuphela?..) Ndaneliseke kakhulu luphuhliso lwam - ...
Njengomthetho, sisebenzisa ezi nqwaba zilandelayo kwii-K8s amaqela (kwizisombululo ezizibambe ngokwazo):
- ;
- .
Nangona kunjalo, andiyi kuhlala kwimiyalelo yofakelo kunye noqwalaselo lwabo. Endaweni yoko, ndiya kugxila kwiintsilelo zabo kunye nezigqibo zehlabathi jikelele malunga nemeko ngeelogi ngokubanzi.
Ziqhelanise neelog kwii-K8s
"Iilog zemihla ngemihla", bangaphi kuni?..
Ukuqokelelwa kweendawo ezikumbindi zokhuni ukusuka kwiziseko ezingundoqo ezifanelekileyo kufuna izibonelelo ezininzi, eziya kuchithwa ekuqokeleleni, ekugcinweni nasekusetyenzweni kweelog. Ngexesha lokusebenza kweeprojekthi ezahlukeneyo, sasijongene neemfuno ezahlukeneyo kunye neengxaki zokusebenza ezivela kuzo.
Makhe sizame iClickHouse
Makhe sijonge ugcino olusembindini kwiprojekthi enesicelo esenza iilog ngokusebenzayo: ngaphezulu kwemigca engama-5000 ngomzuzwana. Masiqale ukusebenza kunye nezigodo zakhe, zongeza kwiClickHouse.
Ngokukhawuleza ukuba ixesha lokwenyani lifunekayo, i-4-core server eneClickHouse iya kube sele ilayishwe ngokugqithisileyo kwisistim esezantsi yediski:
Olu hlobo lokulayisha lubangelwa kukuba sizama ukubhala kwi-ClickHouse ngokukhawuleza. Kwaye i-database iphendula kule nto ngokunyuka komthwalo wediski, onokubangela ezi mpazamo zilandelayo:
DB::Exception: Too many parts (300). Merges are processing significantly slower than inserts
Inyani yile kwiClickHouse (ziqulethe idatha yelog) zineengxaki zazo ngexesha lokubhala imisebenzi. Idatha efakwe kuzo ivelisa ulwahlulo lwethutyana, oluthi ke ludityaniswe netafile ephambili. Ngenxa yoko, ukurekhodwa kubonakala kunzima kakhulu kwidiski, kwaye kuxhomekeke kumda esifumene isaziso malunga ngasentla: akukho ngaphezu kwe-1 subpartitions enokuthi idityaniswe kwi-300 yesibini (enyanisweni, oku kukufakwa kwe-300. ngesekhondi).
Ukunqanda le ndlela yokuziphatha, ngamaqhekeza amakhulu kangangoko kwaye akukho ngaphezu kwexesha eli-1 rhoqo kwimizuzwana emi-2. Nangona kunjalo, ukubhala ngokugqabhuka okukhulu kucebisa ukuba masibhale kancinci rhoqo kwiClickHouse. Oku ke, kunokukhokelela ekuphuphumeni kwesithinteli kunye nokulahleka kweelogi. Isisombululo kukunyusa i-Fluentd buffer, kodwa ke ukusetyenziswa kwememori kuya kwanda.
Qaphela:: Enye inkalo eyingxaki yesisombululo sethu ngeClickHouse yayinxulumene nento yokuba ukwahlula kwimeko yethu (loghouse) kuphunyezwa ngeetafile zangaphandle eziqhagamshelwe. . Oku kukhokelela kwinto yokuba xa kusenziwa iisampulu zamathuba amakhulu, i-RAM eninzi iyafuneka, kuba i-metatable iphinda iphindaphinde kuzo zonke izahlulo- kwanazo ngokucacileyo aziqulathanga datha iyimfuneko. Nangona kunjalo, ngoku le ndlela inokubhengezwa ngokukhuselekileyo ukuba ayisebenzi kwiinguqulelo zangoku zeClickHouse (c ).
Ngenxa yoko, kuyacaca ukuba akusiyo yonke iprojekthi enezixhobo ezaneleyo zokuqokelela izingodo ngexesha langempela kwi-ClickHouse (ngokuchanekileyo, ukusabalalisa kwabo akuyi kufaneleka). Ukongeza, kuya kufuneka usebenzise ibhetri, esiza kubuyela kuyo kamva. Ityala elichazwe ngasentla liyinyani. Kwaye ngelo xesha asikwazanga ukunika isisombululo esithembekileyo nesizinzileyo esiya kuhambelana nomthengi kwaye sivumele ukuba siqokelele izingodo ngokulibaziseka okuncinci ...
Kuthekani nge-Elasticsearch?
I-Elasticsearch iyaziwa ngokuphatha umthwalo onzima. Masiyizame kwiprojekthi enye. Ngoku umthwalo ujongeka ngolu hlobo:
I-Elasticsearch ikwazile ukwetyisa umjelo wedatha, nangona kunjalo, ukubhala imiqulu enjalo kuyo kusebenzisa kakhulu i-CPU. Oku kugqitywa ngokuququzelela iqela. Ngobuchwephesha, oku akuyongxaki, kodwa kuye kwavela ukuba ukusebenzisa inkqubo yokuqokelela ilog esele siyisebenzisa malunga ne-8 cores kwaye sinecandelo elongezelelweyo elilayishwe kakhulu kwinkqubo...
Umgca ophantsi: olu khetho lunokulungiswa, kodwa kuphela ukuba iprojekthi inkulu kwaye ulawulo lwayo lulungele ukuchitha izibonelelo ezibalulekileyo kwinkqubo yokugawulwa kwemithi.
Emva koko kuphakama umbuzo wendalo:
Zeziphi iinkuni ezifuneka ngokwenene?
Makhe sizame ukutshintsha indlela ngokwayo: iilogi kufuneka ngaxeshanye zibe nolwazi kwaye zingagubungeli Zonke isiganeko kwinkqubo.
Masithi sinevenkile ekwi-intanethi eyimpumelelo. Zeziphi iinkuni ezibalulekileyo? Ukuqokelela ulwazi oluninzi kangangoko kunokwenzeka, umzekelo, ukusuka kwisango lokuhlawula, ngumbono omkhulu. Kodwa ayizizo zonke iilogi ezivela kwinkonzo yokuqhawula umfanekiso kwikhathalogu yemveliso zibaluleke kakhulu kuthi: kuphela iimpazamo kunye nokubeka iliso eliphezulu ngokwaneleyo (umzekelo, ipesenti yeempazamo ze-500 eziveliswa leli candelo).
Ngoko sifikelele kwisigqibo sokuba ukugawulwa kwemithi okuphakathi akusoloko kuthetheleleka. Rhoqo umxhasi ufuna ukuqokelela zonke iilog kwindawo enye, nangona eneneni, kuyo yonke ilog, kuphela i-5% enemiqathango yemiyalezo ebalulekileyo kwishishini efunekayo:
- Ngamanye amaxesha kwanele ukuqwalasela, yithi, kuphela ubukhulu belog yesikhongozeli kunye nomqokeleli wephutha (umzekelo, uSentry).
- Isaziso sempazamo kunye nelogi enkulu yendawo ngokwayo inokwanela ukuphanda izehlo.
- Siye saba neeprojekthi ezenza uvavanyo olusebenzayo kuphela kunye neenkqubo zokuqokelela iimpazamo. Umphuhlisi akakhange afune zigodo ngolo hlobo-babone yonke into ukusuka kulandelelwano lwempazamo.
Umfanekiso wobomi
Elinye ibali linokuba ngumzekelo omhle. Sifumene isicelo esivela kwiqela lokhuseleko lomnye wabathengi bethu owayesele esebenzisa isisombululo sorhwebo esaphuhliswa kwakudala ngaphambi kokwaziswa kweKubernetes.
Kwakuyimfuneko "ukwenza abahlobo" benkqubo yokuqokelela log kunye ne-sensor yokufumanisa ingxaki yenkampani - QRadar. Le nkqubo inokufumana iilog nge-syslog protocol kwaye ifumane kwakhona kwi-FTP. Nangona kunjalo, akuzange kwenzeke ngokukhawuleza ukuyidibanisa ne-remote_syslog plugin ukwenzela ukuba bafunde kakuhle (njengoko kwavela, ). Iingxaki ngokuseta i-QRadar yajika yaba kwicala leqela lokhuseleko lomxhasi.
Ngenxa yoko, inxalenye yeelogi zeshishini-ezibalulekileyo zilayishwe kwi-FTP QRadar, kwaye enye inxalenye yathunyelwa nge-syslog ekude ngokuthe ngqo kwii-nodes. Kuba oku sade sabhala - mhlawumbi kuya kunceda umntu asombulule ingxaki efanayo ... Enkosi kwiskimu esisiphumo, umxhasi ngokwakhe wafumana kwaye wahlalutya iinkuni ezibalulekileyo (esebenzisa izixhobo zakhe ezizithandayo), kwaye sakwazi ukunciphisa iindleko zenkqubo yokugawulwa kwemithi, ukugcina kuphela kwinyanga ephelile.
Omnye umzekelo ubonisa into engafanele yenziwe. Omnye wabathengi bethu ukuba baqhubeke wonke umntu iziganeko ezivela kumsebenzisi, ezenziwe nge-multiline imveliso engacwangciswanga ulwazi kwilog. Njengoko unokuthekelela, ezo zikhuni bezingafanelekanga kakhulu ukuba uzifunde kwaye uzigcine.
Iikhrayitheriya zelogi
Imizekelo enjalo ikhokelela kwisigqibo sokuba ngaphezu kokukhetha inkqubo yokuqokelela ilogi, kufuneka kananjalo uyila iinkuni ngokwazo! Ziziphi iimfuno apha?
- Iilogi mazibe kwifomathi efundeka ngomatshini (umzekelo, JSON).
- Iilogi kufuneka zibe compact kwaye zikwazi ukutshintsha inqanaba lokugawulwa kwemithi ukuze kulungiswe iingxaki ezinokubakho. Kwangaxeshanye, kwindawo zemveliso kufuneka uqhube iinkqubo ezinenqanaba lokugawulwa kwemithi njenge isilumkiso okanye imposiso.
- Iilogi kufuneka zibe yinto eqhelekileyo, oko kukuthi, kwinto yelogi, yonke imigca kufuneka ibe nohlobo olufanayo lwentsimi.
Izigodo ezingacwangciswanga zingakhokelela kwiingxaki ngokulayisha izigodo kwindawo yokugcina kunye nokuyeka ngokupheleleyo ekusebenzeni kwazo. Njengomzekeliso, nanku umzekelo onempazamo 400, abaninzi abaye badibana nayo ngokuqinisekileyo kwizigodo ezicacileyo:
2019-10-29 13:10:43 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch"
Impazamo ithetha ukuba uthumela umhlaba oluhlobo olungazinzanga kwisalathiso esinemephu esele yenziwe. Owona mzekelo ulula yintsimi kwilog ye nginx enoguquko $upstream_status. Ingaqulatha nokuba inani okanye umtya. Umzekelo:
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "17ee8a579e833b5ab9843a0aca10b941", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staffs/265.png", "protocol": "HTTP/1.1", "status": "200", "body_size": "906", "referrer": "https://example.com/staff", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.001", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "127.0.0.1:9000", "upstream_status": "200", "upstream_response_length": "906", "location": "staff"}
{ "ip": "1.2.3.4", "http_user": "-", "request_id": "47fe42807f2a7d8d5467511d7d553a1b", "time": "29/Oct/2019:16:18:57 +0300", "method": "GET", "uri": "/staff", "protocol": "HTTP/1.1", "status": "200", "body_size": "2984", "referrer": "-", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36", "request_time": "0.010", "cache_status": "-", "upstream_response_time": "0.001, 0.007", "upstream_addr": "10.100.0.10:9000, 10.100.0.11:9000", "upstream_status": "404, 200", "upstream_response_length": "0, 2984", "location": "staff"}
Iilogi zibonisa ukuba umncedisi we-10.100.0.10 uphendule ngephutha le-404 kwaye isicelo sithunyelwe kwenye indawo yokugcina umxholo. Ngenxa yoko, ixabiso kwiilog libe ngolu hlobo:
"upstream_response_time": "0.001, 0.007"
Le meko ixhaphake kangangokuba ide ifuneke ngokwahlukileyo .
Kuthekani ngokuthembeka?
Kukho amaxesha apho zonke iilogi ngaphandle kokukhetha zibalulekile. Kwaye ngale nto, inkqubo yoqokelelo yelog eqhelekileyo yee K8s ezicetywayo/ezixoxiwe ngasentla zineengxaki.
Umzekelo, umntu okwaziyo ukuthetha ngokutyibilikayo akakwazi ukuqokelela iilogi kwizikhongozeli zexesha elifutshane. Kwenye yeeprojekthi zethu, isikhongozeli sokufuduka sedatabase sihlala ngaphantsi kwemizuzwana emi-4 emva koko sacinywa - ngokuhambelana nesichasiselo esihambelanayo:
"helm.sh/hook-delete-policy": hook-succeeded
Ngenxa yoko, i-log yokwenziwa kokufuduka ayizange ifakwe kwindawo yokugcina. Ezopolitiko zinokunceda kule meko. before-hook-creation.
Omnye umzekelo kukujikeleza kwelogi ye-Docker. Masithi kukho isicelo esibhala ngokusebenzayo kwiilog. Ngaphantsi kweemeko eziqhelekileyo, silawula ukucubungula zonke iilogi, kodwa ngokukhawuleza ukuba kuvela ingxaki - umzekelo, njengoko kuchazwe ngasentla ngefomathi engalunganga - ukumisa ukucubungula, kwaye i-Docker ijikeleza ifayile. Isiphumo kukuba iilogi ezibalulekileyo zeshishini zinokulahleka.
Kungenxa yoko kubalulekile ukwahlula imilambo yelog, ukubethelela ukuthumela ezona zixabisekileyo ngqo kwisicelo ukuqinisekisa ukhuseleko lwabo. Ukongeza, akuyi kuba yinto engafanelekanga ukudala ezinye "i-accumulator" yezigodo, enokusinda ekungafumanekiyo kogcino olufutshane ngelixa ugcina imiyalezo ebalulekileyo.
Okokugqibela, asimele siyilibale loo nto Kubalulekile ukubeka esweni nayiphi na inkqubo engaphantsi ngokufanelekileyo. Ngaphandle koko, kulula ukubaleka kwimeko apho i-fluent ikwimo CrashLoopBackOff kwaye ayithumeli nantoni na, kwaye oku kuthembisa ukulahleka kolwazi olubalulekileyo.
ezifunyanisiweyo
Kweli nqaku, asijongi kwizisombululo ze-SaaS njengeDatadog. Uninzi lweengxaki ezichazwe apha sele zisonjululwe ngendlela enye okanye enye ziinkampani zorhwebo ezikhethekileyo ekuqokeleleni izigodo, kodwa ayinguye wonke umntu onokusebenzisa i-SaaS ngezizathu ezahlukeneyo. (ezona ziphambili ziindleko kunye nokuthotyelwa kwe-152-FZ).
Ingqokelela yelog esembindini ekuqaleni ibonakala njengomsebenzi olula, kodwa akunjalo kwaphela. Kubalulekile ukukhumbula ukuba:
- Kuphela ngamacandelo abalulekileyo kufuneka afakwe kwiinkcukacha, ngelixa ukubeka iliso kunye nokuqokelela iimpazamo kunokumiselwa kwezinye iinkqubo.
- Iilogi kwimveliso kufuneka zigcinwe zincinci ukuze zingafaki umthwalo ongeyomfuneko.
- Iilogi kufuneka zifundeke ngoomatshini, ziqheleke, kwaye zibe nefomathi engqongqo.
- Iilogi ezibaluleke ngokwenene kufuneka zithunyelwe kumjelo owahlukileyo, omele uhlukaniswe kwizinto eziphambili.
- Kuyafaneleka ukuqwalasela i-accumulator yelogi, enokukusindisa ekuqhumeni komthwalo ophezulu kwaye wenze umthwalo kwisitoreji ufanayo.
Le mithetho elula, ukuba isetyenziswe kuyo yonke indawo, iya kuvumela ukuba iisekethe ezichazwe ngasentla zisebenze - nangona zilahlekile amacandelo abalulekileyo (ibhetri). Ukuba awuhambelani nemigaqo enjalo, umsebenzi uya kukukhokelela ngokulula kunye neziseko zoncedo kwelinye elithwele kakhulu (kwaye kwangaxeshanye lingasebenzi) inxalenye yenkqubo.
PS
Funda nakwibhlog yethu:
- Β«";
- Β«";
- «».
umthombo: www.habr.com