Kule mihla, akunakwenzeka ukucinga iprojekthi esekwe ku-Kubernetes ngaphandle kwesitaki se-ELK, egcina iilogi zazo zombini izicelo kunye namacandelo enkqubo yeqela. Kwindlela yethu yokusebenza, sisebenzisa i-EFK stack eneFluentd endaweni yeLogstash.
UFluentd ngumqokeleli welog yangoku, wendalo yonke ofumana ukuthandwa ngakumbi kwaye ujoyine i-Cloud Native Computing Foundation, yiyo loo nto iVector yayo yophuhliso igxile ekusebenziseni ngokubambisana neKubernetes.
Inyani yokusebenzisa i-Fluentd endaweni ye-Logstash ayitshintshi isiseko esiqhelekileyo sephakheji yesoftware, nangona kunjalo, iFluentd ibonakaliswe ngamagqabantshintshi ayo athile ngenxa yokuguquguquka kwayo.
Ngokomzekelo, xa saqala ukusebenzisa i-EFK kwiprojekthi exakekileyo kunye nobukhulu obuphezulu bokugawulwa kwemithi, sasijongene nelokuba e-Kibana eminye imiyalezo iboniswe ngokuphindaphindiweyo ngamaxesha amaninzi. Kweli nqaku siza kukuxelela ukuba kutheni le nto yenzeka kunye nendlela yokusombulula ingxaki.
Ingxaki yokuphindaphinda amaxwebhu
Kwiiprojekthi zethu, i-Fluentd isetyenziswa njenge-DaemonSet (iqaliswe ngokuzenzekelayo kwimeko enye kwindawo nganye yeqela le-Kubernetes) kwaye ibeka iliso kwi-stdout logs kwi-/var/log/containers. Emva kokuqokelela kunye nokucutshungulwa, iilogi ezikwifom ye-JSON zithunyelwa kwi-ElasticSearch, ephakanyiswe kwi-cluster okanye ifom ye-standalone, kuxhomekeke kwisikali seprojekthi kunye neemfuno zokusebenza kunye nokunyamezela impazamo. I-Kibana isetyenziswa njengojongano lomzobo.
Xa usebenzisa i-Fluentd ene-plugin ye-buffering ephumayo, siye sadibana nemeko apho amanye amaxwebhu kwi-ElasticSearch enomxholo ofanayo kwaye ahluke kuphela kwisichongi. Ungaqinisekisa ukuba oku kuphindaphindwa komyalezo usebenzisa ilogi yeNginx njengomzekelo. Kwifayile yelog, lo myalezo ukhona kwikopi enye:
127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -Nangona kunjalo, kukho amaxwebhu amaninzi kwi-ElasticSearch analo myalezo:
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "HgGl_nIBR8C-2_33RlQV",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "IgGm_nIBR8C-2_33e2ST",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}Ngaphezu koko, kunokuphinda-phinda kabini.
Ngelixa ulungisa le ngxaki kwiilog zeFluentd, unokubona inani elikhulu lezilumkiso ezinomxholo olandelayo:
2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"Ezi zilumkiso zenzeka xa i-ElasticSearch ingenako ukubuyisela impendulo kwisicelo ngexesha elichazwe yi-application_timeout parameter, yiyo loo nto iqhekeza lebuffer elithunyelweyo lingenako ukucocwa. Emva koku, i-Fluentd izama ukuthumela iqhekeza le-buffer kwi-ElasticSearch kwakhona kwaye emva kwenani elingenasizathu lenzame, umsebenzi ugqiba ngempumelelo:
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce"
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b"
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"Nangona kunjalo, i-ElasticSearch iphatha iqhekeza ngalinye le-buffer njengeyahlukileyo kwaye ibanike ixabiso elahlukileyo le-id ngexesha lesalathiso. Le yindlela iikopi zemiyalezo ezivela ngayo.
Kwi-Kibana ibonakala ngolu hlobo:
Uluhlu lweengxaki
Kukho iindlela ezininzi zokusombulula le ngxaki. Enye yazo yindlela eyakhelweyo kwiplagi ye-plugin-elasticsearch ephuculweyo ukuze kuveliswe ihashi eyodwa kuxwebhu ngalunye. Ukuba usebenzisa le ndlela, i-ElasticSearch iyakuqaphela uphinda-phindo kwinqanaba lokudlulisela kwaye ithintele amaxwebhu aphindiweyo. Kodwa kufuneka sithathele ingqalelo ukuba le ndlela yokusombulula ingxaki iyasokola kunye nophando kwaye ayiphelisi impazamo ngokungabikho kwexesha, ngoko sikuyekile ukusetyenziswa kwayo.
Sisebenzisa i-plugin ye-buffering kwimveliso ye-Fluentd ukukhusela ilahleko yelogi kwimeko yeengxaki zenethiwekhi zexesha elifutshane okanye ukunyuka kokugawulwa kwemithi. Ukuba ngesizathu esithile i-ElasticSearch ayikwazi ukubhala ngoko nangoko uxwebhu kwisalathiso, uxwebhu lubekwe emgceni kwaye lugcinwe kwidiski. Ke ngoko, kwimeko yethu, ukuze kupheliswe intsusa yengxaki ekhokelela kwimpazamo echazwe ngasentla, kuyafuneka ukuseta amaxabiso achanekileyo eparameters zebuffering, apho i-Fluentd output buffer iya kuba nobungakanani obaneleyo kwaye kwangaxeshanye ikwazi ukucocwa ngexesha elibekiweyo.
Kuyaphawuleka ukuba amaxabiso eeparameters ezixoxwe ngezantsi apha ngabanye kwimeko nganye ethile yokusebenzisa i-buffering kwiiplagi zemveliso, njengoko zixhomekeke kwizinto ezininzi: ubukhulu bokubhala imiyalezo kwilog ngeenkonzo, ukusebenza kwedisk system, network. Umthwalo wetshaneli kunye ne-bandwidth yayo. Ke ngoko, ukuze ufumane useto lwe-buffer olulungele imeko nganye, kodwa olungafunekiyo, ukuthintela uphendlo olude ngokumfameka, ungasebenzisa ulwazi lolungiso lweempazamo olubhalelwa nguFluentd kwilog ngexesha lomsebenzi kwaye ngokukhawuleza ufumane amaxabiso achanekileyo.
Ngexesha ingxaki irekhodwa, ubumbeko bujongeka ngolu hlobo:
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.test.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 8M
queue_limit_length 8
overflow_action block
</buffer>Xa usombulula ingxaki, amaxabiso ezi parameters zilandelayo akhethwa ngesandla:
chunk_limit_size β ubungakanani beqhekeza apho imiyalezo ekwi buffer eyahlulwe khona.
- flush_interval β ixesha lokuphumla emva kokuba isithinteli siyacinywa.
- queue_limit_length - elona nani likhulu lamaqhekeza emgceni.
- request_timeout lixesha apho unxibelelwano phakathi kweFluentd kunye ne-ElasticSearch lusekwe.
Ubungakanani bebuffer buphelele bunokubalwa ngokuphinda-phinda iparameters queue_limit_length kunye ne-chunk_limit_size, enokutolikwa "njengelona nani liphezulu leenqununu ezikumgca, nganye kuzo inobungakanani obunikiweyo." Ukuba ubungakanani besithinteli asanelanga, isilumkiso esilandelayo siya kuvela kwiilog:
2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:blockKuthetha ukuba isithinteli asinalo ixesha lokucocwa ngexesha elibekiweyo kwaye idatha engena kwi-buffer epheleleyo ivaliwe, okuya kukhokelela ekulahlekelweni kwenxalenye yelogi.
Ungasinyusa isithinteli ngeendlela ezimbini: ngokunyusa nokuba kubungakanani bechunk nganye emgceni, okanye inani leenqununu ezinokuba kumgca.
Ukuba useta i-chunk size chunk_limit_size ibe ngaphezulu kwe-32 megabytes, ngoko ke i-ElasticSeacrh ayizukuyamkela, kuba ipakethi engenayo iya kuba nkulu kakhulu. Ke ngoko, ukuba ufuna ukonyusa isithinteli ngakumbi, kungcono ukonyusa ubude bomgca queue_limit_length.
Xa isithinteli siyeka ukuphuphuma kwaye lixesha kuphela elingonelanga umyalezo oseleyo, ungaqala ukwandisa iparameter ye-application_timeout. Nangona kunjalo, ukuba ubeka ixabiso kwimizuzwana engama-20, ezi zilumkiso zilandelayo ziyakuqala ukuvela kwiilog zeFluentd:
2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" Lo myalezo awuchaphazeli ukusebenza kwenkqubo nangayiphi na indlela kwaye ithetha ukuba ixesha lokugungxula ibuffer lithathe ixesha elide kunokumiswa yi slow_flush_log_threshold parameter. Olu lulungiso lweenkcukacha kwaye siyalusebenzisa xa sikhetha ixabiso le-application_timeout parameter.
I-algorithm yokukhetha ngokubanzi imi ngolu hlobo lulandelayo:
- Seta request_timeout kwixabiso eliqinisekisiweyo ukuba likhulu kunemfuneko (amakhulu emizuzwana). Ngexesha lokucwangcisa, inqobo ephambili yolungiselelo oluchanekileyo lwale parameter iya kuba kukunyamalala kwezilumkiso zokungabikho kwexesha.
- Linda imiyalezo emalunga nokugqithisa i-slow_flush_log_threshold threshold. Okubhaliweyo okulumkisayo kwisithuba sexesha elapsed_time sizakubonisa ixesha lokwenyani ibuffer isusiwe.
- Seta request_timeout kwixabiso elikhulu kunobuninzi bexesha elapsed_time elifunyenwe ngexesha lokuqwalaselwa. Sibala ixabiso le-application_timeout njengexesha elidlulileyo + 50%.
- Ukususa izilumkiso malunga nogungxulo olude lwebuffer kwilog, unganyusa ixabiso le slow_flush_log_threshold. Sibala eli xabiso njengexesha elidlulileyo + 25%.
Amaxabiso okugqibela ezi parameters, njengoko kuphawuliwe ngaphambili, afunyanwa ngokwahlukeneyo kwimeko nganye. Ngokulandela i-algorithm engentla, siqinisekisiwe ukuba siyayisusa impazamo ekhokelela kwimiyalezo ephindaphindiweyo.
Itheyibhile engezantsi ibonisa ukuba inani leempazamo ngosuku, elikhokelela ekuphindaphindweni kwemiyalezo, utshintsho kwinkqubo yokukhetha amaxabiso eeparamitha ezichazwe ngasentla:
indawo-1
indawo-2
indawo-3
indawo-4
Ngaphambi mva
Ngaphambi mva
Ngaphambi mva
Ngaphambi mva
ayiphumelelanga ukugungxula isithinteli
1749/2
694/2
47/0
1121/2
uzame kwakhona kuphumelele
410/2
205/1
24/0
241/2
Kuyafaneleka ukongeza ukuba uqaphele ukuba izicwangciso ezibangelwayo zingaphulukana nokubaluleka kwazo njengoko iprojekthi ikhula kwaye, ngokufanelekileyo, inani leelogi landa. Uphawu oluphambili lwexesha elingonelanga lokuphuma kubuyiselo lwemiyalezo emalunga nokugungxulwa kwebuffer ende kwilog yeFluentd, oko kukuthi, ukugqithela kwi-slow_flush_log_threshold threshold. Ukususela kweli nqanaba, kusekho umda omncinci ngaphambi kokuba iparamitha yesicelo_yexesha igqithiswe, ngoko ke kuyimfuneko ukuphendula le miyalezo ngexesha elifanelekileyo kwaye uphinde inkqubo yokukhetha izicwangciso ezifanelekileyo ezichazwe ngasentla.
isiphelo
Ukulungiswa kakuhle kwe-Fluentd output buffer sesinye sezigaba eziphambili zokumisela istaki se-EFK, simisela uzinzo lokusebenza kwayo kunye nokubekwa kwamaxwebhu achanekileyo kwizalathisi. Ngokusekwe kwi-algorithm yoqwalaselo echaziweyo, unokuqiniseka ukuba zonke iilogi ziya kubhalwa kwisalathiso se-ElasticSearch ngendlela echanekileyo, ngaphandle kokuphindaphinda okanye ilahleko.
Funda namanye amanqaku kwibhlog yethu:
umthombo: www.habr.com