Kulezi zinsuku, akunakwenzeka ukucabanga ngephrojekthi esekwe ku-Kubernetes ngaphandle kwesitaki se-ELK, egcina izingodo zazo zombili izinhlelo zokusebenza nezingxenye zesistimu zeqoqo. Ekuzijwayezeni kwethu, sisebenzisa isitaki se-EFK ne-Fluentd esikhundleni se-Logstash.
I-Fluentd ingumqoqi welogi wesimanjemanje, osebenza emhlabeni wonke othola ukuduma okwengeziwe futhi usejoyine i-Cloud Native Computing Foundation, yingakho i-vector yayo yokuthuthukisa igxile ekusetshenzisweni ngokubambisana ne-Kubernetes.
Iqiniso lokusebenzisa i-Fluentd esikhundleni se-Logstash aliguquli ingqikithi evamile yephakheji yesofthiwe, nokho, i-Fluentd ibonakala ngama-nuances ayo akhethekile ngenxa yokuguquguquka kwayo.
Isibonelo, lapho siqala ukusebenzisa i-EFK kuphrojekthi ematasa enomfutho ophezulu wokugawulwa kwemithi, sasibhekene neqiniso lokuthi e-Kibana eminye imilayezo yaboniswa ngokuphindaphindiwe izikhathi ezimbalwa. Kulesi sihloko sizokutshela ukuthi kungani kwenzeka lesi simo nokuthi ungayixazulula kanjani inkinga.
Inkinga yokuphindaphinda idokhumenti
Kumaphrojekthi ethu, i-Fluentd isetshenziswa njenge-DaemonSet (yethulwa ngokuzenzakalelayo endaweni eyodwa endaweni ngayinye yeqoqo le-Kubernetes) futhi iqapha izingodo ze-stdout ku-/var/log/containers. Ngemva kokuqoqwa nokucutshungulwa, amalogi asesimweni samadokhumenti e-JSON athunyelwa ku-ElasticSearch, aphakanyiswe ngeqoqo noma ifomu elizimele, kuye ngesikali sephrojekthi kanye nezimfuneko zokusebenza nokubekezelela amaphutha. I-Kibana isetshenziswa njenge-graphical interface.
Uma sisebenzisa i-Fluentd ene-plugin ye-buffering ephumayo, sihlangabezane nesimo lapho amanye amadokhumenti ku-ElasticSearch anokuqukethwe okufanayo ncamashi futhi ehluke kuphela kusihlonzi. Ungaqinisekisa ukuthi lokhu ukuphindaphinda komlayezo usebenzisa ilogi ye-Nginx njengesibonelo. Kufayela lokungena, lo mlayezo ukhona ekhophini eyodwa:
127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -Nokho, kunemibhalo embalwa ku-ElasticSearch equkethe lo mlayezo:
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "HgGl_nIBR8C-2_33RlQV",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}
{
"_index": "test-custom-prod-example-2020.01.02",
"_type": "_doc",
"_id": "IgGm_nIBR8C-2_33e2ST",
"_version": 1,
"_score": 0,
"_source": {
"service": "test-custom-prod-example",
"container_name": "nginx",
"namespace": "test-prod",
"@timestamp": "2020-01-14T05:29:47.599052886 00:00",
"log": "127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" -",
"tag": "custom-log"
}
}Ngaphezu kwalokho, kungaba nokuphindaphinda okungaphezu kokubili.
Ngenkathi ulungisa le nkinga kulogi lwe-Fluentd, ungabona inani elikhulu lezixwayiso ngokuqukethwe okulandelayo:
2020-01-16 01:46:46 +0000 [warn]: [test-prod] failed to flush the buffer. retry_time=4 next_retry_seconds=2020-01-16 01:46:53 +0000 chunk="59c37fc3fb320608692c352802b973ce" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>"elasticsearch", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}): read timeout reached"Lezi zexwayiso zenzeka lapho i-ElasticSearch ingakwazi ukubuyisela impendulo esicelweni phakathi nesikhathi esishiwo ipharamitha ye-application_timeout, yingakho isiqephu sebhafa esidlulisiwe singakwazi ukusulwa. Ngemva kwalokhu, i-Fluentd izama ukuthumela isinqamu sebhafa ku-ElasticSearch futhi ngemva kwenani elithile lemizamo, umsebenzi uqeda ngempumelelo:
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fc3fb320608692c352802b973ce"
2020-01-16 01:47:05 +0000 [warn]: [test-prod] retry succeeded. chunk_id="59c37fad241ab300518b936e27200747"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fc11f7ab707ca5de72a88321cc2"
2020-01-16 01:47:05 +0000 [warn]: [test-dev] retry succeeded. chunk_id="59c37fb5adb70c06e649d8c108318c9b"
2020-01-16 01:47:15 +0000 [warn]: [kube-system] retry succeeded. chunk_id="59c37f63a9046e6dff7e9987729be66f"Kodwa-ke, i-ElasticSearch iphatha izingcezu zebhafa ngayinye njengehlukile futhi ibanika amanani enkambu _id ahlukile ngesikhathi sokukhonjwa. Lena yindlela amakhophi emilayezo avela ngayo.
E-Kibana kubukeka kanje:
Ukuxazulula izinkinga
Kunezinketho ezimbalwa zokuxazulula le nkinga. Enye yazo indlela eyakhelwe ku-plugin-plugin-elasticsearch plugin yokukhiqiza i-hashi eyingqayizivele yedokhumenti ngayinye. Uma usebenzisa le ndlela, i-ElasticSearch izobona izimpinda esigabeni sokudlulisela futhi ivimbele impinda yamadokhumenti. Kodwa kufanele sicabangele ukuthi le ndlela yokuxazulula inkinga iyazabalaza nophenyo futhi ayiliqedi iphutha ngokuntula isikhathi sokuphelelwa yisikhathi, ngakho-ke sikuyekile ukusetshenziswa kwayo.
Sisebenzisa i-plugin ye-buffering kokuphumayo kwe-Fluentd ukuze sinqande ukulahleka kwelogi uma kuba nezinkinga zenethiwekhi zesikhashana noma ukukhuphuka kokuqina kokungena. Uma ngesizathu esithile i-ElasticSearch ingakwazi ukubhala idokhumenti ngaso leso sikhathi enkombeni, idokhumenti ifakwa kulayini futhi igcinwe kudiski. Ngakho-ke, esimweni sethu, ukuze kuqedwe umthombo wenkinga eholela ephutheni elichazwe ngenhla, kuyadingeka ukusetha amanani alungile wemingcele ye-buffering, lapho i-Fluentd buffer yokukhipha izoba ngosayizi owanele futhi ngesikhathi esifanayo ukuphatha ukusulwa ngesikhathi esinikeziwe.
Kuyaphawuleka ukuthi amanani amapharamitha okuxoxwe ngawo ngezansi angawodwa esimweni ngasinye sokusebenzisa i-buffering kuma-plugin okukhiphayo, njengoba ancike ezintweni eziningi: ukuqina kokubhala imiyalezo kulogi ngamasevisi, ukusebenza kohlelo lwediski, inethiwekhi. ukulayisha kwesiteshi kanye nomkhawulokudonsa waso. Ngakho-ke, ukuze uthole izilungiselelo ze-buffer ezifanele icala ngalinye, kodwa elingadingeki, ugwema ukusesha okude ngokungaboni, ungasebenzisa imininingwane yokususa iphutha i-Fluentd eyibhala kulogi yayo ngesikhathi sokusebenza futhi uthole amanani alungile ngokushesha.
Ngesikhathi inkinga irekhodwa, ukulungiselelwa kwakubukeka kanjena:
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.test.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 8M
queue_limit_length 8
overflow_action block
</buffer>Lapho kuxazulula inkinga, amanani amapharamitha alandelayo akhethwe ngesandla:
chunk_limit_size β usayizi wezingcezu lapho imilayezo ekubhafa ihlukaniswa khona.
- flush_interval β isikhawu sesikhathi ngemva kwalokho isigcinalwazi siyasulwa.
- queue_limit_length β inani eliphezulu lezingcezu kulayini.
- request_timeout yisikhathi lapho uxhumo phakathi kwe-Fluentd ne-ElasticSearch lusungulwa.
Isamba sikasayizi webhafa singabalwa ngokuphindaphinda amapharamitha queue_limit_length kanye ne-chunk_limit_size, okungahunyushwa ngokuthi βinani eliphezulu lamachuki kulayini, ngalinye elinosayizi othile.β Uma usayizi webhafa ungenele, isixwayiso esilandelayo sizovela kulogi:
2020-01-21 10:22:57 +0000 [warn]: [test-prod] failed to write data into buffer by buffer overflow action=:blockKusho ukuthi isigcinalwazi asinaso isikhathi sokusulwa ngesikhathi esinikeziwe futhi idatha engena kubhafa egcwele ivinjiwe, okuzoholela ekulahlekeni kwengxenye yamalogi.
Ungakwazi ukukhuphula isigcinalwazi ngezindlela ezimbili: ngokwandisa noma usayizi wesiqephu ngasinye kulayini, noma inani lezinqamu ezingaba kulayini.
Uma usetha usayizi we-chunk chunk_limit_size ukuba ube ngaphezu kwamamegabhayithi angu-32, i-ElasticSeacrh ngeke iyamukele, njengoba iphakethe elingenayo lizoba likhulu kakhulu. Ngakho-ke, uma udinga ukwandisa isigcinalwazi ngokuqhubekayo, kungcono ukukhulisa umkhawulo wobude bomugqa_umkhawulo_ubude.
Uma isigcinalwazi siyeka ukuchichima futhi kusele umlayezo onganele wokuvala isikhathi kuphela, ungaqala ukukhulisa ipharamitha ye-application_timeout. Nokho, uma usetha inani libe ngaphezu kwamasekhondi angu-20, izexwayiso ezilandelayo zizoqala ukuvela kulogi lwe-Fluentd:
2020-01-21 09:55:33 +0000 [warn]: [test-dev] buffer flush took longer time than slow_flush_log_threshold: elapsed_time=20.85753920301795 slow_flush_log_threshold=20.0 plugin_id="postgresql-dev" Lo mlayezo awuphazamisi ukusebenza kwesistimu nganoma iyiphi indlela futhi usho ukuthi isikhathi sokushaywa kwebhafa sithathe isikhathi eside kunokumiswa yipharamitha ye-slow_flush_log_threshold. Lolu wulwazi lokulungisa iphutha futhi siyalusebenzisa uma sikhetha inani lepharamitha ye-application_timeout.
I-algorithm yokukhetha ejwayelekile imi kanje:
- Setha isikhathi sokuvala isicelo sibe yinani eliqinisekisiwe ukuthi likhulu kunalokho okudingekayo (amakhulu amasekhondi). Phakathi nokusetha, umbandela oyinhloko wokulungiselelwa okulungile kwaleli pharamitha kuzoba ukunyamalala kwezixwayiso ngokuntuleka kwesikhathi sokuvala.
- Linda imilayezo mayelana nokweqa i-slow_flush_log_threshold threshold. Umbhalo oyisixwayiso kunkambu yesikhathi_edlule uzobonisa isikhathi sangempela lapho isikhafa sisulwe ngaso.
- Setha isicelo_sokuvala sibe yinani elikhulu kunenani eliphezulu elapsed_time elitholwe phakathi nenkathi yokubuka. Sibala inani le-application_timeout njengedlule_isikhathi + 50%.
- Ukuze ususe izexwayiso mayelana nokuguquguquka kwebhafa okude kulogi, ungakhuphula inani elithi slow_flush_log_threshold. Sibala leli nani njengenkathi edlule + 25%.
Amanani okugcina alawa mapharamitha, njengoba kuphawuliwe ekuqaleni, atholakala ngawodwana esimweni ngasinye. Ngokulandela i-algorithm engenhla, siqinisekisiwe ukuqeda iphutha eliholela emilayezweni ephindaphindiwe.
Ithebula elingezansi libonisa ukuthi inani lamaphutha ngosuku, okuholela ekuphindaphindweni kwemilayezo, lishintsha kanjani inqubo yokukhetha amanani wemingcele echazwe ngenhla:
iphuzu-1
iphuzu-2
iphuzu-3
iphuzu-4
Ngaphambi emvakwa
Ngaphambi emvakwa
Ngaphambi emvakwa
Ngaphambi emvakwa
yehlulekile ukususa isilondolozi
1749/2
694/2
47/0
1121/2
ukuzama futhi kuphumelele
410/2
205/1
24/0
241/2
Kuyafaneleka futhi ukuqaphela ukuthi izilungiselelo eziwumphumela zingase zilahlekelwe ukuhambisana kwazo njengoba iphrojekthi ikhula futhi, ngokufanele, inani lamalogi liyakhula. Uphawu oluyinhloko lwesikhathi esinganele sokuvala ukubuyiselwa kwemilayezo emayelana nokuguquguquka kwebhafa ende kulogi ye-Fluentd, okungukuthi, ukweqa umkhawulo we-slow_flush_log_threshold. Kusukela kuleli phuzu kuya phambili, kusekhona imajini encane ngaphambi kokuba ipharamitha yokucela_isikhathi sokuphuma idlule, ngakho-ke kuyadingeka ukuthi uphendule le milayezo ngesikhathi esifanele futhi uphinde inqubo yokukhetha izilungiselelo ezifanele ezichazwe ngenhla.
isiphetho
Ukulungisa kahle isigcinalwazi esiphumayo se-Fluentd kungesinye sezigaba eziyinhloko zokumisa isitaki se-EFK, esinquma ukuzinza kokusebenza kwaso kanye nokubekwa kahle kwamadokhumenti kumakhombo. Ngokusekelwe ku-algorithm yokucushwa echazwe, ungaqiniseka ukuthi wonke amalogi azobhalwa kunkomba ye-ElasticSearch ngendlela efanele, ngaphandle kokuphindaphinda noma ukulahlekelwa.
Funda nezinye izindatshana kubhulogi yethu:
Source: www.habr.com